DDPG-ADRC-Based Load Frequency Control for Multi-Region Power Systems with Renewable Energy Sources and Energy Storage Equipment

Abstract

A scheme of load frequency control (LFC) is proposed based on the deep deterministic policy gradient (DDPG) and active disturbance rejection control (ADRC) for multi-region interconnected power systems considering the renewable energy sources (RESs) and energy storage (ES). The dynamic models of multi-region interconnected power systems are analyzed, which provides a basis for the subsequent RES access. Superconducting magnetic energy storage (SMES) and capacitor energy storage (CES) are adopted due to their rapid response capabilities and fast charge–discharge characteristics. To stabilize the frequency fluctuation, a first-order ADRC is designed, utilizing the anti-perturbation estimation capability of the first-order ADRC to achieve effective control. In addition, the system states are estimated using a linear expansion state observer. Based on the output of the observer, the appropriate feedback control law is selected. The DDPG-ADRC parameter optimization model is constructed to adaptively adjust the control parameters of ADRC based on the target frequency deviation and power deviation. The actor and critic networks are continuously updated according to the actual system response to ensure stable system operation. Finally, the experiment demonstrated that the proposed method outperforms traditional methods across all performance indicators, particularly excelling in reducing adjustment time (45.8% decrease) and overshoot (60% reduction).

1. Introduction

With the growing global energy demand and the increasing emphasis on renewable energy sources (RESs), RESs are becoming more prominent in the energy structure and gradually evolving into a crucial component of the power system. The shift significantly contributes to reducing dependence on traditional fossil fuels and protecting the environment. However, due to the inherent instability and intermittency of RESs, their large-scale integration introduces new challenges to the long-term maintenance of the power system. Liu et al. [1] presented a comprehensive description of research in this field. Load frequency control (LFC), as a critical component for ensuring system stability, urgently requires more advanced control strategies to address various challenges. Mi et al. [2] investigated a robust LFC strategy for power systems considering delay and parameter uncertainty. A sampling proportional integration (PI) LFC scheme was explored, which considers variable period sampling of the control signal [3]. Despite these efforts, none of the proposed controllers have been able to significantly improve LFC performance in practical engineering applications. To solve this problem, Xia et al. [4] studied a robust LFC scheme based on disturbance estimation for power systems subject to random time lag attacks. A novel model predictive control strategy was proposed, which can achieve economic dispatch of load and stable frequency control in a decentralized manner between multiple interconnected power regions [5]. In addition, many control strategies were proposed, such as the fractional-order cascaded control method [6], the decentralized dynamic output feedback controller [7], the intelligent variable structure fuzzy controller [8], and the parallel-type load damping factor controller [9]. Although existing LFC strategies demonstrate satisfactory performance under certain conditions, significant limitations remain. The robust LFC strategy employing periodic sampling PI control exhibits limited adaptability to disturbances induced by RES fluctuations. Furthermore, model predictive control (MPC) strategies, while effective in multi-area coordination, require global state information. This dependency can lead to amplified communication delays and imposes substantial computational demands, often exceeding the requirements for practical engineering real-time operation. Strategies such as fractional-order control and fuzzy logic control enhance dynamic response but lack inherent self-learning capabilities. Consequently, they struggle to address the complex nonlinearities inherent in integrated energy storage (ES) and RESs. These limitations highlight a critical gap whereby current approaches not only inherit the integration challenges of combining traditional control with intelligent algorithms but also fail to reconcile the inherent trade-off between control accuracy and engineering practicality.

To address these challenges, this paper proposes a novel distributed reinforcement learning framework based on the deep deterministic policy gradient (DDPG) algorithm. This framework integrates a linear extended state observer (LESO), core to active disturbance rejection control (ADRC), to estimate total system disturbances in real time. Crucially, the observer dynamically adjusts the DDPG agent’s decision-making cycle. By optimizing the DDPG action space through the LESO’s error compensation mechanism, the framework develops a self-learning disturbance rejection capability. Simultaneously, the end-to-end learning process of the dual closed-loop DDPG structure automatically optimizes the key parameters of the ADRC controller. This co-adaptive learning fosters a synergistic relationship between the RL agent and the disturbance observer. To facilitate engineering practice, ADRC was linearized in [10], greatly simplifying its structure and parameters. The linear ADRC technique is applied to many fields such as permanent magnet synchronous motors [11,12], power electronics [13,14], boiler forced ventilation systems [15,16], and power system control and optimization [17,18]. The nonlinear ADRC of power systems was investigated to solve the hybrid problem of microgrid communication delay [19]. Since the system uncertainty and control difficulty increase accordingly, various intelligent algorithms have been proposed, such as the confined iterative rational Krylov algorithm [20], the multiple-step greedy policy [21], and the large-variance genetic algorithm [22]. In this context, many researchers have conducted extensive research in this field. A deep reinforcement learning parameter optimization model has been constructed in [23], which enables ADRC to achieve optimal control performance. Zhu et al. [24] proposed an ADRC derived from a DDPG and verified the robustness of this method in practical applications.

With the deepening of the dual-carbon strategy, the grid connection of RESs has brought new difficulties to the LFC of the interconnected power grid and has also realized the reliable integration of RESs in the power system [25]. The introduction of RESs increases the complexity of the power system, making it more challenging to operate the system in real time and stably. To improve the utilization of RESs, a new control strategy for a storage system was adopted in [26]. Meanwhile, the application of energy storage equipment in RES power systems has also attracted increasing attention [27]. ES devices can be combined with ADRC strategies to smooth out power fluctuations. Wind power is the most widely used form of renewable energy generation, and has been extensively studied in several papers. The linear ADRC method was inspected, which considers the application in the LFC of complex power systems in wind energy exchange systems [28]. Ebrahimi et al. [29] examined an ES coordination approach of advanced uncertainty control for a realistic multi-region power system by integration with wind energy. As highlighted in [30], an integrated and coordinated suppression strategy was proposed to realize fault ride-through at the access points of large-capacity wind energy. Liu et al. [31] investigated the effects of grid robustness and phase-locked loop characteristics on the stability of wind farms. A method was investigated to smooth the undulations of centralized wind energy generation utilizing a grid-connected vehicle system [32]. Rouhanian et al. [33] obtained an improved LFC model by considering the participation of wind farms. Although ADRC demonstrates excellent performance in improving stability, its practical applications need further exploration to ensure the effectiveness and reliability of control strategies, especially in complex multi-region systems involving RESs. Therefore, the introduction of ADRC in RES power systems is expected to effectively enhance the robustness and immunity of the system to interference.

The objective of this text is to discuss the key technologies and utilization prospects of ADRC in LFC. In RES power systems, the introduction of ADRC is expected to provide theoretical guidance and technical support for ensuring stable and secure operation. Additionally, the study of the DDPG-ADRC application contributes to promoting the progress of power system control technology.

The contributions of this paper are as follows:

• In comparison with the schemes in [34,35], the LFC scheme designed in this paper considers the access of ES devices. The ES system is deeply integrated into the framework combining DDPG and ADRC, achieving coordinated frequency control between the ES system and other power generation equipment.
• This paper proposes an ADRC with a simple structure to enhance the system stability. Meanwhile, this article organically combines DDPG with ADRC, utilizing DDPG for the adaptive adjustment of key parameters in ADRC, thereby forming a novel LFC strategy.
• Compared with other methods [36,37,38,39], the method to optimize ADRC with DDPG proposed in this article has better anti-interference capability and is more suitable for dealing with uncertainties in multi-region power systems. In addition, relative to the solution without ES units, this method significantly enhances the transient response speed of the power system in the event of frequency deviation, which is improved by at least 15%.

The rest of this paper is structured as follows. Section 2 analyzes the dynamic model of the power system. Section 3 proposes a first-order ADRC structure, and uses the DDPG algorithm for parameter optimization. In Section 4, by simulation, the adopted method is verified to be effective. Section 5 summarizes the study.

2. System Model

The LFC model for the i-th region is depicted in Figure 1. In multi-region power systems, thermal power plants in each region are the main power generation mode, which can provide a stable power supply. All thermal power unit elements are mathematically described with first-order equations, as shown below. RESs include wind farms and PV power plants. ES devices include two types of elements. Superconducting magnetic energy storage (SMES) can respond rapidly to system frequency variations, allowing it to rapidly absorb or release large amounts of electrical energy, and capacitor energy storage (CES) fine-tunes the system power through fast charging and discharging. SMES and CES can assist in maintaining system frequency stability, effectively enhancing the frequency control performance of multi-region power systems. The implementation of all these models is as follows.

2.1. RESs Modeling

The power of wind turbines is

$$P_{WT}={\displaystyle \frac{1}{2}}\rho A_b{C}_f{V}_s^3,$$

(1)

where $\rho$ is the air density. $A_b$ is the vane swept size. $V_s$ is the actual wind speed, which is the velocity of the airflow passing through the blades of the wind turbine. $C_f$ is the power factor of the wind turbine, which can be expressed as

$$C_f=(0.44-0.0167\alpha )sin\left[{\displaystyle \frac{(\mu -3)\pi }{15-0.3\alpha }}\right]-0.0184(\mu -3)\alpha ,$$

(2)

where $\alpha$ is the pitch angle. $\mu$ is the tip velocity ratio, which can be expressed as

$$\mu ={\displaystyle \frac{R_b{\omega }_r}{V_s}},$$

(3)

where $R_b$ is the blade radius. ${\omega }_r$ is the blade speed.

The output power of PV is

$$P_{pv}={\eta }_{ce}S\varphi [1-0.005(T_{at}+25)],$$

(4)

where S and ${\eta }_{ce}$ are usually constant, S is the effective light area, and ${\eta }_{ce}$ is the conversion efficiency. The output power is determined entirely by two independent variables, namely sunlight intensity $\varphi$ and ambient temperature $T_{at}$.

2.2. ES Equipment Modeling

The linearization framework of SMES is presented in Figure 2. Based on [40,41], the electrical loss in SMES is expressed as

$$P_{el}={\displaystyle \frac{P_{hl}+P_{cll}}{{\eta }_c}}+P_{lb},$$

(5)

$$P_{hl}=0.25\pi \delta h{(d_0-d_i)}^2,$$

(6)

$$P_{cll}=I_s{n}^{\prime }\psi ,P_{lb}=R_{jr}{I}_s^2,$$

(7)

where ${\eta }_c$ is the cooling efficiency. $d_0$ is the outside diameter. $d_i$ is the inner diameter. $\delta$ is the cooling capacity. h is the height of the magnet. $n^{\prime }$ is all quantities of current leads. $\psi$ is the minimum heat leakage. $R_{jr}$ is the joint resistance. $I_s$ is the incremental of the inductor current in SMES. $P_{el}$ is the total electrical loss of the unit. $P_{hl}$ is the low-temperature loss. $P_{cll}$ is the current lead loss. $P_{lb}$ is the loss of interface wiring.

Equations (5)–(7) are expanded with Taylor expansion and linearized as

$$\mathsf{\Delta }{P}_{el}=\mathsf{\Delta }{I}_s(n^{\prime }\psi +2I_0{R}_{jr}),$$

(8)

where $I_0$ is the current initial value. $\mathsf{\Delta }{I}_s$ is the incremental deviation of the inductor current in SMES.

The variational voltage deviation on the cell coil is expressed as

$$\mathsf{\Delta }{E}_i={\displaystyle \frac{1}{1+T_{sc}s}}[AC{E}_i{K}_{\alpha }-K_{if}\mathsf{\Delta }{I}_s],$$

(9)

where $\mathsf{\Delta }{I}_s={\displaystyle \frac{\mathsf{\Delta }{E}_i}{SL}}$. $K_{if}$ is the gain of feedback signal. $T_{sc}$ is the time constant of the converter delay. $K_{\alpha }$ is the amplification of the gain. $\mathsf{\Delta }{E}_i$ is the incremental voltage deviation of the SMES.

The incremental change in the active power of SMES is

$$\mathsf{\Delta }{P}_{SMES}=(\mathsf{\Delta }{I}_s+I_0)\mathsf{\Delta }{E}_i-\mathsf{\Delta }{P}_{el},$$

(10)

where $\mathsf{\Delta }{P}_{SMES}$ is the power deviation caused by superconducting magnetic ES.

The linearization framework of CES is presented in Figure 3. The voltage increment deviation of CES is expressed as

$$\mathsf{\Delta }{E}_{cs}={\displaystyle \frac{1}{1+T_{c}s}}·{\displaystyle \frac{1}{cs+\frac{1}{R^{\prime }}}}(AC{E}_i{K}_{cf}-K_{vd}\mathsf{\Delta }{I}_{cs}),$$

(11)

where c is the capacitor capacitance. $R^{\prime }$ is the equivalent series resistance value. $T_c$ is the characteristic time of capacitor. $K_{cf}$ is the amplification gain. $K_{vd}$ is the feedback loop gain. $\mathsf{\Delta }{E}_{cs}$ is the incremental voitage deviation of CES. $\mathsf{\Delta }{I}_{cs}$ is the incremental deviation of the current in CES.

The amount of power deviation for CES is expressed as

$$\mathsf{\Delta }{P}_{CES}=\mathsf{\Delta }{E}_{cs}\mathsf{\Delta }{I}_{cs},$$

(12)

where $\mathsf{\Delta }{P}_{CES}$ is the power deviation caused by the capacitor ES.

2.3. LFC Model of Power System

The model of the governor is

$$G_{gi}\left(s\right)={\displaystyle \frac{K_g}{1+T_{gi}s}}.$$

(13)

According to [42], the transfer function of the reheat turbines is

$$G_{ti}\left(s\right)={\displaystyle \frac{K_T(1+K_r{T}_{ri}s)}{(1+T_{ti}s)(1+T_{ri}s)}}.$$

(14)

In accordance with [43], the generator frequency–load relationship is

$$G_{pi}\left(s\right)={\displaystyle \frac{K_p}{1+T_{pi}s}}.$$

(15)

The entire loop can be represented as

$$G\left(s\right)={\displaystyle \frac{G_{ti}\left(s\right)G_{pi}\left(s\right)G_{gi}\left(s\right)}{1+G_{ti}\left(s\right)G_{pi}\left(s\right)G_{gi}\left(s\right)/R_i}}.$$

(16)

The variation in active power on the contact line between regions is

$$\mathsf{\Delta }{P}_{tiei}=\sum _{i\ne j}^n{\displaystyle \frac{2\pi K_{ij}}{s}}(\mathsf{\Delta }{f}_i-\mathsf{\Delta }{f}_j).$$

(17)

From Figure 1, $\mathsf{\Delta }{f}_i$ can be deduced as

$$\mathsf{\Delta }{f}_i=G\left(s\right)(u_i-\mathsf{\Delta }{P}_{di}-\mathsf{\Delta }{P}_{ei}-\mathsf{\Delta }{P}_{tiei}).$$

(18)

The expression for area control error (ACE) is

$$\begin{array}{cc}\hfill AC{E}_i& =\mathsf{\Delta }{f}_i{B}_i+\mathsf{\Delta }{P}_{tiei}=(u_i-\mathsf{\Delta }{P}_{di}-\mathsf{\Delta }{P}_{ei})B_{i}G\left(s\right)+(1-G\left(s\right)B_i)\mathsf{\Delta }{P}_{tiei},\hfill \end{array}$$

(19)

where $\mathsf{\Delta }{f}_i$ is the systematic frequency in region i. $\mathsf{\Delta }{P}_{di}$ is the load disturbance in area i, which represents the total change in output power of the wind turbines and PV. $\mathsf{\Delta }{P}_{ei}$ is the power deviation caused by energy storage devices. $\mathsf{\Delta }{P}_{tiei}$ is the contact line power deviation in the i-th region. $B_i$ is the frequency skew factor of region i. $G_{gi}\left(s\right)$ is the transfer function of the governor. $G_{ti}\left(s\right)$ is the transfer function of the reheat turbine. $G_{pi}\left(s\right)$ is the transfer function of the generator. $K_g$ is the governor static gain factor. $K_r$ is the reheat factor. $K_T$ is the turbine gain factor. $K_p$ is the gain of the generator system. $K_{ij}$ is the synchronization factor for liaison lines between regions i and j. $R_i$ is the speed adjustment caused by governor action. $T_{gi}$ is the time constant of the governor. $T_{ti}$ is the time constant of the turbine. $T_{ri}$ is the time constant of the reheat turbine. $T_{pi}$ is the time constant of the generator system.

3. DDPG-ADRC Framework

DDPG is specifically designed to address the problem of continuous action spaces. In LFC, the output of the controller is usually a continuous value, and DDPG can directly handle continuous actions. DDPG has the capability for online learning. When the output fluctuations of RESs and load changes are frequent, DDPG can automatically extract new state information from the system and adjust to the optimal control scheme, while ADRC is responsible for quickly estimating and compensating for disturbances, and DDPG can adapt to dynamic changes in the system. Its combination with ADRC can further enhance the system robustness and dynamic performance.

3.1. Design of First-Order ADRC

For multi-region interconnected power systems, $AC{E}_i$ is applied as the controller input to stabilize the frequency deviation of the contact line and the frequency deviation between areas to 0 [44]. The architecture of the first-order linear ADRC is shown in Figure 4. The input of the ADRC is $AC{E}_i$, corresponding to the reference input v. In Figure 4, the output u of the ADRC is directly utilized as the output $u_i$ of the controller in Figure 1. The ADRC generates control signals $u_i$ directly based on the area control error, adjusting the mechanical power output of the generator and the ES output. The LESO estimates and compensates for comprehensive disturbances, including fluctuations in RESs and variations in load.

In practical power system applications, ignoring $\mathsf{\Delta }{P}_{tiei}$, (19) can be expressed as

$$\begin{array}{c}{Y}_i\left(s\right)=G_{bi}\left(s\right)U_i\left(s\right)+G_{di}\left(s\right)(D_i\left(s\right)+D_{ei}\left(s\right)),\hfill \end{array}$$

(20)

where $Y_i\left(s\right)$, $U_i\left(s\right)$, $D_i\left(s\right)$, and $D_{ei}\left(s\right)$ are the Laplace transmutes of $AC{E}_i$, $u_i$, $\mathsf{\Delta }{P}_{di}$, and $\mathsf{\Delta }{P}_{ei}$.

Integrating with (13), (14), and (15), $G_{bi}\left(s\right)$ and $G_{di}\left(s\right)$ can be written as

$$\left\{\begin{array}{cc}\hfill G_{bi}\left(s\right)& ={\displaystyle \frac{K_p{K}_g{B}_i{R}_i}{(T_{gi}s+1)(T_{ti}s+1)(T_{pi}s+1)R_i+K_p{K}_g}}\hfill \\ \hfill G_{di}\left(s\right)& ={\displaystyle \frac{(T_{gi}s+1)(T_{ti}s+1)K_p{K}_g{B}_i{R}_i}{(T_{gi}s+1)(T_{ti}s+1)(T_{pi}s+1)R_i+K_p{K}_g}}.\hfill \end{array}\right.$$

(21)

By transforming the above equation, we can obtain the differential form as

$$\begin{array}{cc}\hfill & T_{gi}{T}_{ti}{T}_{pi}{y}_i^{\left(3\right)}\left(t\right)+(T_{gi}{T}_{ti}+T_{gi}+T_{pi}+T_{ti}{T}_{pi}){\ddot{y}}_i\left(t\right)+(T_{gi}+T_{ti}+T_{pi}){\dot{y}}_i\left(t\right)+y_i\left(t\right)\hfill \\ \hfill =& K_p{K}_g{B}_i{R}_{i}u\left(t\right)+T_{gi}{T}_{ti}{K}_p{K}_g{B}_i{R}_i\ddot{d}\left(t\right)+(T_{gi}+T_{ti})K_p{K}_g{B}_i{R}_i\dot{d}\left(t\right)+K_p{K}_g{B}_i{R}_{i}d\left(t\right).\hfill \end{array}$$

(22)

Converting the above process to a generic first-order system, we have

$$\dot{y}=k(t,y,\ddot{y},y^{\left(3\right)},d,\xi ,w)+bu,$$

(23)

where k is a synthetic function of $t,y,y^{\left(3\right)},w,$ and $\xi$. y is the system output, and u is the system input. $y^{\left(3\right)}$ is a third-order dynamic. d is the unmeasurable perturbation of the system. w is the system’s dynamic uncertainty, and $\xi$ is the systematic measurement of noise. The actual system gain b can be estimated by applying $b_0$, which gives the following equations:

$$\dot{y}=k(t,y,\ddot{y},y^{\left(3\right)},d,\xi ,w)+(b-b_0)u+b_{0}u=f+b_{0}u,$$

(24)

$$f=k(t,y,\ddot{y},y^{\left(3\right)},d,\xi ,w)+(b-b_0)u,$$

(25)

where f is the total perturbation of the control system.

According to the first-order system, the state can be defined as $y=x_1$, $x_2=f$, where $x={\left[\begin{array}{cc}{x}_1& x_2\end{array}\right]}^{\mathrm{T}}={\left[\begin{array}{cc}y& f\end{array}\right]}^{\mathrm{T}}$.

The sum of the unknown perturbations of the system can be seen as ${\dot{x}}_2=\dot{f}=h(x,d)$. The state-space form can be translated as

$$\left\{\begin{array}{c}{\dot{x}}_1=x_2+b_{0}u\hfill \\ {\dot{x}}_2=h(x,d)\hfill \\ y=x_1.\hfill \end{array}\right.$$

(26)

The state-space function (26) can be presented as

$$\left\{\begin{array}{c}\dot{x}=Ax+Bu+E\dot{f}\hfill \\ y=Cx,\hfill \end{array}\right.$$

(27)

where $A=\left[\begin{array}{cc}0& 1\\ 0& 0\end{array}\right],B=\left[\begin{array}{c}{b}_0\\ 0\end{array}\right],E=\left[\begin{array}{c}0\\ 1\end{array}\right],C=\left[\begin{array}{cc}1& 0\end{array}\right].$

According to modern control theory, the estimates of each order and the total disturbance f can be obtained from the state observer. A LESO is designed as

$$\left\{\begin{array}{c}\dot{z}=Az+Bu+L(x_1-z_1)\hfill \\ \widehat{y}=Cz,\hfill \end{array}\right.$$

(28)

where $z={\left[\begin{array}{cc}{z}_1& z_2\end{array}\right]}^T$ is the observation state. $z_1$ is the observed value of the system output, and $z_2$ is the observed value of the perturbation. $L={\left[\begin{array}{cc}{\beta }_1& {\beta }_2\end{array}\right]}^T$ is the observer gain vector. When L is chosen sensibly, z can track x well.

The output state feedback control law for selecting ADRC is

$$u={\displaystyle \frac{u_0-z_2}{b_0}},$$

(29)

where $u_0=w_c(r-z_1)$. $w_c$ is the bandwidth of the controller. r is the set value of y (the reference input value is 0 under steady-state conditions).

3.2. DDPG-Based Parameter Adjuster

The DDPG adjuster depicted in this paper is defined as a quadruple [$\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R}$]. It accepts the state information of the i-th region, optimizes the control parameters ($w_c,{\beta }_1,{\beta }_2,b_0$), and feeds them back to ADRC. The structure of the LFC method based on DDPG-ADRC is shown in Figure 5.

The state space $\mathcal{S}$ includes the actual frequency deviation $\mathsf{\Delta }{f}_i$, the target frequency deviation $\mathsf{\Delta }{f}_i^{*}$, the actual power deviation $\mathsf{\Delta }{P}_{tiei}$, the target power deviation $\mathsf{\Delta }{P}_{tiei}^{*}$, the actual area control error $AC{E}_i$, and the target area control error $AC{E}_i^{*}$.

The state space $\mathcal{A}$ includes the actual bandwidth $w_c$ of ADRC, the target bandwidth $w_c^{*}$ of ADRC, the actual observer gains ${\beta }_1$ and ${\beta }_2$, the target observer gains ${\beta }_1^{*}$ and ${\beta }_2^{*}$, the actual system gain $b_0$, and the target system gain $b_0^{*}$.

$\mathcal{T}$ is a dynamic update function that summarizes the relationship from the current state to the following state. The output parameters of DDPG are introduced into ADRC to obtain state variables through LESO.

The goal of the DDPG parameter adjuster is to minimize $\mathsf{\Delta }{f}_i^{*},\mathsf{\Delta }{P}_{tiei}^{*},AC{E}_i^{*}$. Therefore, the reward function $\mathcal{R}$ can be defined as

$$r\left(t\right)=-\sum _{i=1}^T[\left|B_i\mathsf{\Delta }{f}_i\right|+\left|\mathsf{\Delta }{P}_{tiei}\right|].$$

(30)

The DDPG used in this paper includes an actor network and a critic network. The actor network can generate control actions, while the critic network can evaluate the value of the actions.

The input layer of the actor network has a state-space dimension of 6. Its hidden layers consist of two fully connected layers with 256 neurons evenly distributed across the two layers, and the activation function used is ReLU. The output layer of the actor network has an action space dimension of 8, with the activation function being tanh, and the output range constrained to [−1, 1]. The input layer of the critic network comprises 14 units, which is the sum of the state-space dimension and the action space dimension. Its hidden layers consist of two fully connected layers, with the first layer containing 192 neurons and the second layer containing 64 neurons, employing ReLU as the activation function. The output layer is one-dimensional with no activation function. A schematic diagram of the network structure is provided in Figure 6.

The actor strategy network is $\lambda \left(s_t\right|{\delta }^{\lambda })$. When updating the actors, noise is also taken into account. The output of the actor network $a_t$ is expressed as

$$a_t=\lambda \left(s_t\right|{\delta }^{\lambda })+noise=\lambda (\mathsf{\Delta }{f}_i,\mathsf{\Delta }{f}_i^{*},\mathsf{\Delta }{P}_{tiei},\mathsf{\Delta }{P}_{tiei}^{*},AC{E}_i,AC{E}_i^{*}|{\delta }^{\lambda })+noise,$$

(31)

where $s_t$ is the current state.

The critic network is $Q(s_t,a_t|{\delta }^Q)$, which can be expressed as

$$Q(s_t,a_t|{\delta }^Q)=Q(\mathsf{\Delta }{f}_i,\mathsf{\Delta }{f}_i^{*},\mathsf{\Delta }{P}_{tiei},\mathsf{\Delta }{P}_{tiei}^{*},AC{E}_i,AC{E}_i^{*},a_t|{\delta }^Q).$$

(32)

The evaluation value $p_i$ of the target Q network is represented as

$$p_i=r_i+\epsilon Q^{\prime }[s_{i+1},{\lambda }^{\prime }\left(s_{i+1}\right|{\delta }^{{\lambda }^{\prime }})|{\delta }^{Q^{\prime }}],$$

(33)

where $\epsilon$ is the discount factor and $\epsilon \ll 1$. $r_i$ represents the reward, which is (30).

The update of the critic network is achieved by minimizing the error between the target network and the actual network. The loss function is

$$D={\displaystyle \frac{1}{M}}\sum _{i=1}^M{[p_i-Q(s_i,a_i|{\delta }^Q)]}^2.$$

(34)

The goal of the actor network update is to maximize $J={\displaystyle \sum _{t=0}^T}r\left(t\right)$. The following policy gradient is used to update the parameters of the actor network:

$${\nabla }_{\delta }J\approx {\displaystyle \frac{1}{M}}\sum _{i=1}^M[{\nabla }_{a_t}Q(s_t,a_t|{\delta }^Q){|}_{s=s_i,a=\lambda \left(s_i\right|{\delta }^{\lambda })}{\nabla }_{{\delta }^{\lambda }}\lambda \left(s_t\right|{\delta }^{\lambda }){|}_{s=s_i}].$$

(35)

3.3. Parameter Optimization

The parameter optimization diagram is presented in Figure 7. The algorithm self-optimization is achieved through updates ${\delta }^{\lambda }$ and ${\delta }^Q$. To make the algorithm learn better, the parameter update process is split with ${\lambda }^{\prime }\left(s_t\right|{\delta }^{{\lambda }^{\prime }})$ and $Q^{\prime }(s_t,{\lambda }^{\prime }\left(s_t\right|{\delta }^{{\lambda }^{\prime }})|{\delta }^{Q^{\prime }})$. The parameters ${\delta }^{\lambda }$ and ${\delta }^Q$ are slowly tracked by the learning network

$$\left\{\begin{array}{c}{\delta }^{Q^{\prime }}\leftarrow \mu {\delta }^Q+(1-\mu ){\delta }^{Q^{\prime }}\hfill \\ {\delta }^{{\lambda }^{\prime }}\leftarrow \mu {\delta }^{\lambda }+(1-\mu ){\delta }^{{\lambda }^{\prime }},\hfill \end{array}\right.$$

(36)

where $\mu$ is the parameter update rate and $\mu \le 1$.

To make the optimized parameters feasible, we have

$$\left\{\begin{array}{c}{\tilde{\delta }}^i={\displaystyle \frac{{\delta }^i-{\delta }_{min}^i}{{\delta }_{max}^i}}\hfill \\ {\delta }^i={\delta }_{max}^i{\tilde{\delta }}^i+{\delta }_{min}^i\hfill \\ {\tilde{\delta }}_{k+1}^i={\tilde{\delta }}_k^i+a_i,\hfill \end{array}\right.$$

(37)

$${\tilde{\delta }}^i=\left\{\begin{array}{c}{\displaystyle \frac{0.05({\tilde{\delta }}^i+1)}{1.05}},\tilde{\delta }<0.05\hfill \\ {\tilde{\delta }}^i,0.05\ge \tilde{\delta }\le 0.95\hfill \\ {\displaystyle \frac{1-0.05(2-{\tilde{\delta }}^i)}{1.05}},0.95<\tilde{\delta },\hfill \end{array}\right.$$

(38)

where ${\tilde{\delta }}^i$ is the correction value of the i-th parameter. ${\delta }^i$ is the initial value of the i-th parameter. ${\delta }_{max}^i$ is the upper limit of the i-th parameter. ${\delta }_{min}^i$ is the lower limit of the i-th parameter. All parameters and parameter correction actions are normalized to the range of [0, 1] to prevent parameters from exceeding boundaries.

The training process terminates when any one of the predefined convergence criteria is satisfied. The first criterion requires the average reward fluctuation to remain below 1% for 50 consecutive training rounds, as calculated by the reward function (30). Alternatively, training stops when the total number of steps reaches 200,000. A supplementary condition mandates that the average control error must maintain a value below 0.05 for 100 consecutive rounds. To ensure thorough optimization and avoid premature convergence to local optima, the training continues for a minimum of 1000 iterations regardless of other conditions being met earlier.

4. Simulation Results

In general, the RES generation part is regarded as a perturbation of the system. However, its large-scale access will generate a significant effect on the power balance and frequency stability. The introduction of ES devices can solve this problem, and ES can provide a buffer when power fluctuates, helping to maintain the stability of the system’s frequency. The load adjustment is compensated through the disturbances observed by the ADRC, the adaptive learning of DDPG, and the collaborative control of ES devices. As presented in Figure 8, the test was performed on an IEEE 39-bus power network with 10-generators. The three-region interconnected power system is composed of thermal power units, RESs, and ES units. The specific composition of the three-region model is the following: region 1 contains thermal power, wind power, and SMES; region 2 contains thermal power, PV power generation, and CES; and region 3 contains thermal power, wind power, PV power generation, CES, and SMES.

This study uses MATLAB/Simulink R2023a as the primary modeling tool, combined with Python 3.8 to implement deep learning algorithms. An IEEE 39-bus power system is established in MATLAB, and an ADRC is constructed based on S-Function. The construction of the DDPG algorithm is realized through the Tensor Flow 2.6 framework in Python, utilizing the Keras interface to build an actor–critic network. Python is utilized for the training of the DDPG algorithm and the output of policies, while Simulink accepts control signals and provides feedback on system status. The two platforms engage in real-time data interchange, creating a closed-loop simulation process.

4.1. Three-Region Interconnected Power System Without ES

The input signal is set as the load disturbance change signal; the output signal is set as $\mathsf{\Delta }{f}_i$ and $\mathsf{\Delta }{P}_{tie}^{ij}$. The system frequency is controlled by DDPG optimizing the parameters of the first-order ADRC. The four methods (i.e., PID, PI, ADRC, GA-ADRC) are compared to illustrate the good performance of the DDPG. For the power system model used in this paper, a 0.01 p.u. load disturbance is used in the three-region power system at $t=0$. The first-order ADRC designed above is used for LFC, where the controller is applied to each power system region and the response of system frequency when the load changes is observed. After 100 iterations, the relevant parameters of the first-order ADRC are listed in

Table 1. Optimized ADRC parameters.

Symbol	${\mathit{\beta }}_{\mathbf{1}}^{\mathbf{*}}$	${\mathit{\beta }}_{\mathbf{2}}^{\mathbf{*}}$	${\mathit{w}}_{\mathit{c}}^{\mathbf{*}}$	${\mathit{b}}_{\mathbf{0}}^{\mathbf{*}}$
Value	75.62	37.21	4	150

. Power system parameters are considered as listed in

Table 2. System parameters.

Symbol	${\mathit{R}}_{\mathit{i}}$	${\mathit{B}}_{\mathit{i}}$	${\mathit{K}}_{\mathit{p}}$	${\mathit{K}}_{\mathit{g}}$	${\mathit{K}}_{\mathit{T}}$	${\mathit{K}}_{\mathit{r}}$	${\mathit{K}}_{\mathit{i}\mathit{j}}$	${\mathit{T}}_{\mathit{p}\mathit{i}}$	${\mathit{T}}_{\mathit{t}\mathit{i}}$	${\mathit{T}}_{\mathit{g}\mathit{i}}$	${\mathit{T}}_{\mathit{r}\mathit{i}}$
Value	2.4	3	120	0.58	10	0.1	0.2	20	0.3	0.08	0.03

. The training parameters of DDPG are outlined in

Table 3. Training parameter values.

Parameters	Learning Rate (Actor)	Learning Rate (Critic)	Noise	Batch Size	$\mathit{\mu }$	$\mathit{\epsilon }$
Value	${10}^{-4}$	$3\times {10}^{-4}$	0.2	64	${10}^{-3}$	0.99

.

Wind speed was measured in the Zafarana region of Egypt for practical application. The wind speed is constantly changing between approximately 5.1 and 15.8 m/s, and the average value is 11 m/s. The output power of PV modules is mainly related to light intensity and temperature, and real PV data is used in the system. Temperature and solar irradiance data were extracted for a full day; these temperature and solar radiation data were collected at Site Benban in Egypt. The power curves of wind turbines and PV are shown in Figure 9. The installed capacity of the synchronous generator was set based on the configuration of conventional thermal power units, with each unit rated at 100 MW. Region 1 has a wind energy installed capacity of 60 MW, region 2 has a PV installed capacity of 30 MW, and region 3 has a wind energy installed capacity of 60 MW and 30 MW of PV. The parameter selection for wind turbines was based on [45], with real-time adjustments according to frequency response, rather than a fixed average value, and needed to comply with the scale of the wind farm set in the research as well as the control strategies employed. The parameter values of wind turbines are listed in

Table 4. Wind turbine parameters.

Symbol	$\mathit{\rho }$	${\mathit{A}}_{\mathit{b}}$	${\mathit{V}}_{\mathit{s}}$	${\mathit{R}}_{\mathit{t}}$	${\mathit{\omega }}_{\mathit{t}}$	$\mathit{\alpha }$	$\mathit{\mu }$	${\mathit{C}}_{\mathit{f}}$
Value	1.225 $\mathrm{kg}/{\mathrm{m}}^3$	5905 ${\mathrm{m}}^2$	10 m/s	50 m	15 rad/min	3.6	8	0.421

. The frequency deviation of each region and the power deviation of the liaison line between each region are shown in Figure 10.

This paper utilizes synchronized phasor measurement units for real-time sampling, employing zero-crossing detection to capture the zero-crossing points of the voltage waveform to calculate the period T, and then determining the frequency value using $f=1/T$. The sampling frequency is set at 10 kHZ, with 200 sampling points per power frequency cycle, meeting the Nyquist sampling theorem requirements for 50/60 HZ systems. According to the cyclic measurement method, the theoretical resolution under the rated frequency is $\mathsf{\Delta }f=1/\left(N{T}_s\right)$, where $N=200$, with a sampling period of $T_s=0.1$ s, resulting in $\mathsf{\Delta }f\approx 0.05$ HZ. The simulation results indicate that the absolute error in frequency measurement is less than or equal to 0.05 HZ, which meets the requirements for measurement accuracy.

From the above experimental results, it can be seen that several methods can accurately respond to the load change when the system load is perturbed. From the dynamic response curve, when RESs are connected, the transient response increases rapidly due to its overproduction. After that, the designed first-order ADRC processes the situation and returns the response to a normal value so that it can play a positive role in the LFC. Each method can effectively adjust the system frequency deviation to 0. However, compared to the other two methods, the designed first-order ADRC can effectively stabilize the deviation between each region. In a short time, the first-order ADRC has a faster response time and less overshooting, showing good control effect and anti-interference ability. Moreover, comparing the ADRC using conventional manual tuning with the ADRC optimized through traditional genetic algorithms, the DDPG-optimized ADRC demonstrates a stronger disturbance suppression capability and the shortest recovery time for stability, which can significantly enhance system robustness. The comparison of steady-state performance and dynamic response performance indicators is shown in

Table 5. Performance comparison of control methods under load disturbance.

Index	Region	Traditional Methods				DDPG-ADRC	Improvement
		PID	PI	ADRC	GA-ADRC		(%)
Rise time (s)	$\mathsf{\Delta }{f}_1$	2.30	2.25	2.10	2.05	1.25	40.5
	$\mathsf{\Delta }{f}_2$	2.45	2.40	2.25	2.20	1.35	40.0
	$\mathsf{\Delta }{f}_3$	2.20	2.15	2.05	1.95	1.20	41.5
	$\mathsf{\Delta }{P}_{tie}^{12}$	0.95	0.90	0.85	0.80	0.45	47.1
Settling time (s)	$\mathsf{\Delta }{f}_1$	9.20	8.80	8.50	8.30	4.20	45.8
	$\mathsf{\Delta }{f}_2$	9.60	9.30	9.00	8.70	4.50	45.8
	$\mathsf{\Delta }{f}_3$	8.80	8.50	8.20	8.00	4.00	46.2
	$\mathsf{\Delta }{P}_{tie}^{13}$	4.20	4.00	3.80	3.60	1.90	45.8
Overshoot (%)	$\mathsf{\Delta }{f}_1$	21.5	20.0	18.5	16.0	4.2	60.0
	$\mathsf{\Delta }{f}_2$	22.2	20.8	19.2	16.5	4.5	60.2
	$\mathsf{\Delta }{f}_3$	20.5	19.0	17.8	15.2	3.8	60.1
	$\mathsf{\Delta }{P}_{tie}^{23}$	14.5	13.2	12.0	10.5	2.5	60.0
ACE	1	0.98	0.92	0.85	0.78	0.32	62.4
	2	1.05	0.98	0.92	0.85	0.35	62.0
	3	0.92	0.85	0.78	0.70	0.28	64.1

. Obviously, the above experimental outcomes indicate that the first-order ADRC optimization approach can be used for the control of RESs and ES units to further verify the control effect.

4.2. Performance Evaluation of the System Under Communication Delays

The aforementioned experimental scenario only considers the impact of the integration of renewable energy on system performance. In this section, we will include additional uncertainty factors for simulation. Considering the impact of communication delays, taking the previous two areas as an example, we introduced a random delay of 100–500 ms into the control loop to test the robustness of DDPG-ADRC. The results of the experiment are illustrated in Figure 11.

The results indicate that the expanded state observer of ADRC can estimate the delay effects in real time. The frequency deviation of DDPG-ADRC is reduced by 45% compared to the other two proposed algorithms. When the time delay is ≤500 ms, the frequency deviation can still be maintained within ±0.02 HZ, indicating that communication delays do not have a significant impact on frequency deviation.

Furthermore, to verify the impact of $\mathsf{\Delta }{P}_{tiei}$ on the experimental results presented in this article, two regions are taken as examples. Scenario 1 is based on the original model, ignoring $\mathsf{\Delta }{P}_{tiei}$. Scenario 2 considers the dynamics of $\mathsf{\Delta }{P}_{tiei}$. The experimental results are illustrated in Figure 12.

The results indicate that in both scenarios, the frequency deviation of DDPG-ADRC is less than 3%, and the overshoot variation does not exceed 5%. This suggests that, under the current system parameters, ignoring the impact of $\mathsf{\Delta }{P}_{tiei}$ on control performance is within an acceptable range.

4.3. System Performance Evaluation with ES

In this subsection, we have added a comparison of four other algorithms (chaos-optimized-FOPID, MPC, generalized-ADRC, SAC-ADRC). The capacity of SMES is 30 MW, and the capacity of CES is 3.5 MW. The capacity selection reference for SMES is not merely an average value [41]. The choice of parameters is the result of adjustments based on the disturbance assumptions and control objectives of the research scenario. The selection of CES parameters is similar to that of SMES. The parameters of ES are listed in

Table 6. ES device parameters.

Symbol	${\mathit{K}}_{\mathit{i}\mathit{f}}$	${\mathit{\eta }}_{\mathit{c}}$	$\mathit{\delta }$	h	${\mathit{d}}_{\mathbf{0}}$	${\mathit{d}}_{\mathit{i}}$
Value	0.4 kV/kA	0.18	1.8 $\mathrm{W}/{\mathrm{m}}^3$	0.625 m	2.412 m	0.517 m
Symbol	$n^{\prime }$	$I_0$	$R_{jr}$	$K_{\alpha }$	L	$T_{sc}$
Value	2	5000 A	150 $\mathsf{\Omega }$	100 KV/p.u. MW	3.4 H	0.3 s
Symbol	c	$R^{\prime }$	$T_c$	$K_{cf}$	$K_{vd}$
Value	1.5 F	80 $\mathsf{\Omega }$	0.05 s	100 KA/p.u. MW	0.25 KA/KV

. After the addition of the ES units, the response fluctuation is always significantly suppressed, and the ACE response curves after adding ES units are shown in Figure 13 and Figure 14. The dynamic response indicators of different algorithms are shown in

Table 7. Performance comparison of control methods under different algorithms.

Index	Region	Other Methods				DDPG-ADRC	Improvement
		Chaos- optimized- FOFID	MPC	Geralized ADRC	SAC- ADRC		(%)
Rise time (s)	$\mathsf{\Delta }{f}_1$	1.95	1.90	1.85	1.80	1.25	30.6
	$\mathsf{\Delta }{f}_2$	2.10	2.05	2.00	1.95	1.35	30.8
	$\mathsf{\Delta }{f}_3$	1.85	1.80	1.75	1.70	1.20	29.4
	$\mathsf{\Delta }{P}_{tie}^{12}$	0.70	0.65	0.60	0.55	0.45	18.2
Settling time (s)	$\mathsf{\Delta }{f}_1$	7.80	7.50	7.20	7.00	4.20	40.0
	$\mathsf{\Delta }{f}_2$	8.10	7.80	7.50	7.30	4.50	38.4
	$\mathsf{\Delta }{f}_3$	7.20	6.90	6.60	6.40	4.00	37.5
	$\mathsf{\Delta }{P}_{tie}^{13}$	3.30	3.10	2.90	2.70	1.90	29.6
Overshoot (%)	$\mathsf{\Delta }{f}_1$	14.5	13.8	13.0	12.2	4.2	65.6
	$\mathsf{\Delta }{f}_2$	15.2	14.5	13.8	13.0	4.5	65.4
	$\mathsf{\Delta }{f}_3$	13.8	13.0	12.2	11.5	3.8	67.0
	$\mathsf{\Delta }{P}_{tie}^{23}$	9.5	9.0	8.5	8.0	2.5	68.8
ACE	1	0.72	0.68	0.65	0.60	0.32	46.7
	2	0.78	0.74	0.70	0.65	0.35	46.2
	3	0.65	0.62	0.58	0.54	0.28	48.1

.

As can be seen from Figure 13, compared to other algorithms, DDPG-ADRC exhibits significant advantages in both dynamic response and steady-state accuracy. Under the same load disturbance conditions, the average rise time of the dynamic response is reduced by 27.25%, and the overshoot is decreased by approximately 66.7%. The dynamic response curve in Figure 14 indicates that without the integration of ES, the frequency fluctuations exhibit a distinct long-term oscillation pattern. However, after integrating ES with an optimized controller, the oscillations rapidly diminish, and the system returns to steady-state operation in a short period. Therefore, the addition of ES devices can improve the entire power system’s performance. The above results fully demonstrate the capability of ES and advanced control strategies to collaboratively address the uncertainties in interconnected power systems.

5. Conclusions

An LFC method for an RES multi-region interconnected power system was studied to improve the power system’s stability and reliability. Specifically, by focusing on system model structure and control strategies, the LFC scheme with RESs was optimized and innovated to enhance system performance. The fundamental modules of the LFC loops were established, and the RESs and ES units were modeled. Throughout the process, a practical ADRC was created based on the power system model and ADRC principles to enhance the system’s anti-interference. The optimal parameters of ADRC were tuned using DDPG. To enhance the LFC ability and support the integration of RESs into the grid, real data were collected from the test site and simulated at the actual nodes of the power network. The simulation results indicate that the proposed method maintains the frequency deviation within 0.02 HZ under a 0.01 p.u. load disturbance, reducing it by 23% compared to the standalone ADRC control and improving it by 18% compared to DDPG. Meanwhile, when integrating RESs, the overshoot remains below 0.15 HZ. This method significantly enhances the robustness of frequency regulation in the system through the coupling mechanism of reinforcement learning and disturbance-resistant control. The method and model studied in this paper assume that the forecasting error of the RES output is within 20%, without considering the excessively large forecasting errors that extreme weather conditions in real scenarios may cause, which could in turn affect the strategy optimization performance of the DDPG algorithm. Currently, validation has only been conducted through software simulation, and the hardware compatibility of the controller has not been tested on a real-time digital simulation platform. In the future, a distributed computing architecture will be adopted to transform the centralized DDPG algorithm into a multi-agent reinforcement learning framework to address more uncertainty issues. Various performances of the controller will be tested on the real-time digital simulation platform.