Adaptive Active Disturbance Rejection Load Frequency Control for Power System with Renewable Energies Using the Lyapunov Reward-Based Twin Delayed Deep Deterministic Policy Gradient Algorithm

Abstract

The substitution of renewable energy sources (RESs) for conventional fossil fuels in electricity generation is essential in addressing environmental pollution and resource depletion. However, the integration of RESs in the load frequency control (LFC) of power systems can have a negative impact on frequency deviation response, resulting in a decline in power quality. Moreover, load disturbances can also affect the stability of frequency deviation. Hence, this paper presents an LFC method that utilizes the Lyapunov reward-based twin delayed deep deterministic policy gradient (LTD3) algorithm to optimize the linear active disturbance rejection control (LADRC). With the advantages of being model-free and mitigating unknown disturbances, LADRC can regulate load disturbances and renewable energy deviations. Additionally, the LTD3 algorithm, based on the Lyapunov reward function, is employed to optimize controller parameters in real-time, resulting in enhanced control performance. Finally, the LADRC-LTD3 is evaluated using a power system containing two areas, comprising thermal, hydro, and gas power plants in each area, as well as RESs such as a noise-based wind turbine and photovoltaic (PV) system. A comparative analysis is conducted between the performance of the proposed controller and other control techniques, such as integral controller (IC), fractional-order proportional integral derivative (FOPID) controller, I-TD, ID-T, and TD3-optimized LADRC. The results indicate that the proposed method effectively addresses the LFC problem.

1. Introduction

Maintaining system frequency within an acceptable range is a crucial requirement in power system operation, and load frequency control (LFC) plays a vital role in achieving this objective. LFC ensures the balance between active power generation and load [1]. Ensuring the safe operation of the system, particularly in the face of load disturbances, necessitates the effective regulation of system frequency. Hence, this study focuses on addressing the load frequency control problem. The evolving landscape of modern power systems has witnessed substantial transformations in recent years. These changes are driven by the growing diversity of loads and the escalating adoption of renewable energy sources (RESs) as alternatives to traditional power plants, addressing their inherent limitations [2]. However, integrating RESs can lead to reduced system stability and increased frequency deviation. As a result, it is crucial to address the challenges posed by load disturbance and RESs and achieve efficient load frequency control.

Currently, there are numerous power system models being studied. These models can be broadly categorized into one area [3,4,5] and multiple areas [6,7,8], based on the number of areas involved. Among these models, thermal plants [9], hydro plants [10], and natural gas plants [11] are the most common. Moreover, in the past few years, RESs, particularly wind turbines [12] and solar energy [13], have been an extensive research subject. Bakeer et al. [14] provides an extensive discussion on the first-order transfer function model [15], second-order transfer function model [16], and noise-based model [17] of photovoltaic (PV) systems. In a similar vein, for wind turbines, researchers have also explored the first-order transfer function [18], the second-order wind farm with high-voltage direct current (HVDC) line [19], the noise-based model [20], and the doubly fed induction generator (DFIG)-driven wind power system [21]. Except for investigating various energy sources, researchers introduced nonlinear components such as governor deadband (GDB) and generation rate constraint (GRC) into the system to enhance its resemblance to real-world power systems.

In recent years, intelligent control methods that integrate the control system with optimization algorithms have gained popularity for load frequency control. Among these methods, the proportional–integral–derivative (PID) control remains the dominant choice for controllers [22,23,24]. However, the performance of the control system obtained by ordinary PID is poor. Therefore, many improved control methods based on PID have been proposed. For example, Guha et al. [21] proposed a cascade fractional-order controller (CC-FOC) consisting of three-degrees-of-freedom PID (3DOF-PID) and tilt–integral–derivative (TID) controllers, and implemented it on a power system containing three areas with DFIG and conducted performance comparison against a conventional PID. In addition, various control strategies such as sliding mode control [25,26], model predictive control [27], and linear active disturbance rejection control (LADRC) [28] have also been presented as effective solutions for LFC. In particular, the LADRC controller features a simple structure, model-free operation, and the capability to leverage the extended state observer (ESO) to reckon undefined disturbances [29]. As a result, it discovered extensive applications surpassing LFC, such as the servo system [30], motion control [31,32], and control of the liquid level [33]. LADRC is primarily employed for load frequency control in power systems that do not incorporate renewable energy sources. However, in this paper, LADRC is applied to power systems that feature renewable energy sources.

The optimization algorithms currently used in load frequency control systems mainly rely on heuristic methods, such as the grasshopper optimization algorithm (GOA) [34], dragonfly search algorithm (DSA) [35], and whale optimization algorithm [36], etc. However, these algorithms usually only provide the optimal parameters for a specific scenario, which implies that they cannot obtain adaptive parameters. The problem can be addressed by utilizing deep reinforcement learning (DRL). Deep learning algorithms have gained prominence in artificial intelligence, primarily due to their ability to acquire optimal policies for solving sequential decision-making problems [37,38]. Numerous research studies have explored the utilization of DRL for LFC in power systems. For example, Khalid et al. [39] tuned the PID controller parameters by a twin delayed deep deterministic policy gradient (TD3) algorithm for a two-area interconnected power system, where each area contains a TD3 agent. Zheng et al. [40] proposed a soft actor–critic (SAC) algorithm that optimized the LADRC method and achieved application to LFC in power systems. In these studies, the design of the reward function lacked the corresponding theoretical analysis. As a result, there is no unified and efficient design rule for the reward function in DRL algorithms. Drawing inspiration from the Lyapunov reward function [41,42], this paper presents the LTD3 based on the Lyapunov function and TD3 algorithm, which enables real-time optimization of LADRC controller parameters.

The main contributions are summarized below,

• A Lyapunov-based reward function is constructed for LFC systems to improve the convergence of the TD3, namely LTD3.
• The design of LADRC aims to mitigate the effects of load disturbances and the integration of RESs in power systems.
• The proposed LTD3 algorithm is utilized to dynamically adjust the parameters ${\omega }_o$ and ${\omega }_c$ of the LADRC by observing the system state. To evaluate its performance, a comparative analysis is conducted with other techniques such as IC, FOPID, and ID-T [20,43,44]. The evaluations are conducted on a two-area power system incorporating a noise-based wind turbine and PV system, considering different scenarios.

The paper is mainly organized as below. Section 2 presents the mathematical model of a general power system. Section 3 explains the process of the LADRC for LFC. Section 4 introduces the LTD3-based LADRC parameter optimization method. Section 5 showcases simulation results to demonstrate the validation of the proposed method. Lastly, Section 6 concludes the paper and offers suggestions for future work.

2. Multi-Source Power System Modeling

Currently, the majority of electricity generation in most countries is derived from thermal, hydro, and gas power plants, which are utilized to fulfill the base load demand. Further, renewable energy sources are also used to meet load demand. Therefore, this paper considers a multi-source power system comprising thermal, hydro, and gas turbines and renewable energies, as illustrated in Figure 1.

Figure 1 exhibits a general structure of one area in an n-area interconnected power system, denoted as i-th area. It illustrates the correlation between $\mathrm{\Delta }{f}_i$, ${\mathit{ACE}}_i$, and $\mathrm{\Delta }{P}_{\mathit{tiei}}$. And the mathematical expression is shown in Equation (1),

$${\mathit{ACE}}_i=B_i\mathrm{\Delta }{f}_i+\mathrm{\Delta }{P}_{\mathit{tiei}}$$

(1)

where

$$\left\{\begin{array}{c}\mathrm{\Delta }{f}_i=\left(\mathrm{\Delta }{P}_{\mathit{tpi}}+\mathrm{\Delta }{P}_{\mathit{hpi}}+\mathrm{\Delta }{P}_{\mathit{gpi}}+\mathrm{\Delta }{P}_{\mathit{rei}}-\mathrm{\Delta }{P}_{\mathit{di}}-\mathrm{\Delta }{P}_{\mathit{tiei}}\right)G_{\mathit{pi}}\hfill \\ \mathrm{\Delta }{\dot{P}}_{\mathit{tiei}}=\sum _{j=1\left(j\ne i\right)}^{n}2\pi T_{\mathit{ij}}\left(\mathrm{\Delta }{f}_i-\mathrm{\Delta }{f}_j\right)\hfill \end{array}\right.$$

(2)

Generally, the generator of the power system is usually presented by the following transfer function:

$$G_{\mathit{pi}}=\frac{1}{2\mathit{Hs}+D}$$

(3)

The transfer functions of thermal power plants $G_{\mathit{th}}$, hydro turbine plants $G_{\mathit{hp}}$, gas turbine plants $G_{\mathit{gp}}$, and renewable energy sources $\mathrm{\Delta }{P}_w$, $\mathrm{\Delta }{P}_{\mathit{Solar}}$ are introduced in Section 2.1, Section 2.2, Section 2.3, Section 2.4 and Section 2.5, respectively, as referenced in [20]. It is worth noting that the mechanism construction process of the power system model is not considered in this article. Additionally, referring to Figure 1, we can derive the following expression:

$$\left\{\begin{array}{c}\mathrm{\Delta }{P}_{\mathit{tpi}}=G_{\mathit{th}}\mathrm{\Delta }{P}_{c1i}\hfill \\ \mathrm{\Delta }{P}_{\mathit{hpi}}=G_{\mathit{hp}}\mathrm{\Delta }{P}_{c2i}\hfill \\ \mathrm{\Delta }{P}_{\mathit{gpi}}=G_{\mathit{gp}}\mathrm{\Delta }{P}_{c3i}\hfill \end{array}\right.$$

(4)

where $\mathrm{\Delta }{P}_{c1i}$, $\mathrm{\Delta }{P}_{c2i}$, and $\mathrm{\Delta }{P}_{c3i}$ are related to the controller output $u_i$, and also according to Figure 1, their expressions can be obtained as

$$\left\{\begin{array}{c}\hfill \mathrm{\Delta }{P}_{c1i}=u_i-\frac{\mathrm{\Delta }{f}_i}{R_{i1}}\\ \hfill \mathrm{\Delta }{P}_{c2i}=u_i-\frac{\mathrm{\Delta }{f}_i}{R_{i2}}\\ \hfill \mathrm{\Delta }{P}_{c3i}=u_i-\frac{\mathrm{\Delta }{f}_i}{R_{i3}}\end{array}\right.$$

(5)

With Equations (2), (4), and (5), the new expression for $\mathrm{\Delta }{f}_i$ can be organized as

$$\mathrm{\Delta }{f}_i=\frac{\left(G_{\mathit{th}}+G_{\mathit{hp}}+G_{\mathit{gp}}\right)G_{\mathit{pi}}}{1+\left(\frac{G_{\mathit{th}}}{R_{i1}}+\frac{G_{\mathit{hp}}}{R_{i2}}+\frac{G_{\mathit{gp}}}{R_{i3}}\right)G_{\mathit{pi}}}{u}_i+\frac{\left(\mathrm{\Delta }{P}_{\mathit{rei}}-\mathrm{\Delta }{P}_{\mathit{di}}-\mathrm{\Delta }{P}_{\mathit{tiei}}\right)G_{\mathit{pi}}}{1+\left(\frac{G_{\mathit{th}}}{R_{i1}}+\frac{G_{\mathit{hp}}}{R_{i2}}+\frac{G_{\mathit{gp}}}{R_{i3}}\right)G_{\mathit{pi}}}$$

(6)

Thus, combining Equations (1) and (6), the expression about ${\mathit{ACE}}_i$ and $u_i$ of the i-$\mathit{th}$ area can be obtained as

$$\begin{array}{c}\hfill {\mathit{ACE}}_i=\frac{B_i\left(G_{\mathit{th}}+G_{\mathit{hp}}+G_{\mathit{gp}}\right)G_{\mathit{pi}}}{1+\left(\frac{G_{\mathit{th}}}{R_{i1}}+\frac{G_{\mathit{hp}}}{R_{i2}}+\frac{G_{\mathit{gp}}}{R_{i3}}\right)G_{\mathit{pi}}}{u}_i+\frac{B_i\left(\mathrm{\Delta }{P}_{\mathit{rei}}-\mathrm{\Delta }{P}_{\mathit{di}}-\mathrm{\Delta }{P}_{\mathit{tiei}}\right)G_{\mathit{pi}}}{1+\left(\frac{G_{\mathit{th}}}{R_{i1}}+\frac{G_{\mathit{hp}}}{R_{i2}}+\frac{G_{\mathit{gp}}}{R_{i3}}\right)G_{\mathit{pi}}}+\mathrm{\Delta }{P}_{\mathit{tiei}}\end{array}$$

(7)

2.1. Thermal Power Plant

In this paper, we consider a thermal power that includes the reheat turbine and additionally incorporates nonlinear links of GDB and GRC, shown in Figure 2. The GRC is taken as $3\%$ pu/min, while the transfer function of GDB is

$$\mathit{GDB}=0.8-\frac{0.2}{\pi }s$$

(8)

And there is the transfer function of the thermal power plant as

$$G_{\mathit{th}}=\frac{\mathrm{\Delta }{P}_{\mathit{tp}}}{\mathrm{\Delta }{P}_{c1}}=\frac{\mathit{KT}\left(0.8-0.2s/\pi \right)\left(K_r{T}_{r}s+1\right)}{\left(T_{g}s+1\right)\left(T_{t}s+1\right)\left(T_{r}s+1\right)}$$

(9)

2.2. Hydro Power Plant

The diagram expressing the transfer function of a hydroelectric power unit is illustrated in Figure 3, consisting of a governor, a transient droop compensation, and a penstock turbine with GRC constraints. The GRC for upward adjustments is 270% pu/min, while for downward adjustments, it is 360% pu/min.

Then, there is the transfer function of the hydropower plant.

$$G_{\mathit{hp}}=\frac{\mathrm{\Delta }{P}_{\mathit{hp}}}{\mathrm{\Delta }{P}_{c2}}=\frac{\mathit{KH}\left(T_{\mathit{rs}}s+1\right)\left(-T_{w}s+1\right)}{\left(T_{\mathit{gh}}s+1\right)\left(T_{\mathit{rh}}s+1\right)\left(0.5T_{w}s+1\right)}$$

(10)

2.3. Gas Power Plant

Similarly, the gas system, shown in Figure 4, has the following expression:

$$G_{\mathit{gp}}=\frac{\mathrm{\Delta }{P}_{\mathit{gp}}}{\mathrm{\Delta }{P}_{c3}}=\frac{\mathit{KG}\left(X_{g}s+1\right)\left(-T_{\mathit{cr}}s+1\right)}{\left(B_{g}s+C_g\right)\left(Y_{g}s+1\right)\left(T_{f}s+1\right)\left(T_{\mathit{cd}}s+1\right)}$$

(11)

2.4. Noise-Based Wind Turbine Generator

This paper examines wind energy as a form of renewable energy. Wind speed ($V_w$ in m/s) affects the performance of wind energy systems. Figure 5a shows the wind speed model based on noise, which is used to simulate unstable wind speed in the actual environment. Based on the noise-based wind speed, the mechanical power output of the wind energy system can be calculated according to Equation (12),

$$P_w=\frac{1}{2}\rho A_r{C}_p{V^{\prime }}_w^3$$

(12)

where $\rho (\mathrm{kg}/{\mathrm{m}}^3)$ refers to air density, $A_r\left({\mathrm{m}}^2\right)$ denotes blade swept area. The rotor power coefficient $C_p$ is expressed by Equation (13).

$$\left\{\begin{array}{c}{C}_p=c_1\left(\frac{c_2}{{\lambda }_I}-c_3\beta -c_4\right)e^{-\frac{c_5}{{\lambda }_I}}+c_6\lambda \hfill \\ \frac{1}{{\lambda }_I}=\frac{1}{\lambda +0.08\beta }-\frac{0.035}{{\beta }^3+1}\hfill \end{array}\right.$$

(13)

Figure 5b illustrates the connection between $C_p$, $\lambda$, and $\beta$. And the diagram highlights a maximum value of $C_p$. Additionally,

Table 1. Parameter settings for wind turbine generator.

Parameter	Value	Parameter	Value
Rated capacity $P_w$	750 Kw	$c_1$	0.5176
$V_w$	15 m/s	$c_2$	116
$\rho$	1.225 $\mathrm{kg}/{\mathrm{m}}^3$	$c_3$	0.4
$A_r$	1684 ${\mathrm{m}}^2$	$c_4$	5
$\lambda$	8.68 $\mathrm{rpm}$	$c_5$	21
$\beta$	1${}^{\circ }$	$c_6$	0.0068

presents the pertinent parameter values for wind turbine power systems.

2.5. Noise-Based Photovoltaic System

This paper considers both wind and photovoltaic (PV) systems as renewable energy sources. In this noise-based PV system model, the power variation considers the variation of uniform and non-uniform insolation, illustrated in Figure 6. As stated in Ref. [14], the power deviations of PV expressed by Equation (14) can be used to present an actual pattern depicting the variation in PV.

$$\mathrm{\Delta }{P}_{\mathit{Solar}}=0.6\sqrt{P_{\mathit{Solar}}}$$

(14)

3. LADRC Design Process for LFC

Generally, it is imperative to employ a control strategy to preserve the stability of $\mathrm{\Delta }{f}_i$ and $\mathrm{\Delta }{P}_{\mathit{tiei}}$ and mitigate the impact of load disturbances. However, this paper neglects the market mechanism, and as such, steady-state conditions should not experience any frequency deviation or tie-line exchanged power. Moreover, Equation (7) demonstrates that each area is not only affected by the load disturbance $\mathrm{\Delta }{P}_{\mathit{di}}$ but also by the frequency deviation of other areas. In essence, LFC must address the impact of load disturbances and counteract inter-area coupling. Additionally, the renewable energy output power deviation $\mathrm{\Delta }{P}_{\mathit{rei}}$ can also be seen as a disturbance in the load frequency system. Thus, theoretically speaking, the LFC for a power system incorporating renewable energy is more challenging.

LADRC is a model-free controller that can effectively overcome the impact of unknown disturbances. Additionally, it can treat coupling information as unknown disturbances, enabling it to achieve decoupling. Therefore, this paper adopts LADRC for the LFC of multi-source power systems.

3.1. System Order Analysis

While LADRC has model-free characteristics, obtaining the system order is essential for designing the LESO. For convenience, $d=\mathrm{\Delta }{P}_{\mathit{rei}}-\mathrm{\Delta }{P}_{\mathit{di}}$ is defined as the disturbance. Then, carrying out Laplace transform on Equation (7), it yields

$$Y\left(s\right)=G\left(s\right)U\left(s\right)+G_d\left(s\right)D\left(s\right)$$

(15)

where $Y\left(s\right)$, $U\left(s\right)$, and $D\left(s\right)$ are the Laplace forms of ${\mathit{ACE}}_i\left(t\right)$, $u_i\left(t\right)$ and disturbance $d\left(t\right)$, respectively. Then, we can obtain

$$G\left(s\right)=\frac{R_{i1}{R}_{i2}{R}_{i3}{B}_i\left(G_{\mathit{th}}+G_{\mathit{hp}}+G_{\mathit{gp}}\right)G_{\mathit{pi}}}{R_{i1}{R}_{i2}{R}_{i3}+\left(R_{i2}{R}_{i3}{G}_{\mathit{th}}+R_{i1}{R}_{i3}{G}_{\mathit{hp}}+R_{i1}{R}_{i2}{G}_{\mathit{gp}}\right)G_{\mathit{pi}}}$$

(16)

As a result, combined with Equations (9), (10), (11), and (16), the system order of Equation (15) can be obtained as 2. That is, Equation (7) can be represented as

$${\ddot{\mathit{ACE}}}_i=f_1\left({\mathit{ACE}}_i,\mathrm{\Delta }{P}_{\mathit{di}},\mathrm{\Delta }{P}_{\mathit{rei}}\right)+{\mathit{bu}}_i$$

(17)

where $f_1$ denotes the total disturbance containing internal modeling information, load disturbance, and renewable energy output power deviation disturbance; b is the constant coefficient. Furthermore, an adjustable parameter $b_0$ can be used to replace b,

$$\begin{array}{cc}\hfill {\ddot{\mathit{ACE}}}_i& =f_1+(b-b_0)u_i+b_0{u}_i\hfill \\ \hfill & =f+b_0{u}_i\hfill \end{array}$$

(18)

Hence, the variable f represents the cumulative disturbance at the final stage, expressed as $f=f_1+(b-b_0)u_i$.

3.2. Disturbance Estimation and Elimination for LADRC

In Section 3.1, the multi-source power system under investigation is transformed into a second-order system with disturbances, where the disturbance information is unknown. Consequently, this paper will use the LADRC controller to eliminate unknown disturbances and realize the load frequency control. Firstly, an LESO is used to estimate the unknown disturbance.

For the system shown in Equation (18), the states can be defined as $x_1={\dot{\mathit{ACE}}}_i$, $x_2={\mathit{ACE}}_i$ and $x_3=f$, allowing for the derivation of the state space equation

$$\left\{\begin{array}{c}{\dot{x}}_1=x_2\hfill \\ {\dot{x}}_2=x_3+b_0{u}_i\hfill \\ {\dot{x}}_3=\dot{f}\hfill \end{array}\right.$$

(19)

Suppose that $z_1$, $z_2$, and $z_3$ are the observed states of $x_1$, $x_2$ and $x_3$, respectively. Then, the full-order LESO, as shown in Equation (20), is designed to obtain the estimation of the above states [45],

$$\left\{\begin{array}{c}{\dot{z}}_1=z_2-l_1\left(z_1-x_1\right)\hfill \\ {\dot{z}}_2=z_3+b_0{u}_i-l_2\left(z_1-x_1\right)\hfill \\ {\dot{z}}_3=-l_3\left(z_1-x_1\right)\hfill \end{array}\right.$$

(20)

where $l_1$, $l_2$, and $l_3$ are observer gains. Denote the observer error vector as $e={\left[e_1,e_2,e_3\right]}^T$ with $e_1=x_1-z_1$, $e_2=x_2-z_2$, and $e_3=x_3-z_3$; then, there is

$$\dot{e}=\mathit{Ae}+\mathit{Eh}$$

(21)

where $A=\left[\begin{array}{ccc}-l_1& 1& 0\\ -l_2& 0& 1\\ -l_3& 0& 0\end{array}\right]$ and $h=\dot{f}$. It can be found that the observer gains affect the stability of the observer errors. And suppose the eigenvalue is ${\lambda }_A$; then, we have $\mathit{det}(A-{\lambda }_{A}I)=0$, resulting in

$${\lambda }_A^3+l_1{\lambda }_A^2+l_2{\lambda }_A+l_3=0$$

(22)

Then, configure the pole at $-{\omega }_o({\omega }_o>0)$, that is, ${\left({\lambda }_A+{\omega }_o\right)}^3=0$, so that the convergence of the observer errors can be guaranteed. As a result, there are $l_1=3{\omega }_o$, $l_2=3{{\omega }_o}^2$, and $l_3={{\omega }_o}^3$. That is to say, the observer’s adjustable parameter count is reduced by the pole placement design method [46]. When ${\omega }_o$ is appropriate, there is $z_3\approx f$; thus, the control law shown in Equation (22) is designed to realize the elimination of disturbance, where $y_r$ is the target value of ${\mathit{ACE}}_i$, i.e., $y_r=0$.

$$u_i=\frac{k_p\left(y_r-z_1\right)-k_d{z}_2-z_3}{b_0}$$

(23)

Substituting Equation (23) into (18), it can be found that the unknown disturbance f is compensated by $z_3$. Similarly, the controller parameters are placed at the pole $-{\omega }_c({\omega }_c>0)$ through the pole placement method, obtaining $k_p={\omega }_c^2$ and $k_d=2{\omega }_c$. In summary, LADRC requires only the system’s order information to design an LESO for estimating unknown disturbances. The control law is then used to eliminate the estimated disturbance, enabling LFC control. Moreover, the LADRC presented in this paper includes three parameters: the observer parameter ${\omega }_o$, the control law parameter ${\omega }_c$, and the parameter $b_0$. And the structure diagram of LADRC is shown in Figure 7.

4. Adaptive LTD3-LADRC Approach

The performance of a controller is directly impacted by its parameters, and thus, it is crucial to carefully select and optimize them to ensure stability and improve robustness. As the number of areas in power systems increases, the parameter tuning process becomes more challenging and time-consuming, often requiring advanced optimization techniques and simulations to achieve satisfactory results. The adaptive tuning of controller parameters is a sequential decision-making process where the parameters are selected in a time-ordered manner. A suitable representation for this problem is a Markov decision process (MDP) denoted by $M=\{\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}\}$. $\mathcal{S}$ represents the state set, $\mathcal{A}$ denotes the action space, $\mathcal{P}$ indicates the state transition probability, and $\mathcal{R}$ represents the reward value.

RL is an effective means to deal with sequential decision-making problems, while DRL combines the computing power of deep neural networks based on RL. However, the training outcomes heavily rely on the reward function’s design, and there is no unified and efficient design rule yet. A Lyapunov-based reward function is provided for LFC problems based on the TD3 algorithm in this paper, namely the LTD3 algorithm. This approach enables the adaptive optimization of LADRC controller parameters.

4.1. The Basic of TD3

TD3 is an improved algorithm on the basis of DDPG, which solves the overestimation issue in the DDPG [47]. Consistent with the idea of reinforcement learning, the TD3 algorithm is also based on the dynamic interplay between the environment and the agent and uses the reward value in the interaction process to guide the agent toward selecting the most optimal strategy. The optimal policy is usually achieved by maximizing the following state–action value function.

$$Q\left(s,a\right)=\mathbb{E}\left[R_c\left|S_t=s,A_t=a\right.\right]$$

(24)

where $R_c$ is known as the cumulative reward, expressed as $R_c={\displaystyle \sum _{t=0}^{\infty }}{\gamma }^t{r}_{t+1}$. Here, $r\in \mathcal{R}$ is the instant reward, and the discount factor ($\gamma$) quantifies the significance of next-moment reward values in relation to the present moment.

Figure 8 showcases the frame diagram of the TD3 algorithm, encompassing essential components such as a replay buffer and six neural networks. The critic and target critic networks share the same structure as the actor and target actor networks. The replay buffer has the function of storing the data required for training the neural networks. For the six neural networks, there are the critic network 1 $Q_{{\theta }_1}(s,a)$, critic network 2 $Q_{{\theta }_2}(s,a)$, actor network ${\pi }_{\varphi }\left(s\right)$, target critic network 1 $Q_{{\theta }_1^{\prime }}(s,a)$, target critic network 2 $Q_{{\theta }_2^{\prime }}(s,a)$, and target actor network ${\pi }_{{\varphi }^{\prime }}\left(s\right)$. As for the meaning of these symbols, taking $Q_{{\theta }_1}(s,a)$ as an instance, it means that the network input includes s and a, the network weight is represented by ${\theta }_1$, and the network output is Q. The network diagram depicted in Figure 8 illustrates the input and output variables. Notably, the critic network is designed to approximate the state–action value Q, while the actor network generates the action value a based on the given state s.

In addition, compared with the DDPG algorithm, TD3 adds the target critic networks, which leverages the DDQN (double deep Q-network) algorithm, thus alleviating the problem of Q value overestimation to a certain extent. That is to say, to enhance the proximity between the critic network’s generated Q value and the actual Q value, the estimation of the state–action value in the subsequent moment utilizes the smaller output from the two critic networks. This process yields the target value as

$$y=r+\gamma \underset{i=1,2}{min}{Q}_{{{\theta }^{\prime }}_i}\left(s^{\prime },a^{\prime }\right)$$

(25)

where $s^{\prime }$ represents the state value in the subsequent time step; the action value $a^{\prime }$ in the subsequent time step is obtained through the target action network output ${\pi }_{{\varphi }^{\prime }}\left(s^{\prime }\right)$: $a^{\prime }={\pi }_{{\varphi }^{\prime }}\left(s^{\prime }\right)+ϵ$, in which the noise $ϵ$ satisfies $ϵ\sim$clip$\left(\mathcal{N}\right(0,\sigma ),-c,c)$. Then, the TD errors can be obtained as

$$L_{\mathit{ci}}={\left(y-Q_{{\theta }_i}\left(s,a\right)\right)}^2, i=1,2$$

(26)

On the basis of Equation (26), the update of ${\theta }_1$ and ${\theta }_2$ can be realized using the gradient descent method. Additionally, the actor network is updated after $\kappa$ steps of the critic network update, and the updated principle is to hope that the action value $a={\pi }_{\varphi }\left(s\right)$ generated by the actor network can maximize the $Q_{{\theta }_1}(s,a)$.

The update of the target networks adopts an exponentially moving average method, and there is

$$\left\{\begin{array}{c}{{\theta }^{\prime }}_i\leftarrow \tau {\theta }_i+\left(1-\tau \right){{\theta }^{\prime }}_i\hfill \\ {\varphi }^{\prime }\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^{\prime }\hfill \end{array}\right.$$

(27)

where $\tau$ is a smoothing factor. And the pseudocode of the TD3 algorithm is shown in Algorithm 1.

Algorithm 1 TD3 Algorithm

Initialize critic networks $Q_{{\theta }_1}(s,a)$, $Q_{{\theta }_2}(s,a)$ and actor network ${\pi }_{\varphi }\left(s\right)$ with random parameters ${\theta }_1$, ${\theta }_2$ and $\varphi$. Initialize target networks $Q_{{\theta }_1^{\prime }}(s,a)$, $Q_{{\theta }_2^{\prime }}(s,a)$ and ${\pi }_{{\varphi }^{\prime }}\left(s\right)$ with weights ${\theta }_1^{\prime }\leftarrow {\theta }_1$, ${\theta }_2^{\prime }\leftarrow {\theta }_2$ and ${\varphi }^{\prime }\leftarrow \varphi$. Initialize replay buffer $\mathcal{D}$. if $t\le T$ then  Select action with exploration noise $a\sim {\pi }_{\varphi }\left(s\right)+ϵ$, $ϵ\sim \mathcal{N}(0,\sigma )$, observe reward r and new states $s^{\prime }$.  Store transition tuple $(s,a,r,s^{\prime })$ in $\mathcal{D}$.  Sample mini-batch of m transitions $(s,a,r,s^{\prime })$ from $\mathcal{D}$.  Obtain the action value by Target ActorNetwork: $a^{\prime }\leftarrow {\pi }_{{\varphi }^{\prime }}\left(s^{\prime }\right)+ϵ$, $ϵ$ is the noise and satisfies: $ϵ\sim$clip$\left(\mathcal{N}\right(0,\sigma ),-c,c)$.  Calculate the target value by Equation (25): $y\leftarrow r+\gamma {min}_{i=1,2}{Q}_{{{\theta }^{\prime }}_i}\left(s^{\prime },a^{\prime }\right)$.  Update critics by TD-errors: ${\theta }_i\leftarrow arg{min}_{{\theta }_i}\frac{1}{m}\sum {\left(y-Q_{{\theta }_i}\left(s,a\right)\right)}^2$.  if t mod $\kappa$ then   Update $\varphi$ by the deterministic policy gradient: ${\nabla }_{\varphi }J\left(\varphi \right)=\frac{1}{m}\sum {\nabla }_a{Q}_{{\theta }_1}\left(s,a\right)\left|{}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right.$.   Update target networks by moving average method: ${\theta }_i^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$, ${\varphi }^{\prime }\leftarrow \tau \varphi +\left(1-\tau \right){\varphi }^{\prime }$.  end if end if

4.2. Lyapunov-Based Reward Function for TD3

The reward function design significantly impacts the effectiveness of DRL. It serves as the feedback signal given by the environment, guiding the agent to learn the correct behavior. The reward function determines what the agent should strive to achieve and what actions should be avoided. A well-designed reward function can help the agent learn the optimal policy faster and more accurately. At the same time, a poorly designed one can lead to incorrect behavior or slow learning.

In this paper, a Lyapunov [48]-based reward function [49] for TD3 is proposed as

$$R^{\mathit{lyap}}=-\mathcal{L}(s,a)+\lambda \left(\mathcal{L}(s,a)-\gamma \mathcal{L}\left(s^{\prime },a^{\prime }\right)\right)$$

(28)

where $\lambda$ represents the weight value, and it can be observed that when $\lambda =0$, the Lyapunov reward function is consistent with the normal reward function. $\mathcal{L}(s,a)=-R\left(s,a\right)$ is the Lyapunov function with decreasing properties, and the notation $R(s,a)$ represents the feedback received when action a is taken in state s. Then, the following analysis can be carried out.

Assumption 1.

γ approximates to 1, that is, $\gamma \approx 1$.

Assumption 2.

There exists a state–action pair $\left(s^{\ast },a^{\ast }\right)$ that maximizes the reward value, denoted as $R(s^{\ast },a^{\ast })={\mathit{max}}_{s,a}R(s,a)$.

Theorem 1.

Under the premise of Assumptions 1 and 2, if $\exists (s^{\prime },a^{\prime })$, the following inequality holds:

$$\mathcal{L}\left(s^{\prime },a^{\prime }\right)\le \mathcal{L}\left(s,a\right)$$

(29)

then the state–action pair $(s,a)$ can converge to the optimum $(s^{\ast },a^{\ast })$.

Proof of Theorem 1.

Equation (28) can be rearranged with Assumption 1 as

$$R^{\mathit{lyap}}=-\left(1-\lambda \right)\mathcal{L}\left(s,a\right)-\lambda \mathcal{L}\left(s^{\prime },a^{\prime }\right)$$

(30)

With Equation (29), Equation (30) leads to

$$R^{\mathit{lyap}}\ge -\mathcal{L}\left(s,a\right)$$

(31)

Moreover, Equation (30) can further obtain the following inequality results:

$$\begin{array}{cc}\hfill R^{\mathit{lyap}}& =\left(1-\lambda \right)R\left(s,a\right)+\lambda R\left(s^{\prime },a^{\prime }\right)\hfill \\ \hfill & \le R\left(s^{\ast },a^{\ast }\right)\hfill \end{array}$$

(32)

Due to the Lyapunov function $\mathcal{L}\left(s,a\right)$ exhibiting descent property [50], as well as the existence of a maximal reward $R(s^{\ast },a^{\ast })$ (Assumption 2), the $\mathcal{L}\left(s,a\right)$ has lower bound $\mathcal{L}\left(s,a\right)\ge -R(s^{\ast },a^{\ast })$. This demonstrates the convergence of $\mathcal{L}\left(s,a\right)$. Since $\mathcal{L}(s,a)=-R\left(s,a\right)$, $\mathcal{L}(s,a)$ converging to $\mathcal{L}(s^{\ast },a^{\ast })$ is equivalent to $R(s,a)$ converging to $R(s^{\ast },a^{\ast })$. As a result, the state–action pair $(s,a)$ will converge to the optimum $(s^{\ast },a^{\ast })$. □

Remark 1.

The conventional reward function is formulated on the premise that the reward value increases as the state quantity approaches the desired value, as illustrated in Ref. [51]. However, this reward function lacks a solid theoretical foundation. In contrast, the reward function presented in this paper incorporates the Lyapunov function, which not only provides a theoretical basis but also exhibits convergence, as validated by Theorem 1.

4.3. Environment and Agent Settings for Multi-Source Power System

As mentioned, DRL algorithms can continuously learn to find the best policy. In this paper, the environment comprises the multi-source power system and the corresponding LADRC, while the agent refers to the decision maker in the TD3 algorithm. Before training, it is necessary to clarify state and action space composition and establish a reward function based on the corresponding state variables.

This paper considers the action space as the parameters to be adjusted in LADRC. According to Section 3, each area contains an LADRC, and the parameters that need to be adjusted in a controller include three parameters: ${\omega }_o$, ${\omega }_c$, and $b_0$. Moreover, only the tuning of the parameters ${\omega }_o$ and ${\omega }_c$ is considered, because it is mentioned in Ref. [52] that $b_0$ has little influence on suppressing disturbances. Moreover, it is conceivable that with an increase in the number of areas, there will be a corresponding increase in the number of controller parameters that require adjustment, so the action space can be defined as

$$\begin{array}{c}\hfill \{a_{11},a_{12},\dots a_{n1},a_{n2}\in \mathcal{A}|a_{11}={\omega }_{o1},a_{12}={\omega }_{c1},\dots a_{n1}={\omega }_{on},a_{n2}={\omega }_{cn}\}\end{array}$$

(33)

n in Equation (33) is the the number of areas. The detected states in each area are $\mathit{ACE}\left(t\right)$ and $\mathit{ACE}\left(t\right)-\mathit{ACE}(t-1)$, which are equivalent to the two variables of error and derivative of error $A\dot{C}E\left(t\right)$. Therefore, the state space is defined as

$$\begin{array}{c}\hfill \{s_{11},s_{12},\dots s_{n1},s_{n2}\in \mathcal{S}|s_{11}={\mathit{ACE}}_1,s_{12}=A\dot{C}{E}_1,\dots s_{n1}={\mathit{ACE}}_n,s_{n2}=A\dot{C}{E}_n\}\end{array}$$

(34)

Based on the observed state, we expect the area control error to converge quickly and smoothly to the desired value of 0. Then, the following two points about reward function need to be satisfied:

(1) The ${\mathit{ACE}}_i$ is as close to 0 as possible, and the smaller the area control error, the larger the reward value should be.

(2) The overshoot and undershoot of the system response should be as slight as possible.

Therefore, the reward feedback $R_t$ based on the state variables is designed as

$$R_t=-2\sum _{i=1}^n\left[{\left(100{\mathit{ACE}}_i\left(t\right)\right)}^2+{\left(10A\dot{C}{E}_i\left(t\right)\right)}^2\right]$$

(35)

where the coefficients of 100 and 10 are utilized to ensure the consistent magnitude of ${\mathit{ACE}}_i$ and $A\dot{C}{E}_i$. This approach prevents the agent from being overly influenced by a specific state during training. Moreover, $R\left(t\right)$ has the maximum value, thus satisfying Assumption 2.

Then, according to Equations (30) and (35), the expression of the reward function based on Lyapunov can be written as

$$R^{\mathit{lyap}}\left(t\right)=R_t+\lambda \left(\gamma R_{t+1}-R_t\right)$$

(36)

For a power system with n areas, the structure diagram of the proposed LTD3-LADRC is shown in Figure 9. It can be found that the purpose of LTD3 is to obtain the adaptive controller parameters shown in Equation (33) through detecting the real-time state in Equation (34), then inputting these parameters into the controller will generate a new state vector. By continuously repeating this process, load frequency control can be effectively achieved.

5. Simulation Verification and Analysis

In this paper, the simulation is performed using the MATLAB/Simulink R2021b platform. Please refer to the Appendix A for the model under study [20] and its corresponding parameters. Then two LADRC controllers are needed, and LTD3 is used to to realize parameter optimization of ${\omega }_{o1}$, ${\omega }_{c1}$, ${\omega }_{o2}$, and ${\omega }_{c2}$. Additionally, Figure 10 illustrates the architecture of the critic and actor networks, as mentioned in Algorithm 1, proposed in this paper. The number of neurons in each hidden layer and the employed activation function are indicated within the dashed box. Network activation functions such as Relu are also shown in Figure 10.

Moreover, the hyperparameters in the LTD3 algorithm are chosen as $\gamma =0.99$, $\lambda =0.3$, and $\tau =0.005$. To showcase the efficacy of the proposed LTD3 algorithm, we conducted experiments on the LADRC load frequency control system. Specifically, we applied the TD3 algorithm with the reward function described in Equation (35) and the LTD3 algorithm with the reward function outlined in Equation (36). Throughout the training process, one episode represents the execution of one loop in Algorithm 1 with a time interval of 0.01 s and a total simulation time of 15 s. For the control system, the parameters of $b_{01}$ and $b_{02}$ are all chosen as 20, and the action space for DRL is

$$\left\{\begin{array}{c}{\omega }_{oi}\in \left[5,15\right],i=1,2\hfill \\ {\omega }_{ci}\in \left[1,6\right],i=1,2\hfill \end{array}\right.$$

(37)

Figure 11 illustrates the progression of the episode reward value for TD3 and LTD3 throughout the training process. The graph reveals that TD3’s reward value approaches stability around 500 episodes, whereas LTD3 achieves stability in approximately 90 episodes. This observation indicates that LTD3 exhibits faster convergence in its training.

To enhance the verification of the agent trained above, simulations are conducted under the following three scenarios:

• Scenario 1: System response performance without RESs;
• Scenario 2: System response performance with wind turbines and photovoltaic systems;
• Scenario 3: System response performance under system parameter variations.

The difference between Scenario 1 and Scenario 2 lies in the variation of the model environment. Scenario 1 involves the model without renewable energy sources, whereas Scenario 2 incorporates wind and solar energy. Scenario 3 simulates parameter perturbation problems that may occur in the actual environment, essentially to verify the robustness of the LTD3-LADRC method.

5.1. Scenario 1: System Response Performance without RESs

Initially, we analyzed scenarios that exclude wind turbines and photovoltaic systems. This scenario is introduced in the following two cases.

Case A: System considers a SLP

In this case, a 0.01 p.u. step load perturbation (SLP) was introduced in Area I at 5 s, i.e., $\mathrm{\Delta }{P}_{d1}=0.01$ p.u. Figure 12 and Figure 13 present the simulation results, including the response curves of various published works, such as integral control (IC) [44], FOPID [43], and the ID-T based Archimedes optimization algorithm (AOA) [20], as well as the curve generated by LADRC-TD3 for comparison. Furthermore,

Table 2. Comparison results of dynamic performance indicators for Scenario 1—Case A.

Control Method		IC	FOPID	ID-T Based AOA	LADRC-TD3	LADRC-LTD3
$\mathrm{\Delta }{f}_1$	US $\left({10}^{-2}\right)$	−4.37	−3.33	−2.78	−2.26	−2.14
	OS $\left({10}^{-3}\right)$	9.8	7.7	8.8	9	4.3
	ST	50	15.3	11.6	15.6	12.75
	IAE $\left({10}^{-2}\right)$	29.4	7.36	3.85	4.51	3.37
$\mathrm{\Delta }{f}_2$	US $\left({10}^{-2}\right)$	−5.83	−3.14	−2.38	−1.64	−1.57
	OS $\left({10}^{-3}\right)$	11.1	7.8	5.4	13.8	6.2
	ST	48.9	15.4	13.0	15.2	13.53
	IAE $\left({10}^{-2}\right)$	28.77	7.45	3.66	3.88	3.20
$\mathrm{\Delta }{P}_{tie}$	US $\left({10}^{-3}\right)$	−8.6	−5.5	−4.2	−3.6	−3.7
	OS $\left({10}^{-4}\right)$	2.68	7.9	14	6.3	7.3
	ST	42.7	9.7	11.0	9.7	12.5
	IAE $\left({10}^{-2}\right)$	7.57	1.27	0.95	0.69	0.89

The abbreviations US, OS, and ST denote the undershoot, overshoot, and settling time, respectively. In this case, the settling time refers to the minimum duration needed for the output to reach and sustain within $\pm 0.0005$.

provides a numerical comparison of performance indicators, including undershoot, overshoot, settling time, and the integral absolute error (IAE) defined in Equation (38).

$$\mathit{IAE}={\int }_0^T\left|e\right(t\left)\right|\mathit{dt}$$

(38)

Specifically, Figure 12 depicts the dynamic response of the frequency deviation in Area I and Area II, along with the tie-line exchanged power under the influence of SLP. It shows that the system exhibits a transient undershoot response under the influence of disturbances. Table 2 further indicates that the LADRC controller based on TD3 and LTD3 algorithms achieves smaller undershoot than other methods. It is worth noting that LTD3 exhibits slightly lower performance in terms of $\mathrm{\Delta }{P}_{\mathit{tie}}$ compared to TD3, as indicated in Table 2. This occurs because the ${\mathit{ACE}}_i$ is derived from the linear combination of $\mathrm{\Delta }{f}_i$ and $\mathrm{\Delta }{P}_{\mathit{tiei}}$, as shown in Equation (1). The algorithm aims to minimize ${\mathit{ACE}}_i$, considering it ideal for ${\mathit{ACE}}_i$ to be as small as possible. However, since $\mathrm{\Delta }{f}_i$ generally has a larger magnitude than $\mathrm{\Delta }{P}_{\mathit{tiei}}$, it exerts a greater influence on ${\mathit{ACE}}_i$. Consequently, the effect of $\mathrm{\Delta }{f}_i$ achieved by LTD3 is superior, while the impact of $\mathrm{\Delta }{P}_{\mathit{tiei}}$ is not significantly different between the two algorithms. However, the difference is negligible, and from the target of frequency deviation, LTD3 still outperforms TD3. The changing process of controller parameters in the LADRC-TD3 and LADRC-LTD3 methods, obtained from the trained TD3 agent and LTD3 agent, respectively, is presented in Figure 13. It can be found that LTD3 and TD3 provide different controller parameter strategies, and from the performance results in Figure 12, the strategy obtained from the proposed LTD3 is better than that of the TD3.

Case B: System considers a multi-step load perturbation (MSLP)

In this case, the multi-step load perturbation (MSLP), expressed in Figure 14, is added to Area I, which denotes the presence of multiple step load perturbations throughout the simulation process. Then, the dynamic response by I-TD [20], ID-T [20], LADRC-TD3, and the proposed LADRC-LTD3 is shown in Figure 15 and Figure 16. All four control methods can attain steady-state responses when faced with various step disturbances. By comparison, LADRC-TD3 and LADRC-LTD3 exhibit significantly less vibration during the transient response process than the other two methods. Furthermore,

Table 3. Comparison results of IAE for Scenario 1—Case B.

Control Method		I-TD	ID-T	LADRC-TD3	LADRC-LTD3
IAE	$\mathrm{\Delta }{f}_1$	0.7772	0.3143	0.2674	0.1722
	$\mathrm{\Delta }{f}_2$	0.8245	0.2834	0.2376	0.1423
	$\mathrm{\Delta }{P}_{\mathit{tie}}$	0.2344	0.0637	0.0540	0.0326
ITAE	$\mathrm{\Delta }{f}_1$	78.88	34.67	26.33	16.03
	$\mathrm{\Delta }{f}_2$	86.91	29.16	23.35	13.25
	$\mathrm{\Delta }{P}_{\mathit{tie}}$	23.46	7.22	5.41	3.00

provides a numerical comparison of the IAE and ITAE integral of time multiplied by absolute error, as expressed in Equation (39)), for the $\mathrm{\Delta }{f}_1$, $\mathrm{\Delta }{f}_2$, and $\mathrm{\Delta }{P}_{\mathit{tie}}$ responses of the four control methods, respectively. It indicates that the proposed method achieves the smallest IAE and ITAE values, which suggests that the proposed method outperforms the other three methods regarding control accuracy and convergence speed. Figure 16 also displays the adaptive parameters of TD3 and LTD3, obtained by observing the system state during the simulation process.

$$\mathit{ITAE}={\int }_0^{T}t\left|e\right(t\left)\right|\mathit{dt}$$

(39)

5.2. Scenario 2: System Response Performance with RESs

In this scenario, the noise-based wind turbine illustrated in Figure 5 and the photovoltaic system depicted in Figure 6 are taken into account based on the two-area interconnected power system described earlier. With the white noise, the deviation output of wind and solar energy exhibits fluctuations, as illustrated in Figure 17. To further evaluate the effectiveness of the proposed method in regulating LFC that incorporate RESs, the MSLP shown in Figure 18 is added to Area I. The wind turbine is connected at 100s, and the photovoltaic system is connected at 250 s.

The simulation results are depicted in Figure 19 and Figure 20. It is noticeable that, in the presence of renewable energy, the system experiences oscillations, leading to a more significant response overshoot than the load frequency system without renewable energy. However, the proposed LADRC-LTD3 method effectively suppresses the oscillations and promptly restores the system to a stable state.

Table 4. Comparison results of IAE for Scenario 2.

Control Method		I-TD	ID-T	LADRC-TD3	LADRC-LTD3
IAE	$\mathrm{\Delta }{f}_1$	2.1498	0.7702	0.5689	0.4230
	$\mathrm{\Delta }{f}_2$	1.9650	0.6334	0.5392	0.4167
	$\mathrm{\Delta }{P}_{\mathit{tie}}$	0.8144	0.3412	0.1767	0.1243
ITAE	$\mathrm{\Delta }{f}_1$	298.95	104.27	72.93	53.58
	$\mathrm{\Delta }{f}_2$	272.86	85.43	69.97	53.11
	$\mathrm{\Delta }{P}_{\mathit{tie}}$	119.59	55.12	22.24	15.39

further displays the performance results of the four methods regarding IAE and ITAE, indicating that the proposed LADRC-LTD3 outperforms the other three methods significantly. Additionally, Figure 20 illustrates the parameters obtained by the TD3 agent and LTD3 agent. The selection of parameters varies significantly when a step disturbance occurs. In addition, white noise also impacts the selection of parameters.

5.3. Scenario 3: System Response Performance Considering System Parameter Variations

When evaluating controller performance, it is crucial to consider both the dynamic response of the system and the controller’s robustness against variations in system parameters. Therefore, the Monte Carlo method is adopted for the robustness test.

Based on Scenario 2, Monte Carlo simulations are conducted for 50 runs with the model parameters of $T_{\mathit{sg}}$, $T_t$, $T_{\mathit{gh}}$, $X_g$, and $Y_g$ varying within $\pm 50\%$ of their nominal values. Considering such uncertainty, Figure 21 displays the simulation results obtained. The results demonstrate that the proposed method retains its ability to regulate load disturbance and renewable energy and achieve a stable response even when the model parameters are modified. This indicates that the proposed method exhibits strong robustness.

6. Conclusions

In conclusion, the paper proposes a Lyapunov reward-based TD3 algorithm, denoted as LTD3, optimized LADRC method for LFC in multi-source systems with RESs. The effectiveness of the LTD3-LADRC is validated through simulations on a nonlinear two-area interconnected power system that includes thermal, hydro, and gas power plants, as well as wind turbine and photovoltaic systems. The proposed LADRC-LTD3 method is compared with existing methods, and the results show that it effectively addresses the LFC problem in the presence of renewable energy and load disturbance. Furthermore, the proposed method exhibits robustness and maintains system stability even in parameter perturbation.

This paper employs the Lyapunov reward function to enhance the convergence speed of the algorithm, and the effectiveness of the proposed LTD3 algorithm is validated through simulations. Further theoretical analysis of the LTD3 algorithm is necessary in future work to improve its robustness and performance. Furthermore, it is noteworthy that the current study does not encompass the involvement of the load side within the load frequency control framework. Thus, in our forthcoming research, we intend to broaden our model and validate our methodologies in this domain.