Load Frequency Active Disturbance Rejection Control for Multi-Source Power System Based on Soft Actor-Critic

Abstract

To ensure the safe operation of an interconnected power system, it is necessary to maintain the stability of the frequency and the tie-line exchanged power. This is one of the hottest issues in the power system field and is usually called load frequency control. To overcome the influences of load disturbances on multi-source power systems containing thermal power plants, hydropower plants, and gas turbine plants, we design a linear active disturbance rejection control (LADRC) based on the tie-line bias control mode. For LADRC, the parameter selection of the controller directly affects the response performance of the entire system, and it is usually not feasible to manually adjust parameters. Therefore, to obtain the optimal controller parameters, we use the Soft Actor-Critic algorithm in reinforcement learning to obtain the controller parameters in real time, and we design the reward function according to the needs of the power system. We carry out simulation experiments to verify the effectiveness of the proposed method. Compared with the results of other proportional–integral–derivative control techniques using optimization algorithms and LADRC with constant parameters, the proposed method shows significant advantages in terms of overshoot, undershoot, and settling time. In addition, by adding different disturbances to different areas of the multi-source power system, we demonstrate the robustness of the proposed control strategy.

1. Introduction

The interconnected power system has been developed in recent years to better meet power needs. To ensure the safe and reliable operation of the power system, the stability of the system’s frequency must be maintained. However, the mismatch between power generation and required power is prone to instability [1]. Therefore, load frequency control (LFC) and optimal power flow (OPF) [2] have become research hotspots in the field of multi-area interconnected power systems. The former mainly considers designing the controller to eliminate the influence of load disturbance on the frequency, and the latter determines the steady-state operating conditions containing the frequency distribution of the power system. In addition, there are multiple sources of power generation in power systems, among which thermal power generation, hydropower generation, and gas turbines are most common, and each has different characteristics. In this study, we mainly investigate the LFC of the multi-source power system.

For the LFC system in the non-deregulated environment, the control strategy needs to meet the performance requirements of frequency deviation and tie-line exchanged power at the planned value. Among various control strategies, proportional–integral–derivative (PID) control [3,4,5] occupies a dominant position inLFC due to its clear principle and simple implementation. However, PID is usually unsuitable for the demanding control requirements of LFC. Other control strategies such as robust control [6], model predictive control (MPC) [7], and adaptive control [8] could also achieve good results in LFC. For example, Zeng et al. [9] proposed an adaptive MPC method for a multi-area interconnected power system with PV generation realizing frequency deviation control. Rajeswari et al. [10] designed an adaptive LFC PID controller based on the generalized Hopfield neural network (GHNN), which can be used in situations where load demand is random, system dynamic modeling is inaccurate, the system model is nonlinear, and system parameter changes are uncertain. Although these methods can achieve higher control accuracy than PID, their design process is complex and they are challenging to implement in engineering applications. In reality, there may be multiple sources of power generation in one area. For example, Sahu [11] et al. applied the differential evolution (DE) optimized fuzzy PID to a two-area interconnected power system with multi-source power generation containing thermal plant, hydro plant, and gas turbine plant in a deregulated environment, which improved the dynamic performance.

Under these demands, active disturbance rejection control (ADRC) was proposed by Han [12]. It has a simple structure and does not rely on model information. At present, the most widely used ADRC is the linear ADRC (LADRC) [13], which linearizes the structure of the extended state observer (ESO) and nonlinear state error feedback (NLSEF) control law in ADRC. LADRC dramatically simplifies the design of the ADRC controller and significantly reduces the number of parameters that need to be adjusted, which greatly promotes the engineering application of ADRC. Currently, LADRC has been successfully applied in many fields, such as ship path following control [14], aircraft control [15], and hydraulic systems [16], to name a few. As for LFC, LADRC has also shown good control effects. For example, Tan et al. [17] applied LADRC to power systems with a deregulated environment to realize the stability of LFC. Tang et al. [18] studied the application of LADRC in the LFC of complex power systems with wind energy conversion systems and verified the effectiveness of the designed controller through simulations.

Although the number of controller parameters of LADRC has been greatly reduced compared with the original ADRC parameters, when the number of controllers increases, the determination of the parameters is still challenging. Therefore, parameter tuning is also a vital part of the controller design process. At present, the most widely studied parameter optimization methods are intelligent optimization algorithms, including particle swarm optimization (PSO) [19], genetic algorithm (GA) [20], and simulated annealing (SA) [21]. However, these algorithms have poor robustness and can only obtain the optimal parameters of the controller under certain operating conditions. Fuzzy control [22] can obtain adaptive parameter values depending on the model information. Neural networks [23] can obtain adaptive control parameters without relying on model information, but they do not cope well with sudden operating conditions. Therefore, an intelligent method that is robust and does not rely on models is urgently needed for tuning parameters.

Reinforcement learning [24] is an algorithm that simulates the decision-making process of the human brain, where the design of the reward function gives the algorithm its decision-making ability. The salient feature of reinforcement learning is to obtain reward feedback through interaction between the environment and the agent (brain) without knowing the environment’s information so as to realize the selection of the optimal action-value according to the environmental state. This realizes a process of self-adaptation in which action-values are selected according to the environmental state. With these advantages, reinforcement learning has achieved excellent results in path planning [25], games [26], and other fields. Recently, research has been conducted on reinforcement learning to adjust controller parameters. For example, Chen [27] and Zheng [28] applied Q learning to parameter tuning of LADRC. Nevertheless, Q learning is more suitable for systems with limited states and limited actions. Soft actor-critic (SAC) is an algorithm based on deep reinforcement learning that uses deep neural networks to remove the finite-dimensional restrictions on states and actions [29]. In addition, compared with the deep deterministic policy gradient (DDPG) algorithm, which is also suitable for continuous state and continuous action in reinforcement learning, the convergence of SAC is proved to be better. Therefore, in this study, we use SAC to solve the problem of LADRC parameters (continuous actions) tuning in LFC (continuous states).

In this study, based on a two-area interconnected power system with thermal power plants, hydropower plants, and gas turbine plants, we design three different LADRC methods for each area. Moreover, we use SAC to adjust the nine parameters of all controllers. The simulation results verify the effectiveness of the proposed method.

The rest of the paper is arranged as follows. Section 2 presents a model description of the two-area six-unit interconnected power system. Section 3 shows the design process of LADRC. Based on introducing SAC, Section 4 describes how SAC optimizes controller parameters. Section 5 demonstrates the superiority of the proposed method through simulation results. Section 6 concludes the paper.

2. Modeling of Two-Area Six-Unit Power System

We consider a two-area six-unit interconnected power system consisting of one thermal power plant, one hydropower plant, and one gas turbine plant in each area, as shown in Figure 1 [30], where each unit consists of a governor $G_{gi},i=1,2,3$ marked in light yellow and a turbine $G_{ti}$ marked in light green. Moreover, the links marked in purple are generators $G_p$. The symbols are defined in

Table 1. Symbol definitions and values.

Symbol	Terminology	Value
$\Delta f$	Frequency variation	-
$\Delta P_{tie}$	Tie-line exchanged power	-
$ACE$	Area control error	-
$B_i$	Frequency bias constant	0.4312 MW/Hz
$R_i$	Speed regulation constant	2.4 Hz/MW
$T_{gi}$	Governor time constant	0.08 s
$T_t$	Turbine time constant	0.3 s
$T_r$	Reheat turbine time constant	10 s
$K_r$	Reheat gain	0.3
$T_2$	Compensator time constant	28.75 s
$T_R$	Compensator time constant	5 s
$T_w$	Hydro turbine time constant	1 s
$b_g$	Valve positioner time constant	0.05 s
$c_g$	Valve positioner time constant	1
$X_c$	Governor lead time constant	0.6 s
$Y_c$	Governor lag time constant	1s
$T_{CR}$	Gas turbine combustion reaction time delay	0.01 s
$T_F$	Fuel system time constant	0.23 s
$T_{CD}$	Compressor discharge time constant	0.2 s
$KT$	Proportional constant	0.543478
$KH$	Proportional constant	0.326084
$KG$	Proportional constant	0.130438
$\Delta P_{di}$	Step load disturbance	-
$T_p$	Generator time constant	11.49 s
$K_p$	Generator gain	68.9566
$T_{12}$	Synchronization time constant	0.086 s

. Generally, the governor mainly controls the speed of the turbine, and the turbine converts mechanical energy into electrical energy. However, to ensure the safe and stable operation of the system, two conditions need to be met: one is that the frequency deviation $\Delta f$ is stable at 0, and the other is that the tie-line exchanged power $\Delta P_{tie}$ is maintained at the planned value.

Then, from Figure 1, taking the first area as an example, the following derivation can be obtained:

$$\Delta f_1=\frac{G_p}{1+{\displaystyle \sum _{i=1}^3}{G}_p{G}_{gi}{G}_{ti}/R_i}·\left(\sum _{i=1}^3{u}_i{G}_{gi}{G}_{ti}-\Delta P_{d1}-\Delta P_{tie}\right)$$

(1)

where $u_i(i=1,2,3)$ are the inputs of the three units, $G_{g3}=\frac{1}{b_{g}s+c_{g}s}$, $G_{gi}=\frac{1}{T_{gi}s+1}(i=1,2)$, $G_{t1}=\frac{KT\left(K_r{T}_{r}s+1\right)}{\left(T_{t}s+1\right)\left(T_{r}s+1\right)}$, $G_{t2}=\frac{KH\left(T_{R}s+1\right)\left(-T_{w}s+1\right)}{\left(T_{2}s+1\right)\left(0.5T_{w}s+1\right)}$, and $G_{t3}=\frac{KG\left(X_{c}s+1\right)\left(-T_{CR}s+1\right)}{\left(Y_{c}s+1\right)\left(T_{F}s+1\right)\left(T_{CD}s+1\right)}$.

$$\begin{array}{cc}\hfill AC{E}_1& =B_1\Delta f_1+\Delta P_{tie}\hfill \\ \hfill & =\frac{B_1{G}_p}{1+{\displaystyle \sum _{i=1}^3}{G}_{gi}{G}_{ti}/R_i}·\left(\sum _{i=1}^3{u}_i{G}_{gi}{G}_{ti}-\Delta P_{d1}-\Delta P_{tie}\right)+\Delta P_{tie}\hfill \end{array}$$

(2)

where $\Delta P_{tie}$ is expressed as

$$\Delta P_{tie}=\frac{2\pi T_{12}}{s}·\left(\Delta f_1-\Delta f_2\right)$$

(3)

In addition, there are many uncertainties in the two-area six-unit power system, such as unmodeled dynamics, load disturbance, and internal parameter perturbations, which lead to model–reality mismatch. Therefore, our research goal in this study is to design controllers to make the system stable despite these uncertainties.

3. Design of LADRC

Because ADRC has a simple design, is model-free, and possesses self-decoupling characteristics, it has been favored by many scholars in recent years. In this study, we design an LADRC controller for the system shown in Figure 1. We adopt the tie-line bias control (TBC) [31] mode to simultaneously adjust the frequency deviation and the tie-line exchanged power, and we select $ACE$ as the controller input. As for the coupling of the system, we adopt a decentralized control method [32], that is, under the premise of ignoring the tie-line exchanged power, we design a controller for each area. As shown in Figure 1, the two areas have the same structure, so the two controllers will be identical.

The design of the LADRC does not require accurate model information of the system, only the order of the controlled plant. Let us take the first area as an example. Removing $\Delta P_{tie}$, we rearrange the expression of $AC{E}_1$ as

$$y_1=AC{E}_1=\frac{B_1{G}_p{\displaystyle \sum _{i=1}^3}{u}_i{G}_{gi}{G}_{ti}}{1+{\displaystyle \sum _{i=1}^3}{G}_{gi}{G}_{ti}/R_i}-\frac{B_1{G}_p}{1+{\displaystyle \sum _{i=1}^3}{G}_{gi}{G}_{ti}/R_i}·\Delta P_{d1}$$

(4)

Then, for the thermal power unit, $u_1$ is the output of the first controller, and $u_2$ and $u_3$ are constants, that is,

$$\frac{y_1}{u_1}=\frac{B_1{G}_p{G}_{g1}{G}_{t1}}{1+{\displaystyle \sum _{i=1}^3}{G}_{gi}{G}_{ti}/R_i}$$

(5)

Therefore, the system order of the thermal power unit is 3. Then, we can express the system as shown in Equation (6). In the same way, the system orders of the hydropower turbine unit and the gas turbine unit are 2 and 3, respectively. Therefore, taking the thermal power unit as an example, we design the LADRC as follows.

$$y_1^{\left(3\right)}=f+b_{01}{u}_1$$

(6)

where f is the total disturbance, including the unmodeled dynamics, load disturbance, and parameter perturbation, and $b_{01}$ is an adjustable parameter. The core idea of LADRC is to estimate the total disturbance f through the linear extended state observer (LESO) and use the PD control combination to eliminate the disturbance.

We define the states as $x_1=y_1$, $x_2={\dot{y}}_1$, $x_3={\ddot{y}}_1$, and $x_4=f$. Then, we design LESO as

$$\left\{\begin{array}{c}e=z_1-y_1\hfill \\ {\dot{z}}_1=z_2-{\beta }_{01}e\hfill \\ {\dot{z}}_2=z_3-{\beta }_{02}e\hfill \\ {\dot{z}}_3=z_4-{\beta }_{03}e+b_{0}u\hfill \\ {\dot{z}}_4=-{\beta }_{04}e\hfill \end{array}\right.$$

(7)

where $z_1$, $z_2$, $z_3$, and $z_4$ denote the estimated values of $x_1$, $x_2$, $x_3$, and $x_4$, respectively. Furthermore, ${\beta }_{01}$, ${\beta }_{02}$, ${\beta }_{03}$, and ${\beta }_{04}$ represent the observer gains. We write the observer gain matrix as $L={\left[{\beta }_{01},{\beta }_{02},{\beta }_{03},{\beta }_{04}\right]}^T$. Then, the suitable L can guarantee the accuracy of the estimation. For convenience, we configure the observer gain at the pole $-{\omega }_{o1}$ [33] using the pole configuration method as

$$s^4+{\beta }_{01}{s}^3+\cdots +{\beta }_{04}={\left(s+{\omega }_{o1}\right)}^4$$

(8)

Then,

$${\beta }_{0i}=C_4^i{\omega }_{o1}^i$$

(9)

In the case of $z_4\approx f$, the disturbance is eliminated. Then, we can take $u_1$ in Equation (6) as

$$u_1=\frac{u_0-z_4}{b_{01}}$$

(10)

where

$$u_0=-k_1{z}_1-k_2{z}_2-k_3{z}_3$$

(11)

where $k_1$, $k_2$, and $k_3$ are the feedback control gains.

Finally, we rearrange Equation (6) into the following form, thereby realizing the compensation of disturbances:

$$y_1^{\left(3\right)}=-k_1{z}_1-k_2{z}_2-k_3{z}_3$$

(12)

Similarly, we configure the feedback control gain to the pole $-{\omega }_{c1}$,

$$s^3+k_3{s}^2+k_{2}s+k_1={\left(s+{\omega }_{c1}\right)}^3$$

(13)

Thus,

$$k_i=C_3^{i-1}{\omega }_{c1}^{3-i+1}$$

(14)

The structure diagram of LADRC is shown in Figure 2. The parameters that need to be adjusted in LADRC are ${\omega }_{o1}$, ${\omega }_{c1}$, and $b_{01}$. Second order or third order linear active disturbance rejection controllers can be designed for the hydropower turbine unit and gas turbine unit following the above process. To sum up, the parameters that need to be adjusted are ${\omega }_{o1}$, ${\omega }_{c1}$, $b_{01}$, ${\omega }_{o2}$, ${\omega }_{c2}$, $b_{02}$, ${\omega }_{o3}$, ${\omega }_{c3}$, and $b_{03}$.

4. LADRC Parameter Optimization Based on SAC

Because it is challenging to adjust nine parameters at the same time manually, we use the SAC algorithm in reinforcement learning to adjust all controller parameters.

4.1. The Basics of SAC

Reinforcement learning is a method of solving sequential decision-making problems, and its process is usually expressed as a Markov decision process (MDP): $\left(S,A,P,R\right)$. S and A represent the state space and action space, respectively, which need to be artificially defined. P is the probability of state transition, which represents the probability of transition from state $s_t$ and action $a_t$ to state $s_{t+1}$, namely $p\left[s_{t+1}\left|s_t,a_t\right.\right]$. R is the reward function, which reflects how close the actual state is to the target state. The goal of reinforcement learning is to find the optimal strategy through continuous iteration between the agent and the environment, that is, the probability of taking action a under the condition of state s: $\pi \left(a\left|s\right.\right)=p\left[a_t=a\left|s_t=s\right.\right]$. Then, based on $\pi$, the agent can choose the best action. In general, the state-action value function is used to evaluate the value of the action:

$$Q_{\pi }\left(s,a\right)=E_{\pi }\left[R_c\left|S_t=s,A_t=a\right.\right]$$

(15)

where $R_c$ is the cumulative reward ($R_c={\displaystyle \sum _{t=k}^{\infty }}{\gamma }^k{R}_{t+k+1}$), and $\gamma \in [0,1]$ is the discount factor reflecting the the importance of the reward value in the future.

SAC is an algorithm in reinforcement learning that can solve continuous state space and continuous action space problems. The structure diagram of SAC is shown in Figure 3. It should be pointed out that, unlike Q in the traditional reinforcement learning algorithm, the Q value in SAC not only maximizes the cumulative reward but also increases entropy [34].

$$Q_{\pi }\left(s,a\right)=\sum _{t=0}^T{E}_{\pi }\left[r\left(s,a\right)+\alpha H\left(\pi \left(·\left|s\right.\right)\right)\left|S_t=s,A_t=a\right.\right]$$

(16)

where $H\left(·\left|s\right.\right)$ represents the entropy under the policy $\pi$, which is a measure that describe the uncertainty of random variables, $H\left(\pi \left(·\left|s\right.\right)\right)=-log\left(\pi \left(a\right|s)\right)$, and $\alpha$ is the weight of entropy.

It can be seen from Figure 3 that there are mainly three parts of SAC: the first is the replay buffer for storing data $\left(s_t,a_t,r_t,s_{t+1}\right)$ to train the networks, the second is the critic network used to generate the Q value, and the third is the actor network to generate the action value.

Four networks contain two Q networks and two target Q networks for the critic networks, where the target Q networks have the same structure as the Q networks. The addition of the target Q network improves the stability of network training. For example, assume that the corresponding weights of the four networks are ${\theta }_1$, ${\theta }_2$, ${\theta }_1^{\prime }$, and ${\theta }_2^{\prime }$, where the first two weights are updated through network training shown in Equation (17), and the latter two are updated through the exponential moving average method on the basis of ${\theta }_1$ and ${\theta }_2$ shown in Equation (19) [35], where $\tau \in (0,1]$ is the target smooth factor.

$${\theta }_i={\theta }_i+\left[\underset{Q}{min}{Q}_{\pi }\left(s_t,a_t;{\theta }_i\right)-Q_s\left(s_t,a_t\right)\right]\nabla Q\left(s_t,a_t;{\theta }_i\right)$$

(17)

where $Q_{\pi }\left(s_t,a_t;{\theta }_i\right)$ represents the Q network output when the network inputs are $s_t$ and $a_t$ and the network weight is ${\theta }_i$. $Q_s$ is expressed in Equation (18).

$$Q_s\left(s_t,a_t\right)=r\left(s_t,a_t\right)+\gamma \left[Q_s\left(s_{t+1},a_{t+1}\right)-\alpha log\left(\pi \left(a_{t+1}\left|s_{t+1}\right.\right)\right)\right]$$

(18)

$${\theta }_i^{\prime }=\tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$$

(19)

For the actor-network, its purpose is to generate the optimal action-value through the input of the state. In SAC, supposing that the weight of the actor-network is $\varphi$, the loss function is [34]

$$J_{\pi }\left(\varphi \right)=D_{KL}\left(\pi \left(·\left|s_t\right.\right)∥\frac{exp\left\{Q_s\left(s_t,·\right)\right\}}{Z\left(s_t\right)}\right)$$

(20)

where $D_{KL}$ is the Kullback–Leibler divergence and $Z\left(s_t\right)$ is the log partition function. Since $Z\left(s_t\right)$ has little effect on the update of the weight $\varphi$, it can be ignored, so Equation (20) can be rearranged as

$$J_{\pi }\left(\varphi \right)=log\left({\pi }_{\varphi }\left(a_t\left|s_t\right.\right)\right)-Q_s\left(s_t,a_t\right)$$

(21)

The action $a_t$ can be obtained by the network input $s_t$ and the weight $\varphi$ [36], which is expressed as

$$a_t=f_{\varphi }\left({\epsilon }_t;s_t\right)=f_{\varphi }^{\mu }\left(s_t\right)+{\epsilon }_t{f}_{\varphi }^{\sigma }\left(s_t\right)$$

(22)

where ${\epsilon }_t$ is sampled from a Gaussian distribution, and $f_{\varphi }^{\mu }\left(s_t\right)$ and $f_{\varphi }^{\sigma }\left(s_t\right)$ denote the mean value and covariance value of the network output.

4.2. The Design of Environment and the Agent

The main idea of SAC optimization is introduced above, and the optimal strategy $\pi {}^{*}$ is obtained by maximizing Q, that is, $\pi {}^{*}=\underset{a}{max}Q\left(s,a\right)$. The whole process is realized by the continuous interaction between the environment and the agent. In this study, the environment refers to the power system, including LADRC, and the agent is similar to the human brain and can make decisions. To optimize the parameters of LADRC, it is necessary to define the states, actions in the environment, and the reward function in the agent.

We define the state as follows:

$$\left\{\begin{array}{c}{s}_1\left(k\right)=AC{E}_1\left(k\right)\hfill \\ s_2\left(k\right)=AC{E}_2\left(k\right)\hfill \end{array}\right.$$

(23)

where k is the simulation step.

The actions are the nine controller parameters to be optimized:

$$\left\{\begin{array}{c}{a}_1={\omega }_{o1},a_2={\omega }_{c1},a_3=b_{01}\hfill \\ a_4={\omega }_{o2},a_5={\omega }_{c2},a_6=b_{02}\hfill \\ a_7={\omega }_{o3},a_8={\omega }_{c3},a_9=b_{03}\hfill \end{array}\right.$$

(24)

The reward function is the primary basis for how the agent reacts, so we establish the reward function according to the state. In this study, our control target is that the $ACE$ in the two areas can be stabilized to 0. Therefore, we give the reward function as

$$r=-\left|s_1\right|-\left|s_2\right|$$

(25)

The whole process is shown in Figure 4, which mainly includes two stages. One is the training stage for training the connected critic networks and the actor-network, and the other is the online stage for using the trained actor-network to select parameters online.

5. Simulation Results and Discussion

To verify the effectiveness of the proposed method, we conducted simulation experiments on the two-area six-unit interconnected power system shown in Figure 1.

Table 2. Parameter settings during SAC training.

Parameter Description	Value
Random mini-batch	32
Learning rate of critic networks	${10}^{-4}$
Learning rate of actor network	${10}^{-4}$
Discount factor	0.9999
Target smooth factor	0.001
Simulation run time	20 s
Simulation sample step	0.01 s
Number of iterations	500

shows the parameter settings during the SAC training process, where these parameters are chosen by multiple attempts, and

Table 3. Upper and lower bounds of controller parameters.

	${\mathit{\omega }}_{\mathit{o}}$	${\mathit{\omega }}_{\mathit{c}}$	${\mathit{b}}_0$
Thermal power plant	[20,80]	[10,30]	[10,50]
Hydropower plant	[20,80]	[10,30]	[10,500]
Gas turbine plant	[20,80]	[10,30]	[10,500]

shows the optimization ranges of the nine parameters to be adjusted.

The critic network consists of a fully connected neural network with two hidden layers, in which the numbers of input and output neurons are 11 and 1, respectively, and the numbers of hidden layer neurons are 400 and 300, respectively. As for the actor-network, the structure is shown in Figure 5.

In addition, to enable the proposed method to better cope with the impact of load disturbances, during the training process, we randomly add the step load disturbance of 0–0.03 p.u. to area 1 and area 2 in each iteration. We present the simulation verification results in the following subsection.

5.1. Performance Study of SAC-LADRC

Assuming that a load step disturbance of $\Delta P_{d1}=0.01$ p.u. is added to the first area at t = 0, the response curve is shown in Figure 6, and the parameters optimized by SAC are shown in Figure 7, Figure 8 and Figure 9. To verify the effectiveness of the proposed method, Figure 7 also shows the control effect of the PID controller optimized by the differential evolution (DE) algorithm [37], the PID controller optimized by the hybrid stochastic fractal search and pattern search (hFSF-PS) algorithm [38], the PID controller optimized by the teaching learning based optimization (TLBO) algorithm [39], and the LADRC controller with conventional parameters. We select the parameters of LADRC as ${\omega }_{o1}={\omega }_{o2}={\omega }_{o3}=50$, ${\omega }_{c1}={\omega }_{c2}={\omega }_{c3}=15$, and $b_{01}=50,b_{02}=b_{03}=500$.

Table 4. Comparison of performance indicators of different control methods.

Control Method	$\mathbf{\Delta }{\mathit{f}}_1$			$\mathbf{\Delta }{\mathit{f}}_2$			$\mathbf{\Delta }{\mathit{P}}_{\mathit{tie}}$			$\mathit{ITAE}\times {10}^{-5}$
	${\mathit{U}}_{\mathit{sh}}$	${\mathit{O}}_{\mathit{sh}}$	${\mathit{T}}_{\mathit{s}}$	${\mathit{U}}_{\mathit{sh}}$	${\mathit{O}}_{\mathit{sh}}$	${\mathit{T}}_{\mathit{s}}$	${\mathit{U}}_{\mathit{sh}}$	${\mathit{O}}_{\mathit{sh}}$	${\mathit{T}}_{\mathit{s}}$
DE-PID	−268	202	13.91	−223	8.11	8.45	−48	1.92	9.4	41,469
hSFS-PS-PID	−217	41	8.59	−145	24	7.34	−34	2.50	3.84	18,289
TLBO-PID	−155	21	6.17	−64	10	6.44	−16	1.87	2.82	9017
LADRC	−42	39	2.52	−7.53	6.84	2.36	−2.11	0.88	0	848.7
SAC-LADRC	−9.97	2.64	0.03	−0.25	0.023	0	−0.07	0.004	0	2.566

The magnitude of overshoot and undershoot is ${10}^{-4}$. The settling time means that the variable reaches and remains at $\pm 0.05\%$.

shows the quantification results of performance indicators, including overshoot $O_{sh}$, undershoot $U_{sh}$, settling time $T_s$, and integral of time-weighted absolute error ($ITAE$) for frequency deviation and tie-line exchanged power. We calculate $ITAE$ based on Equation (26).

$$ITAE={\int }_0^T\left(\left|\Delta f_1\right|+\left|\Delta f_2\right|+\left|\Delta P_{tie}\right|\right)tdt$$

(26)

It can be seen from the partially enlarged view that the proposed method has obvious advantages in dynamic performance compared with other control methods, that is, the performance of SAC-LADRC has the smallest overshoot and the shortest response time. Furthermore, since the output of the actor-network is a distribution of parameters, the parameter results obtained by the proposed method will fluctuate in a small range, which is different from other reinforcement learning algorithms like Q learning, which will obtain stable parameter values when the system is stable.

5.2. Robustness Test of SAC-LADRC

To show the ability of the proposed method to suppress different load disturbances, we add the step load disturbance of $\Delta P_{d1}=0.02$ p.u. to the first area at t = 0 s, the step load disturbance of $\Delta P_{d2}=0.01$ p.u. to the first area at t = 10 s, and the step load disturbances of $\Delta P_{d1}=0.01$ p.u. and $\Delta P_{d2}=0.02$ p.u. to the second area at t = 20 s. The simulation results of the above-mentioned trained actor-network are shown in Figure 10.

It can be seen from Figure 10 that, despite the existence of various disturbances, the proposed method can still overcome the influence of load disturbances so that the frequency deviation and tie-line exchanged power quickly stabilize to 0.

6. Conclusions

We studied the LFC problem for a two-area interconnected power system with thermal power plants, hydropower plants, and gas turbine plants. Because traditional PID controllers are unable to meet the system’s high-efficiency operation requirements, we designed LADRC according to the characteristics of different power supplies. To further improve the control performance of LADRC, we used SAC for adaptive adjustment of multiple parameters. To be specific, according to the requirements of the LFC system, we designed the reward function based on the area control error (ACE). Compared with other methods such as DE-PID, hSFS-PS-PID, TLBO-PID, and traditional LADRC, the proposed method dramatically improves dynamic performance, including overshoot, undershoot, and settling time. Moreover, the response curves obtained by the trained agent under the influence of different step load disturbances show that the proposed method has good robustness. Generally speaking, the proposed method has the advantages of small error, short stabilization time, and good robustness, so it is an effective method to solve the LFC problem.

In future work, we will consider the application of the SAC algorithm in OPF [40,41]. Moreover, the high proportion of renewable energy is a development trend of the current power system. However, the large-scale grid connection of intermittent renewable energy represented by wind power and photovoltaics has brought challenges to the operation of the power system. Thus, the power system containing the renewable energies will be a focus of our research.