Data-Driven Automatic Generation Control Based on Learning to Coordinate and Teach Reinforcement Mechanism

Abstract

Large-scale integration of renewable energy introduces significant random perturbations to the power system, disrupting the symmetry and balance of active power, which complicates the stabilization of the system’s frequency. Inter-regional energy cooperation plays a crucial role in maintaining the symmetry and balance of the overall power system’s active power. However, when the power system area is expanded, automatic generation control (AGC) based on reinforcement learning faces the challenge of not being able to leverage the prior experience of the original system topology to train the new area, making it difficult to quickly develop an effective control strategy. To address these challenges, this paper proposes a novel data-driven AGC method that employs a multi-agent reinforcement learning algorithm with a learning to coordinate and teach reinforcement (LECTR) mechanism. Specifically, under the LECTR mechanism, when the power system region expands, agents in the original region will instruct the agents in the newly merged region by providing demonstration actions. This accelerates the convergence of their strategy networks and improves control accuracy. Additionally, the proposed algorithm introduces a double critic network to mitigate the issue of target critic network value overestimation in reinforcement learning, thereby obtaining higher-quality empirical data and improving algorithm stability. Finally, simulations are conducted to evaluate the method’s effectiveness in scenarios with an increasing number of IEEE interconnected grid areas.

1. Introduction

In order to solve the fossil fuel crisis and environmental pollution, a new power system primarily based on renewable energy has been rapidly developing in recent years [1,2,3]. The integration of a high proportion of large-capacity renewable energy sources into the grid leads to significant fluctuations in grid frequency, which undermines the stability of the grid system.

To ensure the safe and stable operation of the power system, researchers proposed automatic generation control (AGC) [4,5,6,7,8] as a secondary frequency regulation strategy. AGC maintains power system frequency stability by achieving “symmetry” between the amount of power generated by the system and the amount of power required on the load side, ensuring that both the two are fully matched in quantity and highly synchronized in time. The AGC controller typically takes power system frequency deviation and tie-line power deviation as inputs, and outputs power adjustment commands for each frequency-regulating unit. It maintains the frequency stability of the power system by regulating the “symmetry” between the generated power and the load demand, namely that the two are fully matched in quantity and highly synchronized in time. In [9], a new two-degree-of-freedom combined proportional–integral–derivative control scheme is applied to an interconnected power system to minimize frequency deviations. In [10], the authors present a novel sliding mode control method for load frequency control, which offers advantages such as reduced overshoot and faster response compared to traditional PID control. These aforementioned AGCs are generally designed on the basis of accurate physical system models. However, they face challenges in accurately controlling systems with inherent uncertainties [11] and in coordinating multi-area controllers [12].

As an alternative, data-driven algorithms—particularly reinforcement learning [13,14,15]—have been increasingly applied in the field of AGC due to their powerful search and learning capabilities. This data-driven AGC method constructs a decision-making agent centered around neural networks, allowing the agent to dynamically interact with the interconnected power system environment during the offline training phase. By combining a reward function to quantitatively assess the effectiveness of generation control commands, the agent’s parameters are gradually optimized, ultimately approaching the global optimal solution. After training, the agent can directly generate generation control commands upon acquiring data from the interconnected power system environment, significantly enhancing the computational efficiency of online decision-making.

In [16], the authors proposed a data-driven method for continuous-action AGC; however, this approach relies on single-agent reinforcement learning, which neglects cooperation among areas, leading to unstable training outcomes. In [17], the authors analyzed how positioning agents in each area to obtain global information can stabilize the training environment. Consequently, implementing a multi-agent control algorithm to regulate interconnected power systems has become essential. In [18], the authors proposed a multi-agent deep reinforcement learning method based on continuous action space, where multiple agents are trained to be able to independently output optimal power generation control commands so as to effectively respond to the control demands in complex power system environments. In [19], the authors introduced networks with varying exploration rates during the exploration process and employed categorical experience replay to enhance the utilization of experience. In [20], the authors proposed a centralized learning and decentralized execution automatic generation control framework based on a global action-value function multi-agent deep reinforcement learning algorithm, which can adaptively obtain the optimal coordinated control strategy for multiple LFC controllers. These studies demonstrate the successful application of multi-agent reinforcement learning in AGC, yielding significant results.

However, when a new area is integrated into the original interconnected power system, the previously mentioned algorithms become inefficient, and the training time increases as additional areas are incorporated. Meanwhile, the agents discussed in the aforementioned paper utilize only a single critic network. The AGC based on this method encounters challenges related to increased estimation errors [21] and overfitting during the training process. Consequently, the agent may mistakenly perceive certain actions as more beneficial for long-term rewards, leading to the adoption of suboptimal strategies.

To address the challenges associated with data-driven AGC, a MADRL control method based on the LECTR mechanism [22] is proposed for interconnected power systems. The approach introduces a double-target critic network delay update strategy to prevent the overestimation of target critic network value and enhance learning efficiency. Simultaneously, the newly integrated regional agents function as novice agents and interact with the expert agents of the original interconnected power system. The LeCTR mechanism introduces dynamic switching of agent roles, building upon the teach reinforcement mechanism. This enhanced interaction method further facilitates the learning of agents to collaborate and accelerates the optimization of the novice agent network. Ultimately, the trained agents are deployed within the power system environment to generate power commands in real-time.

The main contributions of this paper are as follows:

(1)

A novel multi-agent distributed coordinated control framework based on the LECTR mechanism is proposed. This framework improves the training speed of the newly added area agent when the regions of the power system are expanded.

(2)

During the training of the agents, we employed a double-target critic network architecture in place of a single critic network. This structure reduces the variance in target critic network value estimation, thereby enhancing the learning performance of the multi-agent system and facilitating superior control strategies.

(3)

We developed a cooperative reward function considering the control error of each region within the interconnected system, effectively guiding the training of the reinforcement learning algorithm.

The paper is structured as follows: In Section 2, we provide a detailed description of the interconnected power system model. Section 3 introduces a novel data-driven AGC framework. Section 4 describes the training process of the AGC agent utilizing the LECTR mechanism. Section 5 evaluates the effectiveness of this method in an expandable power system through simulation. Finally, in Section 6, we summarize the conclusions derived from this study.

2. Muti-Area Power System Model

Figure 1 presents a typical linearized interconnected power system model [23], expressed in transfer function form. The power system consists of various components, including generators, governors, steam turbines, renewable energy sources, and loads. The agents in each area manage the total power signal for that region, with the goal of minimizing frequency deviations and tie-line power fluctuations. They achieve this by adjusting the output of each unit within the area to optimize control performance. The dynamics of the system within the i-th area of a multi-area power system can be represented by a set of differential equations.

$$\Delta {\dot{f}}_i={\displaystyle \frac{1}{2H_i}}\left(\Delta P_{\mathrm{m}i}+\Delta P_{\mathrm{R}\mathrm{E}\mathrm{S}i}-\Delta P_{\mathrm{L}i}-\Delta P_{\mathrm{t}\mathrm{i}\mathrm{e},i}\right)-{\displaystyle \frac{D_i}{2H_i}}\Delta f_i$$

(1)

$$\Delta {\dot{P}}_{\mathrm{m}i}={\displaystyle \frac{1}{T_{\mathrm{t}i}}}\Delta P_{\mathrm{g}i}-{\displaystyle \frac{1}{T_{\mathrm{t}i}}}\Delta P_{\mathrm{m}i}$$

(2)

$$\Delta {\dot{P}}_{\mathrm{g}i}={\displaystyle \frac{1}{T_{\mathrm{g}i}}}\Delta P_{\mathrm{c}i}-{\displaystyle \frac{1}{R_i{T}_{\mathrm{g}i}}}\Delta f_i-{\displaystyle \frac{1}{T_{\mathrm{g}i}}}\Delta P_{\mathrm{g}i}$$

(3)

$$\Delta {\dot{P}}_{\mathrm{t}\mathrm{i}\mathrm{e},i}=2\pi {\sum }_{j=1,j\ne i}^N T_{ij}\left(\Delta f_i-\Delta f_j\right)$$

(4)

$$T_{ij}={\displaystyle \frac{\left|V_i{V}_j\right|}{X_{ij}}}cos\left({\delta }_i^0-{\delta }_j^0\right)$$

(5)

where ${\delta }_i^0, V_i$ are power angle and terminal voltage of equivalent machines for area i. $X_{ij}, T_{ij}$ are tie-line reactance and synchronous coefficient between area i and j. ${\Delta P}_{mi}, {\Delta P}_{gi}$ are generator mechanical output and valve position for i-th area. $T_{gi}, T_{ti}$ are governor time constant and turbine time constant for i-th area. $D_i, H_i, R_i$ are the machine damping coefficient, synchronous machine inertia, and speed drop for i-th area. $\Delta P_{\mathrm{L}i}$, $\Delta P_{\mathrm{R}\mathrm{E}\mathrm{S}i}$ represent the load disturbance and the random disturbance caused by the renewable energy. The generation commands, denoted as $P_{\mathrm{c}i}$, are typically determined based on the area control error (ACE) signal, which is denoted by $ACE=B_i∆f_i+∆P_{\mathrm{t}\mathrm{i}\mathrm{e},i}$. The bias factor $B_i$ is determined using the equation $B_i=\left({\displaystyle \frac{1}{R_i}}\right)+D_i$.

In a multi-region interconnected power system, AGC controllers in each region not only regulate locally based on the ACE information within their respective regions, but also collaborate with AGC controllers in other regions. This collaboration ensures optimal cross-region power exchange and frequency regulation through the sharing of information between interconnected regions, effectively addressing the challenges posed by fluctuating and intermittent renewable energy sources.

3. Framework for AGC Based on MADRL Algorithm with the LECTR Mechanism

3.1. Action Space

The actions of each agent determine the output of the unit in the area, and the action space is shown below:

$$a_i=\left[A_{oder-i}\right]$$

(6)

where $a_i$ is the action of the i-th agent and $A_{oder-i}$ is the regulation command of the i-th agent.

3.2. State Space

The agent generates control actions by sensing information regarding the state of the power system environment. The combination of state information constitutes the state space of the agent, which in this paper, includes the following:

$$S_i=[{ACE}_i,{∆f}_i,P_{\mathrm{t}\mathrm{i}\mathrm{e},i}]$$

(7)

where ${ACE}_i$ represents the ACE value for i-th region, ${∆f}_i$ represents the frequency deviation for i-th region, and $P_{tie,i}$ represents the tie line deviation for i-th region.

3.3. Reward Function

To achieve optimal AGC performance in this system, minimizing the ACE is crucial. The reward function is designed to account for both system frequency deviation and tie-line power. If the ACE becomes excessively large during the training process, the reward function value is assigned a more negative value. The scaling factor $\epsilon$ determines the relative impact of ACE and power output variations on the reward and is set to 0.5. Additionally, the term ${{∆P}_{oder-i}}^2={(P_i-P_i^{\prime })}^2$ is introduced as part of the reward to guide the algorithm in avoiding large fluctuations in the generation command.

$$\begin{array}{c}{r}_i^t=-\sum _{i=1}^n\epsilon {\left(B_i{∆f}_i+{∆P}_{\mathrm{t}\mathrm{i}\mathrm{e},i}\right)}^2+\left(1-\epsilon \right){\left(P_i-P_i^{\prime }\right)}^2+A\\ \left\{\begin{array}{c}A=0 \left|{ACE}_i\right|<5 MW\\ A=-10 \left|{ACE}_i\right|\ge 5 MW\end{array}\right.\end{array}$$

(8)

3.4. Overall Training Framework

As shown in Figure 2, the proposed method is divided into two main procedures: the first procedure focuses on centralized training of the agents. The agents include an actor network $\chi (o;{\theta }_i)$ and a critic network $M(o,a;{\phi }_i)$. In the centralized training procedure, the actor network $\chi (o;{\theta }_i)$ observes the state $o_i$ and sends generation commands $a_i,i\in \left[1,n\right]$ to the generators in their respective areas. Subsequently, the actor network receives the updated state of the system following the interaction, which is expressed as $o_i^{\prime },i\in \left[1,n\right]$. At each time step, the reward value $r_i,i\in \left[1,n\right]$ for the action taken in each area is calculated using a specific reward function. Finally, the data ($o_i{,a}_i,r_i,o_i^{\prime })$ will be added to the experience pool. The critic network samples data from the experience pool to evaluate the effectiveness of new actions, which subsequently updates the actor network. This process facilitates the training of multi-agent systems. The second procedure involves agents that completed their training (i.e., the expert agent j) providing action suggestions for power generation commands to the agent in the newly added region (i.e., the novice agent i). This approach accelerates the the novice agent i’s learning of control strategies for cooperative control within the interconnected power system. After offline training optimization, the agents are deployed in an interconnected power system to enable efficient real-time control.

4. Training for Agent Based on the LECTR Mechanism

The data-driven AGC method, based on the LECTR mechanism, incorporates this mechanism to enhance the convergence speed of newly added area agents. Additionally, it introduces a double-target critic network with a delayed update strategy to address the issue of overestimation in the target critic network’s value. In the following subsections, we will present the enhanced MADRL algorithm and outline the specific steps for implementing the LECTR mechanism.

4.1. MADRL with a Double Target Critic Network Delay Update Strategy

The purpose of the improved MADRL algorithm presented in this paper is to facilitate updates to the parameters of the actor network parameters, maximizing the cumulative expected return $J$.

$$J\left({\theta }_i\right)=E_{O~D,a~\chi }\left[{\sum }_{t=0}^T{\gamma }^t{r}_i^t\right]$$

(9)

where $\gamma$ is the reward discount factor, suggesting that agents are more focused on the long-term performance of the overall reward.

In the improved MADRL algorithm, the learning process involves training a centralized critic network that has access to the policies of all agents. This centralized critic network is represented as a centralized state value function.

$$M(o,a_1,\cdots ,a_n;{\phi }_i)$$

(10)

where o represents the overall observation value, and $o_i$ represents the local observations of each i-th agent. $\chi (o;{\theta }_i)$ represents the action computed by the actor network for a given state $o_i$.

During the update of the parameters for the i-th agent, the gradient descent methods are typically employed. The parameters of the actor network are updated by iterating in the direction opposite to the gradient. This approach allows us to gradually improve the performance of the agent.

$${\mathsf{\nabla }}_{{\theta }_i}J=E_{o,a_{i~D}}\left[{{\mathsf{\nabla }}_{{\theta }_i}\chi (o;{\theta }_i){\mathsf{\nabla }}_{a_i}M(O,a_1,\cdots ,a_n;{\phi }_i)}_{{|a}_i=\chi (o;{\theta }_i)}\right]$$

(11)

The experience replay area D here includes $\left(o,a_1,\dots ,a_n,r_1,\dots ,r_n,o^{\prime }\right)$ of all agents.

Then, we minimize the loss function, which can be expressed as the difference between the target value and the evaluation value $M(o,a;{\phi }_i)$. The parameters of the critic network can be adjusted.

$$y_i={r_i+\gamma M^{-}(o^{\prime },a_1^{\prime },\cdots ,a_n^{\prime };{\phi }_i^{-})|}_{a_i^{\prime }={\chi }^{-}(o_i^{\prime };{\theta }_i^{-})}$$

(12)

$$L={\displaystyle \frac{1}{M}}{\sum }_{i=1}^M{(y_i-M(o,a_1,\cdots ,a_n;{\phi }_i\left)\right)}^2$$

(13)

where $M^{-}(o,a;{\phi }_i^{-})$ and $\chi (o_i^{\prime };{\theta }_i^{-})$ are the target critic and actor networks. The target critic network and actor network initially replicate the parameters of the original critic and actor networks during the offline training process. Subsequently, the parameters of the target networks are updated with a delay, which enhances the stability of the updates for both the critic and actor networks.

To address the issue of target critic network value overestimation in the traditional MADRL algorithm, a double-target critic network $M^{-}(o,a;{\phi }_i^{-})$ is employed to mitigate the overestimation of the state value function. It is essential to consider the information from all agents in MADRL. Consequently, during training, all agents should have access to comprehensive state information, allowing them to observe the entire state space.

In addition, regularization techniques are introduced to smooth the target critic network value and prevent overfitting. Normally distributed random noise, denoted as ε, is added during the update of the critic network to enhance stability against fluctuations. The improved method for calculating the target value is presented below:

$$\epsilon ~clip(N\left(0,\sigma \right),-c,c)$$

(14)

$$y_i=r_i+{min}_{j=\mathrm{1,2}}\gamma M^{-}(o^{\prime },a_1^{\prime }+\epsilon ,\cdots ,a_n^{\prime }+\epsilon ;{\phi }_i^{-})$$

(15)

Then, the parameters of the target networks are updated with a delay, which enhances the stability of the updates for both the critic network and the actor network. The updates for the actor network, critic network, target actor network, and target critic network are as follows:

$$\begin{array}{c}{\phi }_i={\phi }_i-{\alpha }^{{\phi }_i}{\mathsf{\nabla }}_{{\phi }_i}L\left({\phi }_i\right)\hspace{1em}\hspace{1em}\hspace{1em}{\phi }_i^{-}=\tau {\phi }_i+(1-\tau ){\phi }_i^{-}\\ {\theta }_i={\theta }_i+{\alpha }^{{\theta }_i}{\mathsf{\nabla }}_{{\theta }_i}J\left({\theta }_i\right)\hspace{1em}\hspace{1em}\hspace{1em}{\theta }_i^{-}=\tau {\theta }_i+(1-\tau ){\theta }_i^{-}\end{array}$$

(16)

where ${\alpha }^{{\phi }_i}$ and ${\alpha }^{{\theta }_i}$ represent the learning rates for the critic and actor networks, while τ denotes the soft update rate for the target networks.

4.2. The Application of the LECTR Mechanism in Expandable Power System

This subsection outlines the procedure for executing a cooperative multi-region load frequency control task after novice agent i received guidance from expert agent j. The entire process is further divided into two phases (see Figure 3).

Phase I: In each learning iteration, the actor networks of the expert agent and the novice agent interact with the power system environment to update their actor network parameters ${\theta }_i$ and ${\theta }_j$ based on the actions suggested by the expert agent. This process utilizes the improved MADRL algorithm $L_i$ and $L_j$ ($L_i$ stands for the learning algorithm of the novice agent algorithm and $L_j$ for the learning algorithm of expert agent algorithm). We call this phase $P_{task}$.

Phase II: This phase primarily focuses on updating the suggestion network of the expert agent and request network of the novice agent based on the advising-level rewards generated during the process. We refer to this phase as $P_{advise}$.

The agent in the LeCTR mechanism can assume the role of a novice agent, an expert agent, or both simultaneously. Additionally, the agent is capable of seeking advice regarding its own state while providing guidance to teammates in different states. The LeCTR mechanism uses distinct advising-level observations for the request network of the novice agent and the suggestion network of the expert agent. The request network of the novice agent ${\tilde{\chi }}_N^i({\tilde{O}}_N^i;{\tilde{\theta }}_N^i)$ for agent i decides when to request advice using observation ${\tilde{O}}_N^i= <o^i,M^i(o^i,a^i)>$ where $s^i$ and $M^i(o^i,a^i;{\phi }_i)$ are calculated by the critic network. Through ${\tilde{O}}_N^i$, agent i observes a measure of its local task-level observation and actor network state. Similarly, the expert agent j’s suggestion network ${\tilde{\chi }}_E^j({\tilde{O}}_E^j;{\tilde{\theta }}_E^j)$ uses advising-level observation ${\tilde{O}}_E^j= <o^i,M^i\left(o^i,a^i;{\phi }_i\right),M^j\left(o^i,a^i;{\phi }_j\right)>$ to decide when/what to advise. ${\tilde{O}}_E^j$ provides expert agent j with a measure of the task-level state/knowledge of novice agent i (via $o^i$ and $M^i\left(o^i,a^i;{\phi }_i\right)$ and of its own task-level knowledge given the student’s context (via $M^j\left(o^i,a^i;{\phi }_j\right)$). The novice perspective action of the agent is ${\tilde{a}}_N^i={\tilde{\chi }}_N^i({\tilde{O}}_N^i;{\tilde{\theta }}_N^i)\in${request advice, do not request advice}. Meanwhile, expert agent j decides what to advise: either an action from action space of novice agent i, or a special no-advice action $\widetilde{a_{\varnothing }}$. Thus, the expert perspective action for agent j is ${\tilde{a}}_E^j={\tilde{\chi }}_E^j({\tilde{O}}_E^j;{\tilde{\theta }}_E^j)\cup \widetilde{a_{\mathrm{\varnothing }}}$. Given no advice, novice agent i executes the originally intended action $a^i$. However, given advice ${\tilde{a}}_E^j$, novice agent i executes action ${\tilde{a}}_E^j$. Following advice execution, agents collect task-level experiences and update their actor network.

In Phase I, we set the reward function to evaluate the contribution of the actor network output actions to stabilize the power system frequency. In Phase II, we need to set up relevant advising-level reward functions to evaluate the contribution of the outputs of the suggestion network and the request network to accelerate the learning of the newly added area agents. Since request network ${\tilde{\chi }}_N^i$ and suggestion network ${\tilde{\chi }}_E^j$ must coordinate to help student i learn, they receive identical advising-level rewards ${\tilde{r}}_E^i$ = ${\tilde{r}}_N^j$. In this paper, we set up the advising-level reward function as follows:

$${max}_a{M}_E\left(o^i,a;{\phi }_j\right)-M_E\left(o^i,a^i;{\phi }_j\right)$$

(17)

The above proposal-level reward function measures the strategic deviation between the action $a_i$ chosen by the novice agent i in state $o_i$ and the optimal action evaluated by the expert agent j. The proposed reward function measures the deviation between the optimal action and the optimal action evaluated by the expert agent j. By quantifying the difference in the value of the critic network, the expert agent learns to estimate the ‘cognitive error’ of its teammates in a given state and decides whether or not to intervene accordingly. This setup allows the expert agent to improve the learning collaboration between them during the teaching process, so that they not only focus on their own task benefits, but also understand and contribute to the learning process of their teammates.

When the expert agent outputs the suggested actions, ${\tilde{\chi }}_E^j$ and ${\tilde{\chi }}_N^i$ will be rewarded, otherwise the reward is zero. The parameter-updating methods of the suggestion network and the request network are referred to in the literature [22].

In order to facilitate collaborative learning among agents, the LeCTR framework employs a novel teaching mechanism that supports dynamic role switching by structurally modeling observations, actions, and strategies at the instructional level. Specifically, $\tilde{\chi }$ is $<{\tilde{\chi }}_N^i,{\tilde{\chi }}_N^j,{\tilde{\chi }}_E^i,{\tilde{\chi }}_E^j>$; $\tilde{O}$ is $<{\tilde{O}}_N^i,{\tilde{O}}_N^j,{\tilde{O}}_E^i,{\tilde{O}}_E^i>$; $\tilde{A}$ is $<{\tilde{a}}_N^i,{\tilde{a}}_N^j,{\tilde{a}}_E^i,{\tilde{a}}_E^j>$; $\tilde{r}$ is $r_E^j+r_N^j$; and $\tilde{\theta }$ is the parameter of strategy $\tilde{\chi }$. $\tilde{N}$ is the experience pool of data related to the training of the request network and the suggestion network; and $M(\tilde{S},\tilde{A};\tilde{\theta })$ is the critic network for evaluating the actions of the suggestion network and the request network. With the unified modeling described above, LeCTR allows agents to simultaneously request assistance as a ‘novice agent’ and provide feedback as an ‘expert agent’ at any time step, enabling bi-directional switching of collaborative behaviors.

The training process of the LECTR mechanism and algorithmic flow of algorithm for training agents are shown in Figure 4.

5. Case Study

In this section, we first establish a simulation of a three-area power system to evaluate the control performance of the improved MADRL algorithm compared to other control algorithms, including deep Q-networks (DQN) [24] and the multi-agent deep deterministic policy gradient (MADDPG) [25]. Subsequently, we demonstrate the advantages of the improved MADRL with the LECTR mechanism in accelerating the learning process of the newly added agent and improving the control performance in an expanded interconnected power system.

5.1. Experimental Environment

The simulations were carried out using the deep learning framework PaddlePaddle, with the following hardware configuration: an AMD Ryzen 7 2700 eight-core processor operating at 3.20 GHz, an NVIDIA GeForce RTX 2060 GPU, and a total of 16 GB of RAM. The sampling period of the controller during the simulation is 4 s, and the total duration of the disturbance is 1500 s. We set the baseline capacity of the system to 5000 MVA. The characteristics of the power system model presented in this paper are listed in

Table 1. Parameters of the interconnected power system.

Parameter	${\mathit{T}}_{\mathit{g}}$ (s)	${\mathit{T}}_{\mathit{t}}$ (s)	$\mathbf{D}$ (p.u./Hz)	$\mathbf{H}$ (p.u./Hz)	${\mathit{T}}_{\mathit{i}\mathit{j}}$ (p.u./Hz)	$\mathbf{R}$ (Hz/p.u.)
Area 1	0.10	0.35	1.7	24	0.014	0.5
Area 2	0.09	0.32	1.5	19	0.07	0.3
Area 3	0.08	0.33	2.7	22	0.03	0.2
Expanded area 4	0.10	0.35	1.4	17	0.019	0.6

.

For the DQN algorithm, the reward discount factor $\gamma$ and the learning rate $\alpha$ were set to 0.99 and 0.0005, respectively. MADDPG utilized a learning rate of 0.001 for both the actor and critic network, a reward discount factor $\gamma$ of 0.99, and a soft update rate $\tau$ of 0.01. In the proposed method, the normally distributed random noise $\epsilon$ was set to 0.1, and the time interval for delayed update was set to 2. The learning rates of the actor network and critic network were set to 0.002, and a reward discount factor $\gamma$ and soft update rate $\tau$ were set to 0.98 and 0.005. The number of training iterations for all algorithms was set to 30,000.

5.2. Three-Area Power System Disturbance Simulation

5.2.1. Step Load Disturbance

As shown in Figure 5, a step load disturbance is introduced to simulate the load in the power system, which experiences irregular and sudden increases and decreases. The transition process of the improved MADRL algorithm occurs in the shortest time, and the ACE of the improved MADRL algorithm remains at 0 after the step change is applied. At 300 s, 500 s, and 900 s, the absolute values of the maximum ACE changes for the three algorithms DQN, MADDPG, and the improved MADRL are (0.007 p.u., 0.0052 p.u., and 0.0032 p.u.), (0.0100 p.u., 0.0082 p.u., and 0.00526 p.u.), and (0.0066 p.u., 0.0035 p.u., and 0.0023 p.u.), respectively. The improved MADRL reduces the absolute values of the maximum ACE changes compared to the other algorithms by (21.28–53.46%), (23.28–56.85%), and (26.53–54.32%).

5.2.2. Random Renewable Energy Disturbance

Renewable energy sources exhibit considerable randomness and fluctuations. To examine the impact of these stochastic variations in renewable energy on the interconnected power system, we treat renewable energy as a random disturbance without accounting for its secondary frequency regulation capabilities. Figure 6 presents the data for renewable energy sources in a three-area system.

Figure 6 shows that the mean absolute ACE of the improved MADRL algorithm is significantly lower than that of other algorithms, while maintaining ACE within a reasonable range across multiple power system areas. The MADDPG algorithm can reduce the ACE to within 0.005 p.u. However, there are still regions exhibiting unsatisfactory performance. This is because the improved MADRL algorithm introduces a double-target critic network, which enhances the stability of the algorithm’s convergence and prevents over-rewarding. Consequently, the improved MADRL algorithm demonstrates greater adaptability to power system environments characterized by a high proportion of renewable energy sources, resulting in superior control performance. In contrast, DQN is a single-agent algorithm that employs discrete generation control commands. As the interconnected power system expands, the dimensionality of the action space increases exponentially, leading to challenges in both exploration and convergence for the DQN algorithm. As a result, the DQN algorithm exhibits the poorest control performance, resulting in the highest average fluctuations in ACE.

Figure 7 shows the power output of the generator with different algorithms in Area 1. It can be seen that the improved MADRL algorithm has the best stability in the face of renewable energy disturbances, followed by MADDPG. Due to the strong stochastic variation of the output power of the renewable energy sources, the DQN has difficulties keeping up with the load for a short period of time.

The comparison of specific data is presented in

Table 2. Numerical results of regional system performance comparisons.

Method	Reward	Mean Absolute ACE [p.u.]	Largest Variation in Absolute ACE [p.u.]
DQN	−1.95	0.003084164	0.013206667
MADDPG	−1.45	0.001495223	0.007463333
Improved MADRL	−0.84	0.000595075	0.002533333

. The power system data indicate that the mean absolute ACE of other algorithms was 2.5 to 6.10 times higher than that of the improved MADRL algorithm. Furthermore, the improved MADRL algorithm reduces the maximum variation in absolute ACE by 66.08% to 80.83% and achieves a reward value that is 72.61% to 132.14% higher than that of the other algorithms. Therefore, the improved MADRL algorithm demonstrates the best control performance.

5.3. Extension of Power System Disturbance Simulation

This section of the simulation extends the power system to a fourth area to evaluate the control and learning performance of the enhanced MADRL with the LECTR mechanism in a highly stochastic power system environment.

5.3.1. Comparison of Offline Training Speed

Figure 8 shows the reward convergence curves of the newly added regional agents trained offline under the guidance of four different algorithms. The proposed method achieves convergence within 6833 rounds, demonstrating improvements in convergence speed of 58.73%, 67.95%, and 75.01% compared to the other three algorithms (improved MADRL, MADDPG, and DQN), respectively.

This is because we use a trained expert agent from the original power system region to provide action suggestions, which the novice agent in the newly joined area selectively accepts during the training process. This method ensures that the novice agent explores the power system environment through stochastic actions while assimilating the suggested actions from the expert agent’s perspective and effectively translating them into local empirical knowledge. Consequently, this enables the novice agent to learn how to make improved control decisions in a scalable power system environment, thereby reducing the time required for independent learning.

For the other algorithms, there are serious challenges in dealing with the expanding interconnected power system. Each agent in the newly joined area must be retrained individually, which increases the training complexity of the multi-agent algorithms. As a result, the convergence of rewards under these algorithms becomes slower.

5.3.2. Step Load Disturbance

Figure 9 illustrates the change in ACE of the four algorithms following a step disturbance in the extended power system. It is evident that the improved MADRL with the LECTR mechanism recovers to 0 p.u. more rapidly after the disturbance. At 200 s, 600 s, and 900 s, the improved MADRL with the LECTR mechanism reduces the absolute values of the maximum ACE change compared to the other algorithms by 21.28% to 53.46%, 23.28% to 56.85%, and 26.53% to 54.32%, respectively.

5.3.3. Random Renewable Energy Disturbance

Figure 10 shows the disturbance data of renewable energy sources, such as wind power, in the newly extended area. It also compares the control strategies developed by the agent in the fourth area using four different algorithms: the improved MADRL with the LECTR mechanism, improved MADRL, MADDPG, and DQN, based on ACE metrics. The results indicate that the control performance of the improved MADRL with the LECTR mechanism is superior not only to that of DQN and MADDPG, but also shows further enhancement compared to the improved MADRL.

The comparative data for each algorithm are presented in

Table 3. Numerical results of four-area system performance comparisons.

Method	Reward	Mean Absolute ACE [p.u.]	Largest Variation in Absolute ACE [p.u.]	Episode Number of Convergence
Improved MADRL with LECTR	−0.75	0.000658	0.00216	6833
Improved MADRL	−0.94	0.00146	0.00435	16558
MADDPG	−1.25	0.00234	0.00669	21324
DQN	−1.88	0.00433	0.00982	27345

. The improved MADRL with the LECTR mechanism reduces the mean absolute ACE by 54.93% to 84.80%, decreases the largest variation in absolute ACE by 50.03% to 78%, and achieves a reward value that is 25.33% to 150.66% higher than that of other algorithms.

During the expansion of power system regions, it is essential to integrate new agents into the existing multi-agent framework to develop innovative coordination control strategies for the expanded power system area. Simulations demonstrate that our proposed improved MADRL with LECTR algorithm does not require re-centralized training of all regional agents, while maintaining excellent convergence and control performance. The improved MADRL with the LECTR mechanism offers a valuable reference for data-driven AGC in more flexibly interconnected power systems.

6. Conclusions

To achieve real-time frequency response control in extended power systems, we propose a MADRL framework based on the LECTR mechanism for the dynamic optimization of AGC.

(1)

The LECTR mechanism is introduced to facilitate the joint development of control strategies when interconnected power system regions are expanded. The LECTR mechanism enhances the learning speed of agents in the newly integrated region by 58.73% to 75.01% and reduces the average ACE by 54.93% to 84.80% by enabling peer-to-peer action suggestions from agents in the original region to those in the new region.

(2)

In a fixed-area power system environment, the proposed method effectively reduces ACE fluctuations and demonstrates superior control performance.