A Neighbor Feature Aggregation-Based Multi-Agent Reinforcement Learning Method for Fast Solution of Distributed Real-Time Power Dispatch Problem

Abstract

To address the challenges posed by the strong uncertainty of high-proportion renewable energy sources (RES) to the secure and stable operation of distributed real-time power dispatch (D-RTPD) in new-type power systems, this paper proposes an integrated solution combining a neighborhood feature aggregation-based graph attention network (NFA-GAT) and multi-agent deep deterministic policy gradient (MADDPG). First, the D-RTPD problem is modeled as a decentralized partially observable Markov decision process (Dec-POMDP), which effectively captures the stochastic game characteristics of multi-regional agents and the partial observability of grid states. Second, the NFA-GAT is designed to enhance agents’ perception of grid operating states: by introducing a spatial discount factor, it realizes rational aggregation of multi-order neighborhood information while modeling the attenuation of electrical quantity influence with topological distance. Third, a prior-guided mechanism is integrated into the MADDPG framework to eliminate constraint-violating actions by setting their actor logits to negative infinity, improving training efficiency and strategy reliability. Simulation validations on the IEEE 118-bus test system (75.2% RES installed capacity ratio) show that the proposed method achieves efficient training convergence. Compared with the multi-layer perceptron (MLP) structure, it attains higher cumulative reward values and scenario win rates. When compared with traditional model-driven (ADMM) and data-driven (Q-MIX) methods, the proposed method balances solution efficiency, operational safety (98.7% maximum line load rate, zero power flow violation rate), and economic performance ($12,845 daily dispatch cost), providing a reliable technical support for D-RTPD under high-proportion RES integration.

1. Introduction

1.1. Background and Motivation

At this stage, China’s power grid primarily adopts a “hierarchical and zonal” dispatching system for management. Generally, provincial-level power grids serve as independent dispatching units, with cross-provincial power transmission corridors and direct current (DC) transmission projects acting as coordination means between provincial-level power grids. However, issues such as tight power flow coupling and weak communication connections exist between provincial-level power grids, failing to fully realize resource complementarity and regional mutual support among multiple provincial-level power grids [1,2,3]. Nevertheless, against the backdrop of “carbon neutrality and carbon peaking”, relying solely on long-distance DC external power transmission projects is insufficient to cope with the integration of a high proportion of RES sources. It is therefore necessary to deeply tap into the absorption capacity of provincial/regional power grids and explore a multi-regional power grid collaborative dispatching mode [4,5,6]. Consequently, solving the problem of distributed real-time power dispatch (D-RTPD) holds significant practical importance.

In response to this, this paper models the D-RTPD problem as a decentralized partially observable Markov decision process and proposes a neighborhood feature aggregation-based D-RTPD solution method, which balances solution efficiency and solution quality under limited information exchange.

1.2. Literature Review

Existing research on D-RTPD can be roughly categorized into two types: mathematical optimization-based methods and multi-agent reinforcement learning (MARL)-based methods. For mathematical optimization methods, researchers have proposed various distributed algorithms to address the global optimization problem under limited information interaction. Reference [7] develops a regional decomposition optimal power flow algorithm based on augmented Lagrangian relaxation, which decomposes large-scale problems into regional subproblems but remains a centralized solution in essence, making it inapplicable to scenarios where only neighborhood information is available. Reference [8] proposes a consensus-based distributed control algorithm for economic dispatch (ED), which achieves optimal solutions through iterative updates of power imbalance estimation among agents, but requires massive iterative calculations that limit real-time performance. Reference [9] further combines consensus mechanisms with graph theory to solve dynamic economic dispatch (DED) problems, yet still faces challenges in balancing computational efficiency and solution accuracy under high-proportion RES penetration. Reference [10] design a new group decision-making method that applies two new consensus methods based on sentiment analysis for application in the analysis of texts on social media. The application to a real case of public investment demonstrates the effectiveness of the approach in collaborative contexts that used natural language.

With the development of artificial intelligence, MARL has emerged as a promising approach for collaborative decision-making in multi-agent systems, and has been widely applied in power system dispatching [11,12,13,14,15,16,17,18]. Reference [19] adopts the MADDPG algorithm to train dispatching agents for active-reactive power coordination, realizing efficient regulation of multiple resources but ignoring the objective inconsistency among agents in collaborative tasks. References [20,21,22] applies stochastic consensus maximum entropy reinforcement learning to voltage and reactive power control (VVC), improving communication efficiency but failing to address the insufficient perception of grid global states by individual agents. Reference [23] converts the dispatching problem into a partially observable Markov decision process (POMDP) and uses MADDPG for centralized training and decentralized execution, yet the lack of effective coordination mechanisms still leads to suboptimal collaborative performance. Reference [24] proposes a pricing economic dispatch method based on local multipliers and conic relaxation, which achieves efficient solution under AC power flow constraints but relies on accurate mathematical modeling and lacks adaptability to high-proportion renewable energy uncertainty. Reference [25] combines model-free prediction and fixed-time control for islanded AC microgrids, focusing on robustness but failing to address distributed collaborative dispatch in large-scale multi-regional power grids. Reference [26] proposes a consensus method for group decision-making that, through sentiment analysis, assesses aggressiveness in comments using generative artificial intelligence, specifically the large language models provided by Google. In summary, mathematical optimization methods suffer from low real-time performance due to excessive iterations, while existing MARL-based methods struggle with inadequate neighborhood information utilization, making them difficult to meet the requirements of solution accuracy and algorithm scalability for D-RTPD under high-proportion RESs integration.

High-proportion renewable energy integration poses two core challenges to distributed real-time power dispatch (D-RTPD): partial observability of grid states due to limited local/neighboring information access and RES/load randomness, and communication constraints from inter-regional latency and bandwidth limits that restrict large-scale information interaction. Conventional model-driven methods (e.g., ADMM) assume global information sharing, ignore partial observability, and suffer from low real-time performance due to massive iterations, while existing MARL methods adopt single-agent POMDP modeling, neglect multi-agent stochastic game characteristics and underutilize constrained neighborhood information, leading to inadequate grid state perception. Moreover, neither approach adapts to the core characteristics of power grids with a high proportion of renewable energy, including partial observability and limited information interaction. To address these issues, this paper models D-RTPD as a decentralized partially observable Markov decision process (Dec-POMDP), which captures the above distributed features and converts the complex source–storage coordinated dispatch problem into a MARL-adaptive sequential decision-making problem, laying a rigorous theoretical foundation for the proposed NFA-GAT-MADDPG method.

1.3. Contribution and Summary

The main contributions of this paper are as follows:

(1)

A decentralized partially observable Markov decision process (Dec-POMDP) model tailored for distributed real-time power dispatch is established. By integrating the stochastic game characteristics of multi-regional agents and the partial observability of grid states, the model effectively converts the complex mathematical optimization problem of source–storage coordinated dispatch into a sequential decision-making problem suitable for reinforcement learning. This provides a rigorous theoretical framework for handling the strong uncertainty of high-proportion RES and limited information interaction among regions.

(2)

A neighborhood feature aggregation-based graph attention network (NFA-GAT) is proposed to enhance the agents’ perception of grid operating states. By introducing a spatial discount factor to model the attenuation of electrical quantity influence with topological distance, the network realizes effective aggregation of multi-order neighborhood information. This enables agents to fully capture the spatial correlation of power grid features under limited information interaction, improving the accuracy of policy decisions.

(3)

A multi-agent reinforcement learning framework integrating MADDPG and a prior-guided mechanism is proposed. The MADDPG algorithm ensures stable training of agents under environmental non-stationarity through centralized criticism and decentralized execution, while the prior-guided mechanism eliminates invalid actions violating constraints by setting their actor logits to negative infinity. This synergy enhances the efficiency and reliability of policy learning, enabling agents to generate feasible and optimal dispatching strategies in complex real-time power grid environments.

(4)

Comprehensive case studies are conducted on the IEEE 118-bus test system with the 75.2% installed capacity ratio of RESs. The proposed method is compared with traditional mathematical optimization methods and MARL methods with different network structures. Additionally, the impact of agent quantity on algorithm convergence is analyzed. Numerical results verify its superior performance in solution efficiency and algorithm scalability, which provide practical guidance for the deployment of multi-agent systems in real-world D-RTPD scenarios.

2. Basic Models and Theoretical Methods

2.1. Mathematic Model of D-RTPD

To suppress the strong uncertainty of RES during the D-RTPD phase [27], the power system needs to dispatch flexible resources with fast regulation speeds, and maintain power balance as well as safe and stable operation through the complementary operation mode of renewable energy sources (RESs) and energy storages (ESs). This section mainly uses “source–storage” coordination to address the D-RTPD challenges of power grids with high-proportion RES integration. Therefore, the objective functions of this section include generator output and rescheduling costs, ES operation costs, and line overload penalties, and the D-RTPD model is as follows:

$$\begin{array}{c}{\displaystyle \min \sum _{t=t_0+1}^{t_0+T}\left[{\displaystyle \sum _i^{N_G}\left[C_i\left(P_{i,t}^{DA}+\mathrm{\Delta }{P}_{i,t}\right)+C_i^{redis}\left(\mathrm{\Delta }{P}_{i,t}\right)\right]}+{\displaystyle \sum _j^{N_{ESS}}{C}_j^{ESS}\left(P_{j,t}^{ESS}\right)}+{\displaystyle \sum _L^{N_L}\beta {\left(P_{L,t}-P_L^{\max }\right)}^2}\right]}\end{array}$$

(1)

$$P_{D,t}={\displaystyle \sum _{i\in N_{Gen}}\left(P_{i,t}^{DA}+\mathrm{\Delta }{P}_{i,t}\right)}+{\displaystyle \sum _{j\in N_{ESS}}{P}_{j,t}^{ESS}}+P_{RES,t}+P_{i,t}^{SB}$$

(1a)

$$P_{j,t}^{ESS}=E_j^{rated}\left({\mu }_{j,t}^{dis}{\eta }_j+{\displaystyle \frac{{\mu }_{j,t}^{cha}}{{\eta }_j}}\right)\left(S_{j,t}-S_{j,t-1}\right)$$

(1b)

$${\mu }_{j,t}^{dis}+{\mu }_{j,t}^{cha}\le 1,\forall j\in N_{ESS}$$

(1c)

$$S_{j,t}^{\min }\le S_{j,t}\le S_{j,t}^{\max }$$

(1d)

$$\left|P_{i,t}^{ESS}\right|\le {\psi }_j^{ramp}{E}_{ESS}^{rated}$$

(1e)

$$P_i^{\min }\le P_{i,t}^{DA}+\mathrm{\Delta }{P}_{i,t}\le P_i^{\max }$$

(1f)

$$P_{SB}^{\min }\le P_{i,t}^{SB}\le P_{SB}^{\max }$$

(1g)

$$\left|\mathrm{\Delta }{P}_{i,t}\right|\le P_i^{ramp}$$

(1h)

$$\left|P_{L,t}\right|\le P_L^{\max }$$

(1i)

where $C_i$ and $C_i^{redis}$ are respectively system operation cost and redispatch operation cost. $P_{i,t}^{DA}$ and $\mathrm{\Delta }{P}_{j,t}$ are respectively day-ahead schedule and real-time adjustment of i-th generator. The operation cost of ES is incorporated into the objective function. Specifically, $C_j^{ESS}\left(P_{j,t}^{ESS}\right)=c_j^{ESS}\left|P_{j,t}^{ESS}\right|$ and $P_{j,t}^{ESS}$ denote the charging/discharging power of ES unit j at time t (the power is positive during discharging and negative during charging), and $c_j^{ESS}$ represents the charging/discharging cost coefficient of ES unit j. $P_{i,t}^{SB}$ refers to the output of the balancing unit, and only the upper and lower limits of the output are considered as constraints for the balancing unit. Constraint (1a) is the power balance constraint. Constraint (1b) is the ES charging/discharging power conversion equation, where $E_j^{rated}$ represents the rated capacity of ES unit j, ${\mu }_{j,t}^{cha}$ and ${\mu }_{j,t}^{dis}$ denote the charging and discharging states of ES unit j at time t respectively (0–1 variables), ${\eta }_j$ is the energy conversion efficiency coefficient of ES unit j, and $S_{j,t}$ refers to the state of charge (SoC) of ES unit j at time t. Constraint (1c) is the ES charging/discharging state constraint. Constraint (1d) is the SoC constraint for ES unit j, which is used to avoid overly discharge or charge, and $S_{j,t}^{\min }$ and $S_{j,t}^{\max }$ represent the minimum and maximum SoC values of ES unit j respectively. Constraint (1e) is the charging/discharging ramping constraint for ES unit j, and ${\psi }_j^{ramp}$ denotes the charging/discharging ramping coefficient of the ES. Constraint (1f) is the generator output constraint, with $P_i^{\max }$ and $P_i^{\min }$ being the maximum and minimum output of generator i respectively. Constraint (1g) is the balancing unit output constraint. Constraint (1h) is the generator ramping limit, where $P_i^{ramp}$ refers to the ramping limit of generator i within a single time period. Constraint (1i) is the line transmission capacity constraint, with $P_{L,t}$ representing the line transmission power at time t and $P_L^{\max }$ being the maximum line transmission capacity. $N_{Gen}$ and $N_{ESS}$ denote the sets of generators and ES units respectively.

2.2. MADDPG Algorithm

MADDPG is a deterministic gradient descent algorithm, where each agent requires a critic that acquires global information and an actor that observes local information. Without loss of generality, $\theta =\left[{\theta }_1,\dots ,{\theta }_n\right]$ is used to represent the parameters of the strategies for n agents. For the i-th agent, its cumulative expected reward is:

$$J\left({\theta }_i\right)=E_{s\sim {\rho }^{\pi },a_i\sim {\pi }_{{\theta }_i}}\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{r}_{i,t}}\right]$$

(2)

where s and a_i denote the state observation and action value of the i-th agent, respectively. ${\rho }^{\pi }$ and ${\pi }_{\theta }$ represent the state transition probability and policy function, respectively. Therefore, the gradient formula of its deterministic policy is:

$${\nabla }_{{\theta }_i}J\left({\pi }_{{\theta }_i}\right)=E_{o,a\sim D}\left[{\left.{\nabla }_{{\theta }_i}{\pi }_i\left(a_i\left|o_i\right.\right){\nabla }_{a_i}{Q}_i^{\pi }\left(o,a\right)\right|}_{a_i={\pi }_i\left(o_i\right)}\right]$$

(3)

where ${\pi }_{{\theta }_i}$ denotes the deterministic policy of the i-th agent. $o_i$ and $a_i$ represent the observation and action value of the i-th agent, respectively. $o$ and $a$ denote the observation vector and action vector of the multi-agent system, respectively. D is the replay buffer that stores the trajectories of the multi-agent system. $Q_i^{\pi }\left(o,a\right)$ represents the centralized state-action value of the i-th agent under the deterministic policy. This gradient formula enables iterative optimization of agent policies, driving policy updates toward maximizing the cumulative reward.

For the update method of the critic in centralized training, Temporal Difference (TD) and Double Q-Network approaches are adopted, which can be expressed as:

$$\left\{\begin{array}{c}L\left({\theta }_i\right)=E_{o,a,r,o^{\prime }}\left[{\left(Q_i^{\pi }\left(o,a\right)-y_i\right)}^2\right]\\ y_i={\left.r_i+\gamma {\overline{Q}}_i^{{\pi }^{\prime }}\left(o^{\prime },a^{\prime }\right)\right|}_{a^{\prime }={\pi }^{\prime }\left(o^{\prime }\right)}\end{array}\right.$$

(4)

where ${\overline{Q}}_i^{{\pi }^{\prime }}$ denotes the centralized state-action value under the policy output by the target network. $y_i$ represents the true state value. ${\pi }^{\prime }=\left[{{\pi }^{\prime }}_1,\dots ,{{\pi }^{\prime }}_n\right]$ stands for the target policy with parameters ${\theta }^{\prime }$. $o^{\prime }$ and $a^{\prime }$ denote the observation vector of the multi-agent system at the next time step and the action vector of the target policy at the next time step, respectively.

2.3. Graph Attention Network

RL algorithms rely on neural networks to process large volumes of input information. The attention mechanism can help neural networks quickly screen key features and improve the training efficiency of neural networks. Assume that there exist N electrical quantities as the input vector $\left[x_1,\dots ,x_N\right]$, for any input vector $X$ and the corresponding query vector $q$. An information selection mechanism can be adopted to aggregate the input information, which is:

$$att\left(X,q\right)={\displaystyle {\sum }_{n=1}^N{\alpha }_n{x}_n}={\mathrm{E}}_{z\sim p\left(z\left|X,q\right.\right)}\left[x_z\right]$$

(5)

where ${\alpha }_n$ is referred to as the attention distribution, that is ${\alpha }_n=\mathrm{softmax}\left(x^{T}q\right)$.

The graph attention layer applies the attention mechanism to graph neural networks. It updates node features through the information of first-order neighborhood nodes, and learns different attention weight coefficients for different neighborhood variables. A shared weight matrix $W\in R^{F\times F^{\prime }}$ is trained for all nodes, which represents the relationship between the F input features and the F′ output features, and can be used to derive the neighborhood weight coefficient of each node. The calculation formulas of the attention value $e_{ij}$ and the attention coefficient $a_{ij}$ are as follows:

$$e_{ij}=\mathrm{LeakyReLU}\left({\overrightarrow{a}}^T\left[W{\overrightarrow{h}}_i‖W{\overrightarrow{h}}_j\right]\right)$$

(6)

$$a_{ij}=\mathrm{softmax}\left(e_{ij}\right)={\displaystyle \frac{\exp \left(e_{ij}\right)}{{\displaystyle {\sum }_{k\in {\mathrm{\Omega }}_i}\exp \left(e_{ik}\right)}}}$$

(7)

After the attention coefficients are calculated, for any node i, the model can adopt a multi-head attention mechanism to enhance its representation capability. For the output features of the intermediate layers of the GAT, a K weight matrix is used to calculate the self-attention, concatenate the K output results for output, and finally adopt a strategy of averaging the output vectors of the attention heads in the output layer. This process can be expressed as follows:

$${\overrightarrow{h^{\prime }}}_i=\sigma \left({\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K{\displaystyle \sum _{j\in {\mathrm{\Omega }}_i}{\alpha }_{ij}^k{W}^k{\overrightarrow{h}}_j}}\right)$$

(8)

where ${\alpha }_{ij}^k$ denotes the attention coefficient of the k-th attention head, and $W^k$ is the weight parameter matrix of the k-th attention head. It is worth noting that the activation function $\sigma$ of the output layer usually takes the form of softmax or logistic sigmoid.

3. Proposed Method

3.1. Decision Process Modeling for D-RTPD

D-RTPD of multi-area power grids is a fully cooperative stochastic game process based on multi-agent systems. It composes the Nash strategies of multiple agents in each state into a joint strategy in a dynamic environment, and updates the reward value of each agent in the game process through continuous interaction with the environment. Considering the mechanism of partitioned power grid scheduling, each scheduling area cannot obtain global information of other areas, so its dispatching problem is actually a decentralized partially observable Markov decision process (Dec-POMDP). Following the stochastic game paradigm, a Dec-POMDP is established for the D-RTPD of multi-area power grids with source–storage coordination, which mainly consists of the following elements:

(1)

Agent: The dispatching agent i in each area. This section mainly considers the generator rescheduling actions and ES regulation actions, and all multi-agents adopt decision-making in a continuous action space. $N_{ag}$ denotes the set of multi-agents;

(2)

State: The system state at time t is $s_t$, which includes the day-ahead predicted active power output $P_t^{W,DA}$ and real-time predicted active power output $P_t^W$ of wind power at time t, the day-ahead predicted active power output $P_t^{S,DA}$ and real-time predicted active power output $P_t^S$ of solar units at time t, the day-ahead scheduled active power output $G_t^{DA}$ of uncontrolled generators, the SoC $S_t$ of all ESs, the day-ahead predicted active load $P_t^{L,DA}$ and real-time predicted active load $P_t^L$ of load nodes, the load rate ${\rho }_t$ of each branch in the system, and the output value $P_{t-1}^G$ of controlled generators at the previous time step. It is worth noting that the action of generators at time 0 is the value specified in the day-ahead scheduling plan.

$$s_t=\left[P_t^{W,DA},P_t^W,P_t^{S,DA},P_t^S,G_t^{DA},S_t,P_t^{L,DA},P_t^L,{\rho }_t,P_{t-1}^G\right]$$

(9)

(3)

Local Observation Information: The observation quantity of an agent is $o_i\subseteq s$. The observation quantity of each agent is divided into two parts: one is the general observation quantity, i.e., the electrical quantities of the node where the agent is located and its k-th order neighborhood; the other is the special observation quantity. Specifically, the agent responsible for controlling generator rescheduling actions focuses on the generator output value of the managed area at the previous time step, while the agent responsible for controlling ES responses focuses on the SoC state at the previous time step, the day-ahead and real-time predicted values of RES, and the dispatching time step t. Without loss of generality, the observation quantities of the two types of agents can be expressed as follows:

$$o_{i,t}^{Gen}=\left[P_{k-hop,t}^{L,DA},P_{k-hop,t}^L,{\rho }_{k-th,t},G_{M,t}^{DA},P_{M,t-1}^G,\right]$$

(10)

$$o_{j,t}^{ESS}=\left[P_{k-hop,t}^{L,DA},P_{k-hop,t}^L,{\rho }_{k-th,t},P_{k-hop,t}^{W,DA},P_{k-hop,t}^W,P_{k-hop,t}^{S,DA},P_{k-hop,t}^S,S_t,t\right]$$

(11)

It is worth noting that the k-th order neighborhood observation quantity of an agent does not include the part beyond its regulation area.

(4)

Action: The actions of the two types of agents are generator rescheduling actions and ES charging/discharging actions respectively. All actions considered in this section are continuous regulation actions; therefore, both the generator rescheduling and ES charging/discharging strategies take values in the continuous action space. The upper and lower limits of the generator’s action space are its maximum and minimum active power output. In addition to satisfying the upper and lower limit constraints of SoC, the ES also needs to meet the requirement that the difference between the initial SoC and the final SoC within each dispatching day is within a certain range, which can be expressed as follows:

$$\left|S_{j,1}-S_{j,T_{\max }}\right|\le \mathrm{\Delta }{S}^{\max },\forall j\in N_{ESS}$$

(12)

where $T^{\max }$ denotes the maximum duration of a dispatching day and $\mathrm{\Delta }{S}^{\max }$ represents the maximum allowable deviation of the SoC.

(5)

Reward Function: This section employs generator rescheduling and ES actions to address the strong uncertainty of high-proportion RES, and the economic costs of generator dispatching also need to be taken into account. Therefore, the design of the reward function in this section should revolve around three aspects: the system line load rate, the generator unit rescheduling cost, and the ES action cost, which can be expressed as follows:

$$\left\{\begin{array}{l}{g}_t^{line}={\displaystyle \sum _{l=1}^{N_l}\left[{\phi }_1^l\max \left(0,{\rho }_l-1\right)+{\phi }_2^l\max \left(0,{\rho }_l-0.95\right)\right]}\hfill \\ g_t^{gen}={\displaystyle \sum _{i=1}^{N_{Gen}}\left({\phi }_1^g\mathrm{\Delta }{p}_g^2+{\phi }_2^g\mathrm{\Delta }{p}_g\right)}\hfill \\ g_t^{stg}={\phi }_1^s{\displaystyle \sum _{j=1}^{N_{ESS}}{C}_j^{ESS}\left(P_{j,t}^{ESS}\right)}+{\phi }_2^s{\psi }_{s,t}{\displaystyle \sum _{j=1}^{N_{ESS}}\left(S_t-S_0\right)}\hfill \end{array}\right.$$

(13)

where $g_t^{line}$, $g_t^{gen}$, and $g_t^{stg}$ denote the line load rate reward term, generator rescheduling cost reward term, and ES action cost reward term respectively. ${\phi }_1^l$ and ${\phi }_2^l$ represent the penalty coefficients for line power flow overload and heavy load respectively, both of which take values less than 0. ${\phi }_1^g$ and ${\phi }_2^g$ are the quadratic coefficient and linear coefficient of the generator rescheduling cost respectively. ${\phi }_1^s$ and ${\phi }_2^s$ are the ES action cost coefficient and SoC deviation penalty coefficient respectively, and ${\psi }_{s,t}$ is the SoC deviation assessment coefficient.

Considering that the SoC deviation of ES should satisfy Formula (5), the value of ${\psi }_{s,t}$ shall follow the principle that it increases as the value approaches the maximum dispatching duration. Therefore, this section adopts a sigmoid function to determine the value of ${\psi }_{s,t}$, which can be expressed as follows:

$${\psi }_{s,t}={\left[1+\exp \left(-\left(\ln 99\right)\left(t-0.95T\right)/0.05T\right)\right]}^{-1}$$

(14)

where t and T represent the current dispatching time step and the maximum dispatching duration respectively, and the curve of the function is shown in Figure 1.

Figure 1. Value curve of SoC deviation assessment coefficient based on sigmoid function.

When formulating strategies, the agents need to take both the current power grid state and the long-term power grid situation into account, and their immediate reward and cumulative reward functions are shown in Formulas (8) and (9), respectively.

$$g_t=\left\{\begin{array}{cc}\kappa & When constraints\\ {\phi }_l{g}_t^{line}+{\phi }_g{g}_t^{gen}+{\phi }_s{g}_t^{stg}+g_t^{con}& Others\end{array}\right.$$

(15)

$$r_t={\displaystyle \sum _{k=t_0}^t{\gamma }_r^{t-k}{g}_k}$$

(16)

where ${\phi }_l$, ${\phi }_g$, and ${\phi }_s$ denote the line load rate reward coefficient, generator rescheduling cost reward coefficient, and ES action cost reward coefficient respectively. $g_t^{con}$ represents the constant reward term for normal system power flow. $t_0$ denotes the initial time of cumulative reward. and ${\gamma }_r$ represents the attenuation factor of cumulative reward.

3.2. Solution Method Based on MADDPG

3.2.1. Policy Optimization

In Equation (4), the training of the critic for each agent requires the policies of other agents. In the MADDPG algorithm, the policy of other agents for each agent can be obtained by means of fitting approximation, without the need for communication and interaction with them. Therefore, each agent can maintain n − 1 policy approximation functions, and the logarithm of the approximated policy can be used as the loss function, which can be expressed as:

$$L\left({\varphi }_i^j\right)=-E_{o_j,a_j}\left[\log {\widehat{\pi }}_{{\varphi }_i^j}\left(a_j\left|o_j\right.\right)+\lambda H\left({\widehat{\pi }}_{{\varphi }_i^j}\right)\right]$$

(17)

where ${\varphi }_i^j$ denotes the policy approximation parameter of the j-th agent for the i-th agent, and ${\widehat{\pi }}_{{\varphi }_i^j}$ is its policy approximation function. By minimizing this loss function, the approximation of the policies of other agents can be obtained. Thus, $y_i$ can be expressed in Equation (4) as follows:

$$y_i=r_i+\gamma {\overline{Q}}_i^{{\pi }^{\prime }}\left(o^{\prime },{{\widehat{\pi }}^{\prime }}_{{\varphi }_i^1}\left(o_1\right),\dots ,{\widehat{{\pi }^{\prime }}}_{{\varphi }_i^n}\left(o_n\right)\right)$$

(18)

Since the policy of each agent is constantly updated and iterated, the environment faced by a single reinforcement learning agent is dynamically unstable. To address this problem, MADDPG adopts a policy set technique, which assumes that the policy of each agent consists of a set containing multiple sub-policies. For the i-th agent, its policy ${\pi }_i$ is composed of k sub-policies, and only one sub-policy ${\pi }_i^{\left(k\right)}$ is used in each training episode. The overall reward of its policy set is expressed as follows:

$$J_e\left({\pi }_i\right)=E_{k\sim \mathrm{unif}\left(1,K\right),s\sim {\rho }^{\pi },a\sim {\pi }_i^{\left(k\right)}}\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{r}_{i,t}}\right]$$

(19)

Therefore, the update gradient for each sub-policy of the i-th agent is expressed as follows:

$${\nabla }_{{\theta }_i^{\left(k\right)}}{J}_e\left({\pi }_i\right)={\displaystyle \frac{1}{K}}{E}_{o,a\sim D}\left[{\left.{\nabla }_{{\theta }_i^{\left(k\right)}}{\pi }_i^{\left(k\right)}\left(a_i\left|o_i\right.\right){\nabla }_{a_i}{Q}_i^{\pi }\left(o,a\right)\right|}_{a_i={\pi }_i^{\left(k\right)}\left(o_i\right)}\right]$$

(20)

3.2.2. Neighborhood Feature Aggregation Based on Graph Attention Network

GAT leverages the multi-head attention mechanism to perform feature aggregation on first-order neighborhood information, while neglecting the impact of high-order neighborhood information on agents. In power grids, line power flow is often strongly correlated with the power injection and outflow of multiple nodes in the network. It is difficult for scheduling agents to accurately and effectively perceive the power flow operation state of the power grid relying solely on first-order neighborhood information [28,29,30,31]. Therefore, this section proposes a graph attention network based on neighborhood feature aggregation (NFA-GAT) through the feature aggregation and superposition of multi-order neighborhood information, so as to expand the perception range of the agent’s GAT layer.

For any power grid node, the influence of the electrical quantities of its neighboring nodes on the power flow of the connected lines will weaken as the distance increases. Therefore, when aggregating the neighborhood features of agents, we should assign different contribution degrees to the nodes in neighborhoods of different orders.

By introducing a spatial discount factor to weight multi-order neighborhood information, Equations (6) and (7) (base GAT attention coefficient calculation) are combined and rewritten as follows:

$$\begin{array}{cc}\hfill {\alpha }_{ij}& =\mathrm{softmax}\left({\tilde{e}}_{ij}\right)={\displaystyle \frac{\exp \left(\eta \left(d_{ij}\right)\times e_{ij}\right)}{{\displaystyle {\sum }_{k\in {\mathrm{\Omega }}_i^K}\exp \left(\eta \left(d_{ik}\right)\times e_{ik}\right)}}}\hfill \\ & ={\displaystyle \frac{\exp \left({\eta }_d^{d_{ij}}\times \mathrm{LeakyReLU}\left({\overrightarrow{a}}^T\left[W{\overrightarrow{h}}_i‖W{\overrightarrow{h}}_j\right]\right)\right)}{{\displaystyle {\sum }_{k\in {\mathrm{\Omega }}_i^K}\exp \left({\eta }_d^{d_{ik}}\times \mathrm{LeakyReLU}\left({\overrightarrow{a}}^T\left[W{\overrightarrow{h}}_i‖W{\overrightarrow{h}}_k\right]\right)\right)}}}\hfill \end{array}$$

(21)

where ${\mathrm{\Omega }}_i^K$ denotes the k-th order neighborhood of the node where agent i is located; $\eta$ represents the distance-dependent SPF; $d_{ij}$ denotes the neighborhood order between node i and node j (i.e., the shortest hop count between node i and node j). Starting from the physical characteristics of power systems, this section illustrates the exponential attenuation law of electrical quantities (power/voltage) with topological distance (hop count), which originates from the voltage and power division effects of line impedances. Therefore, the adoption of an exponential function $\eta \left(d_{ij}\right)={\eta }^{d_{ij}}$ conforms to objective physical laws. The optimality of the spatial discount factor will be investigated in the Section 4.6. By comparing the capabilities of balancing neighborhood information aggregation and invalid information filtering, the performance differences among different parameter settings are verified, and the rationality of the exponential function form is validated.

Similarly, to prevent overfitting, NFA-GAT also adopts the multi-head attention mechanism presented in Equation (8). Taking a power grid as an example, the schematic diagram of its principle for feature aggregation of the second-order neighborhood of a certain node in the graph is shown in Figure 2, which intuitively presents the process of NFA-GAT assigning exponentially attenuated weights to first-order and second-order neighborhood nodes based on the spatial discount factor, then fusing multi-order neighborhood feature information with the local node feature to form the final high-dimensional feature representation of the node.

Figure 2. Schematic diagram of second-order neighborhood feature aggregation for nodes based on NFA-GAT.

3.2.3. Prior-Guided Mechanism

To enhance the effectiveness of the agents’ output strategies, we have incorporated a prior knowledge-guided action validity check into each agent’s policy network. By setting the actor logits corresponding to invalid actions that violate constraints to negative infinity, invalid actions can be eliminated during the agents’ training process, thereby improving the training efficiency of multi-agent systems. The details of action validity check mechanism are: (1) The masking operation acts directly on the actor logits output by the Actor network after feature extraction via NFA-GAT. Specifically, the logits corresponding to invalid actions that violate power grid constraints (e.g., unit output limits, energy storage charging/discharging constraints, line power flow constraints) are set to negative infinity; (2) The masking operation only modifies the logits corresponding to invalid actions without changing the gradient propagation path of the Actor network. After the revised logits are converted into an action probability distribution via softmax, they can still normally participate in the computation and update of the policy gradient, ensuring differentiability throughout the training process.

Additionaly, a discretization sampling is performed on the continuous policies output by the agents deployed with the MADDPG algorithm, which thus requires the use of the Gumbel-max trick. For any n-dimensional probability vector $\pi$, Gumbel noise is added to its corresponding discrete random variable $x_{\pi }$ before sampling is conducted, which can be expressed as follows:

$$x_{\pi }=\arg \max \left(\log \left({\pi }_i\right)+G_i\right)$$

(22)

where $G_i$ denotes an independent and identically distributed (i.i.d.) random variable following the standard Gumbel distribution, and its cumulative distribution function is expressed as follows:

$$G_i=-\log \left(-\log \left(U_i\right)\right),U_i\sim U\left(0,1\right)$$

(23)

Considering that the argmax operation in Equation (23) renders it non-differentiable, we replace the argmax with the softmax function to enable gradient derivation and update of the algorithm. Therefore, Gumbel–softmax approximates the original continuous action probability distribution with the output of the neural network while sampling the maximum probability using argmax. Essentially, it leverages the reparameterization trick to address the non-differentiability issue caused by argmax.

3.3. Training Framework

As a tool to enhance the neighborhood feature extraction capability of agents, NFA-GAT needs to be deployed in the Actor network of agents. Therefore, the structure of the agent Actor network integrated with the NFA-GAT and MADDPG algorithms is illustrated in Figure 3. For the input variable t, it only needs to be fed into the ES agents rather than the generator agents. In this section, ES is dispatched at the grid side. The “centralized training, distributed execution” framework of the multi-agent system does not rely on a communication network, where the neighborhood of each agent refers to the adjacent bus nodes in its physical network. During the centralized training phase of the multi-agent model, the Critic network evaluates the action of each agent based on the information from all agents, thereby ensuring the stability of the environment throughout the training process. After the multi-agent model is trained to convergence, each agent can formulate effective strategies solely based on its local observations during deployment. Ultimately, the real-time scheduling of the power grid is achieved through the coordinated strategies of multiple agents.

Figure 3. Actor network structure of the agent.

Since multi-agents need to utilize the attention mechanism to process neighborhood features, this section selects the electrical characteristic quantities of the power grid to avoid extracting those irrelevant to real-time scheduling. Thus, the environment returns an updated feature matrix $F_t$ before each interaction with the agents. After the environment returns the system state and feature matrix, multiple agents obtain their respective local observations, extract the neighborhood features from the feature matrix using the NFA-GAT layer, and finally output policy actions. To satisfy the inequality constraints of the real-time scheduling model, an action validity verification mechanism that satisfies model constraints is established for each agent. When certain actions in the discrete action space of any agent would cause the generator or ES to exceed limits, this mechanism sets the actor logits of these actions to negative infinity. As a result, the probability density of these actions in the constructed categorical distribution becomes zero, ensuring that all actions output by each agent are valid and executable in the environment.

To implement the prior-guided mechanism, after the policy network outputs the action probability distribution, this distribution is converted into a discrete Gumbel distribution for discrete policy output. The core purposes of adopting the Gumbel-max trick and discretized sampling are to achieve efficient optimization of continuous policies. After the Actor network outputs a continuous policy distribution, Gumbel noise injection is applied to perform discretized sampling of the continuous policy, followed by top-k optimization to extract the top-k optimal continuous action values. This approach does not design the action space as discrete; instead, it discretizes and filters the output results of the continuous policy to improve the decision-making efficiency of real-time scheduling.

At each decision step, the decision trajectories of the multi-agents are combined into a piece of experience ${\left(o_t,a_t,r_t,F_t,o_{t+1},d_t\right)}_{\tau }$ and stored in the experience replay buffer. When an episode ends (either the maximum number of steps is reached or the scenario power flow exceeds limits), the environment is reset again and a new scenario is randomly generated. After a certain number of interaction episodes, we randomly sample a batch of samples from the experience replay buffer and use these samples to train the neural networks of the multi-agents. During the network update process, we first input the local observations, action values, reward values and Done flags of all agents. The centralized Critic evaluates and feeds back the Q-value of each agent based on global information. The Actor training of each agent relies on the local observations, actions of the agent itself and the Q-value returned by the centralized Critic. After training for a certain number of steps, when the loss functions of the multi-agents are observed to converge, we terminate the training process and export the trained model. The pseudocode for training the multi-agent system for real-time scheduling in this section is shown in Algorithm 1.

Algorithm 1: Multi-agent training algorithm for real-time scheduling based on NFA-GAT and MADDPG

1: Initialization: Power grid simulation environment, training data set, neural network structure and parameters of multi-agents, experience replay buffer 2: for episode = 1 to I do 3:    Shuffle the training set, set case index = 0, reset the power grid environment, configure the training scenario, and obtain the initial system operating state $s_t$ 4:    for t = 1 to T do 5:         Obtain the local observations ${\mathbf{O}}_t^1,\dots ,{\mathbf{O}}_t^i,\dots ,{\mathbf{O}}_t^n$ of the multi-agents at time t and the feature matrix ${\mathbf{F}}_t$ of the system from the system operating state $s_t$ 6:       for agent i do 7:            if controlled generator then 8:                Feed the local observation ${\mathbf{O}}_{i,t}^{\mathrm{Gen}}$ and the feature matrix ${\mathbf{F}}_t$ into the MLP and NFA-GAT respectively, and output the action distribution 9:           if controlled ES then 10:               Feed the local observation ${\mathbf{O}}_{i,t}^{\mathrm{ESS}}$ and the feature matrix ${\mathbf{F}}_t$ into the MLP and NFA-GAT respectively, and output the action distribution 11:           Leverage the action validity verification mechanism to sets the actor logits of these actions of invalid actions in the action distribution to negative infinity, construct a categorical distribution function for actions, and sample the top-k optimal actions ${\mathbf{a}}_{i,t}^{top,k}$ 12:           Leverage the prior-guided mechanism to select the optimal action $a_t^i$ 13:       end for 14:       Convert the actions $a_t^1,\dots ,a_t^i,\dots ,a_t^n$ of the multi-agents into a joint action ${\mathbf{a}}_t^{joint}$ and execute this joint action in the power grid environment 15:       Obtain the system operating state $s_{t+1}$, system feature matrix ${\mathbf{F}}_{t+1}$, reward value $r_t$ and episode termination flag done at the next time step 16:       Update the state $s_t=s_{t+1}$ and the system feature matrix ${\mathbf{F}}_t={\mathbf{F}}_{t+1}$ 17:       if done then 18:            Exit the loop 19:       Add the experience $({\mathbf{O}}_t^i,{\mathbf{A}}_t^i,r_t,t,{\mathbf{F}}_t,{\mathbf{O}}_{t+1}^i,done)$ of the multi-agents to the experience replay buffer 20:   end for 21:   Sample a batch of data from the experience replay buffer, update the parameters of the Critic network based on Equation (17), and update the parameters of the Actor network based on Equation (20) 22:   Perform a soft update on the target network parameters using the main network parameters 23:   case index = case index + 1, if case index less than the maximum number of scenarios in the training set else 0 24: end for

4. Case Simulation

4.1. Case Description

In this section, an offline power grid simulation environment for interaction with agents based on the power flow calculation engine PyPower 5.1.4 is built, and the IEEE 118-bus power grid is selected as the object of D-RTPD. Considering that the generator dispatching agents and ES dispatching agents have a large adjustable range, this section divides the target power grid into six regions, and sets six generator agents (orange generators in the figure) in each dispatching region. Meanwhile, to cope with the integration of high-proportion RES, we configured seven distributed ES agents in the power grid to dispatch ES resources, and its topology is shown in Figure 4.

Figure 4. Topology diagram of the IEEE 118-bus system.

The primary task of D-RTPD is to ensure the safe and stable operation of the power grid. Therefore, we track the maximum line load rate of the power grid. Given that D-RTPD requires a response time for generator redispatch, the power grid will experience power flow violations if the line load rate reaches 100% for two consecutive time steps.

The scenarios with differentiated RES penetration rates of the system are established, which integrate a large number of wind and solar units into various regions of the power grid. The output curves of both wind turbines and solar systems are derived from historical measured data. In the case study of this section, the installed capacity proportion of RES reaches 75.2%. However, considering the output characteristics of wind and solar power as well as ultra-short-term forecasting errors, the actual penetration rate fluctuates within the range of 0% to 71% during the D-RTPD period. Based on the day-ahead dispatching plan, the interval distribution of RES penetration rates under single-step operation scenarios of the power grid is shown in Figure 5.

Figure 5. Distribution of RES penetration rate in single-step operation scenarios within the training scenario set.

We collected 1000 daily dispatching scenarios covering different seasons and RESs penetration rates. These scenarios are derived from simulation data of the IEEE 118-bus system, with random fluctuations superimposed to simulate ultra-short-term forecasting errors. Furthermore, 200 scenarios were randomly selected from the same data source as the training set, ensuring consistent distribution characteristics but not overlapping with the training set. The training scenarios in this section are organized on a daily basis. The optimization solver Gurobi 13.0.1 was adopted to generate the day-ahead dispatching plan for each training scenario. Multiple agents are required to formulate 288 D-RTPD plans at 5-min intervals based on the day-ahead dispatching plan; therefore, the maximum time step of each scenario is set to 288 steps. All weight coefficients in this paper have been verified through multiple rounds of simulation. Under a new energy penetration rate of 75.2%, the model ensures compliance with power grid safety constraints while achieving optimal dispatching economy.

Both the policy networks of the generator dispatching agents and ES dispatching agents adopt the discretized DDPG algorithm. In the NFA-GAT layer shown in Figure 3, a hidden layer with 64 neurons and two attention heads are employed, and the aggregated features are 10-dimensional feature vectors within the three-hop neighborhood. The spatial discount factor is fixed at 0.8. For the Actor and Critic networks of each agent, the hidden layers are configured with 128 neurons, and the Leaky ReLU activation function is selected for both networks.

In addition, the batch size sampled from the experience replay buffer is set to 128, and the Adam optimizer is adopted with a learning rate of 0.001 for both the Actor and Critic networks. The correlation coefficients of the reward function are presented in Table 1. To accelerate the training speed, the neural network parameters were shared among all ES agents. The case studies were implemented based on PyPower 5.1.4 and Pytorch-GPU CUDA 11.0, and the training was conducted on a Linux server equipped with 4 NVIDIA RTX 2080Ti graphics cards.

Table 1. Correlation coefficients of the reward function.

${\mathit{\phi }}_1^{\mathit{l}}$	${\mathit{\phi }}_2^{\mathit{l}}$	${\mathit{\phi }}_1^{\mathit{s}}$	${\mathit{\phi }}_2^{\mathit{s}}$	${\mathit{\phi }}_{\mathit{l}}$	${\mathit{\phi }}_{\mathit{g}}$	${\mathit{\phi }}_{\mathit{s}}$	${\mathit{g}}_{\mathit{t}}^{\mathit{c}\mathit{o}\mathit{n}}$	$\mathit{\kappa }$
20	10	1	15	1	0.008	0.5	5	−200

4.2. Validation of Algorithm Learning Capability

During the training process, samples in the training set were randomly selected as D-RTPD scenarios, and the total number of training episodes for the multi-agent system was set to 3500. The first 1000 episodes served as the random exploration phase, after which the neural network parameters of the multi-agents were learned and updated every 100 episodes. After more than 40 h of training, the training curves shown in Figure 6 were obtained. Figure 6a depicts the cumulative reward curve per episode of the multi-agent system deployed with the MADDPG algorithm, where the dark blue curve represents the moving curve and the light blue background corresponds to the original curve. In addition, considering that ES dispatching agents mainly perform coordinated adjustments according to the RES output in the neighboring areas for D-RTPD tasks, all ES agents share the same policy network in the case study of this section.

Figure 6. Curve diagram of multi-agent training process under the algorithm proposed in this section. (a) Moving average cumulative reward curve. (b) Loss value curve of the multi-agent policy network.

Considering the randomness of the alternating occurrence of scenarios with different penetration rates in the training set, the reward curve of the multi-agent system during the random exploration phase is highly unstable, and power flow divergence frequently occurs (with the cumulative reward value hovering around −200). This indicates that the day-ahead dispatching plan based on day-ahead forecasting information struggles to cope with the impact of strong uncertainties. The power grid is prone to power flow violations caused by either transmission line overload or insufficient regulation capacity of the slack bus.

After 2500 training episodes, the cumulative reward curve of the multi-agent system stabilizes at around 400, and its loss curve also converges within a certain range. This indicates that the training algorithm proposed in this section, which is based on NFA-GAT and MADDPG, exhibits favorable convergence performance. It enables the policy networks of the multi-agent system to output a deterministic action distribution, and to formulate effective real-time control strategies under the uncertainty impact of high-proportion RES.

As can be seen from the loss function curves in Figure 6b, the multi-agent training loss curve fluctuates sharply at the beginning of this phase. After a period of network parameter updates, it has started to show a convergence trend. Considering that the deterministic gradient descent algorithm is essentially a conservative search strategy, it can be observed that the loss value of the agents keeps fluctuating within a relatively small range. The loss values of the policy networks of multiple agents converge within a specific threshold range after 200 training iterations (corresponding to 3000 episodes of interaction between the agents and the environment). However, considering the dynamic nature of the multi-agent game process, the loss curves of the policy functions still show slight fluctuations.

In addition, given that the seven ES agents share neural network parameters, it can be observed from the figure that the loss function curves of these seven agents all converge to the same level. Combined with the cumulative reward curves, this demonstrates that the method of sharing neural network parameters does not affect the effectiveness of the D-RTPD strategies.

4.3. Validation of Algorithm Deployment Performance

To verify the deployment performance of the multiple agents trained by the algorithm proposed in this section, we deployed the 13 well-trained agents (six generator agents and seven ES agents) in the offline power grid simulation environment. A total of 100 independent scenarios that never appeared in the training and validation sets were established. These scenarios correspond to actual dispatching days, with distinct load and renewable energy source output characteristics to fully verify the generalization ability of the proposed method. Among them, three test scenarios with different penetration rates were randomly selected. These three scenarios were derived from different dispatching days in December, April, and August, respectively, and their load and penetration rate curves are shown in Figure 7.

Figure 7. Curves of load and RES penetration rate under different test scenarios.

In the scenarios of the training set, we fully considered the fluctuations of sources and loads as well as the uncertainty of RES. The blue curve and the orange curve represent the day-ahead short-term predicted values and real-time short-term predicted values respectively. It can be seen that the magnitude of load fluctuations remains at a relatively low level, while the predicted values of the total RES penetration rate of the system have large deviations, leading to power fluctuations ranging from 100 MW to 500 MW.

As can be seen from Figure 7, due to the characteristic of solar (PV) systems that their output reaches the maximum at noon, RES tends to generate excessive power during the noon period. Day-ahead short-term forecasts may have large deviations during this time window, which makes it easy to encounter challenges such as power balance and power flow regulation when implementing the day-ahead dispatching plan in the noon period.

Although wind and PV generation have a certain degree of complementarity, during the two periods of noon and afternoon when PV output surges sharply and drops drastically, the power grid will face power flow violations and insufficient regulation capacity of the slack bus caused by problems such as inadequate upward and downward ramping capabilities of generators and limited transmission capacity of lines. In such cases, the coordinated dispatching of generators and ESs is required to ensure the safe and reliable operation of the power system.

As can be seen from Figure 8, in all three scenarios, the generators in subfigures (a,c,e) reduced their output during the peak solar (PV) generation period. However, taking into account the constraints on generator ramping capabilities and high redispatch costs, the ES agents implemented corresponding coordinated scheduling. By comparing the operational actions of generators and ESs across the three scenarios, it can be concluded that ES regulation serves as the primary means to cope with RES output fluctuations in this case study, with the following details.

Figure 8. D-RTPD strategies of multi-agents under different scenarios.

In Scenario 1, the RES output exhibited a trend of first rising, then falling, and subsequently surging sharply. The generators made adjustments during the morning hours based on the day-ahead dispatching plan, while no significant redispatch actions were carried out during the noon period of peak PV generation. During noon, the power grid mainly relied on ES actions to absorb the RES output. In the evening, since the ESs needed to discharge to keep the difference in the initial and final SoC within a specific threshold range, their discharge power just offset the decline in RES output.

Scenario 2 and Scenario 3 are roughly similar in terms of scenario characteristics, but Scenario 2 has a higher load level. Therefore, both the generator output level and the depth of ES charging/discharging in Scenario 2 are higher than those in Scenario 3. Considering that the RES output fluctuations in both scenarios show a single-peak characteristic, the dispatching strategies adopted by the ES agents in the two scenarios are generally similar.

Combined with the D-RTPD strategies of multi-agents under different scenarios in Figure 8, an analysis of the maximum line load rate of the system in Figure 9 reveals the following. In December (Scenario 1), when the RES penetration rate is relatively low, the total power fluctuation caused by RES uncertainty remains at a low level, and the maximum line load rate of the power grid is also low. However, due to the sharp drop followed by a sudden surge of RES output in the morning, the day-ahead dispatching plan suffers from insufficient downward regulation capacity of the slack bus. The D-RTPD strategy leverages the charging capability of ESs to absorb the surplus RES output, thus resolving the issue of insufficient downward regulation capacity of the system.

Figure 9. Maximum line load rate of the system after implementing the D-RTPD plan under different scenarios.

In April and August, when the RES penetration rate is relatively high, although the total load during the dispatching period is not large, the system mainly relies on the output of RES with high penetration rate to supply the load. As can be seen from the figure, during the noon period of peak RES generation, the limitations of generators’ downward ramping capability lead to severe line congestion, resulting in power flow violations in the system lines under the day-ahead dispatching plan. In contrast, the D-RTPD strategy utilizes the characteristics of ESs: it charges the ES when RES output surges to avoid power flow violations, and discharges when RES output drops sharply, thereby reducing the redispatch costs of generators.

By observing the line load rate during the evening period across the three scenarios, it can be found that considering the constraint on the initial and final SOC values of ESs in this section, ES stores a large amount of electricity during the noon period of peak RES generation and thus has to discharge in the evening. This complements the output characteristics of RES. Therefore, during the evening dispatching period when the load level is still relatively high, the maximum line load rate of the power grid remains at a high level. Meanwhile, generators reduce their output during the noon period of surging RES output, and only perform minor redispatch operations in the evening when ES takes on the main power supply task.

Finally, combining the agent modeling characteristics, scenario characteristics and D-RTPD strategies, we conduct a macro-analysis of the three scenarios. By comparing the load characteristics, RES characteristics and agent action strategies across the three different scenarios, it can be found that the action strategies of ES agents are mainly focused on source-load balance regulation, whereas the generator agents act primarily based on the SoC of ESs. The specific manifestations are as follows:

In Scenario 1, the RES output experiences a trend of first increasing and then decreasing during the morning dispatching period. During this period, generator agents are mainly responsible for downward and upward output adjustments. However, due to the relatively low level of RES penetration, some ESs remain in a low SOC state. During the noon period with high load levels, the RES output and ES discharge can match the load changes, so the generator agents do not perform any redispatch actions. Finally, when both the load level and RES output decrease simultaneously, the ESs continue discharging to maintain the source-load balance of the system.

By comparing Scenarios 2 and 3 with Scenario 1, it can be observed that since the load peaks in Scenarios 2 and 3 occur earlier and the RES output presents a single-peak characteristic, the ES agents in Scenarios 2 and 3 implement the discharge strategy at the initial stage. Then, during the noon period when RES output surges sharply, the downward adjustment of generators and charging of ESs are utilized to absorb the excess output. In the evening when RES output drops drastically, the ES agents discharge to maintain source-load balance, and the generator agents refrain from redispatch operations after observing the state of ESs. As the time points of RES output decline in Scenarios 2 and 3 are later than that in Scenario 1, the second discharge time of the ES agents in Scenarios 2 and 3 is also significantly later than that in Scenario 1.

4.4. Comparison of Different Networks

To verify the role of the NFA-GAT network layer proposed in this section in improving algorithm effectiveness, we adopted two structures—the fully connected layer (MLP) and NFA-GAT—respectively in the agent policy network of the proposed algorithm. Under the same algorithm framework, hyperparameters, random scenario seeds, and training duration, the two neural network structures were used to test the multi-agent system across 10 random scenarios every 100 training episodes. A “win” was defined as the power grid maintaining safe and reliable operation throughout 288 D-RTPD periods. The curve comparison diagram is shown in Figure 10.

Figure 10. Comparison of moving average reward curves and win rate curves under different neural networks. (a) Comparison of moving average cumulative reward curves. (b) Comparison of win rate curves in the testing process.

As can be seen from Figure 10a, the multi-agent system with the NFA-GAT layer can achieve high reward values and win rates after training. This indicates that NFA-GAT can help multi-agents efficiently capture neighborhood information, which equips them with the ability to effectively perceive power grid states and enables them to reach the Nash equilibrium state. In contrast, the multi-agent system with MLP fails to capture power grid characteristics effectively, resulting in non-convergence even after more than 3000 training episodes, with both its moving average reward values and win rates lower than those of the system with NFA-GAT.

In addition, by comparing Figure 10a,b, it can be observed that the neural network with NFA-GAT suffers from significant instability in the early training stage, where its moving average reward curve fluctuates sharply and the multi-agents fail to achieve satisfactory win rates. This may be attributed to the fact that the learning of attention coefficients increases the training burden of the neural network. After the neural network learns stable attention coefficients, the performance of the multi-agents is significantly improved, which further demonstrates the effectiveness of the NFA-GAT layer in multi-agent D-RTPD tasks.

4.5. Comparison of Different Algorithms

To better highlight the advantages of the proposed algorithm, a comparison of the solution effects among various algorithms is conducted, with specific details as follows: (1) Model-driven method: the Alternating Direction Method of Multipliers (ADMM); (2) Data-driven method: Q-MIX algorithm. All experiments with “10 independent random seeds” are reported as the “mean ± standard deviation” of all metrics. The comparative results are summarized in Table 2.

Table 2. Performance comparison among different algorithms.

	Proposed Method	ADMM	Q-MIX
Average Solution Time per Step (ms)	23.5 ± 0.5	189.2 ± 1.2	21.5 ± 0.8
Maximum Line Load Rate (%)	98.7 ± 0.3	92.3 ± 0.1	130.5 ± 0.5
Total Dispatching Cost ($/Day)	12,845 ± 126	12,320 ± 98	14,768 ± 215
Power Flow Violation Rate (%)	0 ± 0.0	0 ± 0.0	14.3 ± 0.4

In terms of solution efficiency, the proposed method achieves an average solution time of 23.5 ms per step, which is approximately 8.1 times faster than ADMM. While Q-MIX exhibits a slightly shorter solution time (21.5 ms), its performance in operational safety and economy is significantly inferior. Regarding grid operational safety, the proposed method limits the maximum line load rate to 98.7% (well below the 100% violation threshold) and achieves a zero power flow violation rate. Although the model-driven ADMM achieves 0% power flow violation rate, its solution efficiency is low. In contrast, the purely data-driven Q-MIX exhibits high solution efficiency, but the power flow violation rate reaches 14.3%. Our proposed “data-driven + NFA-GAT” method has better performance on power flow constraints satisfaction and solution efficiency than a single data-driven or model-driven method. Q-MIX suffers from severe line overload (130.5%) and a 14.3% violation rate. This gap stems from the proposed method’s coordinated decision-making framework: NFA-GAT enables accurate perception of grid states, while the prior-guided mechanism eliminates constraint-violating actions, ensuring stable power flow constraints satisfaction. Economically, the proposed method reduces the total daily dispatching cost to $12,845 ± 126, representing a nearly 13.0% reduction compared to Q-MIX. This cost advantage is attributed to the optimal coordination between generator rescheduling and energy storage regulation, which minimizes unnecessary redispatch costs and energy storage operation losses. Overall, the proposed method balances solution efficiency, operational safety, and economic performance, outperforming traditional model-driven and data-driven methods in handling the strong uncertainty of high-proportion renewable energy integration.

4.6. Results of Sensitivity Analysis on the Spatial Discount Factor

Table 3 presents the sensitivity analysis results of the spatial discount factor $\eta$ (with values of 0.6, 0.7, 0.8, and 0.9). As observed, the performance of the proposed NFA-GAT model exhibits a significant dependence on $\eta$, with $\eta =0.8$ achieving the optimal comprehensive performance. Specifically, when $\eta =0.8$, the model attains the highest average cumulative reward 405.6 and scenario win rate 92.3%, accompanied by a zero power flow violation rate and a low maximum line load rate 98.7%, which is well below the 100% violation threshold.

Table 3. Performance comparison among different spatial discount factors.

	Spatial Discount Factor
	0.6	0.7	0.8	0.9
Average Cumulative Reward Value	326.8	372.4	405.6	381.2
Average Win rate (%)	78.2	85.7	92.3	87.5
Maximum Line Load Rate (%)	99.3	98.9	98.7	99.1
Power Flow Violation Rate (%)	1.8	0.9	0.0	0.7

For $\eta =0.7$ and $\eta =0.6$, the lower discount factors lead to insufficient attenuation of high-order neighborhood information, resulting in redundant feature aggregation. This not only reduces the model’s ability to capture critical spatial correlations (reflected by lower cumulative rewards and win rates) but also introduces minor power flow violations (1.8% and 0.9% respectively). In contrast, $\eta =0.9$ overemphasizes the contribution of distant neighborhood nodes due to the weak discount effect, which distorts the spatial priority of grid features. Consequently, the cumulative reward 381.2 and win rate 87.5% decline compared to $\eta =0.8$.

These results validate that $\eta =0.8$ strikes an optimal balance between effective multi-order neighborhood feature aggregation and rational spatial weight allocation. It enables the NFA-GAT network to accurately perceive the attenuation of electrical quantity influence with topological distance, thereby enhancing the agents’ understanding of grid operating states and improving the reliability and efficiency of distributed real-time power dispatch.

4.7. Scalability and Engineering Practical Application of the Proposed Method

To verify the deployment value of the proposed NFA-GAT-MADDPG method in actual power systems, this section systematically demonstrates the practical application potential of the method from three dimensions: grid scale scalability, agent quantity scalability and engineering scenario adaptability, combined with quantitative data and comparative analysis, providing theoretical and practical references for subsequent engineering implementation.

4.7.1. Grid Scale Scalability

The NFA-GAT in the proposed method adopts a sparse neighborhood feature aggregation mechanism, which only performs feature extraction and information fusion on the agent’s local and K-order neighborhood nodes without feature calculation for all grid nodes. This makes the computational complexity grow linearly with the number of grid nodes, breaking the bottleneck of the exponential growth of computational complexity of traditional graph neural networks in large-scale power grids. To quantitatively verify the performance of the method under grid scale expansion, three test systems of different scales (IEEE 30-bus, IEEE 118-bus, and IEEE 300-bus) are selected with a consistent renewable energy penetration rate of about 75%. The core performance indicators of the method under different grid scales are tested, and the results are shown in Table 4.

Table 4. Comparison of method performance indicators under different grid scales.

System	Average Solution Time per Step (ms)	Maximum Line Load Rate (%)	Power Flow Violation Rate (%)
IEEE 39-bus	8.2 ± 0.3	97.5 ± 0.4	0
IEEE 118-bus	23.5 ± 0.5	98.7 ± 0.3	0
IEEE 300-bus	56.7 ± 0.8	99.1 ± 0.2	0

It can be seen from Table 4 that as the number of grid nodes increases from 39 to 300 (10 times expansion), the average solution time per step of the method increases from 8.2 ms to 56.7 ms (only 6.9 times expansion), and the 0% power flow violation rate is always maintained. The maximum line load rate is controlled within the safety threshold of 99.2%.

4.7.2. Agent Quantity Scalability

In multi-agent coordinated D-RTPD, the number of agents critically affects scheduling performance and algorithm convergence. Aiming at the influence of agent quantity on algorithm convergence and dispatch effect in multi-agent cooperative dispatch, the proposed method designs a parameter sharing + hierarchical coordination agent configuration mechanism to realize the flexible expansion of agent quantity. Taking the benchmark IEEE 118-bus system with configurations of six generator dispatch agents and seven energy storage (ES) agents, two comparative cases are investigated: Case 1 with three generator dispatch agents and seven ES agents, and Case 2 with 12 generator dispatch agents and seven ES agents.

The influence of the change of agent quantity on the convergence and dispatch performance of the method is quantitatively analyzed, and the results are shown in Table 5, with the convergence trend corresponding to Figure 11.

Table 5. Comparison of method performance under different agent quantities.

	Training Convergence Iterations	Average Cumulative Reward	Maximum Line Load Rate (%)	Power Flow Violation Rate (%)
Benchmark	3850 ± 52	368.2 ± 15.7	99.5 ± 0.2	1.2
Case 1	3200 ± 45	405.6 ± 12.3	98.7 ± 0.3	0.0
Case 2	4520 ± 68	381.5 ± 14.2	98.9 ± 0.2	0.0

Figure 11. Curves of multi-agent training process under different agent quantities. (a) Moving average cumulative reward curve of Case 1. (b) Loss value curve of the multi-agent policy network for Case 1. (c) Moving average cumulative reward curve of Case 2. (d) Loss value curve of the multi-agent policy network for Case 2.

It can be concluded from Table 5 that due to insufficient dispatchable resources, a power flow violation rate of 1.2% occurs, and the maximum line load rate is close to the safety critical value of 99.5%. Although the daily dispatch cost is low, the operational safety of the power grid is sacrificed. The number of agents matches the demand of grid partition dispatch, with the least training convergence iterations (3200 times) and the highest average cumulative reward (405.6), achieving the optimal balance between safety and economy. Although the 0% violation rate is maintained, the game complexity among agents increases, the training convergence iterations increase by 41.2%, the daily dispatch cost rises by 6.3%, and the cumulative reward decreases slightly.

Results in Figure 11 show that both cases exhibit more fluctuating moving average cumulative reward curves and poorer convergence compared with the benchmark. In Case 1, insufficient dispatchable generators cause inadequate scheduling resources under high renewable penetration, leading to power flow violations and insufficient slack bus regulation capacity; nevertheless, its lower redispatch cost in certain scenarios enables its maximum cumulative reward to approach or exceed the benchmark. In Case 2, more generator agents aggravate game instability and reward volatility, with higher redispatch costs, yet its maximum cumulative reward remains close to the benchmark.

Moreover, the policy networks in both cases tend to converge during training, while their policy loss functions oscillate due to scenario-induced reward fluctuations. Compared with Case 1, Case 2 has higher policy network loss values as agent numbers increase. This indicates that more complex coordination strengthens mutual policy interference among agents, forming a bottleneck for applying large-scale multi-agent systems to real-time power grid dispatch.

4.7.3. Engineering Scenario Adaptability

Based on the design idea of distributed decision-making and local neighborhood information interaction, the proposed method is highly adaptable to the operational characteristics and constraint conditions of actual power systems. Combined with the comparison with traditional methods and the requirements of engineering standards, the engineering adaptability is verified from four dimensions: communication constraints, safety standards, modeling requirements, and economic real-time performance. The quantitative comparison results are shown in Table 6 and the core performance indicators all meet the relevant engineering requirements of the Guidelines for Security and Stable Operation of Power Systems.

Table 6. Comparison of method performance under different algorithm.

Evaluation Dimension	Proposed NFA-GAT-MADDPG	Traditional Model-Driven Method (ADMM)	Traditional Data-Driven Method (Q-MIX)	Engineering Standard Requirements of Power Systems
Maximum Line Load Rate (%)	98.7 ± 0.3	92.3 ± 0.1	130.5 ± 0.5	<100%
Power Flow Violation Rate (%)	0.0	0.0	14.3 ± 0.4	0.0%
High-Precision Modeling Required	No	Yes	No	Adapt to model mismatch scenarios
Average Solution Time per Step (ms)	23.5 ± 0.5	189.2 ± 1.2	21.5 ± 0.8	<100 ms (5-min scale dispatch)

Combined with Table 6 and actual engineering requirements, the 0% power flow violation rate and 98.7% maximum line load rate fully meet the safety operation requirements of power systems for “no overload and no violation”, avoiding safety accidents such as line overload tripping. As a data-driven method, it does not need to establish a complex mathematical model of the power grid, and realizes policy learning only through interaction with the grid simulation environment. It effectively avoids the problems of parameter perturbation and model mismatch in actual power grids and has strong adaptability to changes in grid operating states. Balancing economy and real-time performance: The average solution time per step is 23.5 ms, which meets the time requirement of 5-min scale distributed real-time power dispatch; the daily dispatch cost is reduced by 13.0% compared with Q-MIX. Although it is slightly higher than ADMM, the solution efficiency is improved by 8.1 times, achieving a three-dimensional balance of real-time performance, economy and safety, which is in line with the core demand of actual power system dispatch.

5. Conclusions

This study addresses the intense uncertainty brought by high-proportion renewable energy sources (RES) to distributed real-time power dispatch (D-RTPD) in power systems, and proposes an integrated solution that combines the NFA-GAT with MADDPG based on a source–storage coordination paradigm. Validated on the IEEE 118-bus test system with a 75.2% RES installed capacity ratio, the proposed method demonstrates prominent performance in D-RTPD optimization: MADDPG guarantees stable collaborative dispatch across multi-regional power grids through its centralized criticism and decentralized execution mechanism, where ES agents can autonomously learn adaptive control strategies, charging during RES output surges and discharging during output dips, to mitigate RES fluctuations and cut down dispatch costs effectively. The NFA-GAT, augmented with a distance-dependent spatial discount factor, enables efficient aggregation of multi-order neighborhood information, which enhances agents’ perception of grid operating states and outperforms the MLP structure in terms of cumulative reward and scenario win rate. Additionally, the number of dispatch agents is found to exert a significant impact on the algorithm’s performance: an insufficient number of generator agents leads to a shortage of dispatchable resources and subsequent power flow violations, while an excessive number exacerbates the complexity of inter-agent game, causes multiplicative growth of model parameters, and impairs the convergence stability of the algorithm.

Future research will focus on the following issues:

(1)

Online Update of Graph Attention Weights Under Dynamic Topology Changes: Study the online learning mechanism of NFA-GAT weights during line switching/distributed power access without retraining the model;

(2)

Hierarchical Collaborative Scheduling Strategy for Large-Scale Multi-Agents: Design a “Bi-level” hierarchical architecture, where the upper-level agent is responsible for inter-regional coordination and lower=level agents for local scheduling, addressing convergence and computational complexity issues;

(3)

Robust Scheduling Optimization Under Extreme New Energy Fluctuation Scenarios: Adopt GAN to generate extreme scenarios of sudden rise/drop in new energy output, and optimize the constraint processing capability of NFA-GAT in feature extraction and prior guidance mechanisms;

(4)

Adaptive Sparsification Improvement and Computational Efficiency Enhancement of NFA-GAT [32]: Integrate the sparse graph attention mechanism to reduce the computational load of non-critical nodes and improve the real-time performance of ultra-large-scale power grids.