Optimal Power Flow for High Spatial and Temporal Resolution Power Systems with High Renewable Energy Penetration Using Multi-Agent Deep Reinforcement Learning

Abstract

The increasing integration of renewable energy sources (RESs) introduces significant uncertainties in both generation and demand, presenting critical challenges to the convergence, feasibility, and real-time performance of optimal power flow (OPF). To address these challenges, a multi-agent deep reinforcement learning (DRL) model is proposed to solve the OPF while ensuring constraints are satisfied rapidly. A heterogeneous multi-agent proximal policy optimization (H-MAPPO) DRL algorithm is introduced for multi-area power systems. Each agent is responsible for regulating the output of generation units in a specific area, and together, the agents work to achieve the global OPF objective, which reduces the complexity of the DRL model’s training process. Additionally, a graph neural network (GNN) is integrated into the DRL framework to capture spatiotemporal features such as RES fluctuations and power grid topological structures, enhancing input representation and improving the learning efficiency of the DRL model. The proposed DRL model is validated using the RTS-GMLC test system, and its performance is compared to MATPOWER with the interior-point iterative solver. The RTS-GMLC test system is a power system with high spatial–temporal resolution and near-real load profiles and generation curves. Test results demonstrate that the proposed DRL model achieves a 100% convergence and feasibility rate, with an optimal generation cost similar to that provided by MATPOWER. Furthermore, the proposed DRL model significantly accelerates computation, achieving up to 85 times faster processing than MATPOWER.

1. Introduction

The OPF problem refers to minimizing the power generation cost by appropriately scheduling the dispatchable generation while satisfying physical and engineering constraints of the power system [1]. In recent decades, the penetration of RESs such as photovoltaics and wind power has steadily increased. Under high levels of uncertainty, efficiently solving large-scale OPF problems with feasible solutions has emerged as one of the key challenges in modern power systems.

To deal with the intermittent and uncertain nature of RESs relating to their fluctuations in power output, stochastic OPF is introduced and more frequently solved to maintain the power balance [2]. To obtain the stochastic OPF solution, operators generate numerous load scenarios and solve each scenario with the basic OPF procedure. However, conventional iteration-based optimization methods such as the Newton method, simplex method, and interior-point method, and recently proposed advanced methods such as the point estimation method based on Gram–Charlier expansion [3] and sequential linear programming [4], require significant computational complexity and time consumption for large-scale RES power systems [5]. For instance, solving stochastic OPF for 1000 different load scenarios in a power system with hundreds of nodes typically takes >10 min, which is inadequate for real-time management in terms of the rapid RES fluctuations [6]. Meanwhile, most iteration-based optimization methods are model-based. The performance of model-based methods largely depends on the prior knowledge of electrical parameters of the power system model, which may be time-varying, inaccurate, and unavailable in practical scenarios [7]. In addition, due to the non-linear and non-convex nature of the OPF problem, finding feasible OPF solutions is an NP-hard problem and becomes a cornerstone of existing challenges [8].

The above challenges and the advancement of data-driven artificial intelligence (AI) techniques motivate the use of the DRL model to solve large-scale OPF problems in highly RES-penetrated power systems. Unlike existing adaptive methods such as DynaSLAM [9] with parameter adaptation, the DRL is a more effective method for adapting to entire complex and dynamic environments, e.g., the strong uncertainties of RES generation and load demand, through automatically optimizing reward and policies without manual intervention or prior rules. With the help of the online training and offline execution framework, the DRL method greatly reduces the computational burden to meet the real-time OPF requirement [10]. To guarantee the solution’s feasibility, recent works on DRL-based OPF methods have attempted various strategies to address the security constraints. Yan and et al. proposed a Lagrangian-based DRL approach to construct an augmented action-value function with a penalty on constraint violation [11]. Sayed et al. add a novel convex safety layer with the penalty convex–concave procedure into the DRL model [12]. Similarly, hybrid knowledge or expert knowledge safety layers are developed and combined with the DRL model to improve the solution feasibility in [7,13]. However, most existing DRL-based OPF methods adopt a single-agent architecture [14,15,16]. For large-scale RES power systems, the high-dimensional state and action spaces increase the risk of the dimensionality curse. To improve the scalability and mitigate the training complexity of DRL methods, strategies like multi-task attribution map [17] and the scalable graph DRL [18] are introduced. Additionally, the multi-agent DRL framework [19] presents a promising startegy for solving the large-scale OPF in multi-area power systems. For example, sub-regions and/or areas of the power system are identified according to geographical, administrative, or dispatching authority boundaries. Each agent of the multi-agent DRL framework is only responsible for generating the actions for one area of the power system, and all agents collaboratively share information to accomplish the optimization task for the entire system. The multi-agent DRL framework has been proven effective for the energy management of power systems [20,21] and the frequency or voltage control of microgrids or multi-area power systems [22,23,24]. Nevertheless, the multi-agent DRL framework applied specifically to OPF problems in multi-area power systems still warrants further investigation.

Moreover, in RES power systems, extracting critical spatiotemporal information from RES power output and power system structure to provide meaningful feature inputs for the DRL model is one of the key factors in ensuring its decision-making performance. Li et al. use attention neural networks to directly joint load profile and weather patterns with power dispatch and generation [5]. Gao et al. introduce a Bayesian deep neural network and conditional variational auto-encoder to generate random spatiotemporal source–load scenarios [25]. By integrating physics-informed rules or laws into deep learning models, the physics-informed neural network (PINN) [26,27] shows its great potential for enhancing the convergence and feasibility of OPF in RES power systems [28]. Additionally, the power system is a typical graph-structured system, where data are collected from non-Euclidean domains with interdependency among nodes. Graph neural networks (GNNs) demonstrate strong feature extraction capability in graph-structured systems, e.g., power systems [29,30]. Recent works show that the GNN remarkably improves the OPF performance by integrating the spatial features of power system topology and the temporal features of RESs’ power output time series [31,32,33].

In this paper, an H-MAPPO-based DRL model with a GNN feature extraction layer is proposed to solve the OPF. The main contributions are summarized below:

• The heterogeneous MAPPO (H-MAPPO) DRL model is proposed to address the OPF problem in multi-area power systems, significantly reducing training complexity and enhancing decision-making efficiency.
• A GNN layer is incorporated to extract spatiotemporal features of RES fluctuations and power grid topologies, enhancing the convergence and feasibility of OPF solutions.
• The performance of the proposed DRL model is validated using the RTS-GMLC test system, a near-real power system model with high spatial and temporal resolution and substantial RES penetration.

The remainder of this paper is organized as follows. In Section 2, the OPF model and corresponding MDP are introduced. The H-MAPPO-based DRL model with the GNN for solving the OPF is proposed in Section 3. Simulation results are given in Section 4. Conclusions are drawn in Section 5.

2. Problem Formulation

Under the steady-state operation and 3-ph balanced conditions [34], the OPF model can be formulated as follows:

$$\min C_{sys}=\sum _{t=0}^{\infty }\sum _{k=1}^{N_G}{C}_g\left(P_{k,t}^G\right)$$

(1)

$$P_{i,t}=\sum _{j\in {\mathrm{\Omega }}_i}{V}_{i,t}{V}_{j,t}(G_{ij}cos\left({\theta }_{ij,t}\right)+B_{ij}sin\left({\theta }_{ij,t}\right))$$

(2)

$$Q_{i,t}=\sum _{j\in {\mathrm{\Omega }}_i}{V}_{i,t}{V}_{j,t}(G_{ij}sin\left({\theta }_{ij,t}\right)-B_{ij}cos\left({\theta }_{ij,t}\right))$$

(3)

$$P_{ij,t}=V_{i,t}^2{G}_{ij}-V_{i,t}{V}_{j,t}(G_{ij}cos\left({\theta }_{ij,t}\right)+B_{ij,t}sin\left({\theta }_{ij,t}\right))$$

(4)

$$Q_{ij,t}=-V_{i,t}^2{B}_{ij}+V_{i,t}{V}_{j,t}(B_{ij}cos\left({\theta }_{ij,t}\right)-G_{ij,t}sin\left({\theta }_{ij,t}\right))$$

(5)

$$\underline{P_{k,t}^G}\le P_{k,t}^G\le \overline{P_{k,t}^G}$$

(6)

$$\underline{Q_{k,t}^G}\le Q_{k,t}^G\le \overline{Q_{k,t}^G}$$

(7)

$$\underline{P_{ij}}\le P_{ij,t}\le \overline{P_{ij}}$$

(8)

$${\underline{V_i}}^2\le V_{i,t}^2\le {\overline{V_i}}^2$$

(9)

$$\underline{P_z^{tie}}\le \sum _{\left(ij\right)\in {\mathcal{E}}_z^{tie}}{P}_{ij,t}\le \overline{P_z^{tie}},z\in \mathrm{\Phi }$$

(10)

where the quadratic cost function in Equation (1) is $C_g\left(P_{k,t}^G\right)=c_k^2{\left(P_{k,t}^G\right)}^2+c_k^1{P}_{k,t}^G+c_k^0$.

Markov Decision Process Formulation

In an N-bus power system composed of $M\ge 2$ areas, assuming each area of the power system corresponds to an agent, and all agents collaboratively make decisions to achieve OPF for the entire system, the OPF problem can be formulated as an MDP with the main components of state space S, action space A, state transition probability matrix $\mathbf{P}$, and reward function R:

$$\mathcal{M}=(S,A,\mathbf{P},R)$$

(11)

(1) State: At each time step t, the local observation $o_{m,t}$ of area $m=1,2,\dots ,M$ is

$$o_t^m=\left[{\mathbf{P}}_t^{G,m},P_{loss}\right]$$

(12)

where ${\mathbf{P}}_t^{G,m}=\left[P_{1,t}^{G,m},P_{2,t}^{G,m},\dots ,P_{N_m^G,t}^{G,m}\right]$.

The global state space includes the local observation of all areas:

$$S_t=\left[o_t^1,o_t^2,\dots ,o_t^M\right]$$

(13)

(2) Action: The action space is the set of all possible actions that each area of the power system can take under a given state. At each time step t, the local action space $a_t^m$ is defined as

$$a_t^m=\left[\mathrm{\Delta }{P}_{1,t}^{G,m},\mathrm{\Delta }{P}_{2,t}^{G,m},\dots ,\mathrm{\Delta }{P}_{N_m^G,t}^{G,m}\right]$$

(14)

The entire action space is defined as

$$A_t=\left[a_t^1,a_t^2,\dots ,a_t^M\right]$$

(15)

(3) Reward: The reward function serves as a quantitative metric to assess the utility or value associated with a particular state and action pair, which is designed according to the optimization objective. As shown by Equations (1)–(10), the objective is power flow optimization, e.g., minimizing generation costs under system constraints. The corresponding reward function is defined as follows:

$$R_t=\left\{\begin{array}{cc}\hfill {\alpha }_1-{\displaystyle \frac{C_{sys}}{{\alpha }_2}},& \mathrm{if}\mathrm{the}\mathrm{OPF}\mathrm{is}\mathrm{converged}\mathrm{and}\mathrm{feasible}\hfill \\ \hfill {\alpha }_3-D_p,& \mathrm{if}\mathrm{the}\mathrm{OPF}\mathrm{is}\mathrm{converged}\mathrm{but}\mathrm{infeasible}\hfill \\ \hfill -1000,& \mathrm{if}\mathrm{the}\mathrm{OPF}\mathrm{is}\mathrm{non}\text{-}\mathrm{convergent}\hfill \end{array}\right.$$

(16)

In Equation (16), $D_p={\sum }_{i=1}^N{P}_{i,t}$. The OPF solution is converged and feasible when all constraints shown by Equations (2)–(10) are satisfied. The OPF solution is converged but infeasible when constraints (2)–(5) are satisfied and constraints (6)–(10) are violated. The OPF solution is non-convergent when constraints (2)–(5) are violated.

3. Deep Reinforcement Learning Foundations

DRL is an advanced method for solving MDP problems. The fundamental idea of DRL is to maximize the cumulative reward function assigned to a specific task through continuous interactions between the agent and the environment, e.g., power systems, ultimately learning the optimal strategy for performing the task. To address the challenge of efficient and stable DRL training for coordinating multiple agents in the power system’s OPF environment, the concept of a heterogeneous multi-agent system is introduced, along with the MAPPO algorithm. In addition, the GNN is introduced to extract the global spatial–temporal feature from the time series of state variables and the power grid topology, which will be used as part of the input of the DRL model.

3.1. Heterogeneous Multi-Agent System

In practical large-scale power systems, the power flow optimization strategy is often deployed across multiple areas of the power system. Each area includes different numbers of buses, dispatchable generation units, and transmission lines, requiring the adaptivity for non-uniform state and action spaces of individual DRL agents. The corresponding multi-agent system is the heterogeneous multi-agent system (H-MAS). The H-MAS structure is particularly critical for solving the OPF in multi-area power systems. First, compared with the homogeneous multi-agent DRL method, agents in H-MAS can adapt to non-uniform state and action spaces, enabling them to handle specific operational constraints and objectives. For example, each agent can independently adjust the power output of dispatchable generation units within its area while collaboratively optimizing global objectives, i.e., minimizing the total generation cost while satisfying the physical and engineering constraints of the multi-area power system. Second, the H-MAS naturally aligns with the decentralized nature of modern power systems, where each area operates autonomously but must coordinate with neighboring areas to ensure system-wide stability. During the training process, the DRL agents share global information about the multi-area power system and take action with local observations at the execution stage. While the OPF problem of the multi-area power system could theoretically be solved using a single-agent DRL method, the practical training process often suffers from the curse of dimensionality due to the growing state and action spaces in large-scale systems. In contrast, the H-MAS framework decomposes the OPF problem into smaller subproblems for each area, significantly reducing training difficulties and improving scalability for practical multi-area power systems.

3.2. Multi-Agent Proximal Policy Optimization

MAPPO is an extended algorithm of proximal policy optimization for the multi-agent system environment. The proximal policy optimization algorithm is a policy gradient approach, seeking to enhance policy performance by optimizing surrogate objective functions through stochastic gradient ascent and following interactions with the environment and data sampling [35]. By leveraging multiple samples for policy gradient estimation, PPO efficiently utilizes the collected data, enabling faster training and more streamlined update procedures. Furthermore, PPO carefully regulates policy modifications during the update phase to avoid large, destabilizing changes that could compromise training stability. With the MAPPO algorithm, the MDP problem is implemented by modeling each agent’s decision-making as a decentralized MDP with shared observations. During training, agents interact with the environment by observing states (or partial observations), taking actions based on their policies, and receiving rewards, following the MDP structure introduced in Section 2.

The MAPPO algorithm adopts a CTDE framework [36], making it particularly well suited for fully cooperative environments, e.g., the OPF scenario for the power system with multiple areas. In the CTDE framework, the central controller is only responsible for the training, collecting the global environment state information, as shown in Equation (13), to train the actor and critic networks of each agent. During the execution, agent m at time step t can obtain its local observation $o_{m,t}$, as shown in Equation (12), and then, based on its actor network ${\pi }_{{\phi }_m}$, take action $a_{m,t}$, as shown in Equation (15). Due to the cooperative relationship between the agents in this paper, all agents share the same reward $R_t$, as shown in Equation (16). The critic network collects $S_t$ and $A_t$ to generate a Q-value function to maximize the reward function.

The loss function of the actor network is formulated as follows:

$$\mathcal{L}\left({\phi }_m\right)=E\left[min({\rho }_{\phi }·{\widehat{A}}_m(S_t,A_t),clip({\rho }_{\phi },1-\epsilon ,1+\epsilon )){\widehat{A}}_m(S_t,A_t)\right]$$

(17)

In Equation (17), the importance sampling ratio is ${\rho }_{\phi }={\displaystyle \frac{{\pi }_{{\phi }_m}\left(a_{m,t}\right|o_{m,t})}{{\pi }_{{\phi }_m}^{old}\left(a_{m,t}\right|o_{m,t})}}$, representing the ratio of the new policy to the old policy. The clipping function $clip(·)$ limits the change between the old and new policies in the range $\left(1-\epsilon ,1+\epsilon \right)$, aiming to avoid the excessive modification of the objective value. The critic network is trained by the TD-error method [37] to maximize the loss function. The specific loss function is

$${\mathcal{L}}^c=-\sum _{t=1}^{\infty }{(\sum _{t^{\prime }-t}{\gamma }^{t^{\prime }-t}{r}_{t^{\prime }}-V_{{\varphi }_m}\left(o_{m,t}\right))}^2$$

(18)

The proposed DRL-based OPF method deploys the H-MAPPO algorithm. As a comparison baseline, the traditional interior-point method in MATPOWER is used. The interior-point method solves OPF by transforming the constrained optimization problem into a sequence of unconstrained subproblems using barrier functions. It handles inequality constraints, e.g., generation limits, by keeping iterates strictly feasible while gradually reducing a barrier parameter. The interior-point method follows the perturbed Karush–Kuhn–Tucker conditions, balancing optimality and feasibility until convergence.

3.3. Graph Neural Network

The GNN is an effective method for the representation learning of graph-structured data. The power system can be represented as a graph, enabling the use of GNNs to extract its spatiotemporal features. Let $\mathcal{G}=\left(\mathcal{V},\mathcal{E}\right)$ denote the graph of a power system, where $\mathcal{V}$ and $\mathcal{E}$ are the node and edge sets. The node and edge represent the bus and transmission line in the power system. The node feature vectors $X_v$ for $v\in \mathcal{V}$ are the input of the GNN. For each bus in the power system, the node feature $X_v$ includes the generation active power output and active load demand of that bus. In this paper, the GNN shares the same structure of the graph $\mathcal{G}$. GNNs use the graph structure and node features $X_v$ to extract a representation vector of each node, $h_v$, the output of GNN. GNNs employ a neighborhood aggregation approach, where the representation vector of a node is derived by recursively collecting and transforming the representation vectors of its neighboring nodes [38]. Specifically, the GraphSAGE (Graph Sample and Aggregator) [39] is deployed, which is a GNN framework for learning node embeddings in large-scale graphs. The core idea of GraphSAGE is to generate node representations by sampling and aggregating features from a node’s local neighborhood, rather than learning individual embeddings for each node. By recursively aggregating information from sampled neighbors, GraphSAGE captures structural patterns. A modified mean-based aggregator with a convolutional function is used to sample and aggregate features:

$${\mathbf{h}}_v^{\left(l\right)}={\mathbf{W}}_1{\mathbf{h}}_v^{(l-1)}+{\mathbf{W}}_2·\mathrm{MEAN}\left(\left\{{\mathbf{h}}_u^{\left(l\right)},\forall u\in {\mathrm{\Omega }}_v\right\}\right)$$

(19)

In Equation (19), the input of GNN ${\mathbf{h}}_v^{\left(0\right)}$ is a vector that includes the information about the adjacency matrix of the power grid, admittance of transmission lines, and loads of all nodes. The output of GNN ${\mathbf{h}}_v^{\left(L\right)}$ is called the representation vector.

The overall architecture for power flow optimization based on H-MAPPO and GNN is illustrated in Figure 1. With the input including the adjacency matrix of the power grid, admittance of transmission lines, and loads of all nodes, the GNN is employed to extract the topology-embedded spatiotemporal features of the power grid. Subsequently, the features provided by the GNN are multiplied with the local observation from each area to form the inputs for H-MAPPO. Based on the inputs and the trained policy, H-MAPPO determines the action values for the next step, which is fed back into the power system to adjust generator outputs and achieve the OPF.

4. Simulation and Results

4.1. RTS-Grid Modernization Laboratory Consortium System

In 2019, the RTS-GMLC system was introduced as an updated version of the RTS-96 test system. The RTS-GMLC system aims to facilitate the study and analysis of challenges in modern power systems including the variability and uncertainty of generation fleets and the flexibility of dispatch processes such as unit commitment and economic dispatch under uncertainty. As shown in Figure 2, the RTS-GMLC system includes 73 buses, 158 generators, 121 branches, and 3 areas connected by 6 tie lines.

The RTS-GMLC system represents a high spatial and temporal resolution power system with high renewable penetration. For the RTS-GMLC system, monitoring, analysis, and control operations can be carried out with detailed generation, load, and grid topology data across both space and time. The high-resolution capability is essential for addressing the increasing complexity of power systems due to the integration of RESs. As shown in Figure 3a, the geographic location of the power network of the RTS-GMLC model is arbitrarily projected onto nearly 250 square miles in the southwestern United States. With the help of the data from balancing authorities and the Western Wind and Solar Integration Study 2 (WWSIS-2) dataset, the RTS-GMLC model provides a reference for load, solar, wind, rooftop photovoltaic, CSP, and hydrologic time-series [40]. The load and RES time series enable year-long simulations with hourly resolution, e.g., 24 data points each day. A one-week illustration of load and RES time series is shown in Figure 3b–g. Nevertheless, it should be noted that the RTS-GMLC system remains synthetic and does not represent a real-world power system [41].

4.2. Experimental Setup

The effectiveness and performance of the proposed DRL model are validated by the TRS-GMLC system. The proposed DRL model is realized with PyTorch (2.6.0). The training is conducted on a personal laptop with 13th Gen Intel(R) Core(TM) i9-13900H. The power system environment is built in MATLAB2024b with the MATPOWER toolbox. The load profile and power generation time series from 1 January 2021 to 31 December 2021 are available from the RTS-GMLC system repository [42], which are used to generate the training dataset. The training dataset includes 8784 samples, representing the one-year hourly power flow conditions. With the CTDE framework of the MAPPO, three DRL agents are used for the OPF task. Each agent corresponds to one area in the RTS-GMLC system shown in Figure 2. Each agent takes action on generation power adjustment for the corresponding area. The hyperparameters used to train the proposed DRL model are given in

Table 1. Summary of hyperparameters.

Parameters	Type or Value
optimizer	Adam
activation function	tanh
learning rate	5e-5
GAE lambda	1.0
discount factor	0.99
clipping $\epsilon$	0.3
batch size	2160
number of hidden layers in GNN	5
neuron number of each hidden layer	64
length of representation vector	68

.

4.3. Results Analysis

During the training process, the DRL model is updated once per episode with a batch of samples. The reward for an episode is the sum of rewards for all samples in the batch. As shown in Figure 4, the reward for each episode continuously increases and converges to a relatively high reward value after approximately 600 episodes.

Four metrics are used to evaluate the performance: convergence rate, feasibility rate, cost difference, and computing time. The OPF solution is convergent if it satisfies the constraints from Equations (2)–(5). The OPF solution is feasible if it meets all constraints from Equations (6)–(10). Practical power system operation requires OPF solutions that are both converged and feasible. As shown by Equations (20) and (21), the convergence rate ${\mathcal{R}}_c$ and feasibility rate ${\mathcal{R}}_f$ are defined as the ratio of converged and feasible OPF solutions to the total number of samples in the test dataset:

$${\mathcal{R}}_c={\displaystyle \frac{n_c}{n_{test}}}\times 100\%$$

(20)

$${\mathcal{R}}_f={\displaystyle \frac{n_f}{n_{test}}}\times 100\%$$

(21)

The cost difference refers to the difference between the generation costs determined by the DRL model and MATPOWER. Note that the cost difference is normalized by the generation cost given by MATPOWER. The computing time refers to the time for the DRL model or MATPOWER to obtain an OPF solution.

The performance is tested under scenario A and scenario B. For scenario A and scenario B, the test dataset includes $n_{test}$ = 2000 samples based on the load profile sampled within $\left[90\%,110\%\right]$ and $\left[50\%,150\%\right]$ of the default load uniformly at random, respectively. Note that compared with scenario A, the load sampling uncertainty is high in scenario B, increasing the mismatched power $D_p$ and difficulty in finding converged and feasible OPF solutions. In each scenario, the performance of the proposed DRL model after training is compared to that of MATPOWER. It can be found from

Table 2. Performance comparisons under scenario A.

Metrics	Proposed DRL Model	MATPOWER
Convergence rate (%)	100%	100%
Feasibility rate (%)	100%	100%
Computing time (ms)	6.7 ± 3.6	250.0 ± 8.4

that, for scenario A with low load sampling uncertainty, the performance in terms of convergence, feasibility, and generation cost is similar between the proposed DRL model and MATPOWER, e.g., the convergence rate and feasibility rate are 100% for both proposed DRL model and MATPOWER, and the cost difference is 1.4% ± 0.75%. As shown by

Table 3. Performance comparisons under scenario B.

Metrics	Proposed DRL Model	MATPOWER
Convergence rate (%)	100%	68.8%
Feasibility rate (%)	100%	68.8%
Computing time (ms)	6.8 ± 4.0	580.0 ± 12.5

, for scenario B with high load sampling uncertainty, the convergence rate and feasibility rate of the proposed DRL model are still 100% with the similar generation cost given by MATPOWER (cost difference is 1.5% ± 0.75%), while the convergence rate and feasibility rate of the MATPOWER significantly decrease to 68.8%. In addition, for large-scale power systems with high RES penetration, the computational efficiency of OPF methods is essential to maintain power balance under rapid power fluctuations. The proposed DRL model shows a substantially fast decision-making performance. Specifically, for scenario A, the average computing time of the proposed DRL model is 6.7 ms, approximately 37× faster than MATPOWER, and for scenario B, the average computing time of the proposed DRL model is 6.8 ms, approximately 85× faster than MATPOWER. Finally, different random seeds are used to initialize the DRL training process. The corresponding convergence rate and feasibility rate are 100% with the DRL after training, demonstrating robustness against the stochastic nature of DRL algorithms.

5. Conclusions

For multi-area power systems with a high penetration of RESs, a DRL model using the H-MAPPO framework is proposed to solve the OPF problem. In the proposed DRL model, each agent is responsible for determining the power generation in one area of the power system while all agents collaboratively optimize the overall power flow across the whole power system. A GNN layer is integrated into the model to extract spatiotemporal features of RES fluctuations and grid topologies. The performance of the proposed DRL model is validated using the RTS-GMLC test system, and its results are compared to MATPOWER, which solves the OPF using the interior-point method. The proposed DRL model achieves a 100% convergence rate and feasibility rate. The optimized generation cost is comparable between the DRL model and MATPOWER. Furthermore, the DRL model demonstrates robustness in terms of convergence and feasibility even under high load sampling uncertainty. Additionally, the proposed DRL model exhibits significant computational efficiency, being up to 85 times faster than MATPOWER, thus meeting the real-time requirements for OPF tasks.