Distribution Network Optimization with Aggregation and Reinforcement Learning Under Massive Distributed Resources Integration

Abstract

The integration of large-scale distributed energy resources (DERs) into distribution networks (DNs) brings challenges to the effective control of DNs. In traditional approaches, mathematical or reinforcement learning (RL)-based solution algorithms are commonly used. However, the exponential increase in the number of DERs reduces the effectiveness of these strategies. Mathematical methods struggle to cope with the dynamic uncertainty caused by the high penetration of renewable energy, while RL algorithms relying on global data training may violate multi-agent privacy protocols. This paper proposes a DNs cooperative optimization method based on resource aggregation and RL. To reduce optimization dimensionality and ensure the privacy of resource data, a dynamic aggregation strategy is employed to aggregate a large number of distributed energy resources into aggregated entities, and the adjustable active–reactive power boundaries of each aggregated entity are derived. To fully exploit the regulation capability of DNs, data centers (DCs), as novel devices, are considered as flexible loads. To improve the convergence speed of model training and decision-making accuracy, evolution strategies (ES) and prioritized experience replay (PER) are integrated into the Soft Actor-Critic (SAC) algorithm, respectively. The proposed method is validated on the IEEE 33-bus and IEEE 123-bus systems. The results demonstrate the effectiveness and superiority of the proposed method in ensuring the secure operation of DNs.

1. Introduction

1.1. Background

By the end of 2025, China’s distributed photovoltaic installed capacity had reached 530 GW, while the nation’s cumulative installed wind power capacity had reached 640 GW, indicating a significant trend toward large-scale distributed energy integration [1]. However, while high renewable penetration brings green benefits, it also profoundly changes power system operation. In Spain, renewables accounted for 55.5% of generation in 2025, yet declining inertia and weak dynamic voltage support have become serious concerns. On April 28 that year, many inverter-based resources failed to provide reactive/voltage support during high PV output, leading to a major blackout on the Iberian Peninsula. This shows that as renewable penetration rises, operational uncertainty grows, and extreme weather combined with supply–demand stress can easily trigger cascading failures. In contrast, a grid with good voltage levels and low losses under normal operation ages more slowly, operates more healthily, and offers better initial conditions for coping with emergencies and achieving rapid recovery.

In China, as distributed PV penetration keeps increasing, the source–load imbalance in distribution networks worsens, making voltage violations and network losses more prominent. Thus, how to effectively control vast, scattered distributed energy resources and coordinate optimal dispatch among multiple stakeholders has become a key challenge that urgently needs to be solved.

With the widespread adoption of artificial intelligence (AI), the number and load of data centers (DCs) are growing rapidly, presenting new potential for the management of DNs. In 2022, DCs of China consumed 270 billion kWh of electricity, accounting for 3% of the total societal electricity consumption [2]. By 2030, it is projected that the total electricity consumption of DCs of China will reach approximately 525.76 billion kilowatt-hours, accounting for 4.8% of the total societal electricity consumption [3]. Typically, the adjustable load ratio within DCs can exceed 20% [4], and through coordinated optimization of computing, power, and thermal systems, a carbon emission reduction of 38.5% can be achieved [5]. However, a variety of optimization and control methods cannot directly and effectively control a large number of DCs and RES. Meanwhile, since the power consumption values of DCs can reflect a wealth of information about customers, directly reporting their regulation capabilities to the Distribution Systems Operator (DSO) is prone to causing data privacy leakage.

Currently, the control of dispatchable resources in distribution networks remains primarily device-level independent regulation, without the formation of system-level coordinated optimization. Specifically, branch switches are used for network reconfiguration to alter power flow distribution [6]; photovoltaic inverters can regulate reactive power [7]; capacitor banks provide discrete reactive power compensation through switching operations to support voltage [8]; energy storage systems (ESS) leverage their fast bidirectional charging/discharging capabilities for peak shaving and voltage support [9]; and electric vehicle charging stations regulate active power through coordinated charging control [10]. Although device-level independent regulation can effectively adjust local areas, the regulation capability of individual devices is limited. Moreover, with the integration of large-scale devices, existing optimization algorithms cannot rapidly and effectively solve large-scale, complex distribution network optimization problems, making it difficult to achieve global secure and economic operation under high-penetration distributed energy resource integration scenarios.

The coordinated optimization of distribution networks is an optimization problem. To explore its control strategies, the problem must be solved under specific practical conditions. The algorithms for solving optimization problems mainly fall into three categories: mathematical programming, physics-inspired heuristics, and intelligent optimization [11]. Mathematical programming methods, such as nonlinear optimal power flow, often face the curse of dimensionality when dealing with large-scale, non-convex distribution network optimization problems involving mixed discrete variables, making it difficult to obtain a global optimal solution within a limited time, whereas linear optimal power flow struggles to accurately capture the nonlinear characteristics in actual system operation as well as the switching logic of discrete devices. Physics-inspired heuristic algorithms are prone to getting trapped in local optima and are highly dependent on accurate physical models and parameters [12]. In contrast, intelligent algorithms, particularly reinforcement learning, have gradually gained attention. Reinforcement learning offers unique advantages in real-time decision-making and adaptive optimization [13]. However, current reinforcement learning algorithms still suffer from low sample efficiency and convergence difficulties during the training phase, and their solution optimality and computational efficiency require further improvement.

1.2. Previous Research

This part delves into the research status quo and existing problems of DC, VPP aggregation, and multi-agent deep reinforcement learning (MADRL). DCs can be regarded as a type of flexible load that is regulable in both time and space, and they can be incorporated into the operation of power systems as an important demand response resource [14]. Studies on the planning of DCs primarily focus on determining their locations and capacities with a long-term perspective. Ref. [15] takes into account various uncertainties and network constraints during the expansion planning process to determine the locations and capacities of DCs. Ref. [16] proposes the need for expansion planning of DCs to meet the development of data networks. Most studies on the operational optimization of DCs leverage their regulation potential to improve economic benefits [17]. Ref. [18] reduces the operational costs of microgrids based on the demand response capability of DCs. Refs. [19,20] achieve cost and emission reductions by shifting the workloads of DCs to fully absorb renewable energy. Ref. [21] utilizes the flexibility of DCs’ energy consumption to optimize power flow within the system. Specifically, DCs have the potential to reduce network losses, alleviate network congestion, etc. However, the full exploitation of this flexibility in DNs has not been thoroughly explored, nor have methods been established for the effective control of large-scale DCs.

Current MADRL has gained considerable attention and application in real-time DN optimal operation, owing to its model-free nature and adaptive decision-making capability [22]. For instance, Ref. [7] develops a voltage regulation framework incorporating transactive energy, where the reactive power outputs of renewable energy sources (RES) and ESS are controlled to improve the economic performance of DNs under voltage constraints. However, while such studies confirm the feasibility of MADRL in DN control problems, they often neglect the scalability of the proposed methods [23]. To enhance scalability, reference [24] introduces a distributed variant of the proximal policy optimization algorithm specifically designed for electric vehicle charging coordination. However, its effectiveness has only been validated when directly controlling a few devices, and has not been verified under the access of hundreds or thousands of energy resources. Moreover, the training phase still requires the collection of RES data, which does not fundamentally eliminate the potential threat of data leakage [25].

In summary, there is currently a lack of a MADRL-based scheduling framework tailored for large-scale distributed generators. Such a framework would help effectively exploit the regulation potential of distribution networks and achieve fast and efficient solutions in scenarios involving large-scale DERs and DCs connected to DNs.

1.3. Contributions

In the above literature, it can be observed that MADRL methods for real-time active and reactive power cooperative optimization in DNs under large-scale resource integration are relatively new and have not yet considered the regulation capabilities of novel devices such as data centers. To fill these research gaps, this paper proposes a real-time active and reactive power cooperative optimization method for DNs based on large-scale resource aggregation, aiming to reduce the voltage violation rate and network loss in DNs. Compared with existing methods, the main contributions of this paper are summarized as follows:

(1)

A real-time active and reactive power cooperative optimization method for DNs based on large-scale resource aggregation is proposed. The total power aggregated from large-scale resources is used as the input for designing the MADRL model, enabling effective control of large-scale resources within the region, preserving the data privacy of individual resources, and reducing voltage violation rates and network losses without requiring communication.

(2)

A control method is proposed that utilizes data centers as novel control devices. DCs, serving as flexible loads, can further unlock the regulation potential of DNs.

(3)

We propose integrating ES and PER techniques into the SAC algorithm. By incorporating the concept of population evolution, ES enhances global exploration capability to avoid falling into local optima, while PER prioritizes sampling high-value experience samples to improve training efficiency and stability, thereby significantly accelerating convergence and enhancing decision-making quality.

The effectiveness of the proposed model is validated on the IEEE 33-bus and IEEE 123-bus systems. By evaluating metrics such as voltage violation rate and network loss, it is demonstrated that the proposed model outperforms other benchmark models, including the model without aggregation and MADRL models employing various algorithms.

The remainder of this paper is organized as follows. Section 2 introduces the proposed model, including the aggregation of massive distributed resources, the mathematical modeling of multi-region cooperative optimization for DNs, and the formulation as a Partially Observable Markov Decision Process (POMDP). Section 3 presents the improved MADRL algorithm. Section 4 validates the effectiveness of the proposed method in controlling a large number of distributed resources by comparing it with device-level independent control approaches, and also demonstrates the effectiveness and superiority of the improved algorithm through comparison with multiple benchmark algorithms. Section 5 concludes the paper.

2. Optimization Model

The recent power outages in Spain have highlighted critical vulnerabilities in systems with high penetration of renewable energy. The reasons include (1) rapid voltage fluctuations caused by intermittent generation, (2) delayed response of centralized control under emergency conditions, and (3) insufficient coordination among numerous distributed resources. These challenges call for a decentralized, fast-acting, and adaptive voltage regulation scheme. To address (1) rapid voltage fluctuations caused by intermittent generation, the method proposed in this paper is illustrated in Figure 1. First, in the stage of large-scale resource aggregation within a region, DC, PV, and WT are aggregated to obtain the total controllable power of all resources in the region. Meanwhile, instead of reporting all resource data to the DSO, the region only provides its total dispatchable power. Based on the MADRL method, each agent independently controls a different region, and the optimality of decision-making is enhanced through inter-agent cooperation. Finally, based on the decision results, the DSO issues control commands to each region.

Figure 1. MADRL-based cooperative optimization of DNs with massive resource aggregation.

To address (2) delayed response of centralized control under emergency conditions and (3) insufficient coordination among numerous distributed resources, this method leverages aggregation to achieve effective control over large-scale resource integration, and enhances scalability from the following three aspects: (1) computational dimension—aggregation reduces the high-dimensional individual control problem to a low-dimensional regional-level problem, fundamentally alleviating the curse of dimensionality; (2) communication dimension—each region only uploads the aggregated total power rather than individual device data, avoiding frequent data exchange between massive devices and the main station; (3) architectural dimension—the adopted centralized training with decentralized execution (CTDE) framework enables fully parallel decision-making by all agents during online execution, such that the overall computation time does not grow with the number of network bues but depends only on the decision time of a single region.

Furthermore, by providing the dispatchable range of the overall regional resources rather than the data of individual resources, data privacy protection is achieved. Notably, the proposed method only issues dispatch commands to each region without intervening in its internal control, allowing the region to manage its internal resources according to its own specific conditions. Furthermore, when new distributed resources are added within a region, only the aggregation model parameters of that region need to be updated, without altering the overall framework or retraining all agents, which further enhances the engineering scalability of the method.

The method can serve as a last-line defense against cascading voltage collapse, as it operates locally and adapts to real-time conditions, which is exactly what the Spanish system lacked during the event.

2.1. Aggregation of a Large Number of Resources

PVs and WTs are RESs with adjustable potential. Meanwhile, the rapid development of CDs has also made them significant new adjustable loads in DNs. In contrast to existing studies, this paper first aggregates a large number of WTs, PVs, and DCs connected to the same DN bus to obtain their total power, and then employs MADRL to control the aggregated total power, rather than using MADRL to individually control each independent WT, PV, and DC. It is worth noting that, due to the relatively small scale of network connections among DERs compared to the DN system, the network topology and power losses are neglected.

DCs can be either distributed or centralized in type, and they are regarded as time-scale adjustable loads. The effectiveness of the proposed method depends solely on the regulation capacity and response characteristics of the DCs. The power regulation constraints for WTs, PVs, and DCs are presented as follows.

$$\{\begin{array}{c}0\le \sum P_t^{WT}\le \sum P_{t,max}^{WT}\\ \sum Q_{t,min}^{WT}\le \sum Q_t^{WT}\le \sum Q_{t,max}^{WT}\end{array}\forall t\in T$$

(1)

$$\{\begin{array}{c}0\le \sum P_t^{PV}\le \sum P_{t,max}^{PV}\\ \sum Q_{t,min}^{PV}\le \sum Q_t^{PV}\le \sum Q_{t,max}^{PV}\end{array}\forall t\in T$$

(2)

$$\sum L_{t,min}^{DC}\le \sum L_t^{DC}\le \sum L_{t,max}^{DC} t\in T$$

(3)

where $P_t^{WT}$ and $P_t^{PV}$ are the adjusted active power after aggregation at time $t$; $Q_t^{WT}$ and $Q_t^{PV}$ are the adjusted active power after aggregation at time $t$; $P_{t,max}^{WT}$, $Q_{t,max}^{WT}$, $P_{t,max}^{PV}$ and $Q_{t,max}^{PV}$ are the adjusted active maximum power of WTs and PVs; $L_t^{DC}$ is the adjusted active power of DCs, which is constrained by the adjusted active maximum power limits of the DCs, $L_{t,max}^{DC}$ and $L_{t,min}^{DC}$.

$$P_{i,t}^{agg}=\sum P_t^{WT}+\sum P_t^{PV}-\sum L_t^{DC}$$

(4)

$$Q_{i,t}^{agg}=\sum Q_t^{WT}+\sum Q_t^{PV}$$

(5)

where $P_{i,t}^{agg}$ and $Q_{i,t}^{agg}$ are the adjusted active power after aggregation on bus $i$ at time $t$.

The adjustable ranges of active and reactive power after aggregation are expressed as follows.

$${\mathsf{\Omega }}_p=\{[P^{agg}|{\underset{\_}{P}}_t^{agg}\le P_t^{agg}\le {\overline{P}}_t^{agg}]\}$$

(6)

$${\mathsf{\Omega }}_p=\{[Q^{agg}|{\underset{\_}{Q}}_t^{agg}\le Q_t^{agg}\le {\overline{Q}}_t^{agg}]\}$$

(7)

In this section, a large number of WTs, PVs, and DCs are aggregated. Compared to the complex distribution network, the interconnecting lines among these resources are shorter, and the power transmitted is smaller; therefore, the network topology among the resources is neglected. The aggregation of a large number of resources yields the ranges and regulatory constraints of the total active and reactive power, which serve as the action space and constraints for the controlled objects in the MADRL framework, and they vary dynamically.

2.2. Optimization Objectives and Constraints of DNs

This section formulates a cooperative optimization framework for DNs, aiming to avoid voltage violations and reduce the network losses of DNs. The optimization issue is formulated as follows:

$$\underset{P_{i,t}^{\mathit{region}},Q_{i,t}^{\mathit{region}},P_{i,t}^{\mathit{BESS}}}{\min }{\sum }_{t\in T}\left(k_1{P}_t^{loss}+k_2{V}_t^{vio}\right)$$

(8)

$$P_t^{loss}={\sum }_{i,j}{r}_{ij}{l}_{ij,t}$$

(9)

$$C_t^{pro}={\sum }_{i\in region}[{\left(v_{min}-v_{i,t}\right)}_{+}+{\left(v_{i,t}-v_{max}\right)}_{+}]$$

(10)

where $P_t^{loss}$ represents the network loss; $C_t^{pro}$ represents the profit of the transactive energy between DNs and the main grid. $k_1$, $k_2$ are the target coefficients of voltage penalty and profit, respectively.

$$\begin{array}{c}{P}_{i,t}^{agg}-P_{i,t}^{BESS}=e_{i,t}{\sum }_{i,j\in {\Omega }_k}\left(G_{ij,t}{e}_{j,t}-B_{ij,t}{f}_{j,t}\right)\\ +f_{i,t}{\sum }_{i,j\in {\Omega }_k}\left(G_{ij,t}{f}_{j,t}+B_{ij,t}{e}_{j,t}\right)\end{array}$$

(11)

$$\begin{array}{c}{Q}_{i,t}^{agg}-Q_{i,t}^{BESS}=f_{i,t}{\sum }_{i,j\in {\Omega }_k}\left(G_{ij,t}{e}_{j,t}-B_{ij,t}{f}_{j,t}\right)\\ -e_{i,t}{\sum }_{i,j\in {\Omega }_k}\left(G_{ij,t}{f}_{j,t}+B_{ij,t}{e}_{j,t}\right)\end{array}$$

(12)

$$P_{i,t,min}^{agg}\le P_{i,t}^{agg}\le P_{i,t,max}^{agg}\forall i\in {\Omega }_k,t\in T$$

(13)

$$Q_{i,t,min}^{agg}\le Q_{i,t}^{agg}\le Q_{i,t,max}^{agg}\forall i\in {\Omega }_k,t\in T$$

(14)

$$\{\begin{array}{c}{S}_{i,t+1}^{SoC}=S_{i,t}^{SoC}+{\eta }^{charge}{P}_{i,t}^{Bess+}∆t+{\displaystyle \frac{P_{i,t}^{Bess-}∆t}{{\eta }^{discharge}}}\hfill \\ 0\le P_{i,t}^{Bess+}\le P_{i,t,max}^{Bess+},P_{i,t,max}^{Bess-}\le P_{i,t}^{Bess-}\le 0\hfill \\ {SoC}_i^{min}\le S_{i,t}^{SoC}\le {SoC}_i^{max}\end{array}$$

(15)

where the real and complex components of branch $ij$ admittance at time $t$ are expressed by $G_{ij,t}$ and $B_{ij,t}$; $f_{j,t}$ is the complex component of the voltage phasor $v_{i,t}$ and $e_{j,t}$ is the real component of the voltage phasor $v_{i,t}$ on bus $i$ at time $t$. The constraint equations should primarily focus on the regulation range of the regional resource aggregation. $P_{i,t,min}^{agg}$, $P_{i,t,max}^{agg}$, $Q_{i,t,min}^{agg}$, $Q_{i,t,max}^{agg}$ represent the active and reactive power regulation range of the region. $S_{i,t}^{SoC}$ is the state of charge (SoC) of the ESSs installed on bus $i$ at time $t$; the output power of ESSs installed on bus $i$ at time $t$ is represented by $P_{i,t}^{Bess}$; $P_{i,t}^{Bess+}$ indicates that the ESS is charging; otherwise, $P_{i,t}^{Bess-}$ indicates that it is discharging; $P_{i,t}^{Bess}$ is limited by maximum charging power $P_{i,t,max}^{Bess+}$ and maximum discharging power $P_{i,t,max}^{Bess-}$; ${\eta }^{charge}$ and ${\eta }^{discharge}$ are the charging/discharging rates; the upper and lower limits of ESSs are represented by ${SoC}_i^{max}$, ${SoC}_i^{min}$.

The issue is formulated as a nonlinear programming model:

min  (8)
s.t.  (11)–(15)

(16)

Equation (16) incorporates a multi-objective optimization function aimed at reducing network losses and avoiding voltage violations. It is solved under the constraints related to regional resource aggregation, ESSs and network operation. The decision variables of Equation (16) are denoted by $a_{k,t}=\left(a_{k,0},a_{k,1}...a_{k,T-1}\right)$. Therefore, this optimization problem can be transformed into a POMDP model.

2.3. POMDP Modeling for Optimization Strategy

We propose an optimization strategy within the MADRL framework. We model Equation (14) as a POMDP and further determine the action variables and state variables based on the variables and constraints involved. It can be described by a tuple $PO=(K,S,\{O_k\},\{A_k\},R,\mathcal{T},\gamma )$. $K$ is the number of agents, $o_k\in O_{1:K}$ represents the observed state value of agent $k$, $a_k\in A_{1:K}$ is the set of actions, $s\in S$ is the set of global states, $(s,o_{1:k},a_{1:k})$ is the state transition function, denoted as $\mathcal{T}$, and $\gamma$ is the discount factor. Then, the reward $r_k=r_k(s_t,a_t)$ is calculated by $S,$ $A_k$, which can be shared. Meanwhile, let $\omega$ represent the environmental uncertainty, and the next new state $s_{t+1}$ is obtained based on the $\mathcal{T}$ [26]. ${\pi }_k(·|s)$ represents the policy learned by each agent, which transforms the observed state of the environment by the agent into probabilities for each possible action. $\{K,S,A_1,A_2\dots A_K,\mathcal{T},R_1,R_2\dots R_K,\gamma \}$ is the trajectory of POMDP. Maximizing the cumulative discounted reward $R_k=E_{s~\mathcal{T},a_k~{\pi }_k}\left[{\sum }_{t\ge 0}{\gamma }^t{r}_{k,t}\right]$ is the objective for each agent. The cooperative optimization problem for DNs is transformed into a POMDP, with its components defined in detail as follows.

The cooperative optimization strategy of DNs based on MADRL considering a regional feasible region can be divided into two stages: centralized training and distributed execution [27]. The multiple agents are responsible for controlling their respective regions. During the training stage, control strategies are learned through interactions between the agents and the DN environment, along with data sharing. During the execution stage, agents only need to obtain information within their respective regions to make control decisions.

(1)

Observation space: the observation of agent $k$ is presented by:

$${\mathit{o}}_{\mathit{k},\mathit{t}}=\left({\mathit{P}}_{\mathit{t}}^{\mathit{l}\mathit{o}\mathit{s}\mathit{s}},{\mathit{P}}_{\mathit{i},\mathit{t}}^{\mathit{a}\mathit{g}\mathit{g}},{\mathit{Q}}_{\mathit{i},\mathit{t}}^{\mathit{a}\mathit{g}\mathit{g}},{\mathit{S}}_{\mathit{i},\mathit{t}}^{\mathit{S}\mathit{o}\mathit{C}},{\mathit{V}}_{\mathit{i},\mathit{t}}\right)\mathit{i}\in {\mathit{\Omega }}_{\mathit{k}}$$

(17)

(2)

Action space: the action of the regional agent $k$ at time $t$ is represented by $a_{k,t}\in A_{k,t}$, which controls the output power of regional resource aggregation and ESS.

$${\mathit{a}}_{\mathit{k},\mathit{t}}=\left({\mathit{P}}_{\mathit{i},\mathit{t}}^{\mathit{a}\mathit{g}\mathit{g}},{\mathit{Q}}_{\mathit{i},\mathit{t}}^{\mathit{a}\mathit{g}\mathit{g}},{\mathit{P}}_{\mathit{i},\mathit{t}}^{\mathit{E}\mathit{S}\mathit{S}}\right)\forall \mathit{i}\in {\mathit{\Omega }}_{\mathit{k}}$$

(18)

(3)

Constraints: the action space and observation space are constrained by the limitations imposed by regional resource aggregation and ESS.

$$\{\begin{array}{c}{\mathit{P}}_{\mathit{i},\mathit{t}}^{\mathit{a}\mathit{g}\mathit{g}}\in [{\mathit{P}}_{\mathit{i},\mathit{t},\mathit{m}\mathit{i}\mathit{n}}^{\mathit{a}\mathit{g}\mathit{g}},{\mathit{P}}_{\mathit{i},\mathit{t},\mathit{m}\mathit{a}\mathit{x}}^{\mathit{a}\mathit{g}\mathit{g}}]\hfill \\ {\mathit{Q}}_{\mathit{i},\mathit{t}}^{\mathit{a}\mathit{g}\mathit{g}}\in [{\mathit{Q}}_{\mathit{i},\mathit{t},\mathit{m}\mathit{i}\mathit{n}}^{\mathit{a}\mathit{g}\mathit{g}},{\mathit{Q}}_{\mathit{i},\mathit{t},\mathit{m}\mathit{a}\mathit{x}}^{\mathit{a}\mathit{g}\mathit{g}}]\hfill \end{array}$$

(19)

where Equation (19) represents the boundaries of the active and reactive power outputs of the regional resource aggregation.

$$P_{i,t}^{ESS}\in \left[-P_{i,t}^{ESS,max},P_{i,t}^{ESS,max}\right],\forall i\in {\Omega }_k$$

(20)

$$P_{i,t}^{BESS}=\{\begin{array}{c}\min \left(P_{i,t}^{ESS},\left(S_{i,t}^{SoC,max}-S_{i,t-1}^{SoC}\right)\right),P_{i,t}^{ESS}\ge 0\\ \max \left(P_{i,t}^{ESS},\left(S_{i,t}^{SoC,min}-S_{i,t-1}^{SoC}\right)\right),P_{i,t}^{ESS}<0\end{array}$$

(21)

where Equation (20) represents the constraints of ESSs. $P_{i,t}^{ESS}$ is the value of the action directly output by the agent, constrained by $-P_{i,t}^{ESS,max}$ and $P_{i,t}^{ESS,max}$.

(4)

Reward function: the optimization issue in this paper constructs a POMDP for multiple agents, each of which can share a reward. The reward for each agent is expressed as:

$$r_t={-k}_1{P}_t^{loss}{-k}_2{C}_t^v$$

(22)

$$C_t^v={\sum }_{i\in }[{\left(v_{min}-v_{i,t}\right)}_{+}+{\left(v_{i,t}-v_{max}\right)}_{+}]$$

(23)

where the first two terms in Equation (22) constitute the objective function, which fully takes into account regional cooperative optimization to reduce network losses; $k_2$ is the penalty coefficient for voltage violation, ${\left(x\right)}_{+}=\mathrm{m}\mathrm{a}\mathrm{x}(0,x)$, the third term, $C_t^v$, represents the penalty for voltage violations, ensuring compliance with voltage constraints.

3. Proposed Method

(1)

Soft Actor–Critic Method Augmented by ES and PER

The SAC algorithm is particularly well-suited for the DN optimization problem considered in this study, as its two core mechanisms directly address the primary challenges of the problem. On the one hand, due to the high uncertainty in renewable energy generation, e.g., PV output and load demand, conventional gradient-based methods are prone to becoming trapped in suboptimal solutions. The maximum entropy objective of SAC encourages each agent to explore diverse actions during training, enabling more thorough evaluation of control strategies across varying operational scenarios and reducing the risk of converging to local optima [28]. On the other hand, practical distribution network operation inevitably involves communication delays and measurement noise, which can cause deterministic policies to produce volatile and unreliable control actions. The stochastic policy adopted by SAC outputs a probability distribution over actions rather than deterministic instructions [29]. By sampling from this distribution, the agent maintains stable control performance even under small disturbances, thereby ensuring robustness against real-world uncertainties.

While searching for the optimal strategy, another objective of the SAC algorithm is to incorporate an entropy term into the equation and maximize the entropy for each state [28]:

$${\pi }^{*}=\arg {\mathit{max}}_{\pi }{\sum }_{t=0}^T{\mathrm{E}}_{\left(s_t,a_t\right)~{\rho }_{\pi }}[r\left(s_t,a_t\right)+\alpha \mathrm{H}(\pi (·|s_t\left)\right)]$$

(24)

where $\mathcal{H}\left(\pi (·|s_t)\right)$ represents the entropy function; $\alpha$ is the temperature parameter, which controls the stochasticity of the optimal policy; the state-action or state marginal distribution of the trajectory induced by the policy $\pi (a_t|s_t)$ is denoted by ${\rho }_{\pi }$.

Finally, the entropy-regularized Bellman equation is formulated as:

$${\mathcal{T}}^{\pi }{Q}_{\theta }\left(s_t,a_t\right)\triangleq \mathrm{r}(s_{t+1},a_{t+1})+\gamma {\mathbb{E}}_{s_{t+1}~{\rho }_{\pi }}\left[V\left(s_{t+1}\right)\right]$$

(25)

$$V\left(s_{t+1}\right)={\mathbb{E}}_{a_{t+1}~\pi }[Q_{\theta }\left(s_{t+1},a_{t+1}\right)-\alpha \mathrm{l}\mathrm{o}\mathrm{g}{\pi }_{\phi }(a_{t+1}|s_{t+1})]$$

(26)

where the Q-function and the soft state value function are represented by $Q_{\theta }\left(s_t,a_t\right)$ and $V_{\theta }\left(s_{t+1}\right)$, respectively; the value of $\gamma$ is set within the range of (0, 1), and it serves to penalize future rewards.

The update policy is shown as follows:

$${\pi }_{new}=arg {min}_{{\pi }^{\prime }\in }{D}_{KL}({\pi }^{\prime }||{\displaystyle \frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(Q_{\theta }\right(s_t,·)/\alpha )}{Z^{\pi }\left(s_t\right)}})$$

(27)

where $Z^{\pi }\left(s_t\right)$ is the partition function, whose role is to normalize the numerator into a probability distribution; the degree of matching between the two probability distributions is measured using the divergence $D_{KL}$.

The parameters of the actor network and the critic network are denoted as φ and θ, respectively, which are used to adjust the policy ${\pi }_{\phi }(a_t|s_t)$ and the soft Q-function $Q_{\theta }\left(s_t,a_t\right)$. Minimizing the sum of squared residual errors and the soft Bellman residual is the objective for training the soft Q-function, as detailed below:

$$J_Q(\theta )={\mathbb{E}}_{(s_t,a_t)~\mathcal{B}} \left[{\displaystyle \frac{1}{2}}{(Q_{\theta }(s_t,a_t)-{\widehat{Q}}_{\theta }\left(s_t,a_t\right))}^2\right]$$

(28)

$${\widehat{Q}}_{\theta }\left(s_t,a_t\right)=r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}~{\rho }_{\pi }}{\mathrm{V}}_{\overline{\theta }}(s_{t+1})$$

(29)

where $\mathcal{B}$ represents a replay buffer, which is used to store and reuse the experience data from the interactions between the agent and the environment. The update makes use of a target soft Q-function with parameters $\overline{\theta }$.

The gradient of Equation (28) can be expressed as follow.

$$\begin{gathered} {\widehat{\nabla }}_{\theta }{J}_Q\left(\theta \right)={\nabla }_{\theta }{Q}_{\theta }(a_t,s_t)\left(Q_{\theta }\right(s_t,a_t)- \\ \left(r\right(s_t,a_t)+\gamma Q_{\overline{\theta }}(s_{t+1},a_{t+1})-\alpha log({\pi }_{\phi }(a_{t+1}|s_{t+1}\left)\right)\left)\right) \end{gathered}$$

(30)

Finally, the policy parameters φ can be learned by directly minimizing the expected $D_{KL}$.

$$\begin{gathered} J_{\pi }\left(\phi \right)={\mathbb{E}}_{s_t~\mathcal{B},a_t~\pi }\left[\alpha \mathrm{l}\mathrm{o}\mathrm{g}\right({\pi }_{\phi }\left(f_{\phi }\right({\mathsf{ϵ}}_t;s_t)|s_t\left)\right)- \\ {Q}_{\theta }(s_t,f_{\phi }({\mathsf{ϵ}}_t;s_t\left)\right)] \end{gathered}$$

(31)

$$\begin{gathered} {\widehat{\nabla }}_{\phi }{J}_{\pi }\left(\phi \right)={\nabla }_{\phi }\alpha \mathrm{l}\mathrm{o}\mathrm{g}\left({\pi }_{\phi }\right(a_t|s_t))+({\nabla }_{a_t}\alpha \mathrm{l}\mathrm{o}\mathrm{g}\left({\pi }_{\phi }\right(a_t|s_t))- \\ {\nabla }_{a_t}{Q}_{\theta }\left(s_t,{\mathrm{a}}_t\right)\left){\nabla }_{\phi }{f}_{\phi }\right({\mathsf{ϵ}}_t;s_t) \end{gathered}$$

(32)

where $f_{\phi }({\mathsf{ϵ}}_t;s_t)$ is $a_t$ which is obtained by a neural network; ${\mathsf{ϵ}}_t$ is an input noise vector, ${\mathsf{ϵ}}_t~N(0,1)$.

Automatic entropy adjustment dynamically balances exploration and exploitation in the policy by optimizing the α, calculating the gradient of α by minimizing Equation (33):

$$J(\alpha )={\mathbb{E}}_{a_t~{\pi }_{\phi }}[-\alpha log{\pi }_{\phi }(a_t|s_t)-\alpha \overline{\mathcal{H}}]$$

(33)

$${\widehat{\nabla }}_{\alpha }J(\alpha )=-log{\pi }_{\phi }(a_t|s_t)-\overline{\mathcal{H}}$$

(34)

where $\mathcal{H}$ is called the minimum entropy constant.

(2)

ES technique

By combining ES with RL, this paper leverages the strengths of both approaches. ES enhances exploration diversity and sampling efficiency, while the integration of gradient information from RL mitigates ES’s limitation in handling high-dimensional problems. Additionally, ES introduces beneficial redundancy and robustness through elitism. This hybrid strategy not only improves sample diversity and learning stability but also accelerates convergence. Notably, it proves particularly effective in problems with long episodic horizons.

Specifically, we adopt a multivariate Gaussian distribution $\mathcal{N}\left(m,C\right)$ as the sampling distribution for the actor individuals, where $m$ is the mean vector and $C$ is the covariance matrix. The initialization step before training proceeds as follows: $m$ is set to the zero vector 0, and $C$ is set to the identity matrix $I$. This process can be expressed as:

$$\mathcal{N}(\mathit{m},\mathit{C})⟸\mathcal{N}(\mathbf{0},\mathit{I})$$

(35)

The ES draws actor individuals from ${\mathcal{N}}^g(m^g,C^g)$ during generation $g$, with $N_{in}$ representing the population size of the actor. The above sampling operation can be expressed as:

$${\pi }_{\lambda }^{\varphi }~{\mathcal{N}}^g\left(m^g,C^g\right),\lambda \in [1:N_{in}]$$

(36)

During training, all ES actor individuals take actions in the environment, and their fitness values $F$ are evaluated. The population is then updated in two ways: gradient-updated actor individuals are incorporated, and ES actor individuals with the lowest $F$ are removed. Consequently, gradient-based actors ${\pi }^{\varphi }$ and ES actors ${\pi }_{\lambda }^{\varphi }$ jointly populate the actor pool.

Next, from the population, the ES identifies K elite actor individuals that possess the highest fitness values $F$, which are represented as ${\left[{\pi }_{\lambda }^{\varphi }\right]}_{\lambda =1:K}(K<N_{\mathrm{in}})$. Subsequently, the distribution ${\mathcal{N}}^g(m^g,C^g)$ is updated using the fitness weights and these elite individuals according to the following procedure:

$$m^{g+1}={\displaystyle \frac{1}{\mu K}}{\displaystyle {\sum }_{\lambda =1}^K}{F}_{\lambda }^g{\pi }_{\lambda }^{\varphi }$$

(37)

$$C^{g+1}={\displaystyle \frac{1}{nK}}{\displaystyle {\sum }_{\lambda =1}^K}{F}_{\lambda }^g{\left({\pi }_{\lambda }^{\varphi }-m^{g+1}\right)}^2$$

(38)

At the final stage of each generation, interactions with the environment are performed by the RL agent. Through these interactions, a loss-minimization procedure updates the critic network $Q^{\theta }$, and a sampled policy gradient updates the actor network ${\pi }^{\varphi }$. The resulting $Q^{\theta }$, ${\pi }^{\varphi }$, and the updated $\mathcal{N}(m^g,C^g)$ are then used as the starting point for the next generation. This entire process is repeated until convergence is reached.

(3)

PER technique

PER technique assigns larger weights to more significant samples, making them more likely to be selected multiple times during training [30]. By integrating ES and PER into the SAC algorithm, the model’s training speed, decision-making capability, and scalability are enhanced.

The centralized training framework of ES-PER-SAC is depicted in Figure 2.

Figure 2. Centralized training framework of the ES-PER-SAC.

4. Case Studies

In this section, the proposed model is validated on the IEEE 33-bus and 123-bus systems. Subsection A first introduces the number of regional aggregated resources, their access locations, comparative models, and parameter settings. The training process and decision-making results are presented in Subsections B and C, respectively. Subsection D presents the network loss and voltage violation rate, while Subsection E provides an analysis of the training time and decision-making time of different algorithms used in the model.

4.1. Corresponding Setting

The performance of the proposed model was tested on both the IEEE 33-bus and IEEE 123-bus systems. All algorithms were implemented using MATLAB 2022b running on a 64-bit laptop equipped with a 2.20 GHz CPU, 16 GB of RAM, and an Nvidia RTX4050 GPU.

Table 1 lists the total number of aggregated resources and the resource parameters in each region. The primary resources considered in this paper include WTs, PVs, ESSs, and DCs. The detailed curves of the aggregated resources are shown in Figure A1 in Appendix A. Figure 3 illustrates the results of partitioning the DN based on its functional areas, along with the access buses of energy storage systems and aggregated regional resources into the DN.

Table 1. The specific resources aggregated by each region.

The Name of Region	Aggregated Regional Resources	The Name of Resource	Parameters	Capacity/Operation Limits
Region 1	27 WTs, 13 PVs, 37 DCs	WT	Maximum 50 kW	-
Region 2	19 WTs, 27 PVs, 21 DCs	PV	Maximum 25 kW	-
Region 3	17 WTs, 15 PVs, 5 DCs	DC	Maximum 45 kW	-
Region 4	16 WTs, 27 PVs, 16 DCs	ESS	500 kW	0.1–0.8/100 kW

Figure 3. The partitioning of DNs, regional resource aggregation and ESS access buses.

To demonstrate the effectiveness of the proposed model from multiple aspects, we set up four sets of comparative cases: (a) multiple types of resources are not aggregated but are directly connected to the nodes of the DN; (b) resources are aggregated and connected to nodes, but the MADRL adopts the SAC algorithm; (c) resources are aggregated and connected to nodes, but the MADRL adopts the Deep Deterministic Policy Gradient (DDPG) algorithm; (d) resources are aggregated and connected to nodes, but the MADRL adopts the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. Table 2 presents the hyperparameter settings for the different comparative models.

Table 2. Hyperparameter settings.

Alogrithm	Parameters	Values of IEEE 33/123-Bus
ES-PER-SAC	Discount factor	{0.98, 0.99}
	Batch size	{128, 256}
	Experience buffer length	{2 × 105, 1 × 106}
	Learn rate	{0.01, 0.01}
SAC	Discount factor	{0.96, 0.99}
	Batch size	{128, 256}
	Experience buffer length	{2 × 105, 2 × 105}
	Learn rate	{0.01, 0.01}
MADDPG	Discount factor	{0.96, 0.98}
	Batch size	{256, 256}
	Experience buffer length	{2 × 105, 1 × 106}
	Learn rate	{0.01, 0.01}
MATD3	Discount factor	{0.98, 0.99}
	Batch size	{256, 256}
	Experience buffer length	{2 × 105, 2 × 106}
	Learn rate	{0.01, 0.01}

4.2. Training Process and Performance

To demonstrate the convergence, effectiveness, and optimality of the proposed model, Figure 4 and Figure 5 illustrate the training processes of multiple algorithms, namely ES-PER-SAC, ES-PER-SAC (without aggregation), SAC, MATD3, MADDPG and ES-PER-SAC (SADRL) on the IEEE 33-bus and IEEE 123-bus systems, respectively. The following conclusions can be drawn:

Figure 4. Algorithmic training comparisons in IEEE 33-bus.

Figure 5. Algorithmic training comparisons in IEEE 123-bus.

• It can be observed from ES-PER-SAC (SADRL) that the single-agent-based method is still capable of achieving convergence in the relatively small-scale IEEE 33-bus system. However, when applied to the larger IEEE 123-bus system, this single-agent approach fails to converge effectively. In contrast, the MADRL framework adopted in this paper partitions the entire network into multiple subregions, with each agent responsible for a specific region. This decomposition strategy significantly reduces the dimensionality of the action and observation spaces for each individual agent, thereby facilitating more efficient exploration and faster convergence. Moreover, although each agent operates independently, the CTDE paradigm enables inter-agent cooperation during the training phase, allowing the agents to learn coordinated strategies that approximate the global optimum.
• In the absence of resource aggregation, each agent directly controls dozens of resources in the IEEE 33-bus system, and the algorithm fails to achieve effective convergence. In contrast, all other algorithms based on resource aggregation eventually converge. Therefore, in the context of rapidly increasing distributed energy resources, resource aggregation is beneficial for enhancing model scalability. Moreover, since ES-PER-SAC (without aggregation) cannot be effectively controlled in the IEEE 33-bus system, it is not applied to the IEEE 123-bus system.
• Comparing the volatility of the training processes among the algorithms, all four algorithms exhibit a gradually increasing trend. ES-PER-SAC and SAC show a relatively fast increase with stable fluctuations, while MATD3 and MADDPG also display a gradual upward trend but with greater volatility, which stabilizes after 1000 episodes.
• Comparing the final converged optimal values, PER-PS-SAC ultimately converges to approximately −115/−95; SAC converges to around −150/−120; MATD3 and MADDPG converge to about −130/−120 and −140/−150, respectively. These results indicate that the introduction of ES and PER techniques accelerates training convergence and enhances decision-making optimality.

4.3. Results of Decision-Making

The voltage of the DNs is constrained by the reward function. When the voltage exceeds these limits, the reward will be reduced. The voltage at each node at every time step should be maintained within the range of [0.95, 1.05] p.u. Figure 6 presents three-dimensional voltage plots for two distribution networks of different scales. The voltages at all nodes of the DNs remain within the reasonable range over the 24 h period and do not exceed the limits. The maximum voltage values in the two networks occur at bus 11 and bus 81, with values of 1.012/1.046, respectively, while the minimum voltage values are 0.951/0.957. The proposed model effectively achieves voltage regulation.

Figure 6. Hourly voltage data for each bus in IEEE 33/123-bus systems.

Figure 7 illustrates the SoC and charging/discharging power of the ESS in the IEEE 33/123-bus systems. It can be observed that both the charging/discharging actions and the SoC satisfy the constraints. Influenced by the reward function, the ESS operates to further achieve voltage regulation and reduce network losses.

Figure 7. Charging/discharging power and SoC of the ESS.

Figure 8 respectively illustrate the upper and lower limits of the aggregated active power and reactive power of the resources at bus 11 in the IEEE 33-bus system, along with the decision-making results of the model. A negative value after resource aggregation indicates the amount of electricity that can be fed into the DN, while a positive value indicates the amount of electricity absorbed from the network. Since the aggregated resources only include distributed photovoltaics, wind power, and loads without any temporal coupling, both active and reactive power are solely constrained by their respective upper and lower limits. It can be observed that during periods of high voltage, the decision-making results for the aggregated resources decrease significantly, whereas at other times, they are influenced by the combined effects of voltage and network losses.

Figure 8. Decision-making results for active and reactive power outputs of Bus 11.

To further analyze the decision-making values of the aggregated resources, Figure 9 presents the resource aggregation results at bus 58 in the IEEE 123-bus system, along with the resource control results based on MADRL. Similar to Figure 8, the decision-making results for both active and reactive power lie within the upper and lower limits of active/reactive power, demonstrating the accuracy of the model.

Figure 9. Decision-making results for active and reactive power outputs of Bus 58.

4.4. The Network Losses and Voltage Violation Rate of DNs

Table 3 presents the network losses and voltage violation rates obtained from the decision-making results of the proposed model in two different-scale networks over one-day and seven-day horizons. It can be observed that our model is capable of minimizing network losses while effectively avoiding voltage violations. In both the IEEE 33-bus and IEEE 123-bus test systems, the proposed method achieves the lowest network losses and effectively prevents voltage violations, with the network losses over the one-day horizon reduced by approximately 11.7% and 21.5%, respectively, compared to other algorithms. Overall, the decision-making results of our model outperform those of the other comparative algorithms.

Table 3. Comparison of network loss and voltage violation rate in IEEE 33/123-bus using different algorithms.

Test System	Algorithm	One-Day Results		7-Day Results
		Loss (kW)	Voltage Violation Rate (%)	Loss (kW)	Voltage Violation Rate (%)
IEEE 33-bus	MADDPG	2231.77	0	15,147.31	7.2%
	MATD3	2018.19	0	14,372.96	4.7%
	SAC	2093.25	0	14,583.22	2.3%
	ES-PER-SAC	1998.16	0	14,018.79	0
IEEE 123-bus	MADDPG	1169.47	0	8237.93	5.0%
	MATD3	1126.39	0	7916.17	3.9%
	SAC	989.05	0	6832.95	0.8%
	ES-PER-SAC	962.13	0	6719.86	0

4.5. Training and Decision-Making Speed

In addition to the effectiveness and reliability of the model, training speed and computational speed are equally crucial, as they determine the model’s practicality and the time scale of problems it can address. Table 4 presents the training and decision-making times of different algorithms on two distinct systems: the IEEE 33-bus and IEEE 123-bus systems. It can be observed that as the network scale increases, the model generally requires longer training and decision-making times. Compared with various other algorithms, our model integrates ES and PER techniques into the SAC algorithm, thereby improving both training speed and computational speed. Typically, the MADRL decision-making time is within one second, demonstrating the feasibility and effectiveness of the model for real-time optimization and control of DNs across various time scales.

Table 4. Training time and decision-making time of different algorithms on IEEE 33/123-bus systems.

Test System	Algorithm	Training Time (s)	Decision-Making Time (s)
IEEE 33-bus	MADDPG	12,103	0.623
	MATD3	12,377	0.626
	SAC	14,106	0.616
	ES-PER-SAC	10,982	0.591
IEEE 123-bus	MADDPG	21,013	0.827
	MATD3	20,699	0.824
	SAC	22,796	0.803
	ES-PER-SAC	22,138	0.760

Notably, unlike conventional centralized optimization methods whose computation time typically grows exponentially or quadratically with the number of nodes, the online decision time of the proposed method increases only marginally with network scale, as it depends primarily on the inference time of individual agents operating in parallel. This further confirms the scalability of the proposed approach for practical large-scale distribution systems.

4.6. Impact of Different Reward Functions on the Results

As shown in Table 5, when the reward function is designed to consider only voltage violations, the resulting network losses over the seven-day period are as high as 15,963.24 kW and 9374.16 kW, respectively. Conversely, when the objective function is configured to minimize network losses alone, the voltage violation rates reach 12.5% and 6.5%, respectively, over the same period. By contrast, the proposed method achieves a voltage violation rate of 0%, while reducing the network losses to 14,018.79 kW and 6719.86 kW. These results clearly indicate that the reward function design exerts a substantial influence on the overall performance.

Table 5. Results of different reward functions on IEEE 33/123-bus systems.

Test System	Reward Function	Loss (kW)	Voltage Violation Rate (%)
IEEE 33-bus	$r_t={-k}_1{P}_t^{loss}$	14,017.83	12.5%
	$r_t={-k}_2{C}_t^v$	15,963.24	0
	$r_t={-k}_1{P}_t^{loss}{-k}_2{C}_t^v$	14,018.79	0
IEEE 123-bus	$r_t={-k}_1{P}_t^{loss}$	6719.84	6.5%
	$r_t={-k}_2{C}_t^v$	9374.16	0
	$r_t={-k}_1{P}_t^{loss}{-k}_2{C}_t^v$	6719.86	0

Based on the above analysis, we recommend that system operators in regions with high renewable penetration, e.g., Spain, prioritize the deployment of distributed learning-based voltage controllers over centralized model-based ones. Moreover, the aggregation strategy adopted in our framework can be integrated into existing SCADA/EMS systems with only minor hardware upgrades, as it requires only local measurements and sparse inter-agent communication. These measures can significantly reduce the probability of blackouts triggered by voltage issues.

5. Conclusions

This paper proposes a real-time active and reactive power cooperative optimization method for DNs based on large-scale resource aggregation to address the challenge of real-time effective control under large-scale distributed energy resource integration. First, DCs, as novel devices, are considered as flexible loads and incorporated into dispatchable resources to further exploit the regulation potential of DNs. Second, the total power aggregated from a large number of regional distributed resources is used as the input to the MADRL model, which protects the data privacy of individual devices and reduces voltage violation rates and network losses without requiring communication. Finally, ES and PER are integrated into the SAC algorithm. ES enhances global exploration capability through population evolution to avoid falling into local optima, while PER improves training efficiency and stability by prioritizing high-value experience samples, thereby accelerating convergence and enhancing decision-making quality. The results demonstrate that, compared with other methods, the proposed method achieves a voltage violation rate of 0% and achieves an average network loss reduction of 16.6% on both the IEEE 33-bus and IEEE 123-bus systems, confirming its effectiveness and superiority. Future work can further explore the collaborative scheduling potential arising from the spatiotemporal migration of computing tasks in data centers.