Dynamic Collaborative Optimization Strategy for Multiple Area Clusters in Distribution Networks Considering Topology Change

Abstract

To tackle the challenges arising from missing real-time measurement data and dynamic changes in network topology in optimizing and controlling distribution networks, this study proposes a data-driven collaborative optimization strategy tailored for multi-area clusters. Firstly, the distribution network is clustered based on electrical distance modularity and power balance indicators. Next, a collaborative optimization model for multiple area clusters is constructed with the objectives of minimizing node voltage deviations and active power losses. Then, a locally observable Markov decision model within the clusters is developed to characterize the relationship between the temporal operating states of the distribution network and the decision-making instructions. Using the Actor–Critic framework, the cluster agents are trained while considering the changes in cluster boundaries due to topology variations. A Critic network based on an attention encoder is designed to map the dynamically changing cluster observations to a fixed-dimensional space, enabling agents to learn control strategies under topology changes. Finally, case studies show the effectiveness and superiority of the proposed method.

1. Introduction

With the key progress of green energy low-carbon transformation, a large amount of distributed generation (DG) needs wide access to the distribution network. The future Active Distribution Network (ADN) is anticipated to transition from a traditional unidirectional radial configuration to a bidirectional active network. The control mode will be changed from centralized dispatch to hierarchical and clustered control, constituting a hybrid AC–DC distribution network with medium and low voltage interaction and multi-station interconnection. However, due to the large random volatility and strong uncertainty of DG output, which is typically represented by photovoltaic (PV) energy, leading to prominent problems such as voltage overruns and power imbalance in the distribution network, the challenge of new energy consumption is highlighted [1]. Therefore, fully utilizing adjustable resources to implement autonomous operation at the distribution level and collaborative optimization control across multiple distribution areas has become one of the effective means of addressing the aforementioned issues.

Currently, optimal regulation methods for ADN can be classified into two main categories: model-driven [2,3] and data-driven approaches [4]. The model-driven category typically employs mathematical planning methods and heuristic algorithms to formulate the optimal regulation problem of ADN as a nonlinear optimization problem that can be explicitly solved. However, the limited coverage of communication facilities in medium- and low-voltage distribution networks [5], along with frequent changes in line-switching states, results in issues such as incomplete real-time measurement data [6] and fluctuations in network topology. Consequently, the traditional model-driven approach, which relies on comprehensive measurement data, is becoming less applicable.

In recent years, as power grids have undergone intelligent upgrades, data-driven methods, particularly Deep Reinforcement Learning (DRL), have emerged as innovative solutions for the real-time optimization and regulation of distribution networks [6,7,8]. These approaches are leveraged to enhance decision-making processes in dynamic environments by large datasets and adaptive learning techniques.

DRL is essentially a framework of interaction-based learning that models control resources as intelligent agents within the distribution network environment. This integration allows for continuous interaction, enabling the system to learn from the vast amounts of operational data generated during these interactions. Neural networks are utilized to efficiently develop regulation strategies based on the acquired data. As the size of distribution network nodes and the penetration of photovoltaic (PV) systems increase, the expansion of control equipment results in a growing number of intelligent entities. However, this increase can lead to suboptimal synergistic effects among these entities [9]. Additionally, node voltage violations tend to occur in only a limited number of regions, indicating that it is unnecessary to mobilize all resources in order to address localized voltage fluctuations. To tackle these challenges, existing research has explored collaborative optimization of distribution network zones through clustering techniques in [10,11]. This approach aims to enhance the consumption of DG resources in close proximity. For instance, [12] proposes a dynamic clustering method based on a multi-scenario hierarchical clustering approach, which segments the distribution network into clusters. In this framework, intelligent agents are deployed within each region to undergo training, facilitating zonal in situ optimization and improving overall system performance.

However, the aforementioned DRL methods primarily focus on fixed network topologies, which do not account for the dynamic changes that occur in actual distribution networks due to load transfers and the opening and closing of switches. These changes inevitably affect cluster divisions and the dynamic assignment of regulation units to clusters, which involves determining which regulation units are in the same cluster and share regional information. In traditional DRL, the input network dimensions for each agent are fixed, i.e., as the power flow data of all agents within the same cluster at the distribution network nodes. However, when the cluster changes, it leads to changes in the input dimensions of the agent network, rendering traditional methods unsuitable. To address this issue, researchers have concentrated on two main areas: strategy transfer learning and topological feature learning.

In the realm of policy transfer learning, [13] utilized the migration capabilities of neural networks to transfer parameters of the policy network based on a pre-trained intelligent agent model under a specific topological structure. However, this policy transfer method is limited in that it acts merely as a refinement measure after a topology change event has occurred. It also assumes that the state space of the intelligent agent remains unchanged, which introduces certain constraints.

On the other hand, topological feature learning approaches, as discussed in [14,15,16,17], aim to characterize the regional topology of the distribution network using adjacency matrices or one-dimensional vectors. In these methods, neural networks learn the numerical variations of these matrices or vectors to identify anomalies in network topology. Nevertheless, constructing these matrices or vectors still requires a global understanding of the distribution network’s topology or of the entire cluster, which places high demands on the quality of the real-time communication infrastructure.

In summary, this paper presents a data-driven cooperative optimal regulation method designed for multiple area clusters, aimed at enhancing the adaptability of algorithms to changes in distribution network topology. The approach begins by segmenting the distribution network into clusters based on electrical distance modularity and power balance degree. This segmentation allows for the decomposition of the overall distribution network regulation problem into a series of intra-cluster autonomy issues.

Next, the optimization problem for the distribution network is modeled using sequential decision-making principles grounded in the Markov Decision Process (MDP). To address the dynamic nature of cluster boundaries resulting from topology changes, the paper introduces an attention-based Critic network. This network is designed to accommodate dynamic observations, enabling the inputs to the Critic network to be scalable in response to changing conditions.

Finally, the effectiveness and superiority of the proposed algorithms are rigorously evaluated and compared using the IEEE 33-bus test system.

2. Dynamic Clustering of Distribution Networks

The cluster segmentation metric serves as a foundational element for the aggregation of distribution areas, ensuring that clusters are formed in a way that promotes high electrical coupling within the clusters while maintaining approximate decoupling between them. In this paper, two primary indices are utilized for cluster division: electrical distance modularity and power balance.

Electrical Distance Module Degree Metric: The Module Degree is used to measure the strength of the community network structure; the larger the value of the Module Degree, the better the results of the cluster division [18]. The Module Degree function is defined as ρ:

$$\rho ={\displaystyle \frac{1}{2s}}{\displaystyle \sum _i{\displaystyle \sum _j\left[W_{ij}-\frac{K_i{K}_j}{2s}\right]}\lambda \left(C_i,C_j\right)}$$

(1)

where $W_{ij}$ is the adjacency matrix in the modularity metric; $K_i$ is the sum of the weights of all edges connected to node i; s is the sum of the weights of all networks; $C_i$ denotes the cluster where station i is located, $\lambda$ is a 0–1 function, and $\lambda$ = 1 means that node i and node j are classified into a cluster. The formula for $W_{ij}$ is given in [19].

Power Balance Degree Metric: In order to achieve full DG consumption and reduce the power outgoing from the cluster, a power balance degree metric e describing the balance between power supply and demand within the cluster is defined:

$$e={\displaystyle \frac{e_p+e_q}{2}}$$

(2)

where $e_q$ is the total reactive power balance index of the system, and $e_p$ is the total active power balance index of the system. The active power balance index is used to describe the balance relationship between active power demand and supply within the cluster and can be expressed as

$$e_{\mathrm{p}}={\displaystyle \frac{1}{c}}{\displaystyle \sum _{s=1}^c\left[1-\frac{1}{T}{\displaystyle \sum _{i=1}^T\frac{P_{\mathrm{clu},i}(t)}{\max (P_{\mathrm{clu},i}(t))}}\right]}$$

(3)

where $P_{\mathrm{clu},i}$ is the net power characteristic of cluster s at time t; T is the duration of the typical time-varying scenario; and c is the number of clusters.

$$e_{\mathrm{Q}}={\displaystyle \frac{1}{c}}{\displaystyle \sum _{s=1}^c{Q}_i},\hspace{0.33em}{Q}_s=\left\{\begin{array}{lll}{\displaystyle \frac{Q_{\sup ,s}}{Q_{\mathrm{need},s}}}\hfill & \hspace{0.33em}{Q}_{\sup ,s}<Q_{\mathrm{need},s}\hfill & \hfill \\ 1\hfill & \hspace{0.33em}{Q}_{\sup ,s}⩾Q_{\mathrm{need},s}\hfill & \hfill \end{array}\right.$$

(4)

where $Q_{\sup ,s}$, $Q_{\mathrm{need},s}$ are the maximum reactive power output and reactive power demand of cluster s, respectively, and $Q_s$ is the reactive power balance degree of cluster s.

Combining the modularity metric and power balance metric, the cluster composite division metric is expressed as

$$\lambda =a_1\rho +a_{2}e$$

(5)

where a₁ and a₂ are the weighting coefficients of the two cluster indicators.

The distribution network is clustered into multiple area clusters by the Smart Local Moving (SLM) algorithm, which iteratively optimizes the cluster configuration to maximize modularity. Initially, each station area within the distribution network is treated as a distinct cluster. Subsequently, a sequential assignment process is employed, wherein each station area is evaluated for potential reassignment to a cluster containing other substation nodes. The modularity metric is calculated before and after each potential reassignment, and the node exhibiting the largest modularity increment is recorded. This process is repeated until the modularity converges, indicating a local optimum. The resulting clusters are then aggregated to form sub-networks, which are subsequently treated as nodes in a higher-level clustering iteration. This hierarchical clustering process continues until the modularity ceases to increase, thereby yielding an optimal division of the distribution network into multiple area clusters.

3. Co-Optimization Model for Multiple Area Clusters in Distribution Networks

3.1. Equivalent Power Modeling of Distribution Substation

The relationship between the LV distribution substation and the feeder connection is shown in Figure 1, where the station area internally interacts with the overall external active and reactive power at each moment with the feeder layer.

The equipment such as PV, energy storage system (ESS), and DC load is included in the station, and the power balance equation for station i is

$$P_{i,t}^{DG}+P_{i,t}^{disc}-P_{i,t}^{char}-P_{i,t}^{load}=P_{i,t}^{in}$$

(6)

$$Q_{i,t}^{\mathrm{DG}}-Q_{i,t}^{\mathrm{load}}=Q_{i,t}^{\mathrm{in}}$$

(7)

where $P_{i,t}^{DG}$ and $Q_{i,t}^{\mathrm{DG}}$ are the active and reactive power of DG flowing into station i, respectively; $P_{i,t}^{char}$ and $P_{i,t}^{disc}$ are the charging and discharging power of energy storage, respectively; $P_{i,t}^{load}$ and $Q_{i,t}^{\mathrm{load}}$ are the active and reactive power of load in station i, respectively; and $P_{i,t}^{in}$ and $Q_{i,t}^{\mathrm{in}}$ are the active and reactive power flowing into the station, respectively.

3.2. Objective Functions

To fully leverage the regulatory capabilities of adjustable resources within the distribution substation area, a cooperative optimization model designed for multiple distribution stations is presented in this section. The primary objective of this model is to minimize both voltage deviation and active network loss in the distribution network. The model incorporates several critical constraints, including power balance, the operational limits of regulation resources, and voltage security at various nodes. By optimizing the reactive power output of photovoltaic (PV) systems and the charging and discharging power of energy storage systems (ESS), the model aims to enhance both the economic efficiency and operational security of the distribution network. Building upon the clustering results obtained in the preceding section, the scalar objective function is formulated as follows:

$$\min F=\min {\displaystyle \sum _{C\in {\Omega }_{\mathrm{clu}}}{\displaystyle \sum _{t=1}^T{\displaystyle \sum _{i\in C_m}\left(\left|U_{i,t}-U_{\mathrm{ref}}\right|+P_{m,t}^{\mathrm{loss}}\right)}}}$$

(8)

$$P_{m,t}^{\mathrm{loss}}={\displaystyle \sum _{\left(ij\right)\in {\Omega }_{\mathrm{br},m}}{r}_{ij}\frac{P_{ij,t}^2+Q_{ij,t}^2}{U_{i.t}^2}}$$

(9)

where $C$ is the number of the cluster; ${\Omega }_{\mathrm{clu}}$ is the set of all clusters; $C_m$ denotes the set of nodes of cluster m; T denotes the total time period of the scheduling cycle; $U_{i,t}$ is the voltage amplitude of node i at time t, and $U_{\mathrm{ref}}$ is the voltage reference value; $P_{m,t}^{\mathrm{loss}}$ is the active network loss of cluster m at time t; ${\Omega }_{br,m}$ is the set of branch circuits within cluster m; $r_{ij}$ is the resistance of the branch circuit ij; and $P_{ij,t}$ and $Q_{ij,t}$ are the inflow of active and reactive power into the branch circuit ij, respectively.

3.3. Constraints

• Distribution network power balance constraints

$$\left\{\begin{array}{l}{P}_{i,t}=U_{i,t}{\displaystyle \sum _{\begin{array}{l}j=1\\ j\ne i\end{array}}^N{U}_{j,t}\left(G_{ij}\cos {\theta }_{ij}+B_{ij}\sin {\theta }_{ij}\right)}\\ Q_{i,t}=U_{i,t}{\displaystyle \sum _{\begin{array}{l}j=1\\ j\ne i\end{array}}^N{U}_{j,t}\left(G_{ij}\sin {\theta }_{ij}-B_{ij}\cos {\theta }_{ij}\right)}\end{array}\right.$$

(10)

where $G_{ij}$ and $B_{ij}$ are the real and imaginary parts of the line ij conductor, and ${\theta }_{ij}$ is the volt age phase angle difference between nodes i, j.

• Operational security constraints

$$\left\{\begin{array}{l}{U}_{\max }\le U_{i,t}\le U_{\min }\\ I_{ij,t}\le I_{ij,t}^{\max }\end{array}\right.$$

(11)

where $U_{\max }$ and $U_{\min }$ are the safe upper and lower limits of the nodal voltage; and $I_{ij,t}^{\max }$ is the maximum current allowed to flow in the branch ij.

• DG regulation performance constraints

PV active power uses the maximum power tracking mode (Maximum Power Point Tracking, MPPT); this paper considers the full consumption of distributed PV, but only its reactive capacity to regulate. The reactive power regulation range in constant apparent power control mode is

$$-\sqrt{S_{\mathrm{PV},\max }^2-P_{\mathrm{PV}}^2}\le Q_{\mathrm{PV}}\le \sqrt{S_{\mathrm{PV},\max }^2-P_{\mathrm{PV}}^2}$$

(12)

where $S_{\mathrm{PV},\max }$ is the maximum capacity of PV, $P_{\mathrm{PV}}$ is the active output of PV, and $Q_{\mathrm{PV}}$ is the reactive output of PV.

The active power output range of PV system is expressed as

$$0\le P_{\mathrm{PV}}\le P_{\mathrm{MPPT}}$$

(13)

where $P_{\mathrm{MPPT}}$ represents the active power output under the maximum power point tracking control mode.

Meanwhile, The State of Charge (SOC) of the ESS at the current moment is related to the SOC state and charge/discharge power of the previous moment, and the ESS charge/discharge power constraint is

$$E_{i,t+\Delta t}=E_{i.t}+({\eta }_{\mathrm{ch}}{P}_{i,t}^{\mathrm{char}}-{\displaystyle \frac{P_{i,t}^{\mathrm{dis}}}{{\eta }_{\mathrm{dis}}}})\Delta t$$

(14)

$$E_{i.t}\le E_{\max }$$

(15)

$$\left\{\begin{array}{l}0\le S_{i,t}^{\mathrm{char}}{P}_{i,t}^{\mathrm{char}}\le P_{\mathrm{char}}^{\max }\\ 0\le \left(1-S_{i,t}^{\mathrm{char}}\right)P_{i,t}^{\mathrm{dis}}\le P_{\mathrm{dis}}^{\max }\\ S_{i,t}^{\mathrm{char}}\in \left\{0,1\right\}\end{array}\right.$$

(16)

where $E_{i.t}$ denotes the power stored in the ESS in station i at time t, ${\eta }_{\mathrm{ch}}$ and ${\eta }_{\mathrm{dis}}$ are the ESS charging and discharging efficiencies, respectively, $E_{\max }$ is the upper limit of the ESS storage power, $S_{i,t}^{\mathrm{char}}$ is the ESS discharging state identifier, $S_{i,t}^{\mathrm{char}}=1$ denotes that the ESS is in the discharging state, $S_{i,t}^{\mathrm{char}}=0$ denotes that the ESS is in the charging state, and $P_{\mathrm{char}}^{\max }$ and $P_{\mathrm{dis}}^{\max }$ denote the maximum values of the ESS charging and discharging power, respectively.

4. Model Optimal Solving Based on Deep Reinforcement Learning

4.1. Dynamic Clustering Markov Decision Process for Multiple Distribution Areas

The co-optimization of multiple areas within a distribution network involves decision-making that is informed by both the current and prior operating states, thereby characterizing it as a sequential decision-making problem. Given the weak electrical coupling between clusters following the clustering process, autonomous regulation is facilitated within each cluster. Consequently, this paper employs a Markov decision model of multiple intelligences to model the clusters of various station areas, treating each station area as an individual intelligence and the distribution network as the training environment. The principal components of the model are delineated as follows:

• Intelligent agent observation space $O_{i,t}$

The observation space is defined as the set of observational variables that inform the decision-making process of the distribution area agent i. This includes critical parameters, such as the power and voltage measurements at the grid-connected nodes within the distribution area:

$$O_{i,t}=\left[P_{i,t}^{\mathrm{load}},Q_{i,t}^{\mathrm{load}},U_{i,t}\right]$$

(17)

where $P_{i,t}^{\mathrm{load}}$ and $Q_{i,t}^{\mathrm{load}}$ are the active and reactive power of the distribution network node connected to station i at time t.

• Intelligent agent state space $S_{i,t}$

Intelligent agent observations of stations belonging to the same cluster constitute the environmental state of the distribution network, and the state space is represented as

$$S_{i,t}=\left\{P_{i,t}^{\mathrm{load}},Q_{i,t}^{\mathrm{load}},U_{i,t}\right\},i\in {\Omega }_{m,SA}$$

(18)

where ${\Omega }_{m,SA}$ is the set of all station nodes in cluster m.

Considering that, after the topology changes as shown in Figure 2, the state space of each cluster intelligence also changes accordingly with the cluster boundary, taking the three station intelligences i₁, i₂ and i₃ in topology A as examples, their state spaces are, respectively,

$$\left\{\begin{array}{l}{S}_{i_1,t}=\left\{P_{i_1,t}^{\mathrm{load}},P_{j_1,t}^{\mathrm{load}},Q_{i_1,t}^{\mathrm{load}},Q_{j_1,t}^{\mathrm{load}},U_{i_1,t},U_{j_1,t}\right\}\\ S_{i_2,t}=\left\{P_{i_2,t}^{\mathrm{load}},P_{j_2,t}^{\mathrm{load}},Q_{i_2,t}^{\mathrm{load}},Q_{j_2,t}^{\mathrm{load}},U_{i_2,t},U_{j_2,t}\right\}\\ S_{i_3,t}=\left\{P_{i_3,t}^{\mathrm{load}},P_{j_3,t}^{\mathrm{load}},Q_{i_3,t}^{\mathrm{load}},Q_{j_3,t}^{\mathrm{load}},U_{i_3,t},U_{j_3,t}\right\}\end{array}\right.$$

(19)

When the topology is changed from A to B, then the state space of the three station intelligences at this time becomes, respectively,

$$\left\{\begin{array}{l}{S}_{i_1,t}\prime =\left\{P_{i_1,t}^{\mathrm{load}},Q_{i_1,t}^{\mathrm{load}},U_{i_1,t}\right\}\\ S_{i_2,t}\prime =\left\{P_{i_2,t}^{\mathrm{load}},P_{j_2,t}^{\mathrm{load}},P_{j_1,t}^{\mathrm{load}}{Q}_{i_2,t}^{\mathrm{load}},Q_{j_2,t}^{\mathrm{load}},Q_{j_1,t}^{\mathrm{load}}{U}_{i_2,t},U_{j_2,t},U_{j_1,t}\right\}\\ S_{i_3,t}\prime =\left\{P_{i_3,t}^{\mathrm{load}},P_{j_1,t}^{\mathrm{load}},Q_{i_3,t}^{\mathrm{load}},Q_{j_1,t}^{\mathrm{load}},U_{i_3,t},U_{j_1,t}\right\}\end{array}\right.$$

(20)

• Intelligent agent action space $A_{i,t}$

The action space represents the amount of decision making by the intelligent agent in the station based on the observation of $O_{i,t}$, specifically:

$$A_{i,t}=\left\{\Delta Q_{i,t}^{\mathrm{SA}},\Delta P_{i,t}^{\mathrm{SA}}\right\}$$

(21)

where $\Delta P_{i,t}^{\mathrm{SA}}$ and $\Delta Q_{i,t}^{\mathrm{SA}}$ are the active and reactive powers of station area i interacting with the feeder layer, which are realized by the station-side PV and storage inverters, respectively. $\Delta P_{i,t}^{\mathrm{SA}}>0$ denotes the active power emitted by the station, $\Delta P_{i,t}^{\mathrm{SA}}<0$ denotes the active power absorbed, and $\Delta Q_{i,t}^{\mathrm{SA}}$ similarly.

After the execution of the action $A_{i,t}$, the power of the distribution network node connecting station area i is

$$\left\{\begin{array}{l}{P}_{i,t+\Delta t}=P_{i,t}+\Delta P_{i,t}^{\mathrm{SA}}\\ Q_{i,t+\Delta t}=Q_{i,t}+\Delta Q_{i,t}^{\mathrm{SA}}\end{array}\right.$$

(22)

• Reward function $R_{i,t}$

The reward function is used to measure the value of a set of observation–actions of an intelligent agent i. Rewards include objective function rewards and constraint rewards. The larger the reward function, the closer to the optimal goal under the current. The relationship between station areas is one of collaborative cooperation, and the reward function is expressed as the negative of the objective function combined with a penalty term:

$$R_{i,t}=-\left({\displaystyle \sum _{i=1}^{N_{\mathrm{bus}}}\left|U_{i,t}-U_{\mathrm{ref}}\right|}+P_{m,t}^{\mathrm{loss}}\right)-\lambda {\displaystyle \sum _{i=1}^{N_{\mathrm{ESS},\mathrm{m}}}{\Phi }_{i,t}^{\mathrm{ESS}}}$$

(23)

$${\Phi }_{i.t}^{\mathrm{ESS}}=\left\{\begin{array}{l}{\phi }_1{e}^{E_{\min }-E_{i,t}}+{\phi }_2,\hspace{0.33em}\hspace{0.33em}{E}_{i,t}<E_{\min }\\ 0,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}{E}_{\min }<E_{i,t}<E_{\max }\\ {\phi }_3{e}^{E_{i,t}-E_{\max }}+{\phi }_4,\hspace{0.33em}\hspace{0.33em}{E}_{i,t}>E_{\max }\end{array}\right.$$

(24)

where ${\Phi }_{i.t}^{\mathrm{ESS}}$ represents the constraint on the energy storage capacity that is infringed upon during the decision-making process associated with the action variables of intelligent agent j. $\lambda$ is the weight coefficient for balancing the goal reward and the constraint reward, and ${\phi }_1$, ${\phi }_2$, ${\phi }_3$ and ${\phi }_4$ are the ESS penalty function coefficients.

• State transfer

After executing an action based on the current observation, the intelligent agent j obtains the timely reward function and enters the next state to achieve the observation of the next moment. This process is a state transfer, and the whole process starts from the initial moment and transfers states over time sections until there is no feasible solution for the trend or until the end of a scheduling cycle.

4.2. Model Solving Based on the Actor–Critic Framework

The decision variables of the distribution area agent are characterized by the active and reactive power outputs associated with DG and ESS, both of which are continuously adjustable, so the MADDPG algorithm is chosen to solve the continuous action behavior space in multi-intelligence reinforcement learning. For each intelligent agent i, the MADDPG algorithm needs to maintain four neural networks: the Actor network ${\pi }_{i,t}\left(O_{i,t};{\xi }_i\right)$ that executes an action $A_{j,t}$ based on observation $O_{i,t}$ and the target Actor network; the Critic network $Q_{i,t}\left(S_t,A_t;{\eta }_i\right)$ utilized to evaluate the value of actions and the target Critic network, where $S_t$ and $A_t$ denote the set of observations-actions of smart agent i and the rest of the intelligences in the same cluster at the time t, ${\xi }_i$ and ${\eta }_i$ are the corresponding parameters of the networks, respectively. The target network does not participate in the network training, and the parameters of the Actor and Critic networks are periodically copied to the corresponding target network to stabilize the network training process. The output of the Actor network is the action $A_{i,t}$, denoted as

$$A_{i,t}=\left\{\begin{array}{l}{\pi }_{i,t}\left(O_{i,t};{\xi }_i\right)+\sigma \hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}0.1\le p\le 1\\ \mathrm{random}\left(A_{i,t}^{\min },A_{i,t}^{\max }\right)\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}0\le p<0.1\end{array}\right.,p\in \left[0,1\right]$$

(25)

where $\sigma$ is Gaussian noise, $A_{i,t}^{\max }$ and $A_{i,t}^{\min }$ is the upper and lower action limits, i.e., the upper and lower limits of the interaction power between the station and the distribution network; $\mathrm{random}\left(A_{i,t}^{\min },A_{i,t}^{\max }\right)$ represents a random number between the upper and lower action limits.

The Critic network output is the value assessment for $S_t$ and $A_t$, and the network parameters ${\eta }_i$ are updated based on gradient descent, the update formulae detailed in Section 4.3. The Actor network is trained using gradient ascent to maximize the value $Q_j\left(O_{i,t},A_{i,t}\right)$ of current $\left(O_{i,t},A_{i,t}\right)$.

4.3. Attention Critic Network for Counting Dynamic State Quantity Inputs Under Cluster Changes

The Critic network of the traditional MADDPG algorithm consists of a fully connected neural network, where the number of neurons in the input layer corresponds to a fixed dimension of $\left(S_t,A_t\right)$. Once the Critic network is constructed, its input dimensions cannot be modified, which limits the application to scenarios with a fixed number of agents under a fixed topology. When the topology changes, the cluster boundaries also expand or contract, bringing the problem of dynamic changes in the state space of the intelligences as described in Section 4.2. To address this problem, this paper proposes an attention encoder critic network (AECN), which maps the dynamic observation inputs to a fixed dimensional space through encoding, ensuring the dynamic scalability of the network, and the network structure is shown in Figure 3.

The observation–action vector of the distribution substation area agent is encoded into the initial input of the value network, with an initial set of attention weights assigned. The attention network continuously adjusts the attention weight coefficients assigned to each agent during training, enabling the value network to dynamically assess each substation’s contribution to the optimization objective. Based on the adjusted attention weights, the policy network is then guided in making strategy corrections. As shown in Figure 3, for any intelligent agent i, the input to its AECN can be divided into two parts: one part is the environmental information $h_{i,t}=g_i\left(O_{i,t},A_{i,t}\right)$, which represents the agent’s observation-action combination $\left(O_{i,t},A_{i,t}\right)$ encoding of the distribution network environment, and the other part is the attention information $h_{j,t}$, which encodes the observation–action combinations of the other agents within the cluster:

$$h_{j,t}={\displaystyle \sum _{\begin{array}{l}i\ne j\\ \forall j\in {\Omega }_{m,SA}\end{array}}{\omega }_{ij,t}\cdot g_j\left(O_{j,t},A_{j,t}\right)}$$

(26)

$$e_{ij,t}=g_j{\left(O_{j,t},A_{j,t}\right)}^T{\mathbf{W}}_k^T{\mathbf{W}}_q{g}_i\left(O_{i,t},A_{i,t}\right)$$

(27)

$${\omega }_{ij,t}=\exp \left(e_{ij,t}\right)/{\displaystyle \sum _{\begin{array}{l}j\in {\Omega }_{m,SA}\\ j\ne i\end{array}}\exp \left(e_{ij,t}\right)}$$

(28)

In the equation, $g_j\left(O_{j,t},A_{j,t}\right)$ represents the encoding of agent j, where $W_k$ and $W_q$ are the parameters for mapping $g_j\left(O_{j,t},A_{j,t}\right)$ to the key value and for mapping $g_j\left(O_{j,t},A_{j,t}\right)$ to the query, respectively. $e_{ij,t}$ denotes the relevance coefficient that computes the correlation between the observation–action combination of agent i and agent j. ${\omega }_{ij,t}$ is the attention weight coefficient obtained by normalizing $e_{ij,t}$, representing the importance of the actions of other agents as observed by agent i in relation to the task at hand.

The Critic network of agent i concatenates the two input parts, resulting in an output $Q_{i,t}\left(S_t,A_t\right)$, which represents

$$Q_{i,t}(S_t,A_t)=F_i\left(g_i\left(O_{i,t},A_{i,t}\right),{\displaystyle \sum _{j\ne i}{\omega }_{ij,t}}h\left(\mathbf{V}{g}_j\left(O_{j,t},A_{j,t}\right)\right)\right)$$

(29)

In the equation, $F_i$ represents a fully connected neural network, $\mathbf{V}$ is the linear transformation matrix applied to $g_j\left(O_{j,t},A_{j,t}\right)$, and h denotes the activation function.

The Critic network update is implemented based on minimizing the joint regression function, with the loss function given by

$$L_t^Q={\displaystyle \sum _{i=1}^N}\underset{\left(S_t,A_t,r_t,S_{t+1}\right)}{E}\left[{\left(Q_{i,t}(S_t,A_t)-y_{i,t}\right)}^2\right]$$

(30)

$$y_{i,t}=R_{i,t}+\gamma Q_i^{\prime }(S_t^{\prime },A_t^{\prime })$$

(31)

In the equation, E represents the expectation, and $Q_i^{\prime }(S_t^{\prime },A_t^{\prime })$ denotes the output of the target Critic network.

The algorithm training process is shown in Algorithm 1. First, the neural network parameters and the experience replay buffer are initialized. Then, the distribution network environment, based on the actions output by the Actor network, performs power flow simulations to compute the node voltage amplitude and active network loss values. The immediate reward is calculated using the reward function in Equation (22). Following this, the system transitions to the next state based on load and substation output. The resulting data from the interaction is stored in the experience replay buffer, completing one interaction cycle. When the data in the experience replay buffer reaches a certain amount, batch data are sampled to train the Critic and Actor networks. The two networks assist each other in training based on the gradient update formulas until convergence. After training is completed, only the Actor network is deployed to the substation. During the online decision phase, control commands are output in real-time through a feedforward pass of the neural network, enabling millisecond-level decision-making for control instructions.

Algorithm 1 Multi-Agent Reinforcement Learning-Based Coordinated Control Algorithm for Multiple Area Clusters in Distribution Networks

→Initialize: the parameters ${\xi }_i$ and ${\eta }_i$ of the Actor network and Critic of all agents in the area

→Initialize: the experience replay buffer, the agents’ learning rates, and the exploration coefficient $\epsilon$

for epsilon in 1,2,3… do

Initialize the distribution network environment

for t in 1,2,3…T do The area agents obtain local observations $\left[O_{1,t},O_{2,t}\cdots O_{n,t}\right]$ The agents’ Actor networks select action $\left[A_{1,t},A_{2,t}\cdots A_{n,t}\right]$ based on the observation $\left[O_{1,t},O_{2,t}\cdots O_{n,t}\right]$ at time t, resulting in the state St at the current time t The action commands interact with the distribution network environment, performing power flow calculations to obtain the immediate reward Rt and transition to the next state St+1 Store the interaction experience tuple $\left(S_t,A_t,R_t,S_{t+1}\right)$ into the experience replay buffer The Critic network starts training once the data in the replay buffer reaches the specified threshold Randomly sample from the experience pool and update the action value $Q_i(S_t,A_t)$ according to Equation (29) Update the Actor network to output the decision command ${\pi }_i$ Softly update the Actor and Critic networks using target networks until the networks converge

end for

end for

5. Algorithm Analysis

5.1. Algorithm System Setup

To validate the effectiveness of the proposed algorithm, simulation experiments were conducted using an improved IEEE 33-bus system with multiple substations. The system’s base capacity is 10 MVA, and the voltage reference value is 12.66 kV. The voltage safety limits were set to 0.95–1.05 p.u. in this study. All buses in the test system are connected to loads, and photovoltaic systems are connected to buses 14, 15, 20, 24, and 32. The locations of the substations 1–9 connected to the distribution network are shown in Figure 4. The daily load and PV active power output during the test day are illustrated in Figure 5.

The power flow calculation module for the distribution network is based on the PYPOWER library. The load data used for agent training (including active power from photovoltaic systems) is sourced from actual distribution network data collected over a six-month period. From this dataset, 10 randomly selected days from each month, totaling 60 days, were used as the training set data, while the remaining data were used for testing. The hyperparameter settings for the algorithm are provided in

Table 1. Parameters of the algorithm table.

Parameter Name	Parameter Value
Learning rate	1 × 10−4
Discount factor	0.99
Training step size	96
Number of hidden layers	3
Experience replay pool size	100,000
Batch training size	128

.

The cluster partitioning method proposed in Section 1 of this paper was applied to partition the IEEE 33-bus system under different topological structures. The partitioning results are shown in

Table 2. Cluster segmentation results for different topologies.

Topology Number	Line Status Setting	Cluster Number	Number of Nodes Included in the Cluster
1	TS1 and TS2 are disconnected	1	1, 2, 3, 4, 5, 19, 20, 21, 22, 23, 24, 25
		2	6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18
		3	26, 27, 28, 29, 30, 31, 32, 33
2	TS1 is closed, branch 19–20 is disconnected	1	1, 2, 3, 4, 5, 6, 7, 8, 19, 23, 24, 25
		2	9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 20, 21, 22
		3	26, 27, 28, 29, 30, 31, 32, 33
3	TS2 is closed, branch 16–17 is disconnected	1	1, 2, 3, 4, 5, 19, 20, 21, 22, 23, 24, 25
		2	6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
		3	26, 27, 28, 29, 30, 31, 32, 33, 17, 18

.

To evaluate the efficacy of the proposed algorithm, we compare the attention-based action value aggregation approach against two benchmark algorithms:

Comparison Method 1: The action value aggregation method based on the average value, where the attention weight coefficient in Equation (25) is replaced with an average value weight coefficient, as proposed in [20]

Comparison Method 2: The policy transfer learning method, which leverages the transferability of neural networks, and transfers the agent neural network trained in a fixed topology to scenarios with topological changes, as proposed in [21].

5.2. Analysis of the Effects of Regulation

A random day was selected for testing, with the system configured to be in Topology 1 from 00:00 to 10:00, in Topology 2 from 10:00 to 20:00, and in Topology 3 from 20:00 to 24:00. The algorithm is trained for a total of 500 iterations, with validation performed every 10 iterations on the test set to compute the average reward. The reward value curves from 10 experimental runs are shown in Figure 5.

As can be seen from Figure 6, the method proposed in this paper enters the experience accumulation phase during the early stage of training, with the reward values fluctuating significantly. As the training progresses, the agent continuously learns better control strategies from the experience replay pool. After the 50th iteration, the reward gradually increases, and the training converges after 300 iterations.

Figure 7 illustrates the power output of the three clusters throughout the test day. From Figure 7a, it can be seen that each cluster absorbs reactive power between 08:00 and 16:00, when the PV output is high, and provides reactive power during the remaining hours. Cluster 1, with nodes located at the front end of the distribution network, exhibits lower sensitivity to voltage variations compared to the other two clusters. Therefore, it requires less reactive power support at noon, with a daily average reactive power output of 165.99 kVar. Some nodes in Clusters 2 and 3, located at the feeder ends, are more sensitive to voltage changes and require more reactive power support, with daily average reactive power outputs of 146.71 kVar and 186.75 kVar, respectively. Figure 7b shows that the active power regulation has a smaller impact on the distribution network voltage compared to reactive power regulation. Active power is primarily provided for voltage support during the day (08:00–17:00), with daily average active power outputs of 134.89 kW, 50.85 kW, and 104.7 kW for the three clusters, respectively.

Figure 8 presents a comparison of node voltages throughout the day using different control methods. The dotted lines correspond to the safe voltage limits. As shown, before the system optimization, the high PV penetration rate caused a significant voltage violation during the periods of high photovoltaic output (10:00–16:00), with the voltage peaking at 1.0841 p.u. at 14:00. From 20:00 to 24:00, due to heavier load, a voltage violation occurred below the lower limit. The overall voltage deviation was large, severely affecting the system’s safe and stable operation. After applying the method proposed in this paper, the voltage level during the noon period was significantly reduced, and the voltage level during the heavy load periods was increased, with overall voltage fluctuations reduced. In comparison with Method 1, the average action value aggregation method raised the overall voltage during 20:00–24:00, but failed to provide sufficient power support from 10:00–16:00, resulting in voltage violations above the safety limit at nodes 14 and 15 with PV integration. Compared to Method 2, although it also regulated the voltage back into the safe range, the overall voltage deviation was not as small as in the proposed method, and Method 2 required additional offline training after the agent in Topology 1 had been trained, leading to slower real-time decision-making compared to the proposed algorithm. In summary, the voltage control effect of the proposed method is superior, as the attention-based action value aggregation method effectively extracts key information that significantly influences the control of the distribution network. It identifies dynamic changes in the clusters of the substation agents caused by topological changes, thus guiding the Actor network to output strategies that adapt to topology variations.

Table 3. Statistics on composite indicators after regulation by different methods.

Control Method	Average Voltage/p.u.	Maximum Voltage/p.u.	Minimum Voltage/p.u.	Average Active Network Loss/MW
No control strategy	0.9890	1.0842	0.9412	0.5452
Methods in this paper	0.9947	1.0390	0.9501	0.5948
Comparison method 1	0.9898	1.0586	0.9523	0.6441
Comparison method 2	0.9918	1.0467	0.9530	0.6774

presents the voltage indexes and active network loss indexes after applying different control methods. These voltage indexes include the average voltage, maximum voltage, and minimum voltage. As shown in Table 3, the node voltage exceeded the limit severely before regulation, with the maximum voltage reaching 1.0842 p.u. Due to the minimal power flow between nodes under the uncontrolled strategy, the power loss on the transmission lines is relatively low. The proposed method achieves an average voltage closest to the voltage reference value of 1 p.u., and the active network loss only increased by 9.09%, which is less than that of Method 1 and Method 2, indicating better adaptability to topology changes.

Since the proposed method in this paper trains each agent offline, the online execution phase requires only the feedforward computation of the neural network, resulting in a runtime of 3.57 milliseconds, which meets the requirements for real-time control.

6. Conclusions

This paper proposes a multi-area collaborative optimization control method for distribution networks, considering topology changes. Based on the Actor–Critic framework, the method models and solves the voltage regulation and loss optimization problem of distribution networks with multiple areas. A Critic network based on the attention mechanism was designed to enhance the scalability of the agent’s observation space in scenarios with topological cluster changes, allowing dynamic adaptation to changes of the cluster boundary. The main conclusions are as follows:

(1)

Cluster division of distribution network based on electrical distance modularity index and power balance index can effectively simplify the scale of distribution network optimization problem, reduce the power flow between clusters, and realize distribution network voltage sub-cluster regulation.

(2)

The action value aggregation method based on the attention mechanism proposed in this paper can effectively adapt to different topology change scenarios of distribution networks. The simulation results of the improved IEEE33 bus algorithm show that the proposed method significantly reduces voltage deviation and maintains low active network losses, keeping node voltages within the safety range under three topology change scenarios.

Concerning its applicability in distribution networks with a larger number of nodes, future research could apply a transfer-learning-based method. This method could fine-tune the pre-trained intelligent agent network with a small amount of data, thereby enhancing its generalization performance.