Load Restoration Based on Improved Girvan–Newman and QTRAN-Alt in Distribution Networks

Abstract

With the increasing demand for power supply reliability, efficient load restoration in large-scale distribution networks post-outage scenarios has become a critical challenge. However, traditional methods become computationally prohibitive as network expansion leads to exponential growth of decision variables. This study proposes a multi-agent reinforcement learning (MARL) framework enhanced by distribution network partitioning to address this challenge. Firstly, an improved Girvan–Newman algorithm is employed to achieve balanced partitioning of the network, defining the state space of each agent and action boundaries within the multi-agent system (MAS). Subsequently, a counterfactual reasoning framework solved by the QTRAN-alt algorithm is incorporated to refine action selection during training, thereby accelerating convergence and enhancing decision-making efficiency during execution. Experimental validation using a 27-bus system and a 70-bus system demonstrates that the proposed QTRAN-alt with the Girvan–Newman method achieves fast convergence and high returns compared to typical MARL approaches. Furthermore, the proposed methodology significantly improves the success rate of full system restoration without violating constraints.

1. Introduction

As distribution networks expand and face increased complexity, large-scale power outages stemming from upstream grid failures have become a growing concern [1]. In such scenarios, the number of switch operation combinations grows exponentially [2], and traditional load restoration methods relying on experience and manual rules are no longer sufficient to meet the demand for rapid response. As the distribution system directly serves the end users, the system restoration capability of the distribution network not only directly impacts the normal electricity demand of users but also serves as an important indicator of the technical level and service capacity of modern power systems [3,4]. Therefore, the issue of distribution network fault recovery has always been a key research focus in both academia and engineering practice.

To address the issue of load restoration, researchers have explored various approaches. One typical method involves utilizing the graph data structure characteristics of the power grid to optimize the restoration path by spanning tree search [5]. However, as the distribution network scale continues to grow and controllable device categories expand, researchers have increasingly turned to mathematical programming approaches due to the constrained representation capabilities of search algorithms [6]. Programming methods offer greater flexibility, allowing for the dynamic definition of constraints and controlling variables based on specific requirements, and enabling more comprehensive and accurate descriptions of the state changes in the distribution network [7]. Furthermore, some researchers have explored the integration of graph theory with mixed-integer linear programming (MILP) methods, leveraging optimized model structures and algorithm designs to achieve significant improvements in solution efficiency [8].

As distribution networks continue to expand and controllable devices become more diverse, traditional model-based solution methods encounter significant challenges in handling the complexities of real-world operating conditions. Particularly in cases where the impact scope of power outages increases, the computational complexity grows significantly, leading to a considerable reduction in both the solution efficiency and result accuracy. In this context, reinforcement learning (RL), as a model-free learning approach, has gradually gained attention from researchers due to its adaptability and efficiency in complex environments [9,10,11]. However, as state space dimensions and decision complexity become increasingly intricate, RL may face considerable challenges. Specifically, these challenges can cause RL algorithms to become stuck in a local optimum or require excessively long training times to converge. Moreover, these limitations hinder the practical application potential of RL in real-world scenarios and increase the demand for computational resources and time costs [12].

Multi-agent reinforcement learning (MARL) decomposes complex high-dimensional tasks into subproblems of cooperative decision-making among multiple agents, allowing each agent to focus only on its local state and action space. This significantly reduces the state space dimension that a single agent needs to process. Currently, there are preliminary applications of MARL in the system restoration of distribution networks. Wang et al. [13] proposed a hierarchical MARL-based approach, which flexibly mobilizes mobile power sources and repair personnel under extreme weather conditions to improve microgrid load recovery efficiency. Vu et al. [14] introduced invalid action masking in MARL to quickly determine the optimal sequence of switch operations after a fault, and it has been validated in systems of varying scales. Fan et al. [15] employed graph-based RL to design a centralized training method for the multi-agent system (MAS), enhancing cooperative recovery capabilities of the system. The study in [16] proposed a dynamic agent network structure to effectively address the challenges posed by changes in topology and varying input dimensions and filtered out unsafe samples while training MARL.

Existing research on the application of MARL in distribution network restoration primarily focuses on distributed energy optimization within a single feeder or local microgrid. However, there has been limited exploration of multiple feeder outage issues resulting from upstream grid failures. Investigating feeder partitioning strategies is essential to enhance supply reliability under such contingencies [17,18]. The core contributions of the proposed method in applying MARL to load restoration can be summarized as follows:

• A community discovery-based decentralized partially observable Markov decision process (Dec-POMDP) model is proposed, thereby significantly expanding the application capability of MARL in complex distribution network load restoration scenarios.
• A distribution network restoration structure based on the QTRAN-alt algorithm is proposed, where counterfactual analysis is employed to rapidly identify optimal actions during MARL training. This approach effectively accelerates the convergence of MARL in large-scale distribution networks while ensuring the correctness of actions.
• The effectiveness of our approach is demonstrated through practical applications. In terms of key performance indicators such as convergence speed and recovery success rate, our approach surpasses comparison methods.

The structure of the paper is arranged as follows. Section 2 elaborates on the model and its underlying theory. Section 3 delineates the implementation process of the proposed algorithm. Section 4 validates the efficacy of our proposed method through simulation experiments. Section 5 culminates with the presentation of research findings and suggests future research directions.

2. Proposed Load Restoration Method

2.1. Load Restoration Model

Load restoration is one of the core tasks in distribution network fault recovery. Its goal is to minimize the impact of power interruptions on users by reasonably scheduling and optimizing the transfer of loads from power outages to feeders of normally operating substations. In practical distribution networks, there may be interconnections between feeders from different substations. When one substation fails, the feeders served by it can be transferred to other normally operating substations, as shown in Figure 1.

For a distribution network with N unsupplied nodes, the load redistribution target can be formulated as follows:

$$\max {\displaystyle \sum _{n=1}^N{P}_i^{\mathrm{restored}}}$$

(1)

where $P_i^{\mathrm{restored}}$ is the restored load at the power outage node $i$.

Within a distribution network modeled through an undirected graph $G=〈V,E〉$, the load restoration process must satisfy several constraints, including power flow constraints, branch capacity constraints, bus voltage constraints, and radial topology constraints.

$$\left\{\begin{array}{c}{P}_i=U_i{\displaystyle \sum _{j=1}{U}_j\left(G_{ij}\cos {\theta }_{ij}+B_{ij}\sin {\theta }_{ij}\right)}\\ Q_i=U_i{\displaystyle \sum _{j=1}{U}_j\left(G_{ij}\sin {\theta }_{ij}+B_{ij}\cos {\theta }_{ij}\right)}\end{array}\right.$$

(2)

$$I_{ij}\le I_{ij}^{\max }$$

(3)

$$U_i^{\min }\le U_i\le U_i^{\max }$$

(4)

$$\left\{\begin{array}{c}\left|\overline{E}\right|\le \left|\overline{V}\right|-1\\ \mathrm{path}(v_i^{\mathrm{gen}}\leftrightarrow v_j^{\mathrm{gen}})=Ø,i\ne j\end{array}\right.$$

(5)

Equation (2) represents the power flow equation constraint, where $P_i$ and $Q_i$ denote the active and reactive power injections at bus $i$, $U_i$ is the voltage magnitude at bus $i$, $G_{ij}$ and $B_{ij}$ are the conductance and susceptance of the branch connecting bus $i$ and bus $j$, and ${\theta }_{ij}$ represents the phase angle difference between bus $i$ and bus $j$.

Equation (3) expresses the branch current constraint, where $I_{ij}$ and $I_{ij}^{\max }$ represent the current and the maximum allowable current on the branch between bus $i$ and bus $j$.

Equation (4) specifies the bus voltage constraint, where $U_i^{\min }$ and $U_i^{\max }$ represent the minimum and maximum allowable voltage at bus $i$.

Equation (5) defines the radial topology constraint, where $\overline{V}$ and $\overline{E}$ represent the set of operational buses and the set of lines, which form the node and edge sets in the graph. $\mathrm{path}(v_i^{\mathrm{gen}}\leftrightarrow v_j^{\mathrm{gen}})$ represents the path between the equivalent power supply node $v_i^{\mathrm{gen}}$ and the equivalent power supply node $v_j^{\mathrm{gen}}$ in the graph, which is used to prevent the formation of electromagnetic loops in the distribution network caused by the upper grid.

2.2. Improved Girvan–Newman Community Detection Method

As power systems grow larger in scale and distribution networks become more intricate, dividing the distribution network into several geographically independent yet functionally interrelated subregions has become a viable approach. Each subregion is capable of achieving local optimal control while maintaining overall coordination, thus simplifying the decision-making process for load restoration. The partitioning of the distribution network requires converting the physical connectivity of devices into graph-based data for computation. During this process, it is crucial to ensure the connectivity within each subregion. We propose and implement an enhanced Girvan–Newman algorithm for network division.

For a distribution network consisting of $B$ buses and $L$ branches, its equivalent undirected graph $G=〈V,E〉$ can be defined as follows:

$$\begin{array}{c}V=\left\{v_i\right\},\left|V\right|=B\\ E=\left\{\left(v_i,v_j\right)\right\},\left|E\right|=L\end{array}$$

(6)

where $V_i$ represents the bus $i$, and $\left(v_i,v_j\right)$ represents the branch connecting the bus $i$ and the bus $j$, with each edge being undirected. $\left|V\right|$ and $\left|E\right|$ denote the number of nodes and edges in graph $G$.

As a community detection algorithm, the Girvan–Newman method [19] calculates the betweenness of each edge in the graph, which represents the degree of connection between communities. The edge with the highest betweenness is considered a boundary edge between communities and is removed from the graph as shown in Equation (7). This process iteratively divides the graph into distinct communities.

$$e^{\ast }=\arg \underset{e\in E}{\max }B(e)$$

(7)

where $B(e)$ represents the betweenness of an edge, and $e^{*}$ denotes the edge currently exhibiting the highest betweenness. After each edge is removed, the betweenness metrics of the remaining connections are updated, ensuring that boundary changes are dynamically captured throughout the partitioning process. The betweenness $B(v,w)$ of an edge connecting nodes $v$ and $w$ is given by

$$B(v,w)+=\sigma [v]\cdot \mathrm{coeff}$$

(8)

where $\sigma [v]$ represents the number of shortest paths from node $v$ to every other node, and $\sigma [v]$ can be obtained via breadth-first search (BFS). The coefficient is the scaling factor, calculated as follows:

$$\mathrm{coeff}={\displaystyle \frac{1+\delta [v]}{\sigma [v]}}$$

(9)

where $\delta [v]$ represents the dependency contribution of node $v$ to node $w$. Before traversal begins, dependency contributions of all nodes are initialized to zero. When traversing to node $v$, the dependency contributions are updated according to the formula in Equation (10).

$$\delta [w]+=1+\delta [v]$$

(10)

The edge-betweenness partitioning method does not guarantee an equal distribution of nodes across partitions, which may undermine the efficiency of strategy sharing and collaborative optimization among regions.

Specifically, when a partition contains an excessive number of nodes, the load restoration task within that region becomes more complex, which may have an impact on the overall stability of the system due to the imbalanced distribution. Conversely, for partitions with too few nodes, the coupling of strategies increases, thereby significantly raising the complexity of collaborative operations.

Therefore, after computing the partitioning results of the power grid using the Girvan–Newman algorithm, it is necessary to compare the number of nodes in each partition. If the nodes are unevenly distributed, further adjustments are made to redistribute the nodes within the partitions to achieve balance. Specifically, in subgraphs with a larger number of nodes, boundary nodes are traversed and moved from the partition with more nodes to the partition with fewer nodes, until the node count in all partitions is fully balanced. The specific steps of this process are illustrated in Figure 2. This process ensures that the load restoration tasks across all partitions are more evenly distributed. It also reduces the burden on collaborative optimization. This enhancement boosts both system-wide efficiency and operational stability.

2.3. Dec-POMDP Model for Load Restoration

Based on the partitioning of the distribution network, we define each partition as an agent. Each agent within a subregion can only access a subset of observable data from the whole distribution network environment. The load restoration task after partitioning is modeled as a Dec-POMDP. By describing the decision-making process of multiple agents within their local observation spaces, this approach enables collaborative optimization across regions, ultimately achieving the global objective. Typically, a Dec-POMDP can be represented as follows.

$$\left(S,A,T,r,O,N,\gamma \right)$$

(11)

where $S$ is the state space, $A$ is the joint action space, $T$ is the state transition probability distribution, $r$ is the reward function, $O$ is the joint observation space, $N$ is the number of agents, and $\gamma \in [0,1)$ is the discount factor. In the distribution network restoration task, each variable can be described as follows:

1.

State space $S$: the state space must fully represent the information of the environment, which is defined as follows:

$$S=\{S^U,S^I,S^{\mathrm{load}},{\overline{S}}^{\mathrm{load}},S^{\mathrm{gen}}\}$$

(12)

where $S^U$ and $S^I$ represent the bus voltage and line current in the power grid, $S^{\mathrm{load}}$ and ${\overline{S}}^{\mathrm{load}}$ represent the load supplied and the load demand in the grid, and $S^{\mathrm{gen}}$ represents the output of the generators in the network.

2.

Joint action space $A$: with the action space of each agent denoted as $A_i$, the joint action space can be defined as $A=A_1\times A_2\times \cdots \times A_N$. In the distribution network load restoration scenario, the action space for each agent is defined as follows:

$$A_i=\left\{A_{\mathrm{breaker}}\right\}$$

(13)

where $A_{\mathrm{breaker}}$ corresponds to the switches in partition $i$. In practice, considering that switch operations must meet specific switching conditions, agents are not allowed to open a switch with both sides charged if the switch is in a closed state. Similarly, they are not allowed to close a switch with both sides charged if the switch is in an open state.

3.

State transition probability distribution $T$: the probability of the environment transitioning from state $s$ to state $s\prime$ under the joint action $a$ is denoted by $T(s\prime \mid s,a)$. In the distribution network load restoration problem, the next state after applying a given action to the grid is deterministic, so it is fixed as $T(s\prime \mid s,a)=1$.

4.

Reward function $r$: at time $t$, the reward value $r^t=r(s^{t-1},a^{t-1},s^t)$ can be calculated by the state at time $t-1$, the joint action at time $t-1$ and the state at time $t$. According to the Equation (1) in Section 2.1, the reward function can be defined as follows:

$$r^t={\displaystyle \frac{P_{\mathrm{load}}^t-P_{\mathrm{load}}^{t-1}}{P_{\mathrm{base}}}}$$

(14)

where $P_{\mathrm{load}}^t$ and $P_{\mathrm{base}}$ represent the total active power delivered by the system at time $t$ and the predefined baseline. The chosen baseline active power is calculated as the aggregate of individual load baselines. Considering that the system needs to satisfy the constraints, this reward value is valid only when the system does not violate any of the constraints in Section 2.1. If the action taken in the previous time step results in a state that violates a constraint, the reward is set to −1.

5.

Joint observation space $O$: with the observation space of each agent denoted as $O_i$, then the joint observation space can be defined as $O=O_1\times O_2\times \cdots \times O_N$. The observation space for each agent $i$ depends on the state space and joint action space and is represented as $O_i=O(s,a|i)$. Thus, the observation space for each agent can be defined as follows:

$$O_i=\{S_i^U,S_i^I,S_i^{\mathrm{load}},{\overline{S}}_i^{\mathrm{load}},O(A_i)\}$$

(15)

where $S_i^{•}$ represents the corresponding state component in partition $i$, and $O(A_i)$ represents the observed switching status in the corresponding action space.

Due to the escalating computational demands of solving Dec-POMDP, MARL is implemented as a practical approximation method. In this approach, deep neural networks are leveraged to model the decision-making behavior of agents within this framework. In MARL, each agent $i$ selects an action $a_i$ based on its partial observation $o_i$ through its policy ${\pi }_i$, as expressed by

$$\pi \left(a|o\right)={\displaystyle \prod _{i=1}^N{\pi }_i\left(a_i|o_i\right)}$$

(16)

where $\pi$ denotes the joint policy distribution of all agents, $a$ represents the joint action, satisfying $a=\left(a_1,a_2,\dots ,a_N\right)$, and $o$ represents the joint observation, satisfying $o=\left(o_1,o_2,\dots ,o_N\right)$.

The goal of Dec-POMDP in the context of MARL is to find the optimal joint policy $\pi$ that maximizes the expected value of the joint reward, as shown in Equation (17).

$${\pi }^{\ast }=\arg \underset{\pi }{\max }\mathbb{E}\left[{\displaystyle \sum _t{\gamma }^{t}r(s^t,a^t,s^{t+1})}\right]$$

(17)

3. Load Restoration Structure Based on QTRAN-Alt

3.1. MARL Value Functions Decomposition

In MARL, the MAS aims to identify a collaborative strategy that enhances overall performance by maximizing aggregate returns. However, since the joint policy function of a MAS is difficult to represent explicitly, MARL introduces the state-action value function to evaluate potential rewards in future states. In the state-action value function-based solution method, the optimization target is defined by the equation below:

$${\pi }^{\ast }=\arg \underset{\pi }{\max }{Q}^{\pi }(o,a)$$

(18)

where $Q^{\pi }(o,a)$ represents the joint state-action value function of MARL, which can be written as follows:

$$Q^{\pi }(o,a)=\mathbb{E}\left[{\displaystyle \sum _t{\gamma }^t{r}^t}\mid o_0=o,a_0=a\right]$$

(19)

where $o_0$ and $a_0$ represent the joint observation and the joint action of the system at the initial step.

As shown in Equation (16), the joint policy space grows exponentially with the number of agents. In practical applications, the MAS is typically executed in a distributed manner, where each agent independently selects its actions.

$$a_i={\pi }_i\left(o_i\right)$$

(20)

To ensure global optimality during the distributed execution phase while minimizing the computational consumption, the joint state-action value function $Q(o,a)$ in MARL can be decomposed into individual state-action value functions $Q_i(o_i,a_i)$ for each agent. This decomposition simplifies the learning process and significantly reduces the computational cost. In addressing the electricity distribution network recovery operation, the power grid topology provides a clear physical criterion for the decomposition. Additionally, the linear superposition property of independent load restoration operations across regions corresponds well to the mathematical representation of the decomposition.

$$Q^{\prime }(o,a)={\displaystyle \sum _{i=1}^N{Q}_i(o_i,a_i)}$$

(21)

where $Q^{\prime }(o,a)$ is the transformed joint state-action value function, which ensures consistency in the optimal action before and after the transformation, as expressed by:

$$\arg \underset{a}{\max }Q(o,a)=\arg \underset{a}{\max }{Q}^{\prime }(o,a)$$

(22)

The decomposition requires that the combination of individual optimal actions must be equivalent to the joint optimal action. In system restoration tasks without any power deficit, the objective of maximizing load restoration within each partition is entirely consistent to maximize load restoration across the entire network. Therefore, the state-action value function can be linearly decomposed as shown in Equation (23).

$$\arg \underset{a}{\max }Q=\left(\begin{array}{c}\arg \underset{a_1}{\max }{Q}_1\\ \cdots \\ \arg \underset{a_N}{\max }{Q}_N\end{array}\right)$$

(23)

In MARL, each function is approximated by a neural network. To achieve this, an error correction term is introduced to guide the selection of the optimal action. Specifically, when each agent selects a locally optimal action, the following constraint must be satisfied:

$${\displaystyle \sum _{i=1}^N{Q}_i\left(o_i,a_i\right)}-Q\left(o,a\right)+V(o)=0,a=\overline{a}$$

(24)

where $\overline{a}$ represents the set of locally optimal actions for each agent, and $V\left(o\right)$ is the state value function. These two components are expressed as follows:

$$\overline{a}=\left({\overline{a}}_1,{\overline{a}}_2,\dots ,{\overline{a}}_N\right),{\overline{a}}_i=\arg \underset{a_i}{\max }{Q}_i$$

(25)

$$V\left(o\right)=\underset{a}{\max }Q-{\displaystyle \sum _{i=1}^N{Q}_i\left(o_i,{\overline{a}}_i\right)}$$

(26)

Thus, the optimization objective for the optimal actions in MARL is as follows:

$$\min Q^{\prime }\left(o,\overline{a}\right)-\stackrel{⌢}{Q}\left(o,\overline{a}\right)+V(o)$$

(27)

where $\stackrel{⌢}{Q}$ represents the estimated value of the joint state-action value function, which is approximated by a deep neural network.

3.2. QTRAN-Alt Loss Function

In MARL scenarios, the interactions between individual agents and the environment often make it difficult to directly quantify the impact of the action from a single agent on the overall reward. A common approach to addressing this challenge is to fix the actions of other agents while introducing variations in the actions of a specific agent, to assess its marginal contribution to the global reward. This method of analyzing the difference between actual events and those that could occur in a hypothetical scenario is known as counterfactual analysis. Through counterfactual analysis, multi-agent collaborative decision-making can more accurately assign credit, leading to more efficient and robust policy learning. For this study, we employ the QTRAN-alt algorithm [20], integrating it with a counterfactual-based collective architecture to discern optimal and suboptimal actions, thereby reinforcing the constraints on suboptimal actions.

For Equation (24), when the action differs from the local optimum, it can be expressed based on counterfactual analysis as follows:

$$Q^{\prime }\left(o,a_i,a_{-i}\right)-Q\left(o,a_i,a_{-i}\right)+V\ge 0,a\ne \overline{a}$$

(28)

where $a_{-i}$ is the actions of all agents except for agent $i$, which satisfies $a=\left(a_i,a_{-i}\right)$. During the training phase, the current action is generally not considered locally optimal. Therefore, the optimization objective for non-optimal actions is formulated as shown in Equation (29).

$$\underset{a_i\in a}{\min }{Q}^{\prime }\left(o,a_i,a_{-i}\right)-\stackrel{⌢}{Q}\left(o,a_i,a_{-i}\right)+V\left(o\right)$$

(29)

In the QTRAN-alt algorithm, the state-action value function $Q_i(o_i,a_i)$, joint state-action value function $Q(o,a)$, transformed joint state-action value function $Q^{\prime }(o,a)$, and state value function $V\left(o\right)$ are all approximated using neural networks. The loss function is expressed as follows:

$$L=L_{td}+{\lambda }_{opt}{L}_{opt}+{\lambda }_{nopt}{L}_{nopt}$$

(30)

where $L_{td}$ represents the temporal difference loss. $L_{opt}$ and $L_{nopt}$ correspond to the constraint losses for optimal and non-optimal actions, respectively, as defined in Equations (27) and (29). ${\lambda }_{opt}$ and ${\lambda }_{nopt}$ denote the weight coefficients for the optimal and non-optimal loss terms.

3.3. Load Restoration Structure

Based on the methodology described earlier, the distribution network fault restoration task is partitioned using an enhanced Girvan–Newman algorithm and subsequently modeled as a Dec-POMDP model to be solved by the QTRAN-alt algorithm, with its architecture illustrated in Figure 3.

When constructing the distribution network environment, an improved Girvan–Newman algorithm is employed to partition the buses evenly based on the number of feeders. Within each partition, the action and observation spaces for the corresponding agents are established following Equation (13) and (15). Each agent approximates its state-action value function $Q_i(o_i,a_i)$ using a neural network. During the sampling phase, each subregion operates as an independent agent, analyzing local observations to make decisions. The joint action $(a_1,a_2,\dots ,a_N)$ is then applied to the environment, which subsequently updates its state and computes the reward values. The joint observations and rewards, denoted as $(o_1,o_2,\dots ,o_N,r)$, are returned to the MAS. After multiple rounds of interaction, the interaction tuples $\left(o^t,a^t,r^t,o^{t+1}\right)$ are stored in the experience replay buffer.

To accelerate the convergence of MARL, neural networks are used to approximate the joint state-action value function $Q(o,a)$ in Equation (18), the transformed joint state-action value function $Q^{\prime }(o,a)$ in Equation (21), and the state value function $V\left(o\right)$ in Equation (26). During training, a batch of samples $\left(o,a,r,o^{\prime }\right)$ is drawn from the experience buffer, and the output of each neural network is fed into the networks for joint state-action value function, transformed joint state-action value function, and state value function. Finally, the MAS system is updated based on the loss function described in Equation (30).

4. Experimental Study

To validate the proposed method, experiments were conducted on two benchmark distribution networks: a 27-bus system and a 70-bus system, which are both three-phase balanced low-voltage distribution systems with multiple substations. The system topology and component parameters are available in [21], and the equipment scales for both test cases are summarized in

Table 1. Network scale for each case.

Case No.	Bus	Feeder	Line	Load	Generator
1	27	3	30	24	3
2	70	4	76	68	2

.

In the 27-bus system, three low-voltage buses from the upstream substations were excluded from the partitioning process, while in the 70-bus system, two such buses were similarly excluded. Each line in the network is equipped with controllable switches at both ends, which serve as the action space for agents. The load profiles for each test case were constructed by superimposing a 15 min resolution time series curve on the baseline load values, with several representative load profiles illustrated in Figure 4.

For each test case, a fault was randomly set on a low-voltage bus of an upstream substation at a randomly selected time during the day. The MAS was required to complete the load restoration task within 10 steps. Otherwise, it is considered a task failure. The agents interacted with the environment for a total of 100,000 episodes, with evaluations conducted every 100 interactions. The hyperparameters of the MARL algorithm are listed in

Table 2. MARL hyperparameter.

Parameter	Value
$\gamma$	0.99
hidden size for each agent	64
hidden size for mixing network	32
hidden size for state value network	64
learning rate	5 × 10−4
epsilon	0.5
decreased epsilon every episode	0.00064
minimum epsilon	0.02
batch size	32
buffer size	5 × 103
${\lambda }_{opt}$	1
${\lambda }_{nopt}$	1

.

To demonstrate the superiority of the QTRAN-alt algorithm, it was compared against a random action baseline as well as the VDN [22] and QMIX [23]. Both VDN and QMIX share the same hyperparameter settings as QTRAN-alt. The MARL training and testing were based on PyTorch 2.2.1. The modeling and simulation of the power grid environment were carried out using pandapower 2.13.1 [24].

4.1. Load Restoration Experiments in the 27-Bus System

In this experiment, two additional AC lines were incorporated into the original 27-bus system, specifically connecting 2–20 and 4–22. The partitioning results for the 24 buses in the feeder of the distribution network are shown in Figure 5. After partitioning, each agent manages eight buses with load, and under normal operating conditions, the following lines are disconnected: 9–17, 6–14, 13–24, 14–21, 2–20, and 4–22.

To evaluate the acceleration effect of the improved Girvan–Newman algorithm in agent training, a control scenario with random partitioning was also set up during the experiments. This allows for the comparison of the adaptability and learning effectiveness of different algorithms under unstructured partitioning conditions. The reward curves for the four algorithms in the random partitioning scenario are shown in Figure 6. It can be observed that the VDN algorithm converges the fastest and yields the highest reward, followed by QTRAN-alt, which significantly outperforms both QMIX and random actions. QMIX exhibits slower convergence and greater fluctuation in reward values during training. The reward values for random actions fluctuate around 0.17, indicating no learning capability.

Further analysis of the reward changes after partitioning with the Girvan–Newman algorithm is shown in Figure 7. The reward for random actions significantly increases to around 0.8, suggesting that a reasonable partitioning strategy provides a more favorable training environment for the MAS. At this point, QTRAN-alt still converges quickly, while VDN and QMIX exhibit varying degrees of oscillation in their reward values, indicating a certain level of instability in these algorithms.

To systematically assess the final performance of each algorithm in the distribution network recovery task,

Table 3. Successful restoration rate of MARL across partitioning methods in the 27-bus system.

Partitioning Method	Algorithm	Rate (%)
Random	random	35.08
	VDN	98.34
	QMIX	39.41
	QTRAN-alt	96.87
Improved Girvan–Newman	random	83.58
	VDN	97.85
	QMIX	77.91
	QTRAN-alt	99.73

presents the average recovery success rates for each algorithm across all test rounds. The task success rate is set to 100% only when all lost loads have been fully recovered without violating either the time limit or power system constraints. According to the experimental data, in the small-scale grid environment, where the number of agents is relatively low and the decision dimensions are simpler, VDN, relying on its straightforward linear decomposition mechanism, achieves comparable convergence speed and final rewards to QTRAN-alt. Due to the complexity of its value decomposition network, QMIX faces greater optimization challenges during training, resulting in overall poorer performance. In general, the improved Girvan–Newman algorithm combined with QTRAN-alt demonstrates the best performance in recovery tasks. Particularly in the last 50 test rounds, the recovery success rate of QTRAN-alt achieved 100%, further validating the positive impact of a well-structured partitioning strategy on improving agent learning efficiency and system recovery capabilities.

4.2. Load Restoration Experiments in the 70-Bus System

Since the 70-bus distribution network consists of four feeders, the number of partitions was set to four. The partitioning results for the 68 buses within the feeders are shown in Figure 8. After partitioning, each agent manages 17 buses, and under normal operating conditions, the following lines are disconnected: 49–50, 9–15, 38–43, 15–67, 44–45, 26–27, 29–64, and 65–66.

The reward curves for different algorithms under random partitioning are shown in Figure 9. Throughout the training process, the reward for random actions remained around −0.2, indicating that without strategy optimization, random actions fail to effectively address task requirements. Due to the unreasonable partitioning, agents executing random actions frequently violate system constraints, resulting in negative rewards. The rewards for VDN and QMIX were also notably low; although VDN eventually converged to approximately 0.4, its low reward suggests that it only learned partially effective strategies for the task. QTRAN-alt, on the other hand, converged and stabilized around 0.8. These results indicate that under a randomly partitioned grid environment, none of the algorithms were able to explore optimal strategies. Compared to Figure 6, it is evident that in large-scale distribution system restoration tasks, the increase in the number of agents, as well as the dimensional expansion of actions and observations, makes MAS coordination a more challenging task.

Experiments were then conducted in a power grid environment partitioned using the improved Girvan–Newman method, with the reward variations for each algorithm illustrated in Figure 10. As shown, the reward for random actions increased to approximately 0.5, demonstrating that a well-structured partitioning strategy significantly reduces constraint violations caused by agent actions. However, since random actions do not involve policy learning, their reward did not exhibit notable improvements over time. While the rewards for QMIX and VDN were generally higher than those for random actions, their fluctuations were significantly more pronounced. Compared to Figure 7, this suggests that the instability observed in these two algorithms within small-scale grids becomes even more apparent as system dimensionality increases. QTRAN-alt exhibited markedly superior performance, rapidly converging to a high reward value of approximately 1.4 and maintaining steady improvement in later stages. Furthermore, the variance in the reward of QTRAN-alt was substantially lower than that of QMIX and VDN, indicating its higher learning efficiency and better adaptability to large-scale distribution system restoration tasks. Additionally, hyperparameter sensitivity analysis demonstrated that the QTRAN-alt algorithm exhibits superior stability and performance, with detailed comparative results in Appendix A.

Table 4. Successful restoration rate of MARL across partitioning methods in the 70-bus system.

Partitioning Method	Algorithm	Rate (%)
Random	random	1.40
	VDN	6.23
	QMIX	1.04
	QTRAN-alt	23.95
Improved Girvan–Newman	random	31.96
	VDN	58.01
	QMIX	64.57
	QTRAN-alt	96.09

summarizes the success rates of different algorithms across all test rounds under the two partitioning conditions, providing an intuitive comparison of their system restoration performance. From a partitioning perspective, the improved Girvan–Newman method significantly enhanced the success rate, showing the importance of reasonable partitioning strategies in distribution network fault handling. Among the algorithms, QTRAN-alt consistently outperformed the others in both partitioning scenarios. Even in the random partitioning setting, it demonstrated a certain degree of adaptability. In the improved Girvan–Newman partitioned environment, QTRAN-alt was able to ensure successful system restoration in nearly all cases, confirming its strong policy optimization capability and its effectiveness in supporting multi-agent collaboration. These results highlight its superior performance in large-scale distribution system restoration tasks.

5. Conclusions

This paper addresses the system restoration problem in large-scale distribution networks caused by failures at upstream substations and proposes a recovery method based on improved Girvan–Newman and QTRAN-alt. By partitioning the distribution network, each subnet autonomously optimizes its local load restoration strategy within an independent agent framework. The approach leverages value function decomposition and counterfactual analysis to ensure consistency between local and global optimality.

In the small-scale 24-bus distribution network, where interactions of agents and dependencies are relatively simple, the improved Girvan–Newman and QTRAN-alt algorithms did not exhibit significant advantages. However, in the large-scale 70-bus distribution network, the increasing system size and agent count led to a substantial rise in both local and global interaction complexity. Under these conditions, the proposed method achieved a significantly higher success rate in system restoration compared to other baseline algorithms, while also demonstrating faster reward convergence. These results suggest that for large-scale and complex distribution networks, the proposed method offers an effective and feasible solution for system restoration.

Future work will extend this research to address complex fault scenarios in large-scale urban distribution networks with high penetration of renewable energy sources and emerging load types. In an environment with a variety of controllable resources, we consider exploring the MARL algorithm that can handle the joint action spaces with both discrete and continuous. Moreover, the uncertainty quantification framework will be enhanced by integrating advanced stochastic modeling approaches. This uncertainty-aware methodology will be synergistically combined with regional collaboration strategies, leveraging both physical grid characteristics and data-driven measurements from real-world power systems.