A Reinforcement Learning-Based Approach for Distributed Photovoltaic Carrying Capacity Analysis in Distribution Grids

Abstract

Driven by the “double carbon” goals, the penetration rate of distributed photovoltaics (PV) in distribution networks has increased rapidly. However, the continuous growth of distributed PV installed capacity poses significant challenges to the carrying capacity of distribution networks. Reinforcement learning (RL), with its capability to handle high-dimensional nonlinear problems, plays a critical role in analyzing the carrying capacity of distribution networks. This study constructs an evaluation model for distributed PV carrying capacity and proposes a corresponding quantitative evaluation index system by analyzing the core factors influencing it. An optimization scheme based on deep reinforcement learning is adopted, introducing the Deep Deterministic Policy Gradient (DDPG) algorithm to solve the evaluation model. Finally, simulations on the IEEE-33 bus system validate the good feasibility of the reinforcement learning approach for this problem.

1. Introduction

With the advancement of the “dual-carbon” goals, the penetration rate of distributed photovoltaics (PV) in distribution networks has rapidly increased, making it a critical force in energy transition. It offers advantages such as low energy consumption, minimal environmental pollution, and flexible installation locations. However, the continuous growth in distributed PV installation scale poses severe challenges to the carrying capacity of distribution networks. The power output of distributed PV is intermittent and random, and the large-scale integration of distributed PV can lead to intensified voltage fluctuations at distribution network nodes, thereby affecting the safe operation of the distribution network [1]. During periods of abundant noon sunlight, a surge in PV power generation may trigger local voltage violations. At night or on rainy days, when PV power generation is insufficient, reliance on power purchased from the main grid exacerbates the risk of voltage sags. Additionally, the complexity of distribution networks further increases the difficulty of carrying capacity assessment and optimization.

The concept of the distributed PV carrying capacity of a distribution network is defined as the maximum capacity of distributed PV that can be integrated into the network while satisfying the constraints of safe operation [2]. Ref. [3] proposed models for the comprehensive carrying capacity and comprehensive carrying capacity coefficient of distribution networks but did not mention methods for analyzing PV carrying capacity. Ref. [4] analyzed the impacts of distributed PV integration on the power quality and fault current of distribution networks but only considered the static effects of distributed PV integration, ignoring its dynamic characteristics. Ref. [5] constructed a carrying capacity model with the objective of maximizing the carrying capacity of distributed new energy sources but did not consider the impacts of extreme scenarios of distributed generation output on the carrying capacity of distribution networks. Refs. [6,7] improved the operational efficiency of the system and the accommodation capacity of new energy through optimized scheduling of resources, providing new ideas for enhancing the carrying capacity of distributed PV in distribution networks.

Reinforcement learning (RL), an emerging machine learning approach, exhibits robust adaptive and optimization capabilities. Its core principle involves an agent optimizing decision-making strategies by continuously exploring and leveraging reward signals from the environment to maximize long-term cumulative returns [8]. Distinctively, RL does not require extensive pre-labeled data but achieves autonomous learning through dynamic interactions, making it particularly suitable for addressing complex and uncertain decision-making problems. The research and application of RL in power systems provide a foundation for analyzing the distributed PV carrying capacity of distribution networks. Reference [9] combines meta-learning with deep reinforcement learning (DRL) to address voltage violations caused by PV output fluctuations and randomness, yet lacks in-depth analysis of carrying capacity assessment methods. Reference [10] proposes a multi-agent DRL-based local voltage control framework for distributed energy resources, effectively mitigating voltage fluctuations and indirectly enhancing PV hosting capacity. Reference [11] presents a multi-time scale DRL reactive power optimization strategy, offering an efficient approach for reactive power management under high PV penetration. References [12,13,14] summarizes DRL applications in various distribution network scenarios, providing comprehensive theoretical and practical references for RL-based PV hosting capacity improvement, though its case studies are highly specific and lack validation in large-scale grids or multi-user environments. References [15,16,17] optimize PV inverter active/reactive power outputs via RL to regulate voltage, indirectly contributing to hosting capacity enhancement. References [18,19,20] employ DRL for energy coordination and demand response management to reduce electricity costs, but their models exhibit high computational complexity. Reference [21] introduces a chance-constrained programming-based PV hosting capacity assessment method combined with RL for capacity optimization, yet overlooks safety verification under high-penetration scenarios. Collectively, these studies leverage RL for power system optimization but do not directly address the explicit analysis of distribution network PV hosting capacity.

According to the definition in reference [2], determining the distributed PV hosting capacity is inherently an optimization problem. Reinforcement learning, with its robust adaptive and optimization capabilities, offers a promising solution. Therefore, introducing RL technology to optimize distribution network operation strategies can serve as an effective approach to enhance the accommodation capacity of distributed PV, improve power system stability, and reduce economic costs.

2. Analysis of Carrying Capacity of Distributed Photovoltaic Power Generation in Distribution Network

2.1. Influencing Factors of Photovoltaic Carrying Capacity

In the power output structure of distributed photovoltaics (PV) in distribution networks, PV power components generally include PV panels, PV inverters, protection and control systems, and energy storage systems [22].

The typical structure diagram of a distribution network after connecting distributed photovoltaics is shown in Figure 1.

When distributed photovoltaics (PV) are integrated into a distribution network, they alter the voltage distribution and line power flow of the network. Suppose there are n nodes on a distribution network feeder, and the voltage at the k-th node is

$$U_k=U_0-{\displaystyle \sum _{i=1}^k}{\displaystyle \frac{r{l}_i{\displaystyle \sum _{j=i}^n}(P_{\mathrm{L},j}-P_{pv,j})+x{l}_i{\displaystyle \sum _{j=i}^n}(Q_{\mathrm{L},j}-Q_{pv,j})}{U_{i-1}}}$$

(1)

where U₀ represents the voltage at the connection node between the primary grid and the distribution network; U_i−1 is the voltage at node i − 1; P_L,j and Q_L,j denote the active and reactive power of the load at node j, respectively; P_pv,j and Q_pv,j represent the active and reactive power output by the distributed PV system PV_j connected at node j, where j = 1, 2, 3, …, n; l_i is the line length between nodes i − 1 and i; r and x are the resistance and reactance per unit length of the transmission line, respectively.

When the power output of distributed photovoltaics (PV) exceeds the load at a node, reverse power flow occurs. According to Equation (1), the node voltage increases with the increase in PV power output, and the closer to the end of the distribution network feeder, the more significant the voltage rise, which may cause voltage violations. Additionally, the reverse current superimposed on the original load current may exceed the conductor’s current-carrying capacity, accelerating conductor temperature rise and aging. If the reverse power flow passes through a transformer, it may cause reverse overload of the transformer, leading to winding overheating due to long-term reverse power and shortening its service life. Therefore, to ensure the safe and stable operation of the distribution network and prevent transformer overheating and current violations caused by reverse power flow, the following constraints need to be met:

(1)

Line ampacity constraint

$$I_l<I_{l,m\mathit{a}x}$$

(2)

where I_l represents the current flow in line l, while I_l,max denotes the maximum allowable current-carrying capacity of line l.

(2)

Transformer reverse loading ratio constraint

$${\displaystyle \frac{{\displaystyle \sum _{i\in {\mathit{S}}_{\mathrm{N}}}{P}_{pv,i}}-{\displaystyle \sum _{i\in {\mathit{S}}_{\mathrm{N}}}{P}_{\mathrm{L},i}}}{S_{T,\max }}}⩽{\lambda }_{\max }$$

(3)

where S_N is the set of all nodes in the distribution network, S_T,max is the maximum capacity that transformer T can transmit, and ${\lambda }_{m\mathit{a}x}$ is the maximum reverse loading ratio of the transformer, typically set to 80%.

2.2. PV Carrying Capacity Assessment Model

The distributed photovoltaic (PV) carrying capacity of a distribution network generally refers to the maximum installable capacity of distributed PV systems that can be accommodated in a specific area under the premise of safe and stable operation of the distribution network, typically expressed in terms of DC-side peak power or AC-side rated power. Determining the PV carrying capacity requires not only considering factors such as the ampacity of grid equipment, voltage regulation capability, and reverse power flow limitations but also the capacity ratio between PV modules and PV inverters. Therefore, when quantifying the distributed PV carrying capacity of a distribution network, it is essential to first clarify which type of capacity (DC or AC) is being used.

2.2.1. Quantitative Indicators for Photovoltaic Carrying Capacity

In this paper, the photovoltaic (PV) carrying capacity is expressed by the maximum installable capacity of distributed PV systems

$$f_1={\displaystyle \sum _{i\in {\mathit{S}}_{\mathrm{N}}}{C}_{pv,i}}$$

(4)

where f₁ denotes the total capacity of distributed PV integrated into the distribution network; and C_pv,i represents the capacity of distributed PV that can be connected to node i.

Therefore, this paper constructs a distributed PV carrying capacity model for distribution networks with the objectives of maximizing the integration capacity of distributed PV and minimizing operational costs. The objective function is as follows

$$\max f_1$$

(5)

$$\min f_2=f_{\cos \mathrm{t}}+f_{\mathrm{loss}}={\rho }_g\cdot P_g+{\rho }_{\mathrm{P}}{\displaystyle \sum _{l\in {\mathit{S}}_{\mathrm{L}}}{Z}_l {\left|I_l\right|}^2}+{\rho }_{pv}\cdot P_{pvc}$$

(6)

where f_cost is the cost of purchasing electricity from the main grid; f_loss is the sum of the active power loss cost and PV curtailment cost during system operation; $\rho$_g is the unit cost of purchasing electricity from the main grid; $\rho$_P is the unit cost of active power loss; Z_l is the impedance of branch l; I_l is current in branch l; S_L is the set of all branches in the distribution network; $\rho$_pv is the unit cost of PV curtailment; P_pvc is the amount of curtailed PV power.

2.2.2. Restrictive Conditions

(1)

Distributed PV Power Output Constraint

$$\left\{\begin{array}{l}0⩽P_{pv,i}⩽P_{pv,i,\max }\hfill \\ -Q_{pv,i,\max }⩽Q_{pv,i}⩽Q_{pv,i,\max }\hfill \end{array}\right.$$

(7)

where P_pv,i and Q_pv,i are the active and reactive power outputs of the distributed PV connected to node i, respectively; P_pv,i.max and Q_pv,i.max are the upper limits of active and reactive power outputs for the PV at node i.

(2)

Node Voltage Security Constraint

$$U_{i,\min }⩽U_i⩽U_{i,\max }$$

(8)

where U_i is the voltage magnitude at node i; U_i,max and U_i,min are the lower and upper limits of the voltage magnitude at node i.

(3)

Line Ampacity Constraint: as shown in Equation (2).

(4)

Power Flow Balance Constraint

$$\left\{\begin{array}{l}{P}_{pv,i}+P_{g,i}-P_{\mathrm{L},i}-U_i{\displaystyle \sum _{j=1}^N{U}_j}(G_{ij}\cos {\theta }_{ij}+B_{ij}\sin {\theta }_{ij})=0\hfill \\ Q_{pv,i}+Q_{g,i}-Q_{\mathrm{L},i}+Q_{\mathrm{C},i}+U_i{\displaystyle \sum _{j=1}^N{U}_j}(G_{ij}\sin {\theta }_{ij}-B_{ij}\cos {\theta }_{ij})=0\hfill \end{array}\right.$$

(9)

where G_ij and B_ij are the conductance and susceptance between nodes i and j; Q_C,i is the additional reactive power compensation at node i; θ_ij is the voltage phase angle difference between nodes i and j; P_g,i and Q_g,i are the active and reactive power purchased from the primary grid at node i; N is the total number of nodes in the network.

(5)

Primary Grid Power Constraint

$$\left\{\begin{array}{l}{P}_{g,\min }⩽P_g⩽P_{g,\max }\hfill \\ Q_{g,\min }⩽Q_g⩽Q_{g,\max }\hfill \end{array}\right.$$

(10)

where P_g,max and P_g,min are the upper and lower limits of the active power output from the main grid; Q_g,max and Q_g,min are the upper and lower limits of the reactive power output from the primary grid.

(6)

Transformer Reverse Loading Ratio Constraint: as shown in Equation (3).

Based on the above constraints, a distributed PV carrying capacity assessment model for distribution networks can be constructed to quantify the carrying capacity, providing an analytical basis for subsequent capacity improvement strategies.

3. Optimization Solution for Distributed Photovoltaic Carrying Capacity Based on Reinforcement Learning

The PV carrying capacity evaluation model constructed above balances the maximization of distributed PV integration capacity with the minimization of operational costs. However, due to the complex structure of distribution networks and the large number of variables in the optimization model, traditional methods such as Second-Order Cone Programming (SOCP) require tens of minutes for computation, failing to meet the requirements of minute-level dispatching. In contrast, reinforcement learning (RL) models offer rapid response, strong adaptability, and optimization capabilities. They can achieve autonomous learning through dynamic interaction without requiring large amounts of pre-labeled data, making them particularly suitable for addressing complex and uncertain decision-making problems. Therefore, this paper employs an RL model and uses the Deep Deterministic Policy Gradient (DDPG) algorithm to solve the optimization problem.

3.1. Basic Principles of Reinforcement Learning

Reinforcement learning (RL), a crucial branch in the field of machine learning, centers around the concept of maximizing expected rewards by iteratively constructing and refining an action strategy through environmental interaction. This process is inherently dynamic, where an agent continuously receives feedback from the environment, updates its policy based on this feedback, and eventually converges to an optimal strategy after numerous iterations [23]. RL’s distinctive “trial-and-error” learning mechanism grants it unparalleled advantages over traditional optimization methods when dealing with uncertain environments. The basic framework of reinforcement learning is shown in Figure 2.

The framework primarily consists of two key components: the environment and the agent. The agent is further composed of three elements: input I, reward R, and policy P. When performing a task, the agent first uses I to gather the current state from the environment. Based on this state, the agent selects and executes an action according to its current policy P. This action triggers a state transition in the environment, and the agent receives a reward signal as feedback. This interaction cycle repeats, allowing the agent to continuously collect new data and optimize its decision-making. Through iterative learning, the agent gradually refines its policy until an optimal strategy for the task is obtained.

This interaction between the agent and the environment can be formalized using a Markov Decision Process (MDP), defined by the five-tuple (S, A, R, P, γ): state space S encompasses all possible environmental states the agent may encounter; action space A comprises all feasible actions the agent can take in the environment; reward function R provides feedback to the agent, quantifying the desirability of actions; transition probability P describes the probability of transitioning between states given an action; discount factor γ balances immediate and future rewards, reflecting the value of long-term outcomes [23].

3.2. Deep Deterministic Policy Gradient (DDPG) Algorithm

The Deep Deterministic Policy Gradient (DDPG) algorithm combines deep reinforcement learning (DQN) with Deterministic Policy Gradient (DPG) [24,25], fundamentally adopting an Actor–Critic architecture to approximate policies and value functions through two deep neural networks.

3.2.1. Actor–Critic Framework

In high-dimensional spaces, states and actions are represented by high-dimensional vectors s and a. In the DDPG algorithm:, the Actor network (policy network) uses a neural network μ(s|θ^μ) to approximate the policy μ_θ(s). The Critic network (value function network) uses a neural network Q(s,a|θ^Q) to approximate the action-value function Q(s,a). To mitigate instability issues in deep neural network training, such as parameter oscillations and overfitting, DDPG introduces target networks—copied versions of the Actor and Critic networks, denoted as μ’(s|θ^μ’) and Q’(s,a|θ^Q’), where θ^μ’ and θ^Q′ are the parameters of the target networks. The network structure is illustrated in Figure 3.

Based on the above, the four network functions of the DDPG algorithm are as follows:

(1)

Original Actor Network

After each training iteration, the Actor network selects optimal actions based on the current state and interacts with the environment to generate new states and immediate rewards. It outputs the optimal policy with the goal of maximizing the state-action value function, is

$$\max \mathit{Q}{({\mathit{s}}_j,{\mathit{a}}_j\mid {\mathit{\theta }}^Q)}_{{\mathit{a}}_j=\mu ({\mathit{s}}_j\mid {\mathit{\theta }}^{\mu })}$$

(11)

Its loss function is formulated as

$$\min L_{\mu }=-{\displaystyle \frac{1}{N_{\mathrm{batch}}}}Q({\mathit{s}}_j,{\mathit{a}}_j\mid {\mathit{\theta }}^Q){|}_{{\mathit{a}}_j=\mu ({\mathit{s}}_j\mid {\mathit{\theta }}^{\mu })}$$

(12)

where the subscript j denotes the index of N_batch samples (state transition processes) selected from the experience replay buffer.

(2)

Target Actor Network

During the experience replay phase, given the next state s_j+1 from the selected sample (s_j, a_j, r_j+1, s_j+1), the target Actor network outputs the corresponding next optimal action:

$${\mathit{a}}^{\prime }={\mu }^{\prime }({\mathit{s}}_{j+1}\mid {\mathit{\theta }}^{{\mu }^{\prime }})$$

(13)

(3)

Original Critic Network

The original Critic network evaluates the action-value function and gradually approximates the true action-value. Its loss function is expressed as

$$L_Q={\displaystyle \frac{1}{N_{\mathrm{batch}}}}{\displaystyle \sum _j{({\delta }_j)}^2}={\displaystyle \frac{1}{N_{\mathrm{batch}}}}{\displaystyle \sum _j{[y_j-Q({\mathit{s}}_j,{\mathit{a}}_j\mid {\mathit{\theta }}^Q)]}^2}$$

(14)

where δ_j is the temporal difference error:

$${\delta }_j=r_{j+1}+\gamma Q^{\prime }({\mathit{s}}_{j+1},{\mu }^{\prime }({\mathit{s}}_{j+1}\mid {\mathit{\theta }}^{{\mu }^{\prime }})\mid {\mathit{\theta }}^{Q^{\prime }})-Q({\mathit{s}}_j,{\mathit{a}}_j\mid {\mathit{\theta }}^Q)$$

(15)

(4)

Target Critic Network

The target Critic network estimates the target Q-value y_j of the action-value function based on the reward feedback from the environment. The expression for y_j is

$$y_j=r_{j+1}+\gamma {\mathit{Q}}^{\prime }({\mathit{s}}_{j+1},{\mu }^{\prime }({\mathit{s}}_{j+1}\mid {\mathit{\theta }}^{{\mu }^{\prime }})\mid {\mathit{\theta }}^{Q^{\prime }})$$

(16)

where r_j+1 represents the immediate reward obtained after executing the action at step j.

3.2.2. Parameter Updates

The core theoretical foundation of DDPG lies in the deterministic policy gradient theorem. For a deterministic policy a= μ_θ(s), its gradient is

$${\nabla }_{\mathit{\theta }}J(\mathit{\theta })=\mathbb{E}\left[{\nabla }_{\mathit{\theta }}{\mu }_{\mathit{\theta }}(\mathit{s})\cdot {\nabla }_{\mathit{a}}Q(\mathit{s},\mathit{a}){|}_{\mathit{a}={\mu }_{\mathit{\theta }}(\mathit{s})}\right]$$

(17)

This gradient guides the Actor network updates using the Q-value gradient estimated by the Critic network, forming a closed-loop optimization where “the Actor makes decisions, and the Critic evaluates them.” The updates follow gradient descent, with the Actor network’s TD update rule

$${\mathit{\theta }}_{j+1}^{\mu }={\mathit{\theta }}_j^{\mu }+{\alpha }_{\mu }\cdot {\nabla }_{\mathit{a}}Q({\mathit{s}}_j,{\mathit{a}}_j\mid {\mathit{\theta }}^Q){|}_{{\mathit{a}}_j=\mu ({\mathit{s}}_j\mid {\mathit{\theta }}^{\mu })}\cdot {\nabla }_{{\mathit{\theta }}_{\mu }}\mu ({\mathit{s}}_j\mid {\mathit{\theta }}^{\mu })$$

(18)

where α_μ is the learning rate for the Actor network.

The Critic network’s TD update rule is

$${\mathit{\theta }}_{j+1}^Q={\mathit{\theta }}_j^Q+{\alpha }_Q{\delta }_j\cdot {\nabla }_{{\mathit{\theta }}^Q}Q({\mathit{s}}_j,{\mathit{a}}_j\mid {\mathit{\theta }}^Q)$$

(19)

where α_Q is the learning rate for the Critic network.

Target network parameters can be updated in two ways: directly copying weights from the original network after a fixed number of steps and softly updating parameters toward the original network’s values using a small learning rate τ. DDPG employs the second approach

$$\left\{\begin{array}{l}{\mathit{\theta }}_{j+1}^{{\mu }^{\prime }}=\tau {\mathit{\theta }}_j^{\mu }+(1-\tau ){\mathit{\theta }}_j^{{\mu }^{\prime }}\hfill \\ {\mathit{\theta }}_{j+1}^{Q^{\prime }}=\tau {\mathit{\theta }}_j^Q+(1-\tau ){\mathit{\theta }}_j^{Q^{\prime }}\hfill \end{array}\right.$$

(20)

where τ stabilizes training by preventing drastic target value changes. The overall DDPG framework is shown in Figure 4 below.

During each training episode, the original Actor network selects an action a_t based on the current state s_t. Exploration is enhanced by adding random noise ${\mathcal{N}}_t$ to the action, resulting in the executed action a_t. The agent interacts with the environment, receiving reward r_t+1 and next state s_t+1.The transition (s_t, a_t, r_t+1, s_t+1) is stored in the experience replay buffer. Each learning step samples N_batch transitions from the buffer to update network parameters via Equations (11)–(20).

3.2.3. Advantages of the DDPG Algorithm in Optimizing Distributed PV Hosting Capacity in Distribution Networks

The Deep Deterministic Policy Gradient (DDPG) algorithm exhibits significant adaptability to the optimization problem of distributed PV hosting capacity in distribution networks, with its suitability primarily manifested in the following dimensions:

(1)

Sequential Decision-Making Attribute

Evaluating the distributed PV hosting capacity requires integrating load profiles and generation patterns across multiple time intervals, where historical data influences subsequent decisions. As a prominent reinforcement learning method based on the Markov Decision Process (MDP), DDPG inherently possesses sequential decision-making capabilities, making it well-suited for the temporal dynamics of PV hosting capacity analysis in distribution networks.

(2)

Full-Cycle Optimality

The core goal of optimizing distributed PV hosting capacity in distribution networks is to maximize the integrated PV capacity over the entire operating cycle. The training samples of the DDPG algorithm are state transition sequences (s_t, a_t, r_t+1, s_t+1), which exhibit static characteristics of single-time-interval learning in data form. However, its learning process is oriented toward full-cycle optimality, guaranteed by three mechanisms: large discount factor settings, temporal difference (TD) solution methods, and environmental exploration mechanisms.

The discount factor quantifies the weight of future rewards in current decision-making. The action-value function learned by the Critic network is correlated with all subsequent decision behaviors, with its original mathematical expression as follows:

$$Q(s,a)=\mathbb{E}\left[r_{t+1}+\gamma r_{t+2}+{\gamma }^2{r}_{t+3}+\dots \mid s_t=s,a_t=a\right]$$

(21)

From this equation, any state-action value function Q(s,a) is essentially the mathematical expectation of the weighted sum of rewards from all possible subsequent decisions, discounted by the factor γ. Thus, to emphasize global decision effectiveness, a larger discount factor can strengthen the correlation between current and subsequent decisions. As long as the discount factor is non-zero, reinforcement learning inherently incorporates the impact of future actions during action selection, with its specific value only affecting algorithm convergence speed and stability.

The parameter update of Q(s,a) adopts the temporal difference method shown in Equation (16). This method does not require a complete transition path from the current state to the terminal state during iteration; instead, it can complete training using state transition fragments from each time interval, significantly improving training efficiency. Therefore, despite the time-interval-based static nature of DDPG training data, its optimization objective remains focused on full-cycle optimal decisions.

The environmental exploration mechanism ensures the global optimality of solutions. In any state s, the exploration mechanism randomly selects actions from the entire action space with a certain probability, ensuring sufficient traversal of all state and action spaces to avoid local optima and guarantee global optimality.

In summary, the solution results of the DDPG algorithm can fully meet the optimization requirements for distributed PV hosting capacity in distribution networks.

(3)

Differences and Advantages of DDPG Compared to Other Deep Reinforcement Learning Methods

DDPG differs significantly from other deep reinforcement learning methods. Taking PPO (Proximal Policy Optimization) and A2C (Advantage Actor–Critic) as examples—all three belong to the Actor–Critic framework, but their core mechanisms vary, endowing DDPG with unique advantages in solving the distributed PV hosting capacity problem of distribution networks.

In the distribution network PV hosting capacity problem, actions are continuous variables. DDPG adopts a “deterministic policy,” where the Actor network directly outputs optimal action values (rather than action probability distributions). Combined with deep networks, it generates precise continuous actions without discretizing the action space—discretization would reduce precision and lead to suboptimal solutions. In contrast, PPO and A2C inherently rely on “stochastic policies,” outputting action probability distributions. In continuous action spaces, they generate actions through sampling, which may introduce additional errors and lack precision in fine adjustments (e.g., slightly increasing PV capacity at a node to approach constraint boundaries).

The distribution network state space includes voltage, power, and other variables across multiple nodes, forming a high-dimensional continuous state. The relationship between states and actions exhibits strong nonlinearity due to line losses, reactive power compensation, etc. Both the Actor and Critic networks in DDPG use deep neural networks, efficiently approximating nonlinear mappings from high-dimensional states to actions/values. While PPO and A2C can also utilize deep networks, their on-policy nature makes them more sensitive to network structure—for instance, overly deep networks tend to overfit due to sample correlation. DDPG’s experience replay mechanism mitigates this issue.

Distribution network safety constraints (e.g., voltage deviation ≤ ±6%, line load rate ≤ 100%) are “hard constraints.” The algorithm must stably converge to constraint-satisfying solutions, avoiding infeasible solutions (e.g., voltage violations) caused by policy oscillations during training. DDPG introduces “dual target networks,” reducing fluctuations in value estimation by slowly updating target network parameters; meanwhile, the experience replay buffer breaks temporal correlation between samples, preventing oscillations in gradient descent. PPO improves stability through “policy update clipping” (clip mechanism) but remains inherently on-policy, leaving sample correlation unresolved. A2C uses asynchronous multi-thread training, which is prone to parameter oscillations due to policy differences between threads, resulting in poorer stability.

The continuous action characteristics, high-dimensional state space, and strict constraint requirements of the distribution network PV hosting capacity problem make DDPG more advantageous than PPO and A2C: its deterministic policy ensures precise continuous action output; the off-policy mechanism enhances sample efficiency; and dual target networks and experience replay improve training stability. Ultimately, DDPG can more efficiently and accurately solve for the maximum PV integration capacity while satisfying distribution network safety constraints.

3.3. Optimization of Distributed Photovoltaic Carrying Capacity Based on Deep Reinforcement Learning

When solving the distributed PV carrying capacity model for distribution networks using the Deep Deterministic Policy Gradient (DDPG) algorithm, the first step is to transform the PV carrying capacity model into a reinforcement learning (RL) five-tuple (S, A, R, P, γ) framework based on the Markov Decision Process (MDP), fully accounting for the objective function and all constraints. The objective function can be designed as the immediate reward component in the RL process, while a common strategy for handling constraints is to incorporate them into the immediate reward via penalty functions [26]. However, an excessive number of penalty functions can negatively impact the algorithm’s convergence and stability, and accurately determining penalty values is challenging. Therefore, it is more optimal to integrate constraints into the initial definition of the state and action spaces and the design of state transition rules.

(1)

State Space Selection

The state space S should include all factors influencing decision-making. For the problem of enhancing distributed PV carrying capacity in distribution networks studied here, the state space comprises node voltage magnitudes, line current magnitudes, and time-specific features (current hour, light intensity, active/reactive load, and PV power output indicators). The state space is defined in

Table 1. State space definition.

Ingredient	Dimension
Node voltage magnitudes	NU
Line current magnitudes	NI
Time-specific features	5

as follows:

In this study, the analyzed distribution network system satisfies a connected radial structure, so the dimensions in the state space adhere to the following relationship:

$$N_{\mathrm{U}}=N_{\mathrm{I}}+1$$

(22)

The state space S is expressed as

$$S=\left\{S_{\mathrm{DN}},S_{\mathrm{T}}\right\}$$

(23)

where S_DN denotes the set of distribution network elements, including node voltage magnitudes and line current magnitudes; S_T represents time-specific features, which are introduced to ensure that actions taken by the agent can better satisfy all constraints over a 24 h period.

(2)

Action Space Selection

To simplify the problem analysis, this study calculates the maximum installable capacity of distributed PV and minimizes curtailment under the premise of given PV installation locations and quantities. Therefore, the action space A—serving as the decision variable in the model—includes distributed PV capacity and curtailment. The action space is defined in

Table 2. Action space definition.

Ingredient	Dimension
Distributed PV capacity	Npv
Distributed PV curtailment	Npv × T

as follows:

The action space A is expressed as

$$A=\left\{A_{pv},A_{cpv}\right\}$$

(24)

where A_pv denotes the set of distributed PV capacities; A_cpv represents the set of PV curtailment amounts for each installed PV system at each time period t.

During each time period t of the day, based on the current state s_t, the original Actor network in the DDPG algorithm outputs the corresponding action a_t from the predefined action space A according to its decision logic. The Critic network then evaluates the quality of this decision using Q(s,a|θ^Q) based on the current state s_t and the action a_t from the Actor network. The specific architectures of these two networks are shown in Figure 3.

(3)

State Transition

The state transition probability P determines how the next state s_t+1 evolves in the environment after the Actor network outputs action a_t. Following the action a_t, node voltage magnitudes, line current magnitudes, and other state values in s_t+1 are accurately calculated using power flow equations. If the power flow calculation fails to converge, the current episode is terminated, and a large negative reward is assigned to penalize invalid actions. This process relies on physical models to ensure transition accuracy while handling outliers to prevent the agent from learning infeasible strategies, collectively forming the state transition probability P.

(4)

Immediate Reward

For the immediate reward, variables in the objective function that need to be maximized are designed as reward elements in the reinforcement learning process. Meanwhile, voltage and current safety constraints are key constraints limiting the distributed photovoltaic (PV) carrying capacity of distribution networks [27,28]; thus, variables in the objective function that need to be minimized, along with voltage and current violation constraints, are incorporated into the immediate reward through penalty functions. The objective of this study is to maximize distributed PV capacity and minimize operational costs (including main grid power purchase costs, active power loss costs, and PV curtailment costs). Thus, the immediate reward r_t+1 is defined as

$$r_{t+1}=R_1{f}_1-R_2{f}_2-R_3{V}_{over}-R_4{I}_{over}$$

(25)

where R₁ is the reward coefficient for PV capacity; R₂ is the penalty coefficient for operational costs; R₃ and R₄ are penalty coefficients for voltage and current violations, respectively; and V_over and I_over are the absolute values of voltage and current violations.

(5)

Discount Factor

The sum of immediate rewards over an episode constitutes the total return, where the discount factor γ (0 ≤ γ ≤ 1) balances the importance of immediate and future rewards. A value closer to 1 emphasizes long-term cumulative rewards. In this study, maximizing PV capacity over an episode—with current decisions influencing future choices—requires prioritizing cumulative rewards. Therefore, γ is set to 0.99 to highlight long-term benefits.

All objective functions, constraints, and improvement measures for the distributed PV carrying capacity model in distribution networks are thus integrated into the proposed RL framework.

4. Simulation Examples and Analysis

4.1. Example Environment Description

This paper uses the IEEE-33 bus distribution system as an example to calculate the distributed PV carrying capacity of the distribution network based on the reinforcement learning model and verify the effectiveness of the PV carrying capacity improvement measures adopted in this paper. Since in the Python context of reinforcement learning, the index starts from 0, the node numbers in the IEEE-33 bus distribution network are uniformly reduced by one in this paper, and the changed numbers are used uniformly in the following text. The IEEE-33 bus distribution network system is shown in Figure 5 [29]. To simulate the situation of a distribution network with a high proportion of distributed PV, in this paper, a distributed PV is connected to every other node from node 2 to node 32 in the distribution network, and the capacity of the distributed PV is set as a variable to be optimized and is in a state to be solved.

The lower limit of the node voltage in the distribution network is set to 0.94 p.u., and the upper limit is set to 1.06 p.u.; the maximum allowable ampacity of the transmission line is set to 600A, which is related to the material used for the transmission line; the light intensity is fitted according to the PV processing curve of a certain place, and 24 points are evenly taken as the light intensity within 24 h of a day; the branch impedance data is the same as that in reference [30]; the node load data is multiplied by the 24 h load demand of a certain place based on the data in reference [30] to obtain the 24 h load data of the IEEE-33 bus distribution system; the power reference value is 1MW, and the voltage reference value is 12.66kV.

The DDPG network structure is shown in Figure 3. The Actor network has two hidden layers, with 256 neurons in the first layer and 128 neurons in the second layer; the Critic network also has two hidden layers, with 256 neurons in the first layer and 128 neurons in the second layer. The activation function of the hidden layer is the ReLU function. The learning rate of the Actor network α_μ = 0.0001, and the learning rate of the Critic network α_Q = 0.001. The learning rate of the target network τ = 0.005, and only a small number of new parameters are mixed during each update to maintain stability. The capacity of the experience replay buffer is 10⁵, and the sampling method is random sampling, with a length of 64 each time.

Based on the above parameters, this paper uses the Python language and Visual Studio Code 1.100.3 based on the Python 3.9.0 environment to calculate and solve on a computer platform with an 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz, 2.80 GHz processor, and the number of algorithm iterations is 1500 times.

4.2. Example Results and Analysis

Based on the previously established reinforcement learning theoretical framework, the Deep Deterministic Policy Gradient (DDPG) algorithm is employed to iteratively train the model and optimize its parameters. The total training time amounts to 1995.47 s, and the average decision-making time is 0.17 milliseconds.

Within the DDPG algorithm architecture, the Critic network’s loss function serves as a quantitative metric for evaluating the model’s learning performance and convergence status, providing core feedback during the iterative training process. The dynamic evolution of the Critic network’s loss function is shown in Figure 6.

The analysis of the results in the figure shows that the constructed model exhibits excellent learning performance and convergence characteristics. In the initial stage of the DDPG algorithm, the focus is mainly on environment exploration, and then the algorithm gradually converges as the number of steps increases. It can be seen that the model trains well, achieving relatively stable convergence at about half the total number of steps.

Additionally, the variation in the total reward value during the training process is illustrated in Figure 7. As shown in the figure, the total reward gradually increases and stabilizes over time, indicating that the model has been effectively trained. The total reward reaches a high level when it reaches half of the total number of steps, and then stabilizes at a high level, although there are occasional small fluctuations, which do not significantly affect the overall effect.

The trained model is used to solve the distributed PV carrying capacity model for the distribution network, with results presented in Figure 8 and Figure 9.

Figure 8 shows the 24 h curves of total power output and curtailment for PV systems connected to 16 nodes, as well as the variation in primary grid power purchase and active power loss. Figure 9 displays the optimal capacity configuration for each PV node after training. The maximum installable distributed PV capacity in the distribution network is 6.5680 p.u., with a total daily curtailment of 0.6489 p.u. (1.59% of total generation), main grid power purchase of 27.2667 p.u., and total active power loss of 22.3294 p.u.

Analyzing the results obtained above, it is clear from the bar chart that node 24 has the highest value for PV capacity, which is close to 1.2 p.u. Because node 24 is located at the end of the distribution network, the voltage of the end nodes in the distribution network is relatively less affected by the upstream power supply, and when accessing PV, the impact on the overall network voltage is relatively controllable. A large amount of power injection is less likely to cause problems such as voltage overruns, so the upper limit of the PV capacity that can be accessed is higher.

Node 10 has a PV capacity of about 0.7 p.u., node 26 has a PV capacity of about 1.0 p.u., node 28 has a PV capacity of about 0.8 p.u., and node 30 has a PV capacity of about 0.7 p.u., which is at a higher level among all nodes with access to PV. The location of these nodes has some advantages in the network topology. Among them, nodes 28 and 30 are located on the feeder branches, which are at a certain distance from the power point, making them have relatively flexible power regulation space. Under the conditions of satisfying the voltage constraints and power balance, it can accommodate a relatively large amount of PV capacity. Nodes 4, 6, 8, 12, 14, 16, and 32 have smaller and similar PV capacities, between 0.2 and 0.4 p.u., with values closer to each other. Nodes 2, 18, 20, and 22 have the smallest PV capacity, almost close to 0, and are at the low end of the overall range.

4.3. Rationality Analysis of Results

The rationality of the results is analyzed from the perspective of voltage constraints based on the constructed distributed PV carrying capacity model for the distribution network. The voltage variations in all nodes in the distribution system over 24 h under the above capacity configuration are shown in Figure 10.

The following is shown in Figure 10:

• 0:00–6:00: Node voltages remain stable at 0.96–1.00 p.u. with minimal fluctuations. During this nighttime period, PV systems generate little to no power, and the distribution network is primarily supplied by traditional power sources with stable loads, resulting in steady voltages;
• 6:00–10:00: Node voltages gradually rise, with some exceeding 1.00 p.u. and reaching relatively high levels around 10:00 (most at 1.02–1.06 p.u.). As daylight increases, PV systems start generating power, injecting electricity into the distribution network. According to Equation (1), increased power injection at nodes—with constant or slightly changing loads—raises voltages;
• 10:00–14:00: Voltages remain high, with minor fluctuations at some nodes. This period coincides with the peak PV output, where continuous power injection sustains elevated voltages;
• 14:00–18:00: Voltages gradually decline from high levels to near 1.00 p.u. As light intensity weakens, PV output decreases, reducing power injection and causing voltages to drop;
• 18:00–23:00: Voltages continue to decline and stabilize at 0.96–1.00 p.u., similar to nighttime levels. PV systems stop generating power, and the network relies again on traditional power sources. Stable loads lead to stable voltages.

The analysis of the voltage curves reveals that during peak PV output, node voltages approach the allowable upper limit. With the current PV capacity configuration, further increasing PV integration under constant curtailment rates could easily trigger voltage violations, threatening power system security. Conversely, expanding PV capacity while maintaining voltage within safe thresholds (under unchanged conditions) would increase curtailment and operational costs, contradicting the study’s optimization objectives. These results align with actual operational patterns, demonstrating high rationality and reliability.

5. Conclusions

This paper addresses the problem of enhancing the carrying capacity of distributed photovoltaic (PV) systems in distribution networks, focusing on the maximum installable PV capacity under safe operation constraints (i.e., distributed PV carrying capacity) in networks with high PV penetration. Deep reinforcement learning (DRL) is employed for carrying capacity analysis. The main conclusions are as follows:

(1) A methodology for analyzing distributed PV carrying capacity in distribution networks is proposed. By examining PV power output structures, key factors influencing carrying capacity are identified. A quantitative model for distributed PV carrying capacity in distribution networks is established, along with metrics to quantify this capacity.

(2) Reinforcement learning is adopted to optimize and solve the distributed photovoltaic (PV) carrying capacity problem. First, the basic principles of reinforcement learning and the DDPG algorithm are introduced. Then, a corresponding reinforcement learning model is constructed based on the objective function and constraints of the distributed PV carrying capacity model for distribution networks. Furthermore, by analyzing the differences between the DDPG algorithm and other deep reinforcement learning methods, it is concluded that the DDPG algorithm is highly suitable for solving the distributed PV carrying capacity of distribution networks. Finally, the IEEE-33 bus distribution system is taken as an example to verify the feasibility of the DDPG algorithm proposed in this paper for the research content.

In addition, although this paper proposes an innovative method based on deep reinforcement learning and achieves phased results in theoretical analysis, due to limitations in research time and objective conditions, the existing research still has several directions for optimization and expansion, as follows:

(1) Practical promotion of the proposed method in larger-scale power systems. The model established in this paper is only verified through simulation in the IEEE-33 bus distribution system. Whether the method is applicable to larger-scale power grids remains to be confirmed.

(2) Excessively long algorithm training time and occasional poor convergence. Although the decision-making time after training is short (at the millisecond level), the average training time for a single run exceeds 30 min, and it will be even longer for larger-scale power grids, indicating that the learning efficiency needs to be improved.

(3) Transfer application of deep reinforcement learning. The algorithm adopted in this paper is only studied and applied to the analysis of distributed PV carrying capacity in distribution networks. It is a solution for specific problems under specific constraints, lacking research on its universality.