Enhancing PV Hosting Capacity of Electricity Distribution Networks Using Deep Reinforcement Learning-Based Coordinated Voltage Control

Abstract

Coordinated voltage control enables the active management of voltage levels throughout electricity distribution networks by leveraging the voltage support capabilities of existing grid-connected PV inverters. The efficient management of power flows and precise voltage regulation through coordinated voltage control schemes facilitate the increased adoption of rooftop PV systems and enhance the hosting capacity of electricity distribution networks. The research work presented in this paper proposes a coordinated voltage control scheme and evaluates the enhanced hosting capacity utilizing a deep reinforcement learning-based approach. A comparative analysis of the proposed algorithm is presented, and the performance is benchmarked against existing local voltage control schemes. The proposed coordinated voltage control scheme in this paper is evaluated using simulations on a real-world low-voltage electricity distribution network. The evaluation involves quasi-static time series power flow simulations for assessing performance. Furthermore, a discussion is presented that reflects on the strengths and limitations of the proposed scheme based on the results observed from the case study.

1. Introduction

Surging rates of rooftop PV system integration into electricity distribution networks play a crucial part in the decarbonization of the energy sector and its transition toward a greener and more sustainable system. Despite the benefits of adopting PV systems, such as reduced costs of electricity for customers and lesser reliance on fossil fuels, they can cause various challenges to the electricity distribution system if suitable planning and management approaches are not adopted. Some of the adverse impacts of high PV penetration are overvoltage, revenue loss due to excessive curtailments, increased network unbalance, and thermal overloading of feeders and transformers. The term hosting capacity (HC) is commonly defined as the maximum PV generation capacity that can be accommodated on a specific electricity distribution network without resulting in any adverse impacts [1]. An accurate assessment of HC can guide distribution network service providers (DNSPs) and various stakeholders to make investment decisions for future grid reinforcements or expansions while minimizing economic costs and ensuring network reliability.

Before undertaking an HC assessment, performance indices are defined to provide quantitative measures to evaluate the HC of networks. In [2], the performance indices most commonly used in the literature are categorized as overvoltage problems, power quality problems, protection problems, and overloading/power loss problems. The operational constraint violations of the performance indices restrict the HC of a network. Typically, for low-voltage (LV) distribution networks, the overvoltage and thermal capacity of network elements are the most restrictive performance indices in HC assessments [3].

Numerous distinct methods are proposed across various literature sources for the assessment of HC in electricity distribution networks [4,5,6]. HC quantification methods are classified into three groups in [7]: deterministic methods, probabilistic load flow methods, and quasi-static time series (QSTS) methods. Deterministic methods do not require high computational capabilities and offer a quick rough estimate for HC, utilizing fixed-input data models such as customer power consumption and PV generation [8,9,10,11,12]. Probabilistic load flow methods use the probability density functions of stochastic input variables to model the uncertainties of the distribution network and estimate the HC [13,14,15,16,17]. The QSTS methods proposed in various research works utilize a sequence of steady-state power flow simulations to accurately quantify the HC [18,19,20]. QSTS approaches are considered the most accurate method since they incorporate the system dynamics of voltage regulation devices into the HC assessment. The accuracy of the HC quantified using a QSTS simulation depends on factors such as simulation time step resolution, simulation length, and input data resolution [21]. A higher time step resolution guarantees a higher accuracy for the evaluated HC, but the computational burden of the QSTS simulation is significantly increased. Most QSTS simulations proposed in the literature are static methods that consider only the infrequent worst-case snapshots in time to evaluate the HC. These traditional approaches often underestimate the HC, since they do not consider the instances in which operational constraint violations are temporarily acceptable. The study presented in [22] proposes a QSTS simulation with time-aware metrics to evaluate the HC considering the durations of operational constraint violations and dynamic events such as the control actions of Volt-VAr response modes.

In most literature, the HC is evaluated as the aggregate PV generation capacity of a LV feeder and the HC is assumed to be evenly distributed among the integrated PV systems of the LV feeder. The individual HC of each PV system connected to a LV feeder depends on various factors such as the distance from the distribution transformer, type of phase connection, and network unbalance. HC assessment of individual PV systems provides a more accurate estimation of the HC of LV distribution networks even though it has not been extensively studied in the recent literature. Therefore, the HC assessment methodology presented in this paper evaluates the HC of individual PV systems for a 100% PV penetration scenario to arrive at the most accurate HC estimate. Furthermore, HC assessments in most literature using yearlong QSTS simulations define the HC of a LV distribution network as a single value despite the actual HC varying at each time step. The HC assessment presented in this study evaluates the HC in real time at each time step of the yearlong QSTS simulation, showcasing all possible HC values for each individual PV system throughout the year. Such an assessment of HC is more useful for PV systems that are able to regulate their power output to the grid, as it provides operational limits for the PV inverter power output at each time step.

The implementation of effective voltage regulation techniques can significantly enhance the HC of electricity distribution networks since HC is mostly restricted by overvoltage [6]. Coordinated voltage control schemes with sophisticated communication infrastructure are widely recognized to be more efficient in voltage regulation than traditional local voltage control schemes. In [23], several coordinated control schemes proposed in the recent literature are classified into groups based on their communication structure: local control, centralized control, distributed control, and decentralized control.

Optimal power flow (OPF) is at the heart of every coordinated voltage control problem. Some of the widely used algorithms in the recent literature to solve OPF and implement coordinated voltage control include rule-based algorithms [24,25,26], analytical methods for OPF [27,28,29], heuristic algorithms [30,31,32], model-predictive control [33,34,35], and deep reinforcement learning (DRL) algorithms [36,37,38]. Due to the recent advancements in the fields of artificial intelligence and machine learning, DRL algorithms for coordinated voltage control have garnered heightened interest from researchers in recent years. DRL algorithms offer model-free control and exhibit superior performance beyond traditional algorithms, such as model-predictive control, due to their ability to learn and adapt to complex systems with high-dimensional control spaces [39].

Prevalent DRL algorithms utilized for coordinated voltage control in the majority of the literature can be classified into three distinct categories: value-based methods, policy-based methods, and actor–critic algorithms. Value-based methods are primarily Q-learning-based algorithms such as deep Q-networks (DQN) that learn the optimal policy indirectly by optimizing the action-value function (Q-function). In [40], a DQN-based algorithm is utilized to control the reactive power outputs of smart inverters and the switching of capacitor banks to regulate voltage. One of the limitations of such DQN-based algorithms is that they can only perform discrete control actions which may lead to suboptimal performance levels when applied to a continuous control problem such as inverter reactive power control. Policy-based methods optimize the control policy directly in contrast to the value-based methods. Proximal policy optimization (PPO) algorithms are the most commonly used policy-based method for coordinated voltage control in the recent literature [41,42]. PPO algorithms are generally more stable and are well suited for continuous control problems. However, such policy-based algorithms suffer from low sample efficiency since the policy updates are performed using the learning samples generated by following the current policy. Actor–critic algorithms offer a powerful and flexible framework for DRL combining the merits of value-based and policy-based algorithms. Commonly used actor–critic methods for coordinated voltage control include deep deterministic policy gradient (DDPG) [39,43], twin delayed deep deterministic policy gradient (TD3) [44], and soft actor–critic (SAC) [45,46]. High sample efficiency, stability, and efficient exploration–exploitation balance are some of the advantageous characteristics of actor–critic algorithms. In [39] and [47], model-free coordinated voltage control schemes are proposed, utilizing the DDPG algorithm to realize the optimal control policies for network elements such as PV inverters, static VAr compensator (SVC) systems, and smart transformers. However, the proposed schemes are evaluated using only balanced medium-voltage (MV) distribution networks and the performance of such schemes may vary when tested on LV distribution networks where a higher degree of network unbalance exists. Furthermore, the TD3 algorithm [48] may yield superior performance for deterministic policies since it is an enhanced version of the DDPG algorithm with improved stability, and better exploitation and efficient utilization of training samples. The coordinated Volt-VAr control schemes proposed in [45] and [46] utilize the SAC algorithm that trains a stochastic policy to realize optimal control actions. An advantage of the SAC algorithm is that it is less sensitive to hyperparameters and generally less brittle than the TD3 algorithm. However, any persistent exploration noise of stochastic algorithms such as SAC may lead to undesirable outcomes and increase costs for DNSPs. Therefore, deterministic policy algorithms such as TD3 with properly tuned hyperparameters are identified to be more suited for power system applications and are utilized in this paper.

Deep learning methods for HC quantification have recently become a hot topic among researchers as they provide real-time HC assessments for large electricity distribution systems in contrast to traditional HC quantification techniques. In [49], long short-term memory (LSTM) networks were utilized to develop a deep learning-based real-time HC assessment method. The proposed method, namely spatial–temporal LSTM, is a regression model that identifies the mapping rule from the power flow data to the HC data and it does not specifically consider the control elements of the distribution network for the HC assessment. In [50], a cooperative multi-agent deep reinforcement learning algorithm is proposed for the analysis of dynamic HC in distribution networks which also considers network control elements such as SVCs, on-load tap changers (OLTCs), and renewable energy generators in the HC assessment.

An HC assessment is essential before deploying any coordinated voltage control scheme to accurately present a cost–benefit analysis to investors and stakeholders [51]. Therefore, a key feature of the work presented in this paper is the quantification of the enhanced HC due to the proposed coordinated voltage control scheme, which has frequently been overlooked in many of the scholarly publications identified in this paper. Several other crucial attributes of the current work are summarized and compared with past research works in

Table 1. Key features that differentiate the current work from past research works.

Features	[39,45,46,47]	[37]	[49]	[50]	[51]	Current Work
DRL-based coordinated voltage control	✓	✓	×	✓	×	✓
Comparative analysis with other voltage control algorithms	✓	✓	×	✓	✓	✓
Implementation on real-world unbalanced LV network models	×	✓	✓	×	×	✓
Deep learning-based real-time HC assessment	×	×	✓	✓	×	✓
Quantification of enhanced HC	×	×	×	×	✓	✓
A discussion on performance level and limitations	✓	✓	✓	✓	✓	✓

✓: included, ×: not included.

.

The research work undertaken in this paper proposes a coordinated voltage control scheme and a methodology to quantify the real-time HC of LV electricity distribution using the TD3 DRL algorithm. The key contributions of the research work presented in this paper are summarized as flows:

• A methodology to implement coordinated voltage control using the TD3 algorithm.
• Quantification of enhanced HC due to the proposed TD3-based coordinated voltage control scheme.
• Performance evaluation of the proposed scheme using a model of a real-world unbalanced LV distribution network.
• Comparative analysis with other control algorithms such as Volt-VAr/Volt-Watt and DRL algorithms such as DDPG and SAC.
• A discussion detailing the implementation safety, scalability, sample efficiency, and constraint satisfaction of the proposed coordinated control scheme.

The remaining sections of this paper are structured as follows. Section 2 provides a detailed explanation of the formulation of the TD3 algorithm with a brief introduction to the basic concepts behind voltage control and deep reinforcement learning. Section 3 presents the application of the TD3 algorithm for HC assessment and the coordinated voltage control problem. In Section 4, a numerical study is undertaken and a comparative performance evaluation of the proposed scheme is presented. Finally, the critical discussion and conclusions are presented in Section 5 and Section 6, respectively.

2. Problem Formulation

This section introduces the key concepts of voltage regulation and deep reinforcement learning. A brief description of the formulation of the TD3 algorithm is provided with the aid of a diagrammatic representation and the pseudo-code.

2.1. Preliminaries

Voltage is a critical factor when assuring the power quality and reliability of electricity distribution networks. Hence, voltage control is regarded as one of the basic operational requirements at the LV distribution level where uncertainties in load and power generation can cause significant voltage fluctuations. The high resistivity of LV feeder conductors and voltage rises due to high PV generation make the voltage control problem in LV networks distinct from transmission networks. Consider a simplified radial N-bus LV feeder as illustrated in Figure 1, in which the voltage drop ${\mathsf{\Delta }V}_{j,j+1}=(V_{j+1}-V_j)$ from bus $j$ to bus $j+1$ can be described according to Equation (1).

$${\mathsf{\Delta }V}_{j,j+1}={\displaystyle \frac{({\sum }_{i=j+1}^N{(P}_i)-j {\sum }_{i=j+1}^N{(Q}_i))}{V_{j+1}}} · {(R}^{\prime }·L_{j,j+1}+j X^{\prime }· L_{j,j+1})$$

(1)

where $R^{\prime }$ and $X^{\prime }$ are the resistance and reactance of the LV feeder per metre, respectively. A further simplification of Equation (1) is expressed in Equation (2).

$${\mathsf{\Delta }V}_{j,j+1}={\displaystyle \frac{({\sum }_{i=j+1}^N{(P}_i){·R}^{\prime }· L_{j,j+1}+{\sum }_{i=j+1}^N{(Q}_i)·X^{\prime }· L_{j,j+1})}{V_{j+1}}}+j {\displaystyle \frac{({\sum }_{i=j+1}^N{(P}_i){·X}^{\prime }· L_{j,j+1}-{\sum }_{i=j+1}^N{(Q}_i)·R^{\prime }· L_{j,j+1})}{V_{j+1}}}$$

(2)

In LV distribution networks, the X/R ratio is low and the voltage drop can be further simplified and expressed according to Equation (3) by ignoring the imaginary component in Equation (2).

$${\mathsf{\Delta }V}_{j,j+1}={\displaystyle \frac{({\sum }_{i=j+1}^N{(P}_i){·R}^{\prime }· L_{j,j+1}+{\sum }_{i=j+1}^N{(Q}_i)·X^{\prime }· L_{j,j+1})}{V_{j+1}}}$$

(3)

It is evident from Equation (3) that an increase in active power injection ${(P}_{j+1})$ would automatically increase the voltage difference, ${\mathsf{\Delta }V}_{j,j+1}$. Conversely, an application of the negative reactive power increment ${(-Q}_{j+1})$, i.e., the absorption of reactive power, would reduce the voltage magnitude difference between bus $j$ and bus $j+1$. Therefore, voltage regulation can be performed locally at the customer point of connection (POC) busbar by active power curtailments and the absorption of reactive power using customer PV inverters. This method of voltage regulation utilizes the existing resources of the distribution network without the burden of installation and maintenance costs of voltage regulation devices for the DNSP.

It is recommended that the PV inverter systems connecting to the Australian LV distribution networks must be capable of Volt-Watt and Volt-VAr response modes with VAr priority for voltage regulation. The AS/NZS 4777.2:2020 standard [52] provides the recommended Volt-Watt and Volt-VAr control function settings as illustrated in Figure 2 and Figure 3, respectively.

Local voltage control functions, such as Volt-VAr and Volt-Watt, regulate the voltage at the POC based on the principles outlined in Equation (3). However, Equation (3) does not account for phase unbalance in LV distribution networks. As a result, these traditional voltage control methods, while effective in regulating voltage at the POC, may lead to suboptimal outcomes when applied across the entire LV network. The phase unbalance of the network can be described by the percentage voltage unbalance factor ($VUR\%$) as defined in Equation (4) according to the IEC/TR 61000-3-14 standard [53], where ${\underset{\_}{U}}_2$ and ${\underset{\_}{U}}_1$ are the negative and positive sequence voltages, respectively. The compatibility level of $VUR\%$ is below 2% for LV distribution networks.

$$VUR\%={\displaystyle \frac{\left|{\underset{\_}{U}}_2\right|}{\left|{\underset{\_}{U}}_1\right|}}\times 100$$

(4)

Coordinated voltage control offers a host of benefits in contrast to local voltage control schemes when it comes to the management of operational constraints in LV power distribution networks. Coordinated voltage control takes a holistic approach by maintaining a desired voltage profile throughout the power system by optimizing the reactive power flow and active power generation. The work presented in this paper proposes a coordinated voltage control scheme using deep reinforcement learning. A Markov decision process provides the basis for the DRL algorithm implemented in this paper and the following subsections provide a brief introduction to the key concepts of the TD3 DRL algorithm utilized to solve the coordinated voltage control problem.

2.2. Markov Decision Process

A Markov decision process (MDP) is a mathematical framework used to formalize the sequential decision-making process of a decision maker or an agent in an uncertain environment. A MDP can be defined as a 5-tuple $\{S, \mathcal{A},P, R, \gamma \}$ where

• $S$ represents the state space $\left(s\in S\right)$.
• $\mathcal{A}$ represents the action space $\left(a\in \mathcal{A}\right)$.
• $P:S\times \mathcal{A}\times S\to {\mathbb{R}}^{+}$ is the transition probability function, with $P\left(s^{\prime }|s,a\right)$ being the probability of transitioning into the next state $s^{\prime }$ due to an action $a$ taken in the current state $s$.
• $R:S\times \mathcal{A}\times S\to \mathbb{R}$ is the reward function, with $r\left(s,a,s^{\prime }\right)$ being the immediate reward received by an agent after transitioning to the state $s^{\prime }$ from the state $s$ due to an action $a$.
• $\gamma$ is the discount factor $\gamma \in \left(0,1\right)$.

Given the current state of the environment $s_t$ in a standard MDP setting, an agent interacts with the environment at each time step $t=\{0, 1,\dots \}$ by taking an action $a_t$. Consequently, the agent receives an immediate reward $r\left(s_t,a_t,s_{t+1}\right)$ and the environment is then transitioned into the next state $s_{t+1}$. The trajectory $\tau$ of an agent is the sequence of states and actions due to its interactions with the environment. The sum of discounted rewards $G_{t=0}^T$ that an agent receives for a trajectory $\tau$ of episode length $T$ is given in Equation (5).

$$G_{t=0}^T={\sum }_{t=0}^T{\gamma }^{t}r\left(s_t,a_t,s_{t+1}\right)=R_{\left(\tau \right)}$$

(5)

The policy function $\pi$ represents the behaviour of an agent based on its observations in the environment. The policy $\pi$ of an agent can be either stochastic or deterministic. A stochastic policy maps each state $\left(s\in S\right)$ and action $\left(a\in \mathcal{A}\right)$ to the probability distribution $\pi \left(a|s\right)$. A deterministic policy maps each state to an action directly, which implies that $a=\pi \left(s\right)$. Deterministic policies were adopted for the work undertaken in this paper. The expected return $J\left(\pi \right)$ defined in Equation (6) is the expectation of the total reward that an agent receives over all possible trajectories under a policy. In deep reinforcement learning, the goal of an agent is to maximize its expected return by optimizing the policy. Therefore, the optimal policy ${\pi }^{\ast }$ of an agent can be expressed according to Equation (7).

$$J\left(\pi \right)={\mathbb{E}}_{\tau ~\pi }\left[R_{\left(\tau \right)}\right]$$

(6)

$${\pi }^{\ast }=\arg \underset{\pi }{\max }J\left(\pi \right)$$

(7)

The action value function $Q^{\pi }\left(s,a\right)$ given in Equation (8) represents the expected return that an agent can receive under a state $s$ taking an action $a$, following a policy $\pi$. According to the Bellman theorem and by leveraging the Markov property which states that future events are only dependent on the current state and not past events, the action value function can be recursively derived as in Equation (9).

$$Q^{\pi }\left(s,a\right)={\mathbb{E}}_{\tau ~\pi }\left[{\sum }_{t=0}^T{\gamma }^{t}r\left(s_t,a_t,s_{t+1}\right)|s_0=s, { a}_0=a\right]$$

(8)

$$Q^{\pi }\left(s,a\right)=\mathbb{E}\left[r\left(s,a,s^{\prime }\right)+\gamma \mathbb{E}\left[Q^{\pi }\left(s^{\prime },a^{\prime }\right)\right]\right]$$

(9)

2.3. Twin Delayed Deep Deterministic Policy Gradient Algorithm

TD3 is a state-of-the-art off-policy RL algorithm that can be used to solve MDPs with continuous action spaces. It is implemented in the actor–critic framework that simultaneously optimizes a deterministic policy $\pi \left(s\right)$ represented by the actor and two action value functions Q$\left(s,a\right)$ represented by two critics. The policy function maps the state $s_t$ of the environment to the desired action $a_t$. The action value function maps the state action pairs $\left(s_t,a_t\right)$ to the expected cumulative reward. In TD3, deep neural networks (DNNs) are used to approximate the policy function and the two action value functions with the parameters $\varphi$, ${\theta }_1$, and ${\theta }_2$, respectively. To enhance the stability of learning, TD3 employs a target actor function and the two target critic functions approximated by DNNs with the parameters ${\varphi }^{\prime }$, ${\theta }_1^{\prime }$, and ${\theta }_2^{\prime }$, respectively. A total of six DNNs are used in the TD3 algorithm. All standard RL algorithms, including TD3, which employs DNNs as function approximators, make use of an experience replay buffer $\mathcal{B}$. Training the TD3 agent using randomly sampled past experiences stored in the replay buffer breaks any correlations between consecutive experiences, preventing overfitting to a specific experience sequence.

Since the TD3 algorithm trains a deterministic policy, noise $ϵ~\mathcal{N}(0,\sigma )$ is added to the action $a$ taken by the policy function ${ \pi }_{\varphi }\left(s\right)$ to encourage exploration. After the addition of noise, the action needs to be clipped to lie in the valid action range ${(a}_{Low}\le a\le a_{High})$ as given in Equation (10).

$$a \leftarrow clip\left(\left({ \pi }_{\varphi }\left(s\right)+ϵ\right),a_{Low}, { a}_{High}\right), ϵ~\mathcal{N}(0,\sigma )$$

(10)

To minimize the exploitation of action value function approximation errors by the policy function, the TD3 algorithm employs the trick of target policy smoothening, in which clipped noise is added to the target action $\tilde{a}$ given by the target policy function ${\pi }_{{\varphi }^{\prime }}\left(s^{\prime }\right)$. The target action is then clipped to lie in the valid action range as given in Equation (11).

$$\tilde{a} \leftarrow clip\left(\left({ \pi }_{{\varphi }^{\prime }}\left(s^{\prime }\right)+ϵ\right),a_{Low}, { a}_{High}\right), ϵ ~ clip(\mathcal{N}\left(0,\tilde{\sigma }\right),-c, c)$$

(11)

The parameters of the two critic functions are optimized by using gradient descent to minimize their respective loss functions as given in Equations (12) and (13), where $\mathcal{D}$ is a batch of experiences sampled from the replay buffer $\mathcal{B}$. To address the overestimation bias that generally persists in all DRL algorithms, TD3 employs the clipped double-Q learning trick that uses a single target $y\left(r,s^{\prime }\right)$ to calculate the loss functions of both critic functions. The target $y\left(r,s^{\prime }\right)$ is calculated using the minimum of two target critics $Q_{{\theta }_1^{\prime }}$ and $Q_{{\theta }_2^{\prime }}$ as given in Equation (14).

$$L\left({\theta }_1\right)={\mathbb{E}}_{\left(s, a, r, s^{\prime }\right)~\mathcal{D}}\left[{\left(Q_{{\theta }_1}\left(s,a\right)-y\left(r,s^{\prime }\right)\right)}^2\right]$$

(12)

$$L\left({\theta }_2\right)={\mathbb{E}}_{\left(s, a, r, s^{\prime }\right)~\mathcal{D}}\left[{\left(Q_{{\theta }_2}\left(s,a\right)-y\left(r,s^{\prime }\right)\right)}^2\right]$$

(13)

where

$$y\left(r,s^{\prime }\right)=r\left(s,a\right)+\gamma \underset{i=\mathrm{1,2}}{\min }{Q}_{{\theta }_i^{\prime }}(s^{\prime },\tilde{a})$$

(14)

The parameterized policy network ${\pi }_{\varphi }$ can be updated by taking the gradient of the expected return $J\left(\varphi \right)$ according to Equation (15), where ${\mathcal{P}}^{\pi }$ is the discounted state distribution. Another trick adopted by the TD3 algorithm is the delayed policy update, in which the policy and the target networks are updated less frequently than the critic networks. This improves stability and reduces the variance of the policy updates, leading to faster convergence and better performance. The policy and the target networks are generally updated once for every two critic network updates.

$${\nabla }_{\varphi } J\left(\varphi \right)={\mathbb{E}}_{s~{\mathcal{P}}^{\pi }}\left[{{\nabla }_a{Q}_{{\theta }_1}^{\pi }\left(s,a\right)|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]$$

(15)

The parameters of the target critic networks and target actor network are updated according to the soft update rules given in Equations (16) and (17), respectively. Rather than directly copying the parameters of the main networks to the target networks, the hyperparameter $\rho \ll 1$ ensures that the target networks are updated slowly and smoothly, which improves the learning stability and overall performance of the algorithm. The TD3 algorithm is summarized in Algorithm 1 and a diagrammatic illustration is given in Figure 4.

$${\theta }_i^{\prime }{\leftarrow \rho \theta }_i+(1-\rho ){\theta }_i^{\prime }$$

(16)

$${\varphi }^{\prime }\leftarrow \rho \varphi +(1-\rho ){\varphi }^{\prime }$$

(17)

Algorithm 1. TD3 Algorithm-Twin delayed deep deterministic policy gradient.
1:	$\mathrm{Initialize} \mathrm{critic} \mathrm{networks} Q_{{\theta }_1}$$, Q_{{\theta }_2}$$\mathrm{and} \mathrm{actor} \mathrm{network} {\pi }_{\varphi }$$\mathrm{with} \mathrm{random} \mathrm{parameters} {\theta }_1$$,{\theta }_2$$\mathrm{and} \varphi$ respectively.
2:	$\mathrm{Initialize} \mathrm{target} \mathrm{networks} Q_{{\theta }_1^{\prime }}$$, Q_{{\theta }_2^{\prime }}$$\mathrm{and} {\pi }_{{\varphi }^{\prime }}$$\mathrm{setting} \mathrm{target} \mathrm{parameters} \mathrm{equal} \mathrm{to} \mathrm{main} \mathrm{network} \mathrm{parameters} {\theta }_1^{\prime }{\leftarrow \theta }_1$$, {\theta }_2^{\prime }{\leftarrow \theta }_2$$\mathrm{and} {\varphi }^{\prime }\leftarrow \varphi$
3:	$\mathrm{Initialize} \mathrm{the} \mathrm{empty} \mathrm{replay} \mathrm{buffer} \mathcal{B}$$, \mathrm{set} update\_delay=2$$, \mathrm{set} batch\_size=size\left(\mathcal{D}\right)=N$
4:	for $t=1$ to $T$ do:
5:	Observe state $s$$\mathrm{and} \mathrm{select} \mathrm{action} a \leftarrow clip\left(\left({ \pi }_{\varphi }\left(s\right) ϵ\right),a_{Low}, { a}_{High}\right)$$\mathrm{with} \mathrm{exploration} \mathrm{noise} ϵ~\mathcal{N}(0,\sigma )$
6:	Execute action $a$ in the environment. Then $\mathrm{attain} \mathrm{reward} r$$\mathrm{and} \mathrm{observe} \mathrm{new} \mathrm{state} s^{\prime }$
7:	Store experience tuple $\{s, a, r, s^{\prime }\}$$\mathrm{in} \mathrm{replay} \mathrm{buffer} \mathcal{B}$
8:	if $number\_of\_transitions\_in\_$$\mathcal{B}\ge size\left(\mathcal{D}\right)$:
9:	Randomly sample a batch $\mathcal{D}$$\mathrm{of} N$$\mathrm{transitions} \{s, a, r, s^{\prime }\}$$\mathrm{from} \mathcal{B}$
10:	Calculate target actions $\tilde{a}$ with clipped noise $ϵ ~ clip(\mathcal{N}\left(0,\tilde{\sigma }\right),-c, c)$    $\tilde{a} \leftarrow clip\left(\left({ \pi }_{{\varphi }^{\prime }}\left(s^{\prime }\right)+ϵ\right),a_{Low}, { a}_{High}\right)$
11:	Calculate the target $y\left(r,s^{\prime }\right)$ for the critic update    $y\left(r,s^{\prime }\right)=r\left(s,a\right)+\gamma \underset{i=\mathrm{1,2}}{\min }{Q}_{{\theta }_i^{\prime }}(s^{\prime },\tilde{a})$
12:	Update critics $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$$\mathrm{by} \mathrm{gradient} \mathrm{decent} \mathrm{using} L\left({\theta }_1\right)$$\mathrm{and} L\left({\theta }_2\right)$ respectively    $L\left({\theta }_i\right)={\mathbb{E}}_{\left(s, a, r, s^{\prime }\right)~\mathcal{D}}\left[{\left(Q_{{\theta }_i}\left(s,a\right)-y\left(r,s^{\prime }\right)\right)}^2\right]$$\mathrm{for} i=1, 2$
13:	if $t \mathit{m}\mathit{o}\mathit{d} update\_delay=0$ then:
14:	Update $\varphi$ by the deterministic policy gradient:     ${\nabla }_{\varphi }J\left(\varphi \right)={\mathbb{E}}_{s~{\mathcal{P}}^{\pi }}\left[{{\nabla }_a{Q}_{{\theta }_1}^{\pi }\left(s,a\right)|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)\right]$
15:	Update target networks with $\rho \ll 1$:     ${\theta }_i^{\prime }{\leftarrow \rho \theta }_i+(1-\rho ){\theta }_i^{\prime }$$\mathrm{for} i=1, 2$     ${\varphi }^{\prime }\leftarrow \rho \varphi +(1-\rho ){\varphi }^{\prime }$
16:	end if
17:	end if
18:	end for

3. Design of the TD3 Agent

In this paper, two different TD3 agents were developed: one to assess the network HC, and another one to perform coordinated voltage control in the electricity distribution network. This section details the application of DRL for coordinated voltage control and the parameters of the two TD3 agents, including their state space, action space, and reward function.

3.1. Hosting Capacity Assessment

The main objective of the TD3 agent used for the HC assessment is to identify the maximum possible PV rating for each customer allowed to operate in the electricity distribution network without any constraint violations under a coordinated voltage control scheme.

Environment: The environment that the TD3 agent interacts with is the actual electricity distribution network. A modified DIgSILENT PowerFactory model of a real LV distribution network is used as the environment for the agent to interact with. For the HC assessment, 100% PV penetration is assumed in terms of number of customers.

State: The TD3 agent for the hosting capacity assessment makes the following set of two observations at each time step for each of the $N$ number of customers connected to the LV network.

• ${{GHI}_i}_{\left(i\in N\right)}$ = global horizontal irradiance ($GHI\in \left[0, 1\right])$ at customer $i$;
• $V_{i \left(i\in N\right)}^{CCP}$ = voltage at the customer connection point (CCP) of customer $i$.

The total number of states observed by the TD3 agent at each time step is determined to be $2N$. Solar irradiation data are essential for calculating the available active power of each customer inverter at each time step. If the modelled network only covers a small geographical area, the global horizontal irradiance (GHI) can be assumed to be the same for all the customers. The forecasted GHI data are used as input states for the HC assessment. The voltage $V_i^{CCP}$ at the CCP of customer $i$ is acquired from the load flow simulation of the modelled network. The states are given as inputs to the actor and the critic networks as illustrated in Figure 5 and Figure 6, respectively.

Action: The actions are approximated by the actor network of the TD3 agent. At each time step, for each of the $N$ numbers of PV inverters connected to the LV network, the actor network approximates three sets of actions $a=\left\{a^s,a^P, a^Q\right\}$ described below:

• $a^s\to$ actor network output to determine the maximum PV inverter rating for customers $a^s=\left\{{a^s}_1,{a^s}_2,{a^s}_3,\dots ,{a^s}_N|{a^s}_i\in \mathbb{R},0\le {a^s}_i\le 1,\mathrm{f}\mathrm{o}\mathrm{r} i=\mathrm{1,2},3,\dots ,N\right\}$;
• $a^P\to$ actor network output to determine the active power output of the inverters $a^P=\left\{{a^P}_1,{a^P}_2,{a^P}_3,\dots ,{a^P}_N|{a^P}_i\in \mathbb{R},-1\le {a^P}_i\le 1,\mathrm{f}\mathrm{o}\mathrm{r} i=\mathrm{1,2},3,\dots ,N\right\}$;
• $a^Q\to$ actor network output to determine the reactive power output of the inverters $a^Q=\left\{{a^Q}_1,{a^Q}_2,{a^Q}_3,\dots ,{a^Q}_N|{a^Q}_i\in \mathbb{R},-1\le {a^Q}_i\le 1,\mathrm{f}\mathrm{o}\mathrm{r} i=\mathrm{1,2},3,\dots ,N\right\}$.

At each time step, the TD3 agent takes $3N$ actions, where each action is designed to individually control the active power injection and reactive power absorption/injection. These actions aim to regulate the voltage at the respective POCs based on the principles outlined in Equation (3), while simultaneously ensuring an optimal coordinated voltage control outcome across the entire LV network. As illustrated in Figure 5 and Figure 6, actions are the outputs of the actor network and are given as inputs to the critic network along with the states to approximate the Q-value or the expected cumulative reward.

The maximum PV inverter ratings of customers $S_{PV}$ can be determined by using the action set $a^s$ and the scaling factor ${HC}_{max}$ given in Equation (18). The scaling factor ${HC}_{max}$ can be an extremely high estimate for a PV inverter rating of a customer that is unlikely to be reached at any point in time in the future.

$$S_{PV}={HC}_{max}{\times a}^s$$

(18)

To ensure fairness for all customers when determining their maximum PV inverter ratings $S_{PV}$ and the active power outputs $P_{out}$ in the hosting capacity assessment, two fairness parameters, ${\lambda }_1$ and ${\lambda }_2$, were introduced. The PV inverter ratings of customers are fairly allocated by the TD3 algorithm by clipping $S_{PV}$ using ${\lambda }_1\in \left[0, 1\right]$ according to the range specified in Equation (19), where $\mu \left(S_{PV}\right)$ is the mean PV inverter rating of $N$ customers as given in Equation (20). A low ${\lambda }_1$ value assigns similar PV ratings for all customers with a high degree of fairness and a high ${\lambda }_1$ value means an unfair assignment with high variation between the allocated PV inverter ratings for customers.

$$\left(1-{\lambda }_1\right)·\mu \left(S_{PV}\right) \le S_{PV}\le \left(1+{\lambda }_1\right)· \mu \left(S_{PV}\right)$$

(19)

$$\mu \left(S_{PV}\right)={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{S_{PV}}_i$$

(20)

The available active power $P_{available}$ for an inverter $i$ is calculated according to Equation (21). The active power outputs $P_{out}$ of customer inverters are derived using the parameter ${\lambda }_2\in \left[0, 1\right]$ as given in Equation (22).

$$P_{available}=S_{PV}\times GHI$$

(21)

$$P_{out}=P_{available}\left[(1-{\lambda }_2)+{(\lambda }_2·a^P)\right]$$

(22)

The parameter ${\lambda }_2$ dictates the fairness of the assigned $P_{out}$ values by clipping the minimum possible $P_{out}$ assigned to any customer to be $P_{available}\left(1-{\lambda }_2\right)$. A lower value for ${\lambda }_2$ means that the variance in $P_{out}$ between customers is lower and a fair opportunity is given to all the customers to export active power to the electricity distribution grid. A higher ${\lambda }_2$ value may cause the TD3 algorithm to determine the $P_{out}$ of customers in a more unfair manner, giving precedence to maximizing the aggregate active power output of the LV network rather than the individual active power outputs of the customers. For example, a higher ${\lambda }_2$ may cause the customers that are electrically located further from the distribution transformer to export significantly less active power than the customers that are located electrically closer to the distribution transformer.

The reactive power output $Q_{out}$ of the customer inverters is determined according to Equation (23). Before undertaking the HC assessment, the parameters ${\lambda }_1, {\lambda }_2$, and ${HC}_{max}$ must be defined in a way that leads to fair monetary gain for all the customers and the DNSP participating in the proposed coordinated voltage control scheme.

$$Q_{out}=a^Q\times \sqrt{{S_{PV}}^2-{P_{out}}^2}$$

(23)

Actor and critic networks: As illustrated in Figure 4, the TD3 algorithm utilizes a total of six feed-forward neural networks with two actor networks and four critic networks. Diagrammatic illustrations of the actor and critic networks are given in Figure 5 and Figure 6, respectively. Both the actor and critic networks consist of an input layer followed by five hidden layers, containing {256, 512, 1024, 512, 256} neurons, respectively, each utilizing ReLU activation functions. The output layer employs a tanh activation function to introduce nonlinearity into the model. A summary of the TD3 algorithm parameter settings for HC assessment is given in

Table 2. Parameter settings of the TD3 algorithm used for the HC assessment.

Actor learning rate $\left(\alpha \right)$	0.001
Critic learning rate $\left(\beta \right)$	0.001
Discount factor $\left(\gamma \right)$	0.99
Batch size	750
Standard deviation of the added Gaussian noise for $\sigma \mathrm{a}\mathrm{n}\mathrm{d} \tilde{\sigma }$	0.01
Target network update rate $\left(\rho \right)$	0.005
Activation function of hidden layers	ReLU
Activation function of output layer	tanh
Input layer size	2$N$
Output layer size	3$N$
Size of hidden layers	{256, 512, 1024, 512, 256}

. The learning rates for both the actor and critic, along with the batch size, were optimized through exhaustive testing of various combinations.

Reward: The reward function is a core component of the TD3 algorithm that provides clear informative signals to guide the agent’s learning process effectively. An intuitive design of a reward function requires a cautious analysis of an agent’s objectives, environment dynamics, operational constraints, and potential trade-offs. The reward function should assign higher rewards to the actions that lead to the desired outcomes while penalizing actions that lead to undesirable outcomes. The reward function $R_{HC}$ of the HC assessment for action $a$ at time step $t$ is formulated as given in Equation (24).

$$R_{HC}=-\left[ {\displaystyle \frac{\delta }{N}}{\sum }_{i=1}^N\left[ {\displaystyle \frac{P_{curtailed}\left(i,t\right)}{{HC}_{max}\times GHI\left(i,t\right)}} \right]+\left|V_{max}\left(t\right)-V_{Nom}\right|+\left|V_{min}\left(t\right)-V_{Nom}\right|+\eta \right]$$

(24)

where

$P_{curtailed}\left(i,t\right)=\left[{HC}_{max}\times GHI\left(i,t\right)\right]-P_{out}\left(i,t\right)$

$V_{Nom}=$ nominal bus voltage of the LV distribution network;

$V_{max}\left(t\right)=$ maximum voltage of the voltages observed at the PCCs of customers at time $t$;

$V_{min}\left(t\right)=$ minimum voltage of the voltages observed at the PCCs of customers at time $t$.

The parameter $\delta \in {\mathbb{R}}^{+}$ of Equation (24) is the weight that regulates the degree of reward given between voltage regulation and minimizing active power curtailment. A higher value for $\delta$ results in a higher reward given for minimizing active power curtailment over voltage regulation and a lower value for $\delta$ results in a higher reward given for voltage regulation over minimizing active power curtailment. The parameter $\eta$ is the penalty factor that determines the reward in an instance of operational constraint violation. The voltage constraint violations for this study are defined in Equations (25) and (26). The distribution transformer loading constraint is defined in Equation (27). The parameter $\eta$ typically takes a very large value in the event of operational constraint violations and $\eta =0$ in the event of no operational constraint violations.

$$V_{max}\left(t\right)<V_{\mathrm{m}\mathrm{a}\mathrm{x}\_limit}$$

(25)

$$V_{min}\left(t\right)>V_{\mathrm{m}\mathrm{i}\mathrm{n}\_limit}$$

(26)

$$S_{Trf}\left(t\right)<S_{Trf\_rated}$$

(27)

where

$V_{\mathrm{m}\mathrm{a}\mathrm{x}\_limit}=$ maximum voltage limit (typically, the voltage that an inverter trips);

$V_{\mathrm{m}\mathrm{i}\mathrm{n}\_limit}=$ minimum voltage limit;

$S_{Trf\_rated}=$ rated capacity of the distribution transformer;

$S_{Trf}\left(t\right)=$ distribution transformer load at time $t$.

3.2. Coordinated Voltage Control

A separate TD3 agent is required to be designed, and its parameters must be fine-tuned to attain the best performance for coordinated voltage control. The main objective of the TD3 agent for coordinated voltage control is to determine the active and reactive power outputs of the customer PV inverters. The design of the TD3 agent for coordinated voltage control is similar to the HC assessment with several alterations.

Environment: The environment used for training the TD3 agent is the same as the HC assessment, which is a modified DIgSILENT PowerFactory model of a real LV distribution network. The customer PV ratings and the percentage PV penetration in the LV distribution network must be identified before training the TD3 algorithm. Once the training using the simulation model is complete, the TD3 agent can be implemented in the real LV distribution network for coordinated voltage control.

State: The TD3 agent makes a set of four observations described below at every time step for each of the $N$ number of customers. Therefore, the total number of states observed at each time step is $4N$.

• ${{GHI}_i}_{\left(i\in N\right)}=$ global horizontal irradiance ($GHI\in \left[0, 1\right])$ at customer $i$;
• $V_{i \left(i\in N\right)}^{CCP}=$ voltage at CCP of customer $i$;
• $P_{i \left(i\in N\right)}^{CCP}=$ active power at CCP of customer $i$;
• $Q_{i \left(i\in N\right)}^{CCP}=$ reactive power at CCP of customer $i$.

For training, either forecasted or historical GHI data can be used, and for online implementation, it is recommended that real-time GHI data are used as inputs to the TD3 agent. During training, the states $V_i^{CCP},P_i^{CCP}$, and $Q_i^{CCP}$ are acquired by the load flow software, but for online implementation these inputs can be procured by real-time smart meter measurements.

Action: At each time step, for each of the $N$ number of PV inverters, the actor network approximates two sets of actions $a=\left\{a^P, a^Q\right\}$ as described below. The total number of actions taken by the TD3 agent at each time step is $2N$.

• $a^P\to$ actor network output to determine the active power output of the inverters $a^P=\left\{{a^P}_1,{a^P}_2,{a^P}_3,\dots ,{a^P}_N|{a^P}_i\in \mathbb{R},-1\le {a^P}_i\le 1,\mathrm{f}\mathrm{o}\mathrm{r} i=\mathrm{1,2},3,\dots ,N\right\}$;
• $a^Q\to$ actor network output to determine the reactive power output of the inverters $a^Q=\left\{{a^Q}_1,{a^Q}_2,{a^Q}_3,\dots ,{a^Q}_N|{a^Q}_i\in \mathbb{R},-1\le {a^Q}_i\le 1,\mathrm{f}\mathrm{o}\mathrm{r} i=\mathrm{1,2},3,\dots ,N\right\}$.

The active power output $P_{out}$ of customers is calculated according to Equations (21) and (22). However, the rated capacity of the customer PV inverters $S_{PV}$ in Equation (21) is a fixed value, unlike for the HC assessment. The reactive power outputs of the PV inverters are calculated similar to the HC assessment, which is given in Equation (23). It should be noted that in the proposed coordinated voltage control scheme, priority is given to the active power output of the PV inverters over the reactive power output.

Actor and critic networks: The hidden layers of the actor and critic networks for coordinated voltage control are similar in design to the TD3 agent of the HC algorithm. The input and output layers of the actor and critic networks are modified according to the actions and states defined in the above section. To maintain a certain level of consistency for the HC assessment and the proposed coordinated voltage control method, both TD3 agents must be designed with more or less similar parameters.

Table 3. Parameter settings of the TD3 algorithm used for coordinated voltage control.

$\mathrm{Actor} \mathrm{learning} \mathrm{rate} \left(\alpha \right)$	0.001
$\mathrm{Critic} \mathrm{learning} \mathrm{rate} \left(\beta \right)$	0.001
$\mathrm{Discount} \mathrm{factor} \left(\gamma \right)$	0.99
Batch size	500
$\mathrm{Standard} \mathrm{deviation} \mathrm{of} \mathrm{the} \mathrm{added} \mathrm{Gaussian} \mathrm{noise} \mathrm{for} \sigma \mathrm{a}\mathrm{n}\mathrm{d} \tilde{\sigma }$	0.1
$\mathrm{Target} \mathrm{network} \mathrm{update} \mathrm{rate} \left(\rho \right)$	0.005
Activation function of hidden layers	ReLU
Activation function of output layer	tanh
Input layer size	$4N$
Output layer size	$2N$
Size of hidden layers	{256, 512, 1024, 512, 256}

provides a summary of the parameter settings for the TD3 agent. Similarly to the hosting capacity assessment, these parameters were optimized through exhaustive testing of various combinations, considering performance, memory efficiency, and computation time.

Reward: The reward function for coordinated voltage control $R_{VC}$ follows a similar design to the HC assessment with slight deviations and is given in Equation (28). The values for parameters $\delta$ and $\eta$ need to be redefined for coordinated voltage control to guarantee peak performance in voltage regulation and the optimal minimization of curtailments by the TD3 agent.

$$R_{VC}=-\left[ {\displaystyle \frac{\delta }{N}}{\sum }_{i=1}^N\left[ {\displaystyle \frac{P_{curtailed}\left(i,t\right)}{S_{PV}\left(i\right)\times GHI\left(i,t\right)}} \right]+\left|V_{max}\left(t\right)-V_{Nom}\right|+\left|V_{min}\left(t\right)-V_{Nom}\right|+\eta \right]$$

(28)

where

$P_{curtailed}\left(i,t\right)=\left[S_{PV}\left(i\right)\times GHI\left(i,t\right)\right]-P_{out}\left(i,t\right)$

4. Case Study

Simulations were carried out on a DIgSILENT PowerFactory model of a real 3-phase LV distribution network that supplies electricity to 28 customers. In this section, the extent of HC enhancement of the modelled LV network due to the proposed TD3 coordinated voltage control scheme is quantified and comparative results with various benchmark voltage control methods are presented.

4.1. Experimental Setup

A schematic diagram of the modelled LV distribution network is given in Figure 7. The MV section of the network was modelled as an equivalent impedance and a variable voltage source. The main feeders of the network were 3-phase and the service feeders that connect to the CCP were single-phase. This particular LV network was selected for the case study as it represents a typical LV distribution network with a significant level of phase unbalance. The electrical characteristics and operational constraints of the modelled LV network are given in

Table 4. LV network electrical characteristics.

	R1 Ω/km	X1 Ω/km		R0 Ω/km		X0 Ω/km
Main Feeder	0.298557	0.259633		1.132508		0.945961
Service Feeder	1.480003	0.088		-		-
Network Constraints
${\mathit{V}}_{\mathit{N}\mathit{o}\mathit{m}}=\mathbf{230} \mathbf{V}$	$V_{max\_limit}=258\mathrm{V}$		$V_{min\_limit}=218\mathrm{V}$		$S_{Trf\_rated}=1\mathrm{M}\mathrm{V}\mathrm{A}$

.

To simulate the maximum level of PV penetration, every customer was assumed to own a PV system that connects to the LV distribution network. Customer load data were acquired from historical smart meter measurements and each customer load was modelled with a constant power factor of 0.95. Historical and forecasted GHI data were attained through data providers such as Solcast. For training and evaluation of the TD3 agents, 3 separate load and GHI time series data sets were used as given in

Table 5. Simulation data sets.

Data Set No.	Number of Days	Time Step Resolution	Total Time Steps	Simulation
1	120	30 min	5760	Training of HC and VC agents
2	120	30 min	5760	Evaluation of HC agent
3	1	5 s	17,280	Evaluation of VC agent

. Data Set 1 consisted of time series data of 120 days at 30 min time steps that equally captured all seasonal variations from the past year. Data Set 2 was similar in size and resolution to Data Set 1 but consisted of time series data for the forecasted year. The TD3 agents for the HC assessment and coordinated voltage control were trained using Data Set 1. The HC of the LV network was evaluated using Data Set 2 and the performance of the trained TD3 voltage control agent was evaluated using Data Set 3, which consisted of the time series data of a single day at 5 s resolution.

The proposed TD3 agents were built in Python using TensorFlow 2 and due to the relatively small size of the TD3 networks, only the CPU was utilized for training instead of a GPU that may suit larger networks. The hyperparameters of the TD3 algorithms were optimized through a series of exhaustive simulations and only the tuned hyperparameters were used to demonstrate the results.

4.2. Quantification of the Enhanced HC

The HC assessed in this section indicates the extent of HC enhancement through coordinated voltage control. The proposed methodology to quantify the enhanced HC is summarized in Algorithm 2. The TD3 agent for HC assessment was trained using the time series Data Set 1 that consisted of 5760 time steps. For each time step, 5 episodes were evaluated, resulting in a total of 28,800 learning episodes. Upon successful training of the TD3 agent, the cumulative reward of every 48 time steps (1 day) converged to a high value. Once the agent learned an optimal policy at the end of the learning process, the weight parameters of the TD3 networks were saved. The learning curve of the TD3 agent used for the HC assessment is given in Figure 8.

The HC for this study was evaluated individually for each customer at every time step. In other terms, the HC for a customer was defined as the maximum PV inverter rating that a customer is allowed to connect to the network without causing any network operational constraint violations. To evaluate the HC of all the customers, a time series power flow simulation was undertaken using Data Set 2, which consisted of forecasted data, and the TD3 agent was initialized with the saved weight parameters from the learning process. At every 30 min time step of the simulation, the actor network of the TD3 agent predicted the PV inverter rating, active power output, and reactive power output of all the customers. The PV inverter rating predicted by the TD3 agent was identified as the HC of a particular customer at a given instance of time. For the simplicity of analysis, the minimum of the set of PV inverter rating sizes decided by the TD3 agent throughout the time series simulation can be taken as a single quantified HC value for a customer. The identified HC values for each customer by taking the minimum of the PV ratings throughout the time series simulation are given in Figure 9.

Algorithm 2. Summary of the enhanced HC assessment methodology-Training and evaluation process of TD3 agent for HC assessment
1:	Collect data sets 1 and 2. Define ${\lambda }_1,{\lambda }_2=0.1$
2:	Define the values for reward parameters $\{\delta ,\eta \}$ and hyperparameters $\{\alpha ,\beta ,\gamma ,\sigma ,\rho ,Batch size\}$
3:	Initialize ${TD3}_{HC}$ agent for HC assessment with random weights $w_{HC}=\{{\theta }_1$$,{\theta }_2,\varphi \}$
4:	Using Data Set 1: for $t=1$$\mathrm{to} T$ do:
5:	Take action $a \leftarrow { \pi }_{\varphi }\left(s\right)$ and execute power flow calculation for time step $t$
6:	Train the ${TD3}_{HC}$ agent (steps 6 to 17 in Algorithm 1)
7:	end for
8:	$\mathrm{Save} \mathrm{trained} \mathrm{weights} w_{HC}$
9:	$\mathrm{Initialize} \mathrm{a} \mathrm{new} {TD3}_{HC}$$\mathrm{agent} \mathrm{with} \mathrm{trained} \mathrm{weights} w_{HC}$ for HC evaluation
10:	$\mathrm{Using} \mathrm{Data} \mathrm{Set} 2: \mathbf{for} t=1$$\mathbf{to} T$ do:
11:	Take action $a \leftarrow { \pi }_{\varphi }\left(s\right)$ and execute power flow calculation for time step $t$
12:	$\mathrm{Check} \mathrm{for} \mathrm{operational} \mathrm{constraint} \mathrm{violations} \mathrm{and} \mathrm{save} S_{PV}\left(i,t\right)$$\mathrm{where}; i=1, 2, 3,\dots ,N$
13:	end for
14:	if total operational constraint violations = 0 then:
15:	Identify $\mathrm{the} \mathrm{hosting} \mathrm{capacity} \mathrm{for} \mathrm{each} \mathrm{customer} i$ ${HC}_i=\underset{t\in [1,T]}{\min }{S}_{PV}\left(i,t\right)$
16:	else: go back to step 2
17:	end if

For this case study, the two fairness parameters ${\lambda }_1$ and ${\lambda }_2$ were both defined as 0.1, which ensures that the PV ratings of customers did not vary more than 10% from the mean of all customer PV ratings, and the minimum possible active power output of the PV inverters was clipped to 90% of the available active power. The PV ratings and the active power outputs of the customers estimated by the TD3 algorithm for a single day are illustrated in Figure 10. It should be noted that during periods of low solar irradiation, the PV ratings estimated by the TD3 agent reach the ${HC}_{max}$ value defined in the reward function. The ${HC}_{max}$ parameter prevents the TD3 agent from estimating unrealistically high PV ratings during periods of low solar irradiation to maximize its active power output. The enhanced HC quantified in this study with coordinated voltage control is defined by the first violation of operational constraints. The TD3 agent attempts to learn a policy that does not violate any operational constraints at any given instance of time. The evaluation of the trained TD3 agent was performed by checking for any operational constraint violations throughout the time series simulation that used Data Set 2. If no operational constraint violations were observed, the TD3 agent was considered to be trained successfully. If the actions taken by the TD3 agent led to one or more operational constraint violations, the training procedure had to be performed again with different parameters.

4.3. Training and Performance Testing of the TD3 Agent Used for Coordinated Voltage Control

Another TD3 agent was developed to accurately simulate the performance of the proposed coordinated voltage control scheme in a real-world LV distribution network. For the simulations performed in this section, the PV inverter ratings of customers identified through the HC assessment in Section 4.2 are used to further demonstrate the performance of the proposed algorithm when the LV network is operating at its HC limit. A summary of the methodology used for the training and evaluation process of the proposed TD3 agent for coordinated voltage control is given in Algorithm 3.

Algorithm 3. Summary of the training and performance evaluation methodology for the proposed coordinated voltage control scheme-Training and evaluation process of TD3 agent for coordinated voltage control
1:	$\mathrm{Collect} \mathrm{data} \mathrm{sets} 1 \mathrm{and} 3. \mathrm{Define} {\lambda }_2=0.1$
2:	Set the PV ratings of customers to their HC limit identified in step 15 of algorithm 2
3:	$\mathrm{Define} \mathrm{the} \mathrm{values} \mathrm{for} \mathrm{reward} \mathrm{parameters} \{\delta ,\eta \}$$\mathrm{and} \mathrm{hyperparameters} \{\alpha ,\beta ,\gamma ,\sigma ,\rho ,Batch size\}$
4:	$\mathrm{Initialize} {TD3}_{VC}$$\mathrm{agent} \mathrm{for} \mathrm{coordinated} \mathrm{voltage} \mathrm{control} \mathrm{with} \mathrm{random} \mathrm{weights} w_{VC}=\{{\theta }_1$$,{\theta }_2, \varphi \}$
5:	$\mathrm{Using} \mathrm{Data} \mathrm{Set} 1: \mathbf{for} t=1$$\mathbf{to} T$ do:
6:	Take action $a \leftarrow { \pi }_{\varphi }\left(s\right)$$\mathrm{and} \mathrm{execute} \mathrm{power} \mathrm{flow} \mathrm{calculation} \mathrm{for} \mathrm{time} \mathrm{step} t$
7:	Train the ${TD3}_{VC}$ agent (steps 6 to 17 in Algorithm 1)
8:	end for
9:	$\mathrm{Save} \mathrm{trained} \mathrm{weights} w_{VC}$
10:	$\mathrm{Initialize} \mathrm{a} \mathrm{new} {TD3}_{VC}$$\mathrm{agent} \mathrm{with} \mathrm{trained} \mathrm{weights} w_{VC}$ for performance evaluation
11:	$\mathrm{Using} \mathrm{Data} \mathrm{Set} 3: \mathbf{for} t=1$$\mathbf{to} T$ do:
12:	Take action $a \leftarrow { \pi }_{\varphi }\left(s\right)$$\mathrm{and} \mathrm{execute} \mathrm{power} \mathrm{flow} \mathrm{calculation} \mathrm{for} \mathrm{time} \mathrm{step} t$
13:	Check for operational constraint violations
14:	end for
15:	if total operational constraint violations $\ne$ 0, then: go back to step 3, end if

To attain the optimal performance for coordinated voltage control, the values for the TD3 hyperparameters $\{\alpha ,\beta ,\gamma ,\sigma ,\rho ,Batch size\}$ and the reward function parameters $\{\delta ,\eta \}$ were reconfigured. The TD3 agent was trained using Data Set 1, as it is essential to experience a wide range of states that includes seasonal variations in customer loads and solar irradiation to learn a robust and an optimal policy. The neural network weights of the TD3 agent were saved once the training simulation was completed. A new TD3 agent was initialized using the saved neural network weights to evaluate the performance of the learned policy by undertaking a time series simulation using Data Set 3. A real-world implementation of the proposed coordinated voltage control scheme requires control actions at a very high time step resolution. However, undertaking a time series simulation with a high time step resolution is often associated with a heavy computational burden. Therefore, a compromise was reached between computational capabilities and simulation accuracy by selecting Data Set 3 with a 5 s time step resolution to analyze the performance. If no operational constraint violations were observed throughout the time series simulation, the training process of the TD3 agent was determined to be successful. If one or more operational constraint violations existed throughout the time series simulation, the TD3 agent had to be trained again with a different set of values for hyperparameters and reward function parameters.

4.4. Comparative Analysis of Different Voltage Control Methods

In this section, a comparative analysis of the TD3 algorithm with several other control algorithms is presented using the HC limits defined in Section 4.2. The performance of the proposed TD3-based coordinated voltage control scheme is compared with the following five cases:

Base case: The base case considered in this analysis represents a hypothetical scenario in which PV inverters are operated without any control mechanisms. The PV inverters were allowed to output all available active power without considering any operational constraint violations. The base-case scenario was used to benchmark the TD3 algorithm and other control algorithms presented in the comparative study.

Base case with inverter trip enabled: The control method implemented for this scenario is the tripping action of the PV inverters when the average voltage of any 10 min window reaches higher than the overvoltage limit at the POC. The inverter trip timeout delay was considered to be 2 min. At normal conditions, the inverter outputs maximum available active power.

Local Volt-Watt and Volt-VAr control: Volt-Watt and Volt-VAr controls were implemented with settings from the AS/NZS 4777.2:2020 standard. In this case, the local voltage control was set to reactive power priority mode. In essence, the Volt-VAr function is prioritized over the Volt-Watt function during periods of high solar irradiation.

Coordinated voltage control using DDPG: A deep deterministic policy gradient (DDPG) is an RL algorithm that employs a deterministic policy to find the optimal solutions. DDPG is an actor–critic algorithm that consists of a total of 4 neural networks: actor, target actor, critic, and target critic.

Coordinated voltage control using SAC: Soft actor critic (SAC) is a state-of-the-art RL algorithm that is used to perform continuous control in various domains. SAC learns a stochastic policy which outputs a probability distribution over actions given a state. Entropy regularization is a key feature of SAC which encourages more diverse exploratory actions, in contrast to the DDPG and TD3 algorithms, which encourage exploration by adding noise to the actions. The SAC algorithm used for this study consisted of a total of 5 feed-forward neural networks, each representing actor, critic-1, critic-2, the value network, and the target value network.

A time series simulation using Data Set 3 was carried out to evaluate the performance of the above-mentioned algorithms. The RL algorithms SAC and DDPG were trained using Data Set 1 to capture all seasonal variations in the environment. The learning curves of RL algorithms can be used to analyze their performance level given that the reward function has been strongly formulated. The learning curves for the TD3, DDPG, and SAC algorithms are given in Figure 11, demonstrating the progression of the daily reward received by the agent due to its control actions. The DDPG algorithm gives the worst performance of the three RL algorithms as it converges to a significantly lower reward at the end of the training simulation. The algorithms SAC and TD3 show a better performance, with TD3 converging to a slightly higher reward than SAC at the end of the training simulation.

The customer voltage profiles of the time series simulations given in Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 illustrate which control algorithms are more effective at voltage regulation. The red dotted line indicates the maximum voltage limit of 258 V, the continuous lines correspond to each customer 10 min moving average voltage, and the blue hue represents the instantaneous voltage spread among customers. The remaining coloured lines in Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17 are the voltage profiles of each individual customer. The base case, base case with inverter tripping enabled, Volt-VAr/Volt-Watt control, and DDPG control scenarios resulted in voltage constraint violations during periods of high solar irradiation. The inverter tripping can be seen in the voltage profile of the scenario of the base case with the inverter trip enabled. It can clearly be seen from the Volt-VAr/Vol-Watt scenario voltage profile that local voltage control becomes less effective during periods of high PV generation due to its inability to prevent the tripping of inverters, leading to significant curtailments. The SAC and TD3 DRL algorithms did not cause any overvoltage conditions and managed to reduce the deviation of voltage from its nominal value as compared to the base case scenario. The TD3 control scenario has the best voltage profile out of the six algorithms. The maximum instantaneous customer voltage reached for the TD3 control scenario was approximately 256 V, which is well below the overvoltage limit of 258 V.

The magnitude of total active power curtailments provides another indication for evaluating the performance of algorithms. Figure 18 illustrates the total energy curtailed for all scenarios throughout the time series simulation. It shows that the TD3 algorithm outperforms other control algorithms, resulting in zero energy curtailed throughout the day. The SAC algorithm results in the maximum amount of energy curtailed despite having no instances of inverter tripping. The DDPG algorithm also results in significantly high curtailments followed by the Volt-VAr/Volt-Watt control functions.

Voltage unbalance is a vital aspect of the power quality of the distribution network which must be scrutinized in the time series simulations for each scenario. The 10 min moving average of the maximum percentage voltage unbalance ($VUR\%$) throughout the LV distribution network for the time series simulation of each scenario is given in Figure 19. Each control algorithm significantly increases the network unbalance compared with the base-case scenario. The maximum $VUR\%$ appears to occur for the TD3 control scenario; however, it is lower than the compatibility limit of 2% specified by the IEC/TR 61000-3-14 standard [53]. It is desirable to explicitly factor in network unbalance for the proposed coordinated voltage control approach. However, data on the phase connections of customers in every feeder are not always readily available when implementing such control schemes in real-world electricity distribution networks. Therefore, further research is needed to develop effective solutions that address this limitation and ensure that the $VUR\%$ is maintained within acceptable limits.

5. Discussion

The TD3 algorithm presented in this paper has been subjected to rigorous tuning of hyperparameters and different formulations of the reward function to achieve the performance level presented in the above section. Deterministic algorithms such as DDPG and TD3 are sensitive to hyperparameters and incorrect assignments of hyperparameters will cause divergence in the learning process. Contrarily, algorithms that train stochastic policies such as SAC are less sensitive to hyperparameters and experience faster convergence. However, the SAC algorithm excels primarily in domains that require a high degree of exploration with control actions leading to sparse rewards. In power distribution systems, the cost of suboptimal control actions is significantly high and deterministic algorithms such as TD3 outperform SAC in the aspects of control action precision and fine-tuning capabilities.

The reward function parameter $\delta$ that regulates the balance between voltage regulation and active power curtailments, and the parameter $\eta$, which is the penalty factor for operational constraint violations, are fine-tuned for the TD3 algorithm to ensure convergence to a higher reward during the learning process. It should be noted that the reward function parameters $\delta$ and $\eta$ were not finely tuned specifically for the SAC and DDPG algorithms, but rather the tuned values for the TD3 algorithm were used instead to represent a fair comparison between the learning curves of each DRL algorithm. The inclusion of the penalty factor $\eta$ in the reward function generally caused the DRL algorithms to converge to a more conservative policy, as seen in the performance results of SAC and DDPG algorithms. However, this effect can be minimized by fine tuning of the parameter $\delta$, which is evident from the performance results of TD3 that led to zero active power curtailments and zero operational constraint violations. Provided that the reward function parameters are finely tuned, the SAC and DDPG algorithms may result in better voltage regulation and minimal active power curtailments than the results presented in Section 4.

The main advantage of off-policy algorithms such as TD3 is their high sample efficiency, since the training samples are generated from a different behavioural policy rather than the current policy. A high sample efficiency leads to reduced training data requirements, faster convergence, and improved generalization, capturing the dynamics of the environment more effectively. The data from smart meter measurements are not always possible to obtain without any gaps in the time series data. The impact of missing time series data points is minimized by experience replay in DRL algorithms. Experience replay is a commonly used technique in almost every DRL algorithm that eliminates the temporal correlations present in consecutive training data sets. Prioritized experience replay is an improvement of the traditional experience replay technique that enhances learning efficiency by assigning a high probability to important training samples in the replay buffer. However, the case study presented in this paper only considered a relatively small training data set and prioritized experience replay did not result in any significant improvements to the learning process.

Coordinated voltage control approaches such as the one proposed in this paper offer numerous advantages for distribution networks. However, they face challenges in implementation and operation stages, including the need for communication infrastructure, and possible communication breakdowns or interruptions that may lead to unnecessary curtailments and operational constraint violations. Also, a real-world implementation of coordinated voltage control is dependent on the accuracy of smart meter measurements.

6. Conclusions

The research work presented in this paper consists of two components: a coordinated voltage control scheme implemented using the TD3 algorithm and the quantification of the enhanced HC due to the proposed coordinated voltage control scheme. The presented methodology for HC assessment utilizes a TD3 agent to evaluate the enhanced HC in real time for each individual PV customer connected to the electricity distribution network. The proposed TD3 agent for coordinated voltage control is rigorously tuned to achieve its maximum performance level and follows a different design from that of the HC assessment. The performance of the proposed TD3 algorithm is evaluated using a QSTS simulation of a 28-customer LV distribution network with 100% PV penetration. A comparative analysis of six different voltage control schemes is presented featuring other DRL algorithms, existing local voltage control functions, and the scenarios in which no voltage regulation is performed. In the comparative analysis, a QSTS simulation is utilized to evaluate the customer voltage profiles, total active power curtailments, and maximum percentage of voltage unbalance of each algorithm.

Evaluating the enhanced HC due to a coordinated voltage control scheme provides a deeper perspective on the techno-economic benefits of such schemes. Future work for the research presented in this paper will include a case study of a larger combined MV and LV distribution network. This study will incorporate additional control elements of the distribution network, such as OLTCs, battery energy storage systems, and static VAR systems, into the proposed coordinated voltage control scheme.