Deep Reinforcement Learning-Based Voltage Regulation Using Electric Springs in Active Distribution Networks

Abstract

The increasing penetration of distributed generation in active distribution networks (ADNs) introduces significant voltage regulation challenges due to the intermittent nature of renewable energy sources. Electric springs (ESs) have emerged as a cost-effective alternative to conventional FACTS devices for voltage regulation, requiring minimal energy storage while providing fast, flexible reactive power compensation. This paper proposes a deep reinforcement learning (DRL)-based approach for voltage regulation in balanced active distribution networks with distributed generation. Electric springs are deployed at selected buses in series with noncritical loads to provide flexible voltage support. The main contributions of this work are: (1) a novel region-based penalized reward function that effectively guides the DRL agent to minimize voltage deviations; (2) a coordinated control strategy for multiple ESs using the Deep Deterministic Policy Gradient (DDPG) algorithm, representing the first application of DRL to ES-based voltage regulation; (3) a systematic hyperparameter tuning methodology that significantly improves controller performance; and (4) comprehensive validation demonstrating an approximately 40% reduction in mean voltage deviation relative to the no-control baseline. Three well-known continuous-control DRL algorithms, Twin Delayed Deep Deterministic Policy Gradient (TD3), Proximal Policy Optimization (PPO), and DDPG, are first evaluated using the default hyperparameter configurations provided by MATLAB R2022b.Based on this baseline comparison, a dedicated hyperparameter-tuning procedure is then applied to DDPG to improve the robustness and performance of the resulting controller. The proposed approach is evaluated through simulation studies on the IEEE 33-bus and IEEE 69-bus test systems with time-varying load profiles and fluctuating renewable generation scenarios.

1. Introduction

Traditional power grids have undergone significant evolution in recent years due to the increasing integration of distributed generation (DG) resources, transforming them into ADNs characterized by bidirectional power flows and dynamic, time-varying operating conditions. Among the key advantages of this new paradigm are reduced transmission losses, enhanced resilience to power outages, and lower greenhouse gas emissions [1].

However, the high penetration of renewable energy sources introduces significant voltage regulation challenges due to their inherent intermittency and variability. Voltage regulation in ADNs has been extensively investigated using various control strategies, each characterized by distinct levels of coordination and communication architecture [2]. In addition, many of these schemes incorporate intelligent control techniques to cope with the increasing complexity and uncertainty of modern power systems. Among the most widely used approaches are metaheuristic algorithms and reinforcement learning methods, which enable real-time system optimization under dynamic, nonlinear operating conditions. The controlled elements include renewable energy sources, Flexible AC Transmission Systems/Distributed Flexible AC Transmission Systems (FACTS/D-FACTS), compensation devices, electric vehicles, and energy storage systems (ESSs), all aimed at maintaining voltage stability and enhancing overall system performance [3].

In particular, metaheuristic algorithms have been applied in both local and centralized voltage-control frameworks to tune reactive-power compensator setpoints and transformer tap positions [4,5]. Nevertheless, these methods exhibit significant limitations in real-time applications. They rely on static parameters, lack online feedback, incur high computational costs, and require manual tuning of algorithm hyperparameters. These factors reduce their robustness in the face of changing scenarios.

1.1. Deep Reinforcement Learning for Voltage Control

Deep reinforcement learning algorithms have gained relevance in power systems applications due to their capacity to learn optimal policies directly from interaction with the environment, without requiring an explicit model of the network [3]. Recent studies in energy applications have also reported model-free learning-based control strategies designed to handle uncertainties [6,7,8]. Unlike traditional optimization methods, DRL agents can adapt to changing conditions in real time and handle the high-dimensional state and action spaces typical of modern ADNs.

Several DRL algorithms have been successfully applied to voltage regulation problems. The DDPG algorithm [9] enables continuous control by combining deep neural networks with actor–critic architectures, making it suitable for reactive power dispatch applications. Cao et al. [2] demonstrated the effectiveness of multi-agent DRL for decentralized voltage control with high PV penetration. The TD3 algorithm [10] addresses overestimation issues in DDPG via twin critics and delayed policy updates, improving learning stability. More recently, the Soft Actor–Critic (SAC) algorithm has been applied to hierarchical Volt/VAR control, achieving robust performance under varying network conditions [11]. Additionally, PPO [12] has emerged as an alternative to policy optimization with improved sample efficiency and stability.

DRL has been successfully applied to tasks such as demand response management [13], ESS management [14], voltage control in ADNs [15], and hybrid schemes such as hierarchical Volt/VAR control using the SAC algorithm [11]. Perera and Kamalaruban [16] provide a comprehensive review of DRL applications in energy systems, highlighting the growing adoption of these techniques for real-time control problems.

1.2. Voltage Regulation Devices: From FACTS to Electric Springs

Along with these methodological advances, innovations in voltage regulation devices have emerged, with the electric spring (ES) standing out in recent years. This device enables flexible, real-time voltage regulation that adapts to load and generation variability [17]. From an operating-principle perspective, ESs can provide fast-acting, distributed voltage support suitable for real-time operation, similar to devices such as ESSs and Static Synchronous Compensators (STATCOMs) [14,17]. The ES operates by modulating the voltage across a series-connected noncritical load, effectively creating a “smart load” that can exchange reactive power as needed. This operating principle enables voltage regulation at the point of common coupling (PCC) and supports demand-side management. Note that this work does not aim to compare different grid-support devices (STATCOMs, D-FACTS, ESSs), but rather to systematically compare and tune different deep reinforcement learning techniques. In this context, the ES is adopted as a representative case study due to its nonlinear dynamics and demand-side implementation.

Table 1. Comparison of voltage regulation approaches in ADNs.

Characteristic	STATCOM [17]	ESS [14]	Electric Spring [17]
Energy storage required	No	Yes (large)	No (minimal DC-link)
Capital cost	High	Very high	Moderate
Response time	Fast	Medium	Fast
Scalability	Limited	Limited	High
Active power capability	Limited	Yes	No (reactive only)
Installation complexity	High	Moderate	Low

summarizes the main characteristics of different voltage regulation approaches, highlighting the advantages of ES-based solutions.

Cooperative and centralized schemes have been proposed using multiple ESs, employing algorithms such as Particle Swarm Optimization (PSO) and Genetic Algorithm (GA) to optimize their operation and mitigate voltage deviations [18]. Chen et al. [19] developed distributed cooperative control strategies for multiple DC electric springs, demonstrating the feasibility of coordinated ES operation in distribution networks. More recently, Saha and Dutta [20] extended ES applications to combined voltage–frequency regulation in multi-area power systems, demonstrating the versatility of this technology.

Akhtar et al. [21] demonstrated that smart loads with ESs can contribute to primary frequency control through reactive compensation, while Luo et al. [17] provided a detailed comparison between ES-based distributed voltage control and STATCOM, concluding that ESs offer comparable performance at lower cost for distribution network applications.

1.3. Research Gap and Contributions

Despite the advances achieved, most existing approaches using metaheuristic algorithms and ES devices for voltage control still exhibit significant limitations for real-time operation and in high-uncertainty contexts.

In contrast, DRL algorithms offer clear advantages in dynamic environments, providing faster response times and more precise voltage control. However, despite advances in both DRL-based control and ES technology, no prior studies have addressed their combined application for voltage control in ADNs.

This gap is significant because: (1) ESs offer a cost-effective alternative to conventional FACTS devices for voltage regulation, (2) DRL provides the adaptive, model-free control capability needed to handle the uncertainty of renewable generation, and (3) the combination of both technologies has the potential to achieve superior voltage regulation performance while minimizing infrastructure costs.

This paper presents a centralized voltage control scheme based on the DDPG algorithm for the coordinated operation of multiple ESs in an ADN. The main contributions of this work are as follows:

• A novel region-based penalized reward function that effectively guides the DRL agent to minimize voltage deviations while avoiding aggressive control actions.
• A coordinated control strategy for multiple ESs using the DDPG algorithm, representing the first application of DRL techniques to ES-based voltage regulation in ADNs.
• A systematic hyperparameter tuning methodology based on grid search that significantly improves the performance and robustness of the DDPG controller compared to default configurations.
• Comprehensive validation through simulation studies on the IEEE 33-bus and IEEE 69-bus test systems, demonstrating approximately a 40% reduction in mean voltage deviation compared to the no-control baseline.

The control agent is trained and validated through offline simulations under multiple load variations and active-power-injection scenarios. The reward function is explicitly formulated to minimize voltage deviation at the buses. The results demonstrate the feasibility and effectiveness of the proposed method, highlighting its ability to provide fast, robust regulation in the face of fluctuations and dynamic changes in the system.

The remainder of this paper is organized as follows: Section 2 presents the theoretical background on ES modeling and DRL algorithms. Section 3 describes the proposed framework and methodology. Section 4 details the case study setup. Section 5 presents and analyzes the simulation results. Section 6 discusses the findings, limitations, and future directions. Finally, Section 7 concludes the paper.

2. Modeling and Theoretical Background of the Electric Spring and the DDPG Algorithm

This section establishes the theoretical basis of this work. First, it presents the steady-state mathematical model of the ES, followed by a concise overview of the DDPG algorithm equations.

2.1. Electric Spring Operation for Voltage Regulation in Active Distribution Networks

The power stage of the ES comprises a DC/AC PWM inverter, a DC-link capacitor, and a low-pass output filter. Since the DC-link capacitor is the sole energy buffering element, the system operates exclusively through reactive power exchange. For analytical simplicity, the ES is modeled as an ideal controlled AC voltage source.

The ES is connected in series with a constant-impedance load $Z_{\mathrm{NC},j}$, referred to as the noncritical load (NCL), forming the smart load (SL). The capacity of this SL depends on the ES output voltage $V_{\mathrm{ES},j}$ and the impedance $Z_{\mathrm{NC},j}$. Figure 1 illustrates the connection of the SL to an arbitrary bus j of the ADN, in parallel with a ZIP load representing the critical load (CL). These elements are connected to the slack bus represented by a Thevenin equivalent with voltage (${\mathit{V}}_G$), while $Z_{ij}$ denotes the impedance of the distribution line connecting buses i and j. The current phasors include the ZIP load current (${\mathit{I}}_{ZIP}$), the distribution line current (${\mathit{I}}_{ij}$), the current in the NCL (${\mathit{I}}_{N{C}_j}$), and the current flowing to a further connection bus (${\mathit{I}}_{jk}$). Furthermore, ${\mathit{V}}_j$ represents the PCC voltage for the SL. ES activation is defined by the binary variable $A_j\in \{0,1\}$, where $A_j=1$ denotes activated and $A_j=0$ denotes deactivated.

Reference [23] outlines the procedure for establishing the steady-state single-phase ES model, demonstrating high reliability when integrated with the backward–forward sweep method (BFSM). This process aims to solve the ADN iteratively by updating the voltage and current values, computing the ES model, and performing power flow analysis. The accuracy of the proposed model is not evaluated in this work, as it has been addressed and validated in prior studies [23,24]. In this study, the ES is used as a test platform to evaluate the performance and adaptability of different DRL-based control strategies, rather than as a means to assess or benchmark grid-support technologies.

This work uses a ZIP load model for the CL. The equations for active power ($P_j$) and reactive power ($Q_j$) are given in (1) and (2), respectively. In particular, this model employs parameters $f_{1j}-f_{6j}$ as weighting factors for the ZIP load components (namely, constant-impedance, constant-current, and constant-power), subject to the normalization constraints $f_{1j}+f_{2j}+f_{3j}=1$ and $f_{4j}+f_{5j}+f_{6j}$ = 1.

For this model, a constant-power ZIP load is used. The parameters $P_{Oj}$, and $Q_{Oj}$ denote the rated active and reactive power, respectively, while $V_F$ specifies the nominal voltage magnitude of the CL.

$$P_j=P_{Oj}\left[f_{1j}+f_{2j}\left({\displaystyle \frac{V_j}{V_F}}\right)+f_{3j}{\left({\displaystyle \frac{V_j}{V_F}}\right)}^2\right],$$

(1)

$$Q_j=Q_{Oj}\left[f_{4j}+f_{5j}\left({\displaystyle \frac{V_j}{V_F}}\right)+f_{6j}{\left({\displaystyle \frac{V_j}{V_F}}\right)}^2\right].$$

(2)

From the apparent power components of the CL, the current phasor ${\mathit{I}}_{ZIP}$ is obtained as shown in Equation (3), in which $Y_{ZIP}$ denotes the overall admittance of the ZIP load.

$${\mathit{I}}_{ZIP}=\left({\displaystyle \frac{P_j-j{Q}_j}{{\mathit{V}}_j{\left({\mathit{V}}_j\right)}^{*}}}\right){\mathit{V}}_j=Y_{ZIP}{\mathit{V}}_j.$$

(3)

Similarly, ${\mathit{I}}_{jk}$ is computed using the admittance form, as depicted in Equation (4), where the equivalent admittance of the line segment $j-k$ is represented by $Y_{jk}$.

$${\mathit{I}}_{jk}=\left({\displaystyle \frac{{\mathit{I}}_{jk}}{{\mathit{V}}_j}}\right){\mathit{V}}_j=Y_{jk}{\mathit{V}}_j.$$

(4)

During the iterative solution process, Equation (5) computes the phasor ${\mathit{V}}_G$. Both its magnitude and angle are dynamically adjusted in response to voltage deviations at the PCC, variations in the distribution line parameters, and changes in the equivalent admittances $Y_{ZIP}$ and $Y_{jk}$.

$${\mathit{V}}_G={\mathit{V}}_j\left|1+Z_{ij}\left(Y_{ZIP}+Y_{jk}\right)\right|.$$

(5)

Equation (6) computes the phase angle ${\phi }_{1j}$ between ${\mathit{V}}_j$ and ${\mathit{V}}_G$ as stated in the geometric relationships shown in [22]. The angle ${\phi }_{0j}$ associated with the series impedance of the distribution line is calculated in Equation (7) as follows:

$${\phi }_{1j}=arg\left(1+Z_{ij}\left(Y_{ZIP}+Y_{jk}\right)\right),$$

(6)

$${\phi }_{0j}=ta{n}^{-1}\left({\displaystyle \frac{X_{ij}}{R_{ij}}}\right).$$

(7)

The complex impedance $Z_{ij}$ consists of resistive and inductive components, $R_{ij}$ and $X_{ij}$, respectively. Additionally, Equation (8) incorporates $a_j$ and $b_j$ from Equations (9)–(11) to compute the angle ${\theta }_j$. This angle is essential for satisfying the analytical model and ensuring that the ES operates in reactive mode, along with the operating conditions ($a_j$ and $b_j$) at the PCC.

These operational conditions define the ES’s output voltage limits. Specifically, the primary operational boundary for the ES is governed by the control variable $m_j$, constrained to the range −1 to 1, where $m_j=-1$ corresponds to full capacitive mode and $m_j=1$ denotes full inductive mode. Depending on the active and reactive power at the bus, a portion of the load is classified as NCL, which, in turn, sets the maximum power the ES can supply.

$${\theta }_j=si{n}^{-1}\left(m_j\right)-ta{n}^{-1}\left({\displaystyle \frac{b_j}{a_j}}\right),$$

(8)

$$a_j={\displaystyle \frac{V_G{V}_j}{R_{N{C}_j}}}{K}_{j}cos({\phi }_{0j}+{\phi }_{1j}),$$

(9)

$$b_j={\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{V_j{K}_j}{R_{N{C}_j}}}\right)}^2+{\displaystyle \frac{V_G{V}_j}{R_{N{C}_j}}}{K}_{j}sin({\phi }_{0j}+{\phi }_{1j}),$$

(10)

$$K_j=\sqrt{{\left(R_{jk}\right)}^2+{\left(X_{jk}\right)}^2}.$$

(11)

To correctly integrate the SL with the BFSM, both the magnitudes and phase angles of ${\mathit{I}}_{{NC}_j}$ and ${\mathit{V}}_{E{S}_j}$ must be determined. First, the magnitude is computed, as shown in (12) and (13).

$$I_{{NC}_j}={\displaystyle \frac{V_j}{R_{N{C}_j}}}cos({\theta }_j/2),$$

(12)

$$V_{E{S}_j}=V_{j}sin({\theta }_j/2)\pm I_{N{C}_j}{X}_{N{C}_j}.$$

(13)

The resistance and reactance of $Z_{NCj}$ are represented by $R_{NCj}$ and $X_{NCj}$, respectively. In (13), the result changes sign depending on the power factor of the NCL: it is positive when ${\mathit{I}}_{{NC}_j}$ lags its voltage and negative when it leads. The phase angles are determined using the angle of ${\mathit{V}}_j$ (${\lambda }_j$). For ${\mathit{I}}_{N{C}_j}$, the phase angle is calculated as ${\theta }_{I_{N{C}_j}}={\lambda }_j-{\displaystyle \frac{{\theta }_j}{2}}$ and for ${\mathit{V}}_{E{S}_j}$, the phase angle is ${\theta }_{E{S}_j}={\lambda }_j-{\displaystyle \frac{{\theta }_j}{2}}+sign\left({\theta }_j\right){\displaystyle \frac{\pi }{2}}$. These correlations ensure that the ES operates in reactive compensation mode and confirm the validity of the mathematical model. The electrical variables of the ES model are calculated so that it can be integrated into the BFSM, enabling the calculation of the current ${\mathit{I}}_{ij}$, as shown in (14), through the backward sweep.

$${\mathit{I}}_{ij}={\mathit{I}}_{N{C}_j}+{\mathit{I}}_{jk}+{\mathit{I}}_{ZIP}.$$

(14)

2.2. Reinforcement Learning DDPG Algorithm

An RL system trains one or more agents to interact with an environment E iteratively at discrete time steps t. At each step, the agent receives an observation $x_t$, selects an action $a_t$, and receives a scalar reward $r_t$.

In power systems, DRL is particularly well-suited to complex, nonlinear, and time-varying control problems, such as voltage regulation in ADNs. In this work, we employ a deterministic policy within the DDPG framework [9]. This policy allows the direct adoption of the corresponding deterministic Bellman recursion [25]. Thus, for a policy $\mu :\mathcal{S}\to \mathcal{A}$, the action–value function $Q^{\mu }(s,a)$ satisfies (15):

$$Q^{\mu }(s_t,a_t)={\mathbb{E}}_{r_t,s_{t+1}\sim E}\left[r(s_t,a_t)+\gamma Q^{\mu }\left(s_{t+1},\mu \left(s_{t+1}\right)\right)\right].$$

(15)

As a result, the expectation depends only on the environment dynamics, enabling off-policy learning of $Q^{\mu }$ using transitions generated by a different stochastic behavior policy $\beta$.

In the DPG framework, a deterministic policy $\mu \left(s\right|{\theta }_{\mu })$ represents the actor, mapping each state to a specific action, while the critic $Q(s,a|{\theta }^Q)$ is trained using the Bellman equation above. The actor is then updated by performing a gradient descent on the expected return. The deterministic policy gradient is given by (16):

$${\nabla }_{{\theta }_{\mu }}J\approx {\mathbb{E}}_{s_t\sim {\rho }^{\beta }}\left[{\nabla }_{{\theta }_{\mu }}Q\left(s,a|{\theta }^Q\right)|s=s_t,a=\mu \left(s_t\right|{\theta }^{\mu })\right].$$

(16)

2.2.1. DDPG Architecture and Key Components

The DDPG algorithm [9] extends the Deterministic Policy Gradient (DPG) framework by incorporating deep neural networks to approximate both the actor and critic functions. The key components of DDPG are:

• Actor Network $\mu \left(s\right|{\theta }^{\mu })$: A neural network that maps states to continuous actions. The output layer uses a tanh activation function to bound actions within the valid range $[-1,1]$, which in this application corresponds to the ES control variable $m_j$.
• Critic Network $Q(s,a|{\theta }^Q)$: A neural network that estimates the action-value function. The critic takes both the state and action as inputs and outputs a scalar Q-value representing the expected cumulative reward.
• Experience Replay Buffer: A finite-sized buffer that stores transition tuples $(s_t,a_t,r_t,s_{t+1})$ from agent-environment interactions. During training, mini-batches are uniformly sampled from this buffer to break temporal correlations and improve learning stability.
• Target Networks: Separate copies of the actor (${\mu }^{\prime }$) and critic ($Q^{\prime }$) networks with parameters ${\theta }^{{\mu }^{\prime }}$ and ${\theta }^{Q^{\prime }}$, respectively. These target networks are updated slowly using soft updates:

  $${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime },$$

  (17)

  where $\tau \ll 1$ is the target smoothing factor. This mechanism stabilizes learning by providing consistent target values for the Bellman backup.
• Exploration Noise: To encourage exploration, noise is added to the actor’s output during training. DDPG typically uses an Ornstein–Uhlenbeck (OU) process to generate temporally correlated noise suitable for physical control tasks.

The critic is trained by minimizing the mean squared Bellman error:

$$L\left({\theta }^Q\right)={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[{\left(Q(s,a|{\theta }^Q)-y\right)}^2\right],$$

(18)

where $\mathcal{D}$ is the replay buffer and the target y is computed as:

$$y=r+\gamma Q^{\prime }(s^{\prime },{\mu }^{\prime }\left(s^{\prime }\right|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }}).$$

(19)

2.2.2. Twin Delayed DDPG (TD3) Algorithm

The TD3 algorithm [10] addresses several limitations of DDPG, particularly the tendency to overestimate Q-values, which can lead to suboptimal policies. TD3 introduces three key modifications:

• Clipped Double Q-Learning: TD3 maintains two critic networks $Q_1$ and $Q_2$, each with its own target networks. The target value uses the minimum of the two critics to reduce overestimation:

  $$y=r+\gamma \underset{i=1,2}{min}{Q}_i^{\prime }(s^{\prime },{\tilde{a}}^{\prime }|{\theta }^{Q_i^{\prime }}),$$

  (20)

  where ${\tilde{a}}^{\prime }={\mu }^{\prime }\left(s^{\prime }\right|{\theta }^{{\mu }^{\prime }})+ϵ$ and $ϵ\sim \mathrm{clip}\left(\mathcal{N}\right(0,\sigma ),-c,c)$ is clipped noise added to the target action.
• Delayed Policy Updates: The actor network is updated less frequently than the critics (typically once every two critic updates). Updating the actor less frequently allows the critic estimates to stabilize before being used to update the actor, reducing the accumulation of errors.
• Target Policy Smoothing: Noise is added to the target policy’s actions (as shown in (20)) to smooth the Q-function estimates and prevent the policy from exploiting narrow peaks in the Q-landscape.

Table 2. Comparison between DDPG and TD3 algorithms.

Feature	DDPG	TD3
Number of critics	1	2 (twin critics)
Q-value estimation	Single critic	Minimum of two critics
Policy update frequency	Every step	Delayed (every d steps)
Target action noise	None	Clipped Gaussian
Overestimation bias	Prone to overestimation	Reduced through clipping
Training stability	Moderate	Improved

summarizes the main differences between DDPG and TD3.

2.2.3. Comparison with Other DRL Algorithms

Beyond DDPG and TD3, other DRL algorithms have been applied to voltage regulation problems. The PPO algorithm [12] is a policy gradient method that uses a clipped surrogate objective to ensure stable updates without the need for trust-region constraints. While PPO has shown robust performance across various domains, it typically requires more samples than off-policy methods such as DDPG. The SAC algorithm [11] incorporates entropy regularization to encourage exploration and has demonstrated strong performance in continuous control tasks.

DDPG was selected as the primary algorithm due to: (1) its proven effectiveness in continuous control problems with deterministic optimal policies, (2) its compatibility with the reactive power dispatch problem, where actions are continuous and bounded, and (3) its relatively straightforward hyperparameter tuning compared to entropy-regularized methods. TD3 and PPO are included as baselines to evaluate whether their enhancements provide benefits for this specific application.

3. Framework of the Proposed Approach

This section provides a detailed examination of the proposed framework for optimizing ES operation within the ADN. The framework leverages a DRL algorithm to mitigate voltage deviations caused by fluctuating user consumption patterns and DG active power injections.

3.1. Reinforcement Learning Environment

The architecture of the proposed DRL-based approach for voltage regulation is shown in Figure 2.

In the proposed framework, a set of ESs is controlled by a single agent that interacts with the environment, which is represented as an ADN using the IEEE 33-bus system, as shown in Figure 2. At each time step, stochastic variations in load and distributed generation modify the network’s operating point. The environment provides an observation consisting of the vector of dynamic voltage deviations $\Delta V_D\left(t\right)$ with respect to the dynamic voltage profile $V_D$. This vector constructs the state $s_t$, which is then fed to the DDPG algorithm to compute the reactive-power setpoints for the ESs. The resulting voltages are then used to calculate the reward according to (21) and (22), explicitly encouraging the agent to reduce $\overline{\Delta V_D}$. The time-delay block in Figure 2 closes the discrete-time loop: after the ESs inject or absorb reactive power, the ADN voltages are updated, a new $\Delta V_D(t+1)$ value is obtained, and the next state–action–reward tuple is generated.

The architecture of this control scheme also includes the following components:

• State: $s_t^b\in S_t$ represents the information obtained after the actor applies an action in the environment at a time t. In this work, the voltage deviation of each bus is defined as $s_t^b$, where b denotes the bus index. $S_t$ depicts the state information of all the buses at time t.
• Action: $m_t^c\in M_t$ denotes the control signal issued by the agent, which determines the reactive power injection or absorption of ES number c (where c identifies each ES in the system) at iteration t. The set $M_t$ represents the actions of all ESs at t.
• Reward: $R_E$ represents the cumulative reward obtained in the current episode ($Ep$) and is computed as follows in (21):

  $$R_E\left(Ep\right)=\sum _{t=0}^T{R}_D\left(t\right),$$

  (21)

  where T is the total number of time steps, and $R_D\left(t\right)$ is the reward at t. This work assigns $R_D\left(t\right)$ based on the penalization region, as defined in (22).

$$R_D\left(t\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{\displaystyle \frac{\overline{\Delta V_D}}{\overline{\Delta V_B}}}\le 0.7,\hfill \\ -15\hfill & \mathrm{if}0.7<{\displaystyle \frac{\overline{\Delta V_D}}{\overline{\Delta V_B}}}\le 0.8,\hfill \\ -30\hfill & \mathrm{if}0.8<{\displaystyle \frac{\overline{\Delta V_D}}{\overline{\Delta V_B}}}\le 0.9,\hfill \\ -45\hfill & \mathrm{if}0.9<{\displaystyle \frac{\overline{\Delta V_D}}{\overline{\Delta V_B}}}\le 1,\hfill \\ -60\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(22)

The piecewise reward $R_D\left(t\right)$ in (22) is based on the deviation ratio $\overline{\Delta V_D}/\overline{\Delta V_B}$, which compares the mean dynamic voltage deviation obtained with the ES control action $\overline{\Delta V_D}$ against the mean deviation of the base case without any control $\overline{\Delta V_B}$. The neutral threshold value (0.7) was selected after preliminary parametric studies, as shown in the Sensitivity Analysis of Reward Function Threshold Section. A uniformly stepped penalty is applied, increasing by 15 units for each additional 10% degradation in the deviation ratio. This shaping guides the agent progressively toward the target region while keeping the reward bounded to avoid excessively large penalties. By scaling the penalty with the ratio interval, the agent is strongly incentivized to learn strategies that keep the voltage deviation ratio as low as possible.

Sensitivity Analysis of Reward Function Threshold

To justify the selection of the 0.7 threshold, a systematic sensitivity analysis was conducted by varying $\alpha$ from 0.5 to 0.9 in increments of 0.1. For each threshold value, the DDPG agent was trained for 250 episodes under identical conditions, and the resulting performance was evaluated based on: (1) final mean voltage deviation $\overline{\Delta V_D}$, (2) convergence speed, (3) control action variability, and (4) training time.

The results in

Table 3. Sensitivity analysis of reward function threshold $\alpha$.

Threshold $\mathit{\alpha }$	Mean $\overline{\mathbf{\Delta }{\mathit{V}}_{\mathit{D}}}$ (p.u.)	Convergence (Episodes)	Action Std. Dev.	Training Time (s)
0.5	-	-	-	-
0.6	0.0157	232	0.7386	913.7078
0.7	0.0159	100	0.5570	225.3243
0.8	0.0211	182	0.2182	152.0885
0.9	0.0222	<5	0.1705	160.4302

reveal that the lowest threshold ($\alpha =0.5$) failed to converge within 250 episodes. Although $\alpha =0.6$ yields a marginally lower mean voltage deviation, it converges more slowly and requires substantially longer training time. Additionally, it exhibits the highest action variability, suggesting more aggressive control behavior. Higher thresholds ($\alpha =0.8,0.9$) reduce the elapsed training time and produce smoother actions, but at the cost of poorer voltage regulation performance. Overall, $\alpha =0.7$ offers the best trade-off, achieving low voltage deviation with reasonable convergence speed and moderate action variability. This value also corresponds to a physically meaningful target of 30% improvement over the base case, a challenging yet attainable goal for the ES configuration used in this study.

The value of the ratio changes constantly due to the update of $\overline{\Delta V_D}$ every t, as shown in (23):

$$\overline{\Delta V_D}\left(t\right)={\displaystyle \frac{1}{n}}\sum _{b=1}^n\left|V_{Bus}\left(b\right)-1\right|.$$

(23)

In this equation, n denotes the total number of buses, and $V_{\mathrm{Bus}}\left(b\right)$ represents the voltage at bus b. Similarly, $\overline{\Delta V_B}$ is calculated using its corresponding bus voltages.

The loop continues until the maximum number of time steps is reached or the reset condition in (24) is met.

$$Reset=\overline{\Delta V_D}<0.7\left(\overline{\Delta V_B}\right).$$

(24)

This stopping criterion is based on the same target threshold, $\alpha =0.7$, as the reward design. An episode is considered successful once the controller achieves at least a 30% reduction in mean voltage deviation relative to the no-control baseline. To avoid premature termination due to transient improvements, a minimum of 10 steps is enforced before an episode can terminate.

3.2. Training and Evaluation

To evaluate the proposed methodology, the DDPG algorithm is trained using MATLAB’s Reinforcement Learning Toolbox and custom code. The first step of training, as shown in Figure 3, consists of obtaining an initial DG and load variation from the real profiles used in this work, specifically at $Ep=1$.

The ADN solution is obtained by solving the power flow of the IEEE 33 bus grid using the BFSM algorithm. After this sequence, all the voltages at each bus are available, enabling the computation of $\overline{\Delta V_D}$ which is used to compute the observations for the state $S_t$. With this information, the algorithm can take a set of actions $M_t$ to regulate the operational state of the ESs. After applying the corresponding actions to the ADN, the power flow solution is computed, yielding a reward $R_D$, which is then evaluated to determine whether the reset condition is met or the T condition is satisfied. If the reset condition is met, the episode terminates, and t is reset to 1 to start the next episode ($Ep+1$). This threshold reflects the point at which the agent has successfully learned to maintain the voltage deviation within acceptable operational limits. At this point, the agent receives a cumulative reward of $R_D=0$ for remaining within the desired operating region, as shown in (22). This indicates convergence of the training process and satisfactory control performance.

On the other hand, if the T condition has not yet been reached, the algorithm will repeat the solution process, proposing a new set of actions every t. Training is complete when the maximum number of episodes has been reached.

3.3. Hyperparameter Tuning

In this work, an experimental tuning strategy based on the grid search method is used to adjust key hyperparameters, including the number of training episodes, the actor and critic learning rates, the decay rate, and the episode length [26].

To implement this strategy, each hyperparameter was first evaluated independently. Specifically, a set of five candidate values was defined for each hyperparameter, uniformly distributed within a ±30% interval around its default value. These defaults correspond to widely adopted configurations in continuous-control DRL algorithms and are consistently reported in prior studies.

Table 4. Hyperparameters used for each DRL agent.

Hyperparameter	TD3	DDPG-A	DDPG-B
Discount factor	0.99	0.99	0.9
Experience buffer length	$1\times {10}^4$	$1\times {10}^4$	$1\times {10}^7$
Mini-batch size	64	64	128
Target smooth factor $\tau$	$0.0050$	$1\times {10}^{-3}$	$1\times {10}^{-3}$
Sample time	1	1	1
Exploration noise	Gaussian	Ornstein–Uhlenbeck (0.3)	Ornstein–Uhlenbeck (0.1)
Actor learning rate	$1\times {10}^{-6}$	$1\times {10}^{-6}$	$5\times {10}^{-6}$
Critic learning rate	$1\times {10}^{-5}$	$1\times {10}^{-5}$	$5\times {10}^{-5}$
Variance decay rate	0	0	${10}^{-5}$
Hidden layer 1 (critic)	256	256	256
Hidden layer 2 (critic)	256	256	128
Hidden layer 1 (actor)	256	256	400
Hidden layer 2 (actor)	256	256	300
Number of episodes	500	500	200
Time steps per episode	500	500	200

(column DDPG-B) summarizes the hyperparameters considered in this study. The impact of each candidate value was assessed by analyzing the evolution of the reward function throughout training, enabling the identification of hyperparameters with high sensitivity, as evidenced by significant variations in convergence speed, reward stability, and steady-state performance. Hyperparameters with limited influence on these metrics were retained at their default values.

Subsequently, a subset of highly sensitive hyperparameters was fixed for further tuning. For these parameters, the search intervals were expanded to broader ranges derived from the literature on DDPG-based control and voltage regulation. A sequential tuning procedure was then applied: for each hyperparameter, all other parameters were held constant while candidate values were evaluated individually. For every configuration, ten independent training runs were performed to mitigate the effects of stochasticity inherent to neural network initialization and exploration noise. The parameter value that achieved the highest average reward while simultaneously ensuring stable voltage profiles within acceptable operational limits was retained before proceeding to the next hyperparameter.

This sequential process was repeated across the subset of sensitive hyperparameters until further adjustments produced negligible changes in convergence behavior and voltage regulation performance within the explored ranges.

4. Case Study

This section describes the experimental framework used to evaluate the proposed strategy on the IEEE 33-bus network, comparing a no-control base case against the controlled case.

Test Scenarios Description

The proposed methodology optimizes the operation of the ESs in the IEEE 33-bus radial distribution feeder, depicted in Figure 4. Line impedances and bus load data are from [27], and grid properties are listed in

Table 5. Properties of the IEEE 33-bus radial distribution feeder.

Property	Value
Number of buses	33
Number of loads	32
Total active power demand (kW)	3715
Total reactive power demand (kVAr)	2300
Number of photovoltaic units	4
Number of wind turbine units	3
Total DG active power generation capacity (kW)	2229
Number of ES units	12

. All users connected to the ADN have a residential consumer profile, as described in [28].

The consumption trend of the load profiles is analyzed in Figure 5, which shows detailed curves for the 24 h retrieved from the public ERCOT database [29]. These load profiles are assigned to different grid branches, as indicated by the color coding in Figure 4. Seven DG sources are distributed across the grid, comprising four photovoltaic panels (PVs) and three wind turbines (WTs), and accounting for 60% of the total demand. The behavior of the DG source profiles is depicted in Figure 6, where the PV and WT generation profiles are from the public NREL dataset [30]. The DG sources are placed in a distributed configuration to efficiently deliver injected active power to elements farther from the main grid. To evaluate the proposed methodology, 12 ESs are distributed across the ADN. These ESs perform voltage regulation via reactive power compensation and are controlled using the DDPG algorithm. Each ES operates at 80% of the CL’s active power capacity. This value lies within the range of smart-load participation levels explored in previous ES-based studies [17,21]. To implement the control strategy, a specific neural network architecture and a set of experimentally tuned hyperparameters are adopted for the DDPG-B configuration, as summarized in Table 4. For comparison, the same table reports the hyperparameters used for DDPG-A and TD3, which follow the default MATLAB settings. The actor and critic learning rates were adjusted experimentally to ensure stable training for this specific application. PPO hyperparameters are not listed because, as an on-policy method, several TD3/DDPG-specific parameters (e.g., replay buffer size, target-network smoothing) do not apply.

Once trained, the DRL agents regulate the grid voltage at 15-min intervals over 24 h to reduce $\overline{\Delta V_D}$ below 70% of $\overline{\Delta V_B}$.

Finally, to assess the robustness of the proposed DRL controller, the three DRL configurations (tuned DDPG-B, default DDPG-A, and TD3) are benchmarked against a no-control base case. The simulations are performed on a PC with a 13th Gen Intel(R) Core(TM) i7-13650HX 2.60 GHz processor (Santa Clara, CA, USA) running MATLAB R2022b.

5. Results

This section is divided into two parts: the key results from training the DDPG controller and the performance evaluation during testing for both the base and controlled cases.

5.1. Training Results

Figure 7 illustrates the average reward (blue line) and episode reward (purple line) of the training across 200 episodes for the DDPG agent. With the current parameters and reward function, the maximum achievable reward is zero. At the beginning of the training, the reward shows low, fluctuating values, indicating high variability in the agent´s decisions. This behavior is expected, as the agent undergoes an exploratory phase without prior knowledge of the environment or of how to properly regulate the grid voltage based on ES operation. As training progresses, the episode rewards show a clear trend toward convergence, with gradually increasing values. Nevertheless, after episode 40, sudden drops in the episode reward are observed. These drops are likely due to grid conditions the agent had not previously encountered, such as load fluctuations and active power injections. In response, the DDPG agent takes atypical exploration actions to find a suitable solution. As training continues, these fluctuations decrease, and by episode 60, the agent begins to show steady episode reward behavior. Around episode 100, both curves stabilize, indicating that the agent has successfully learned an effective policy. The average reward approaches zero, indicating that the agent is finding the desired solution specified by the reward function, corresponding to a decrease in the voltage deviation $\overline{\Delta V_D}$ below 70% of $\overline{\Delta V_B}$.

5.2. Testing Results on the IEEE 33-Bus Network

The voltage regulation results are presented in Figure 8, which provides a comprehensive view of the system performance through two complementary representations. Figure 8a displays the minimum bus voltage recorded at each hour, enabling a direct assessment of ANSI C84.1 compliance [31]. The results indicate that the proposed strategy using DDPG-B increases the minimum hourly voltage level, thereby reducing the frequency of undervoltage events compared to the base case. Figure 8b shows the temporal evolution of bus voltages, where shaded regions represent the voltage range (minimum to maximum) across all 33 buses at each 15-min interval. In the base case, the feeder exhibits significant voltage regulation challenges during two critical periods. The first undervoltage event occurs around 6:15 h (point A), where the bus voltage drops to 0.9492 p.u. This behavior is attributed to the morning load ramp-up while DG output remains minimal. Between 8:00 h and 17:00 h, the voltage profile recovers as DG participation increases. The most severe undervoltage condition occurs at point B (21:15 h), where the minimum voltage drops to 0.9203 p.u., representing a 7.97% deviation from the nominal value. In contrast, the implementation of ESs controlled by the DDPG-B algorithm effectively mitigates undervoltage conditions. The shaded orange region in Figure 8b is notably narrower than the blue region, indicating reduced voltage variability. The controlled case exhibits only a single undervoltage event at point C (21:15 h), with the minimum voltage reaches 0.9445 p.u. This event affects buses 12–17 at the feeder end, where cumulative line losses and limited ES capacity create challenging conditions. Nevertheless, the DDPG-B controller reduces the maximum voltage deviation from 7.97% to 5.55%, representing a 30% improvement.

To better understand these results, the agent’s actions are analyzed in more detail. Figure 9a shows the ES reactive power ($Q_{ES}$) across the testing scenarios, while Figure 9b displays the voltage distribution at each bus across all the test cases, linking ES operation to its impact on system voltage. The agent’s control actions exhibit diversity; this is evident in the boxplots in Figure 9a, where each of the 12 ESs shows different reactive power patterns. This diversity in $Q_{ES}$ magnitudes demonstrates that, regardless of load fluctuations, the algorithm enables coordinated ES operation to determine an overall solution, thereby reducing $\overline{\Delta V_D}$. Notably, the boxplot of bus 19 in Figure 9a is the only one whose reactive-power values are mostly positive, indicating that this ES operates in inductive mode most of the time and absorbs reactive power, which reduces the local voltage. This behavior is mainly explained by its electrical location: the ES is connected to the branch supplied from bus 2, one of the closest buses to the slack bus, so the voltage drop along this feeder is smaller, and the voltage at bus 19 is higher than at downstream buses. Consequently, the loads connected to this bus experience voltages closer to 1 p.u., as shown in Figure 9b.

A similar case is observed at bus 22, where the boxplot shows a limited spread, indicating that its voltage values remain consistent throughout the simulation, close to 1 p.u., even though the ES does not operate in inductive mode, unlike the ES at bus 19. Although connected to the same branch, the ES’s location at the distal end results in a voltage reduction. In contrast, for the ESs located at buses 7, 8, 14, 18, 29, 30, 31, and 32, the voltage is lower and, consequently, exhibits a high voltage deviation. This effect is associated with the distance from the slack bus, which shifts the ES’s operating range into capacitive mode.

On the other hand, the ESs connected to buses 24 and 25 exhibit higher operating magnitudes in capacitive mode. Although the connection buses exhibit low-voltage deviations and operate close to the nominal voltage, these ESs still contribute significantly to capacitive compensation. This behavior is explained by the fact that these devices have the most significant compensation capacity, enabling a broader impact on the ADN. As a result, they not only compensate local loads but also provide voltage support to other connected loads. The simulation results obtained with the DDPG algorithm confirm that the training process was effective, enabling the agent to adapt to different operating scenarios while maintaining voltage stability.

Figure 10 compares the no-control base case with the four DRL configurations TD3, PPO, DDPG-A, and DDPG-B. The boxplots show the distribution of the total voltage deviation over a 24-h period, summarizing the median, interquartile range, and extreme values, thereby enabling a detailed comparison of how effectively each controller keeps the bus voltages close to the nominal value.

In the no-control base case, the boxplot shows the widest spread and the most considerable voltage deviations, with a few values exceeding 0.045 p.u. By contrast, TD3 and DDPG-A exhibit intermediate performance, with mean deviations close to 0.02 p.u. In comparison, PPO yields a lower mean deviation (0.0184 p.u.), achieving the best performance among the default configurations. Nevertheless, the proposed DDPG setting (DDPG-B) yields a much more concentrated distribution and the lowest mean value of $\overline{\Delta V_D}$ among all tested agents. In this case, the 25th and 75th percentiles lie closer to the median, and the minimum deviations are lower, indicating more consistent voltage regulation and more reliable control action. The mean value $\overline{\Delta V_D}$ is approximatly 40% lower than in the base case. Regarding the training statistics, DDPG-B achieves the lowest training time (3.76 min), whereas TD3, PPO, and DDPG-A require more than 6 min. A similar trend is observed in the convergence behavior: DDPG-B converges around episode 100, while TD3 and DDPG-A converge only after roughly 200 episodes. PPO stabilizes earlier, despite occasional reward drops. These results support the effectiveness of the proposed hyperparameter tuning.

Although all DRL techniques effectively mitigate voltage deviations, these findings confirm that the tuned DDPG-B agent significantly reduces $\overline{\Delta V_D}$ and provides more uniform behavior across different operating scenarios, outperforming DDPG-A and TD3 in terms of both voltage deviation and consistency.

5.3. Comparison with Metaheuristic-Based Centralized Voltage Control

A comparative performance analysis between the proposed DDPG-B-based control strategy and metaheuristic-based centralized voltage regulation approaches, including GA, PSO, and Grey Wolf Optimizer (GWO), is conducted in this subsection. These techniques are commonly adopted in centralized voltage control schemes and therefore constitute suitable benchmarks for evaluating the effectiveness of the proposed DRL-based approach. For implementation, GWO follows the open-source reference code in [32], while GA and PSO use the MATLAB toolbox provided in [33]. For all three metaheuristics, the default hyperparameter settings are used.

The comparison is conducted under the same network configuration, operating conditions, and scenarios defined in the proposed case study, ensuring a fair and consistent evaluation across all methods. Two key performance indicators are considered: $\overline{\Delta V_D}$ and the computational execution time required to obtain a solution.

The metaheuristic-based methods effectively solve the IEEE 33-bus active distribution network. Their mean voltage deviations, $\overline{\Delta V_D}$, are comparable to those obtained with the DDPG-B-based control strategy, as reported in

Table 6. Solving methodologies comparison.

Algorithm	Max $\overline{\mathbf{\Delta }{\mathit{V}}_{\mathit{D}}}$ (p.u.)	Min $\overline{\mathbf{\Delta }{\mathit{V}}_{\mathit{D}}}$ (p.u.)	Execution Time (min)
DDPG-B	0.0306	0.0013	0.0159
GA	0.0288	0.0006	2.3629
GWO	0.0290	0.0006	2.3898
PSO	0.0289	0.0006	3.5140

, indicating similar voltage regulation performance under the considered operating conditions. However, the computational performance differs significantly. On average, GA, PSO, and GWO require more than 2 min per scenario, while the DDPG-B-based DRL approach requires only 0.0159 min per scenario. This difference arises because the DRL approach exploits a pre-trained policy to generate control actions through a single forward pass. The substantial reduction in execution time highlights the superior computational efficiency of the proposed method, which is particularly relevant for real-time voltage regulation applications.

5.4. Scalability of the Centralized Approach

To assess the scalability of the proposed centralized DRL control framework, the experimental evaluation is extended to the IEEE 69-bus system. In this setup, five ES units are installed at buses with high active power demand, as listed in

Table 7. IEEE 69 bus scalability summary.

Case	Device	No.	Location (Bus)	Max Power (kW)	$\overline{\mathbf{\Delta }{\mathit{V}}_{\mathit{D}}}$ (p.u.)	Reduction (%)
Base case	PV gen.	3	11, 49, 59	760.4, 760.4, 760.4	0.0076	–
DDPG-B	ES	5	12, 21, 50, 61, 64	116, 91.2, 307.7, 995.2, 181.6	0.0046	38.49
	PV gen.	3	11, 49, 59	760.4, 760.4, 760.4

, and each ES operates at 80% of the CL active power capacity. In addition, three PV units are included using the same PV generation profiles (PV1–PV3) as in the previous case study, accounting for 60% of the total active power demand. Additionally, the residential load profile used in the 33-bus case is retained and appropriately assigned across the 69-bus network. The proposed reward function is kept unchanged, following the same criteria defined in (22). Likewise, the tuned DDPG-B hyperparameter configuration is retained.

Despite the increased network size and expanded observation space, DDPG-B achieved a $\overline{\Delta V_D}$ value of 0.0046 p.u., representing a 38.49% improvement over the base case and demonstrating the scalability of the proposed DRL configuration. Further performance improvements can be achieved by retuning the hyperparameters specifically for the IEEE 69-bus system.

6. Discussion

This section provides a critical analysis of the proposed DRL-based voltage control strategy, discussing its advantages and limitations, comparing it with existing approaches, and identifying practical implementation considerations.

6.1. Performance Analysis and Comparison

The proposed controller was trained to coordinate the ES units under stochastic load and distributed generation variations, achieving an average reduction of about 40% in the mean voltage deviation compared to the no-control base case, while keeping all bus voltages within the ANSI limits across 96 test scenarios. In addition, the DDPG-B configuration outperformed DDPG-A, TD3, and PPO, exhibiting a lower mean voltage deviation and tighter boxplots, indicating more consistent and reliable voltage regulation.

The superior performance of DDPG-B over TD3 and PPO in this application is noteworthy, as these algorithms generally outperform DDPG in continuous-control benchmarks, an advantage commonly attributed to TD3’s more conservative Q-value estimation and PPO’s conservative policy updates. However, the relatively simple reward structure and bounded action space in this voltage regulation problem may reduce these advantages. Furthermore, the hyperparameter tuning applied to DDPG-B but not to TD3 or PPO (which used default hyperparameters except for learning rates in TD3) likely contributed to this performance gap.

6.2. Advantages of the Proposed Approach

The proposed DRL-ES voltage regulation scheme offers several advantages over conventional methods:

• Model-free adaptation: Unlike optimization-based approaches that require accurate network models, the DDPG agent learns directly from interactions with the environment, enabling adaptation to model uncertainties and unmodeled dynamics.
• Real-time operation: Once trained, the DDPG agent can compute control actions in milliseconds, making it suitable for real-time voltage regulation under rapidly changing conditions.
• Coordinated multi-ES control: The centralized agent implicitly learns the interactions between multiple ESs, enabling coordinated operation without requiring explicit communication protocols.
• Cost-effective infrastructure: ESs require minimal energy storage compared to battery-based ESSs, reducing capital costs while achieving comparable voltage regulation performance.
• Flexibility: The trained agent can handle diverse operating scenarios without retraining, as demonstrated by consistent performance across 96 test cases with varying load and generation profiles.

6.3. Limitations and Challenges

Despite its promising results, the proposed approach has several limitations that should be acknowledged:

• Centralized architecture: The current implementation relies on a centralized agent with full observability of all bus voltages. This architecture requires a reliable communication infrastructure and introduces a single point of failure. For larger networks, communication delays and bandwidth constraints may become significant concerns.
• Training data requirements: The DDPG agent requires extensive offline training with representative scenarios. If operational conditions deviate significantly from the training data (e.g., extreme weather events or major network reconfigurations), performance may degrade.
• Idealized ES model: The ES is modeled as an ideal controlled AC voltage source, neglecting converter losses, DC-link voltage dynamics, and harmonic distortion. While this simplification is common in planning studies, practical implementations may exhibit different behavior.
• Reactive power only: The ES configuration used in this work provides only reactive power compensation. For severe voltage deviations, active power injection from ESSs or curtailment of DG sources may be necessary.
• Scalability concerns: The state and action spaces grow linearly with the number of buses and ESs. For large-scale networks with hundreds of buses, the current architecture may face scalability challenges that require distributed or hierarchical approaches.

6.4. Practical Implementation Considerations

For practical deployment of the proposed strategy, several considerations should be addressed:

• Communication infrastructure: A supervisory control and data acquisition (SCADA) system or advanced metering infrastructure (AMI) is needed to collect real-time voltage measurements and dispatch control signals to ESs.
• Safety constraints: Additional constraints should be incorporated to prevent the ESs from operating beyond their physical limits and to ensure fail-safe behavior during communication failures.
• Online adaptation: Implementing online fine-tuning mechanisms would allow the agent to adapt to slow drifts in network characteristics without complete retraining.
• Regulatory compliance: The control strategy should be validated against utility standards and regulations governing voltage regulation and power quality.

6.5. Future Research Directions

These results highlight the potential of appropriately tuned DRL-controlled ESs as an effective and robust solution for voltage regulation in ADNs with high renewable penetration and variable load profiles. Future research will focus on extending this framework to multi-agent DRL architectures, considering larger and more complex distribution networks, and integrating ESs with other compensation devices to further improve scalability and resilience.

Specific future directions include:

• Developing distributed multi-agent DRL frameworks where each ES is controlled by a local agent with limited communication, thereby improving scalability and robustness.
• Integration of more detailed ES models, including converter dynamics, losses, and protection schemes.
• Combination of ES-based reactive compensation with battery ESSs for coordinated voltage-and-frequency regulation.
• Extension to unbalanced three-phase networks and investigation of phase-specific voltage regulation.
• Transfer learning approaches to enable rapid adaptation when deploying trained agents to new network configurations.

7. Conclusions

This paper presented a novel deep reinforcement learning-based voltage regulation strategy using electric springs in active distribution networks. The main conclusions of this work are as follows:

• The proposed DDPG-based controller successfully coordinates 12 electric springs to regulate voltage in the IEEE 33-bus test system under time-varying load and distributed generation conditions. The trained agent achieves an approximately 40% reduction in mean voltage deviation compared to the no-control baseline. In addition, a scalability study was conducted on the IEEE 69-bus test system using the same load and distributed generation conditions, with 5 electric springs installed, while keeping the reward design and the tuned DDPG-B hyperparameter configuration unchanged. Under this setting, the controller achieves a 38.49% improvement over the no-control case.
• The region-based penalized reward function effectively guides the agent toward maintaining voltage deviations below the target threshold while avoiding overly aggressive control actions. The sensitivity analysis confirms that the selected threshold of 0.7 provides an effective trade-off between voltage regulation performance and training stability.
• Systematic hyperparameter tuning of the DDPG algorithm (DDPG-B configuration) yields significant performance improvements over the default DDPG configuration (DDPG-A), TD3, and PPO, highlighting the importance of algorithm tuning for specific applications rather than relying on default parameters.
• The combination of DRL and electric springs represents a promising approach for cost-effective voltage regulation in distribution networks with high renewable penetration. Electric springs provide flexible reactive compensation without requiring large energy storage elements, while DRL enables adaptive, model-free control that can handle the uncertainty of renewable generation.
• The proposed approach keeps all bus voltages within ANSI C84.1 limits (0.95–1.05 p.u.) in 95 out of 96 test scenarios, with only minor violations occurring at remote buses during peak demand periods.

While the centralized architecture and idealized ES model represent limitations that should be addressed in future work, the results demonstrate the feasibility and effectiveness of DRL-based ES coordination for voltage regulation. The methodology developed in this work provides a foundation for practical implementation and further research on distributed control architectures and more realistic ES models.