Volt–Var Self-Optimizing Control of Distribution Networks Based on the BOST-GRPO Algorithm Under Stability Constraints

Abstract

High penetration of distributed photovoltaic (PV) generation has intensified voltage violations and stochastic voltage fluctuations in distribution networks, while existing voltage–var control methods still have limitations in terms of communication dependence, scalability, and edge deployment. To address these issues, this paper proposes a stability-constrained voltage–var self-optimizing control method for distribution networks based on the Bandit-Guided Online Self-Tuning Group Relative Policy Optimization (BOST-GRPO) algorithm. First, based on the LinDistFlow linearized power-flow model, a communication-free, decentralized, and locally observable reinforcement learning control environment is constructed, enabling each node to independently generate reactive power regulation commands using only local voltage measurements. Second, a contraction-mapping-based stability constraint is embedded into the policy output layer, theoretically guaranteeing the local exponential convergence of nodal voltage deviations around the equilibrium point and reducing the risk of voltage instability caused by overly aggressive policy actions. Meanwhile, device capacity constraints are incorporated into the policy output through a tanh-based action mapping, ensuring the physical feasibility of control commands. On this basis, BOST-GRPO realizes the online self-tuning of key hyperparameters within a single training process through a Bandit-guided mechanism, thereby avoiding the repeated training overhead caused by traditional offline hyperparameter tuning. Simulation results on the IEEE 33-bus system show that the proposed method outperforms benchmark reinforcement learning algorithms in final test cost, voltage deviation suppression, steady-state error, and regulation speed. Further tests under sensitivity matrix mismatch, different initial voltage disturbance intensities, and the extended IEEE 69-bus system demonstrate that the proposed method achieves good robustness and scalability.

1. Introduction

Driven by the goals of carbon peaking and carbon neutrality, power systems are undergoing a transition from fossil-fuel-dominated generation toward renewable-energy-dominated generation [1,2]. By the end of 2025, China’s installed solar energy capacity had reached approximately 1.20 billion kW and continued to grow [3]. This trend has made distribution networks a major platform for accommodating high shares of renewable energy [4]. However, the high penetration of distributed photovoltaic (PV) generation has further increased the operational complexity of distribution networks [5]. The stochastic and intermittent characteristics of PV output may lead to bidirectional power flows [6], thereby causing nodal voltage violations, increased network losses, and other problems, which severely constrain power supply quality and economic operation [7,8]. To address these challenges, volt–var control (VVC) in active distribution networks has gradually become one of the key technologies [9]. VVC exploits the reactive power regulation capability of distributed resources, such as PV inverters, particularly their fast-response characteristics, to provide voltage support [10]. Against this background, this study proposes an efficient control strategy to achieve coordinated multi-objective optimization of voltage stability control and network loss reduction.

Existing VVC methods can generally be classified into model-driven and data-driven approaches. Among model-driven methods, centralized optimal power flow or optimal reactive power dispatch methods can usually obtain globally optimal or high-quality optimal solutions when the system model is accurate and complete information is available [11]. However, such methods heavily rely on accurate topology and parameter information, and their performance is limited under insufficient observability and incomplete modeling conditions [12]. Meanwhile, as the scale of controllable resources increases, the problem dimension and nonlinearity grow sharply, making centralized solution methods difficult to satisfy real-time requirements [13]. Distributed optimization methods, such as alternating the direction method of multipliers-based algorithms, improve computational efficiency through decomposition and parallelization [14]. Nevertheless, they usually require communication among neighboring nodes to reach consensus, and are therefore constrained by communication reliability and latency [15]. Local control has fast response and requires no communication [16], but it lacks global coordination and may lead to insufficient control performance or even oscillations [5,17]. These limitations indicate the need for more adaptive control mechanisms under time-varying operating conditions. Adaptive event-triggered control via switching functions also provides useful insights into handling unmodeled dynamics and external disturbances [18], suggesting the potential of adaptive mechanisms for voltage control under uncertain operating conditions.

With the development of artificial intelligence technologies, deep reinforcement learning (DRL), as a representative data-driven method, can reduce the dependence on accurate physical models to a certain extent [19]. To alleviate the reliance on explicit physical models, studies [20,21] verified the online adaptive capability of DRL in voltage control. Studies [22,23] further extended reinforcement learning to scenarios such as hierarchical control of active distribution networks and multi-timescale voltage control, demonstrating that data-driven methods can still provide good real-time regulation capability under weak-model conditions and complex operating scenarios. However, single-agent approaches are prone to the curse of dimensionality in large-scale systems [5]. Therefore, multi-agent reinforcement learning with distributed execution capability has gradually become a mainstream approach. In addition, studies [24,25] improved the coordination and generalization capability of voltage control in large-scale distribution networks from the perspectives of enhanced collaboration mechanisms and graph-structured modeling, respectively. Studies [26,27] verified the cooperative effectiveness of actor–critic and proximal policy optimization algorithms, respectively, while study [28] addressed the credit assignment problem through value decomposition. Although multi-agent reinforcement learning has made progress in distribution network voltage control, power systems impose stringent requirements on physical closed-loop stability. Studies [29,30] introduced reward constraints or Lyapunov-based methods, but they lack rigorous theoretical proof and hard-constraint design, making it difficult to fully cover instability risks. Furthermore, studies [31,32] constrained policy behavior through safe reinforcement learning mechanisms and physics-shielding mechanisms, which can reduce the risk of voltage violations to a certain extent. However, these methods still mainly focus on constraints within the safe feasible region, and explicit theoretical guarantees for voltage deviation convergence remain insufficient. To further summarize the methodological differences,

Table 1. Comparison between representative DRL-based VVC methods and the proposed BOST-GRPO framework.

Method Category	Observation/Coordination Requirement	Stability or Safety Treatment	Hyperparameter Adaptation
Model-free or adaptive DRL-based VVC	Learns voltage control policies from interaction data and may use system-level training information	Mainly improves empirical voltage regulation performance	Usually manually tuned or fixed
Hierarchical or multi-timescale DRL control	Improves coordination through hierarchical or multi-timescale decision structures	Focuses on coordinated voltage regulation under complex operating conditions	Usually predefined
Collaborative or graph-based multi-agent DRL	Enhances coordination through collaboration, attention, or graph-structured information	Improves multi-agent coordination and generalization	Usually selected offline or manually tuned
Safe or physics-informed DRL methods	Incorporates physical knowledge, safety constraints, or shielding mechanisms	Reduces unsafe actions and improves closed-loop safety	Usually not self-tuned online
Proposed BOST-GRPO	Uses only local voltage measurements during execution	Embeds contraction-based stability and action-bound constraints	Performs bandit-guided online self-tuning within a single training process

compares representative DRL-based VVC methods with the proposed BOST-GRPO framework from the perspectives of observation requirements, stability treatment, communication dependence, and hyperparameter adaptation.

Despite the significant progress of multi-agent reinforcement learning in voltage control, four key bottlenecks remain. First, the training stage often relies on a centralized critic to aggregate global information, and the communication dependence has not been completely eliminated [5,33]. Second, the centralized training and decentralized execution (CTDE) architecture has high computational complexity, making it difficult to meet the plug-and-play requirements of low-computing-power edge devices [34]. Third, although existing studies have introduced physical and stability constraints, closed-loop stability proofs and hard-constraint designs remain insufficient [35,36]. Fourth, these algorithms are highly sensitive to hyperparameters, while traditional manual tuning is inefficient and lacks robustness [37].

To address the above challenges, this paper proposes a stability-constrained voltage–var self-optimizing control method for distribution networks based on a Bandit-Guided Online Self-Tuning Group Relative Policy Optimization (BOST-GRPO) algorithm. First, based on the LinDistFlow linearized power flow model, a joint optimization objective is formulated, and a communication-free, decentralized, locally observable reinforcement learning environment is constructed, enabling each node to make independent decisions using only local voltage measurements. Second, to address the insufficient safety and verifiability of reinforcement learning control policies, a contraction-mapping-based stability constraint is imposed on the policy output layer. This theoretically guarantees the local exponential convergence of nodal voltage deviations to zero and prevents voltage instability caused by overly aggressive reinforcement learning policies. Third, device capacity constraints are embedded into the policy output through a tanh-based action mapping, ensuring the physical feasibility of control commands. Finally, a BOST-GRPO-based control framework is developed by combining the lightweight, value-function-free GRPO policy-optimization backbone with stability-constrained policy projection and bandit-guided online hyperparameter self-tuning. In this framework, the contraction-mapping-based gain constraint keeps the learned decentralized policies within a locally stable region, while the BOST mechanism adaptively selects key hyperparameters during a single training process. Through this integration, the proposed method improves the stability, training adaptability, and engineering applicability of communication-free decentralized volt–var control.

2. Optimization Model for Volt–Var Control in Distribution Networks

Driven by the goals of carbon peaking and carbon neutrality, the penetration of renewable energy sources, such as distributed photovoltaic (PV) generation, continues to increase in distribution networks. The stochastic and fluctuating characteristics of their outputs make distribution networks more prone to voltage violations, increased network losses, and other operational problems. Deep reinforcement learning (DRL) can learn control policies through “state–action–reward” interactions without relying on accurate analytical models. However, its training process usually requires a large number of iterations and extensive interactions with the environment. To address this challenge and ensure the safety and reliability of control, this paper proposes a voltage–var self-optimizing control method for distribution networks based on the BOST-GRPO algorithm. The overall framework is shown in Figure 1.

2.1. Linearized Power Flow Model of Distribution Networks

The voltage and power flow distributions in distribution networks can be accurately described by the DistFlow model [38]. However, its AC power flow formulation exhibits significant nonlinearity and nonconvexity. If it is directly used as the state transition model of the DRL interaction environment, the simulation cost of each step will be relatively high, which is unfavorable for large-scale training iterations. To balance physical consistency and simulation efficiency, second-order terms such as branch power losses are neglected, and the LinDistFlow linearized model is adopted to describe the mapping relationship between nodal voltages and power injections [2].

$$\mathit{v}=R\mathit{p}+X\mathit{q}+{\mathit{v}}_{\mathrm{ref}}$$

(1)

In Equation (1), v, p, and q denote the column vectors of voltage magnitudes, net active power injections, and net reactive power injections of the N controllable nodes, respectively; v_ref denotes the column vector of reference voltage values in per unit; and the N-dimensional matrices R and X are the per-unit voltage–power sensitivity matrices determined by the system topology and line parameters. To unify physical dimensions and reduce training instability, the per-unit system is adopted in this paper.

2.2. Optimization Objective

The objective of VVC in distribution networks is to improve voltage quality while considering regulation economy under operational constraints. The optimization objective formulated in this paper is to minimize the weighted sum of the total system voltage deviation and control cost over the scheduling horizon T, as expressed in Equation (2).

$$min{\displaystyle {\displaystyle \sum }_{t=1}^T}\left(F_V\left({\mathit{v}}_t\right)+\alpha \cdot F_L\left({\mathit{u}}_t\right)\right)$$

(2)

In Equation (2), α is the weighting coefficient used to balance voltage quality and control cost; v_t denotes the column vector of voltage deviations in the N nodes at time t; and u_t denotes the column vector of reactive power absorption of the N nodes at time t. F_V(v_t) is the total system voltage deviation penalty at time t, which represents the L1 norm of the deviation between the voltage magnitudes of all non-slack nodes and the ideal reference value v_ref = 1. F_L(u_t) is the control cost term at time t, which is defined using the L1 norm of the control actions, as shown in Equations (3) and (4).

$$F_V\left({\mathit{v}}_t\right)={\displaystyle \sum |v_{i,t}-v_{\mathrm{ref}}|={\parallel {\mathit{v}}_t-{\mathit{v}}_{\mathrm{ref}}\parallel }_1}$$

(3)

$$F_L\left({\mathit{u}}_t\right)={\displaystyle \sum \left|u_{j,t}\right|={\parallel {\mathit{u}}_t\parallel }_1}$$

(4)

Here, v_t,j and u_t,j denote the per-unit voltage deviation and reactive power absorption of the j-th agent, respectively. The L1 norm is adopted to provide a proportional penalty on the aggregate voltage deviation and the total reactive-power regulation effort. This formulation is consistent with the MAE-type performance evaluation and is less sensitive than a quadratic penalty to occasional large stochastic deviations caused by PV/load fluctuations. Meanwhile, voltage safety and closed-loop stability are enforced separately through the voltage constraints, the tanh-based action-capacity mapping, and the contraction-based gain constraint.

2.3. Constraints

VVC decisions must comply with the physical laws of power networks and operational safety limits. First, the system state transition is determined by the LinDistFlow linearized power flow model shown in Equation (1). Therefore, during the DRL interaction process, each voltage update naturally satisfies the power flow constraints.

Second, to ensure power quality and equipment safety, nodal voltages should remain within the allowable range, as shown in Equation (5).

$$v_{\min }\le v_{t,j}\le v_{\max }$$

(5)

Based on practical engineering experience, this paper sets v_min = 0.95 p.u. and v_max = 1.05 p.u. Considering the stochastic nature of PV and load disturbances, voltage violations are significantly penalized in the cost function during training to guide the policy through soft constraints, thereby improving its robustness under uncertain disturbances.

Finally, the control actions must satisfy the equipment capacity limits, as shown in Equation (6).

$$-u_{j,\max }\le u_{t,j}\le u_{j,\max }$$

(6)

To ensure physical feasibility during online execution, tanh-based action compression and amplitude scaling are introduced at the output end of the policy network. The control commands are mapped into the allowable range, thereby avoiding infeasible commands and improving execution safety and reliability.

3. Stability Constraint Satisfying Contraction Mapping and Proof of Exponential Stability

In fully decentralized volt–var control, each nodal controller independently generates reactive power regulation commands based only on the local voltage deviation. Since the control policy is learned through data-driven methods, the online closed-loop system may diverge or oscillate under disturbances or parameter variations if verifiable stability constraints are absent. To address this issue, this section constructs a closed-loop voltage regulation operator based on the LinDistFlow model. By using the fixed-point theorem for contraction mappings, a set of stability constraints is proposed and theoretically proven. Specifically, for any decentralized control policy satisfying the proposed constraints, the corresponding closed-loop operator satisfies the contraction mapping condition in a neighborhood of the equilibrium point, and any initial voltage deviation within this neighborhood converges exponentially to the unique equilibrium point.

3.1. Closed-Loop Dynamics of Iterative Voltage Control

Consider the voltage regulation process of a distribution network on a millisecond timescale. Within a fast regulation window where disturbances are approximately static, active power fluctuations and external disturbances vary relatively slowly, whereas reactive power devices such as inverters can rapidly update reactive power commands and drive voltage convergence. Let the reactive power vector of the N nodes at time t be denoted by q_t. The iterative update can then be expressed as Equation (7).

$${\mathit{q}}_{t+1}={\mathit{q}}_t-{\mathit{u}}_t$$

(7)

Under the fully decentralized framework, each agent relies only on its local voltage deviation, and the control law can be expressed as Equation (8).

$${\mathit{u}}_t=\mathit{u}\left({\mathit{v}}_t\right)={\left[u_1\left(v_1\right),\dots ,u_N\left(v_N\right)\right]}^{\mathrm{T}}$$

(8)

Combining Equations (1), (7) and (8), the iterative expression of voltage deviation can be obtained as Equation (9).

$${\mathit{v}}_{t+1}={\mathit{v}}_t-X\mathit{u}\left({\mathit{v}}_t\right)$$

(9)

Here, X is a positive definite reactance matrix, and u(v_t) is the control-law vector function composed of all nodal controllers. At the voltage-deviation equilibrium point v_t = 0, the condition u(0) = 0 is satisfied.

3.2. Stability Constraint Satisfying Contraction Mapping

According to the fixed-point theorem for contraction mappings, to ensure that the closed-loop operator F(v_t) = v_t+1 is exponentially stable within a neighborhood Ω of the voltage-deviation equilibrium point v_t = 0, there must exist a contraction factor γ ∈ (0, 1) such that Equation (10) holds for any v_a, v_b ∈ Ω.

$$\parallel F\left(v_a\right)-F\left(v_b\right)\parallel \le \gamma \parallel v_a-v_b\parallel$$

(10)

For a differentiable mapping F(·), a sufficient condition for satisfying the contraction mapping property is given by Equation (11).

$$\underset{v\in \mathrm{\Omega }}{\sup }\parallel J\left(v\right){\parallel }_2 = \underset{v\in \mathrm{\Omega }}{\sup }\parallel {\displaystyle \frac{\partial F}{\partial v}}{\parallel }_2 < 1$$

(11)

From the update equation F(v_t) = v_t+1 = v_t− Xu(v_t), the corresponding Jacobian matrix can be expressed as Equation (12).

$$\mathrm{J}\left({\mathit{v}}_t\right)=I-X{\nabla }_v\mathit{u}\left({\mathit{v}}_t\right)$$

(12)

Under fully decentralized control, the reactive power absorption u_t,j of the j-th agent depends only on the local voltage v_t,j. Therefore, Equation (13) can be obtained.

$${\nabla }_{\mathit{v}}\mathit{u}\left({\mathit{v}}_t\right)=\mathrm{diag}\left({\displaystyle \frac{d{u}_1\left(v_1\right)}{d{v}_1}},\dots ,{\displaystyle \frac{d{u}_N\left(v_N\right)}{d{v}_N}}\right)=\mathrm{diag}\left(\mathit{k}\right)$$

(13)

In Equation (13), k = [k₁, k₂, …, k_N]^T denotes the policy slope vector of the nodal controllers. Let D = ∇_vu(v_t). Since D ≻ 0 and X ≻ 0, the matrix (I − XD) has the same eigenvalues as (I − D^1/2XD^1/2), and the latter is symmetric. To ensure that J(v_t) = I − XD satisfies the condition in Equation (11), its eigenvalues must lie within (−1, 1), as shown in Equation (14).

$$-I\prec I-D^{1/2}X{D}^{1/2}\prec I\iff 0\prec D^{1/2}X{D}^{1/2}\prec 2I\iff D\prec 2X^{-1}$$

(14)

Accordingly, the controller gain constraint that ensures F(v_t) satisfies the contraction mapping condition in a neighborhood of v_t = 0 can be derived as Equation (15).

$$diag\left(\mathit{k}\right)\prec 2X^{-1}$$

(15)

When Equation (15) holds within the specified neighborhood, the nodal voltage deviations converge to zero at a geometric rate, thereby guaranteeing the local exponential stability of the system at the zero voltage-deviation equilibrium point.

The stability constraint in Equation (15) is a sufficient local condition derived for radial or radially operated distribution networks under the LinDistFlow voltage sensitivity model. In many practical distribution systems, although tie lines or ring-main structures may exist for reliability and service restoration, feeders are commonly operated in an open-loop radial configuration. Under this condition, after removing the slack bus, the reduced voltage–reactive power sensitivity matrix X is generally positive definite in the considered operating region, which is consistent with the systems used in this study.

Under stronger electrical coupling, the off-diagonal entries of the voltage–reactive power sensitivity matrix X increase the closed-loop interaction among local agents and may enlarge λ_max(X), thereby reducing the admissible gain bound k_max. Thus, Equation (15) should be interpreted as a sufficient local stability condition rather than a necessary one. When this condition is satisfied, local stability can be preserved, although the controller may become more conservative and less aggressive in voltage regulation. For large-scale or highly coupled feeders, a smaller k_max, online sensitivity estimation, feeder partitioning, or limited neighboring voltage measurements can be adopted to improve coordination while maintaining the stability margin.

3.3. Effect of Model Mismatch and AC Power-Flow Nonlinearities

The contraction condition in Equation (15) is derived under the nominal LinDistFlow model and therefore serves as a sufficient local stability condition for the linearized closed-loop voltage dynamics. In practical systems, the actual voltage–reactive power sensitivity may deviate from X due to branch losses, nonlinear AC power-flow effects, and operating-point variations. The resulting local voltage update can be expressed as

$${\mathit{v}}_{t+1}={\mathit{v}}_t-(X+\Delta X)\mathit{u}({\mathit{v}}_t)+\mathit{r}({\mathit{v}}_t,{\mathit{u}}_t)$$

(16)

where ΔX denotes the sensitivity mismatch and r(v_t, u_t) collects the higher-order nonlinear residuals. To rigorously bound the impact of these unmodeled dynamics, L_r is introduced as the Lipschitz constant of the residual function r(·) with respect to the voltage deviation, which satisfies ||r(v_a, u) − r(v_b, u)||₂ ≤ L_r ||v_a − v_b||₂. Local contraction is preserved if the perturbation terms are sufficiently small compared with the nominal contraction margin, e.g.,

$$\Vert I-XD(\mathit{v}){\Vert }_2+\Vert \Delta XD(\mathit{v}){\Vert }_2+L_r<1, v\in \mathrm{\Omega }$$

(17)

where D(v) = ∇_vu(v) represents the Jacobian matrix of the decentralized control law evaluated at voltage v. Therefore, uncertainty in X does not invalidate the proposed constraint as long as the resulting perturbation remains within the available contraction margin. In practice, this margin can be enlarged by conservatively reducing the admissible policy-slope bound k_max, which improves robustness to parameter uncertainty but may lead to a slower voltage regulation response. For the full AC power-flow model, Equation (15) does not automatically constitute a stability certificate. The corresponding condition should be verified using the local AC sensitivity matrix:

$$X_{\mathrm{ac}}(v)={\displaystyle \frac{\partial v}{\partial q}}$$

(18)

If ${\sup }_{v\in \mathrm{\Omega }}\Vert I-X_{\mathrm{ac}}(v)D(v){\Vert }_2<1$, the closed-loop AC voltage dynamics remain locally contractive in Ω.

Therefore, the local exponential stability conclusion holds for radial or radially operated feeders under the LinDistFlow sensitivity model, when the voltage deviation remains within the neighborhood Ω, the disturbance varies slowly within the fast reactive-power regulation window, and the implemented decentralized policy satisfies the gain constraint in Equation (15). If the required compensation exceeds the available device capacity, the tanh mapping still guarantees action feasibility, but exact zero-error convergence is no longer guaranteed by the local contraction proof.

4. Stability-Constrained Voltage–Var Self-Optimizing Control Method for Distribution Networks Based on the BOST-GRPO Algorithm

4.1. Formulation of a Partially Observable Markov Decision Process for Distributed Reinforcement Learning

Under a fully decentralized architecture, various reactive power regulation devices in the distribution network are modeled as independent agents. Each agent makes adaptive decisions based only on the real-time voltage measurement at its corresponding node. Considering the uncertainties of power flow equations and PV/load disturbances, this paper formulates the online voltage regulation process as a sequential decision-making problem with local observations: at time t, agent j receives the local observation o_t,j = v_t,j and outputs the action a_t,j = u_t,j. It should be noted that this local-observation setting does not indicate that each agent is trained in an isolated single-bus environment. Instead, all agents interact with the coupled distribution network environment, where the voltage response at each bus is affected by the joint reactive power actions of all agents through the sensitivity matrix X. Therefore, although only local voltage measurements are used during online execution, the learned local policy implicitly reflects the influence of spatial coupling on the local voltage response.

To improve training stability and sample efficiency while maintaining communication-free execution, the GRPO algorithm is adopted for policy iteration. This algorithm inherits the clipped update framework of proximal policy optimization and constructs the advantage function by using intra-group relative returns as the baseline, thereby eliminating the dependence on a value network. Meanwhile, Kullback–Leibler (KL) divergence regularization is introduced to constrain policy drift, significantly reducing computational overhead. Therefore, it is particularly suitable for distributed control scenarios with limited computing resources at the edge devices.

During the m-th update, for each training scenario, we sample M interaction trajectories $\{{\mathrm{\tau }}_j^{\left(i\right)}{\}}_{i=1}^M$ and compute the corresponding returns $\left\{R_j^{\left(i\right)}\right\}$. For the j-th agent, the group-relative advantage is constructed as Equation (19).

$${\widehat{A}}_j^{\left(i\right)}={\displaystyle \frac{R_j^{\left(i\right)}-{\overline{R}}_j}{{\sigma }_R+\delta }}$$

(19)

In Equation (19), ${\overline{R}}_j$ and σ_R denote the intra-group mean and standard deviation, respectively, and δ > 0 is a numerical stability term. The policy parameters θ_j of each agent are then updated independently using the GRPO objective with a clipping term and KL regularization, as shown in Equation (20).

$$\underset{{\theta }_j}{max}{\mathbb{E}}_t\left[min\left(r_t\left({\theta }_j\right){\widehat{A}}_t,\mathrm{clip}\left(r_t\left({\theta }_j\right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right]-\beta D_{\mathrm{KL}}\left({\pi }_{{\theta }_j}\parallel {\pi }_{\mathrm{ref}}\right)$$

(20)

In Equation (20), ε is the clipping coefficient, β is the KL regularization weight, and π_ref denotes the reference policy. The corresponding KL regularization term is given in Equation (21).

$$r_t\left({\theta }_j\right)={\displaystyle \frac{{\pi }_{{\theta }_j}\left(a_{t,j}\mid o_{t,j}\right)}{{\pi }_{{\theta }_j^m}\left(a_{t,j}\mid o_{t,j}\right)}}$$

(21)

In the above optimization process, the policy parameters θj of different agents are decoupled. Therefore, advantage estimation and policy updating can be performed independently and in parallel for each agent. During execution, no communication is required, and the reactive power control command can be generated solely based on local voltage deviation measurements.

4.2. Embedding of Safety Constraints

The reinforcement learning policy directly acts on distribution network voltages in a closed-loop manner. If the policy is driven only by the reward signal, it may generate out-of-limit control actions, or cause iterative oscillations and even divergence due to excessive equivalent local gains. Therefore, this paper embeds the aforementioned sufficient stability condition satisfying contraction mapping and the maximum equipment output constraint as hard constraints into the policy output layer. As a result, the policy is restricted by the same constraint set during both training and online operation, thereby ensuring the feasibility of control actions and improving closed-loop stability by design.

4.2.1. Safety Constraint Satisfying Contraction Mapping

As derived above, when the control law of each agent satisfies Equation (15), the voltage deviation is exponentially stable within a neighborhood of v_t = 0. To implement this condition in the policy structure, the gain parameters of all agents are projected during both training and execution, as shown in Equation (22).

$$\mathit{k}\leftarrow \mathrm{clip}\left(\mathit{k},{\mathit{k}}_{\min },{\mathit{k}}_{\max }\right)$$

(22)

In Equation (22), k_max = [k_max, …, k_max]^T, k_max = 2/λ_max(X), and k_min > 0.

In the implemented policy, each local actor takes the scalar local voltage deviation as input and outputs the corresponding reactive-power command. Therefore, the policy gain k in Equation (13) corresponds to the local slope of the learned actor. In this paper, the gain projection in Equation (22) is applied during both training and execution, so that the learned policy parameters are restricted to satisfy the contraction-based gain bound. Hence, the stability condition is directly embedded into the actual policy output rather than being used only as an offline verification condition.

4.2.2. Maximum Equipment Output Constraint

To satisfy the maximum output limit of the equipment, the unconstrained intermediate variable generated by the policy network is compressed through the tanh function and scaled according to the amplitude limit to obtain the actual action, as shown in Equation (23).

$$a_{t,j}=u_{t,j}=a_{t,j}^{\max }\tan \mathrm{h}\left({\displaystyle \frac{{\widehat{u}}_{t,j}}{a_{t,j}^{\max }}}\right)$$

(23)

Here, ${\widehat{u}}_{t,j}$ denotes the unconstrained intermediate variable output by the policy network, and $a_{t,j}^{\mathrm{m}\mathrm{a}\mathrm{x}}$ represents the maximum reactive power regulation amplitude allowed by the device. This mapping naturally satisfies $\left|a_{t,j}\right|\le a_{t,j}^{\mathrm{m}\mathrm{a}\mathrm{x}}$ hereby avoiding the boundary non-smoothness and gradient issues caused by hard clipping. While ensuring the feasibility of control commands, it facilitates policy-gradient updates and stable training, and further improves the safety and reliability of online execution.

4.3. Group Relative Policy Optimization Control Method Based on the Bandit-Guided Online Self-Tuning Algorithm

To further improve the training stability and sample efficiency of GRPO under multiple operating conditions, this paper proposes the BOST algorithm and integrates it into the GRPO update process as an embedded outer loop, forming the BOST-GRPO training mechanism. Traditional hyperparameter optimization methods, such as Bayesian optimization, often require multiple restarts and cross-validation after complete training, resulting in substantial computational costs. In contrast, the proposed BOST algorithm enables online adaptive tuning of key hyperparameters within a single training sequence by using a multi-armed bandit (MAB) model and an upper confidence bound (UCB) strategy.

The BOST algorithm predefines discrete candidate sets for key hyperparameters, including the learning rate, exploration noise scaling factor, entropy regularization coefficient, clipping coefficient, and learning rate decay base. Specifically, BOST uses the following candidate sets: actor learning rate {0.002, 0.0025, 0.003}, initial exploration-noise scaling factor {1.0, 0.8, 1.2}, entropy coefficient {0.0005, 0.0008, 0.001}, clipping coefficient {0.15, 0.2, 0.25}, and learning-rate decay base {0.5, 0.6, 0.7}. These values are selected online within a single training process. After a fixed number of policy parameter update rounds, the BOST outer loop is activated, and the actual performance of the selected hyperparameters is evaluated according to the test return of the current policy. To eliminate the evaluation bias caused by the natural increasing trend during reinforcement learning training, an exponential moving average (EMA) is used to perform detrended smoothing on the return values, as shown in Equations (24) and (25).

$$U_t^{\mathrm{ema}}={\beta }_{\mathrm{ema}}{U}_{t-1}^{\mathrm{ema}}+\left(1-{\beta }_{\mathrm{ema}}\right)U_t$$

(24)

$${\tilde{U}}_t=U_t-U_t^{\mathrm{ema}}$$

(25)

In Equations (21) and (22), U_t denotes the test return obtained in the current round, β_ema is the smoothing coefficient, and ${\tilde{U}}_t$ is the detrended relative utility value, which is used to accurately reflect the marginal contribution of the current hyperparameter to performance improvement.

When selecting the hyperparameter for the next stage, BOST first prioritizes unexplored candidate values that are closest to the current value. Once all candidate values have been evaluated, the hyperparameter value h* with the highest score is selected according to the UCB criterion, as shown in Equation (26).

$$h^{*}=arg \underset{h}{max}\left({\overline{U}}_h+C\sqrt{{\displaystyle \frac{ln\left(K+1\right)}{K_h}}}\right)$$

(26)

In Equation (26), ${\overline{U}}_h$ denotes the average historical relative utility of the hyperparameter candidate h, K_h denotes the total number of times that this value has been selected, K denotes the total number of tuning steps for this hyperparameter, and C is the UCB adjustment coefficient controlling the degree of exploration. In addition, to ensure convergence stability and physical safety in the later stage of training, a patience-based early stopping and rollback mechanism is introduced into BOST. If the overall system cost does not decrease significantly over multiple consecutive evaluation windows, the algorithm determines that the current model performance has reached a plateau. It then actively terminates the outer-loop hyperparameter exploration and rolls back all hyperparameters to the historically best-performing configuration, which is kept fixed during the remaining training rounds to complete the final policy refinement. The main procedure of the proposed BOST-GRPO algorithm is summarized in Algorithm 1.

Algorithm 1. Main procedure of the proposed BOST-GRPO algorithm

Input: Sensitivity matrix (X), local observations, BOST candidate hyperparameter sets, trajectory group size (M), EMA coefficient, UCB coefficient, and patience threshold. Output: Trained stability-constrained local policies.

1: Initialize the local policy of each agent and compute the admissible gain bound from X.

2: Collect trajectory groups under the current decentralized policies and compute the corresponding returns.

3: Estimate the group-relative advantages according to Equation (19).

4: Update each local policy independently using the GRPO objective in Equation (20).

5: Apply the contraction-based gain projection in Equation (22) and the tanh-based action mapping in Equation (23).

6: Periodically evaluate the current policy and update the EMA-smoothed utility according to Equations (24) and (25).

7: Select the next hyperparameter value using the nearest-unvisited rule or the UCB criterion in Equation (26).

8: Roll back to the historically best hyperparameter configuration when the patience threshold is reached, and keep it fixed for final policy refinement.

9: Return the trained local policies for decentralized online execution.

This process does not require resetting policy parameters or restarting trajectory sampling. It significantly shortens the hyperparameter tuning cycle and completely avoids the large overhead caused by repeated trial-and-error training in traditional methods, making it highly suitable for distributed control scenarios with limited edge-side computing resources.

5. Case Study Analysis

5.1. Test System and Parameter Configuration

The IEEE 33-bus distribution network is selected as the test system for case verification. The system consists of one slack bus and 32 controllable buses, with base power and base voltage set to 100 MVA and 12.66 kV, respectively. According to distribution network operation standards, the safe operating range of nodal voltage is set to [0.95, 1.05] p.u. The reactive power compensation devices at each controllable bus are modeled as independent agents. Each agent takes the local voltage deviation as its state variable and interacts with the environment based on the linearized power flow model. The control action bound u_j,max is generated within the range of [−0.05, 0.05] p.u.

The algorithm simulation is implemented based on the PyTorch framework (version 2.7.1) and conducted on a computing platform equipped with an NVIDIA RTX 4050 GPU. The total number of training episodes is uniformly set to 300, the batch size is 256, the policy network contains 20 neurons, and the coordination coefficient α in the reward function is set to 0.01.

To comprehensively evaluate the superiority of the proposed BOST-GRPO control strategy in terms of training convergence, voltage control accuracy, and physical safety, the proposed method is compared with the following three algorithms.

Comparison method 1: To verify the performance improvement of GRPO over basic distributed reinforcement learning algorithms, the classical Monte Carlo policy gradient algorithm, Reinforce, is selected as the basic benchmark for distributed reinforcement learning.

Comparison method 2: To verify the necessity of BOST for improving model performance and to serve as a basic ablation benchmark, the standard GRPO algorithm without hyperparameter optimization is selected for comparison.

Comparison method 3: To verify the superiority of the proposed BOST-GRPO over conventional tuning methods, GRPO with a two-step hyperparameter optimization (THO) mechanism is selected as a benchmark. This method uses Bayesian optimization for hyperparameter search, with 40 independent trials, each consisting of 200 complete training episodes.

The basic hyperparameter configurations of all algorithms are shown in

Table 2. Basic hyperparameter configurations of different algorithms.

Hyperparameter	Reinforce	GRPO	THO-GRPO	BOST-GRPO
Initial learning rate	5 × 10−4	5 × 10−4	2.1577 × 10−4	5 × 10−4
Learning-rate decay base	1 × 10−3	1 × 10−3	1 × 10−3	1 × 10−3
Entropy coefficient	1 × 10−3	1 × 10−3	3 × 10−3	1 × 10−3
Exploration noise	0.5	0.5	0.5	0.5
Sampling size	1024	1024	1024	1024
Clipping coefficient	–	0.2	0.2277	0.2

. All experiments are conducted under the same hardware and simulation environment settings.

In the implementation, each controllable node is equipped with an independent local actor, which takes the local voltage deviation as input and outputs the corresponding reactive power control action. Each actor contains 20 internal units and four groups of trainable parameters, with each group having a size of 1 × 20. Therefore, each local actor has only 80 trainable parameters. In the IEEE 33-bus test system, there are 32 controllable nodes, and the total raw BOST-GRPO actor weights require approximately 10.24 KB when stored in single-precision floating-point format, corresponding to about 0.32 KB per local actor. Since the actor structure of each node is fixed, the model storage requirement increases approximately linearly with the number of controllable nodes.

5.2. Training Convergence Performance Analysis

Figure 2 shows the cost convergence curves of the four control algorithms during training. The cost function is composed of a weighted sum of the voltage deviation penalty and the control action cost. A lower value indicates better overall control performance.

As shown in Figure 2 and the experimental statistics, the four methods exhibit a clear progressive improvement in performance. First, in comparison method 1, the test cost of the Reinforce algorithm gradually decreases from 0.8883 but finally remains at a relatively high level of 0.7956. This indicates that using only a basic policy gradient method is insufficient to achieve a better overall trade-off in this complex control task.

Second, when the optimization framework is replaced by the standard GRPO algorithm in comparison method 2, the final average test cost decreases to 0.6853, representing a 13.86% reduction compared with Reinforce. This demonstrates that GRPO can significantly improve training effectiveness and raise the upper bound of final performance even without online self-tuning.

Furthermore, after introducing THO in comparison method 3, the final test cost reaches 0.5590, which is 18.44% lower than that of standard GRPO. This verifies the substantial benefit of combining hyperparameter optimization with group relative optimization for the overall objective. It also indicates that this scenario is highly sensitive to hyperparameters, making parameter tuning necessary. However, this performance gain comes from an additional search process, accompanied by substantial extra time and computational overhead.

Finally, after introducing BOST into standard GRPO to form BOST-GRPO, the final test cost decreases to 0.4872, which is 28.91% lower than that of standard GRPO and 12.84% lower than that of THO-GRPO, achieving the best performance among the four methods. These results show that, under the experimental settings of this paper, BOST-GRPO not only achieves better final performance than conventional offline hyperparameter optimization, but also continuously adjusts the hyperparameters according to the policy update state during training, thereby improving convergence quality and stability.

In terms of computational overhead, the training times for 300 episodes of Reinforce, GRPO, THO-GRPO, and BOST-GRPO are 24.25 s, 29.35 s, 30.49 s, and 32.33 s, respectively. The results indicate that BOST-GRPO introduces only limited additional training overhead while achieving a lower test cost. In addition, during online execution, the inference time of all four agent-based algorithms for a single time step is within 0.4 ms, which is far below the millisecond-level control cycle requirement of distribution networks.

By contrast, the main cost of THO-GRPO is usually incurred in the hyperparameter search stage, which involves multiple trials and repeated training. Its total time and computational overhead increase linearly with the search budget, and the single-training time alone cannot reflect its actual tuning cost. Therefore, by integrating hyperparameter adjustment into a single training process, BOST-GRPO avoids the repeated training and additional trial overhead required by conventional offline optimization while maintaining a lightweight implementation. This makes it more suitable for edge-side scenarios with limited computational resources.

Overall, the comparison results show that replacing Reinforce with GRPO significantly improves convergence performance. Introducing THO can further reduce the test cost but incurs additional hyperparameter search overhead. In contrast, BOST-GRPO achieves a better final convergence level and an acceptable training computational burden without offline search or repeated training, demonstrating a stronger overall advantage in balancing performance and overhead.

To further evaluate the dependence of BOST-GRPO on key hyperparameters, a one-factor-at-a-time sensitivity analysis is conducted on the IEEE 33-bus system. In each test, only one parameter is changed, while the other settings remain the same as those in the main experiment. The actor learning rate, exploration-noise scaling factor, learning-rate decay base, UCB adjustment coefficient, and EMA smoothing factor are selected to analyze their effects on algorithm performance and convergence behavior.

Table 3. Sensitivity Analysis Results of Key Hyperparameters.

Parameter	Values	Final Test Cost	Voltage RMSE	Steady-State Error
actor_lr	2.0 × 10−3/2.5 × 10−3/3.0 × 10−3	0.4668–0.4877	1.93 × 10−2–1.98 × 10−2	3.84 × 10−3–4.07 × 10−3
sigma_e0_scale	0.8/1/1.2	0.4806–0.4943	1.97 × 10−2–1.98 × 10−2	4.01 × 10−3–4.09 × 10−3
lr_decay_base	5/6/7	0.4801–0.4939	1.95 × 10−2–1.99 × 10−2	3.95 × 10−3–4.15 × 10−3
UCB_coef	0/0.5/1	0.4868–0.4886	1.98 × 10−2–1.98 × 10−2	4.04 × 10−3–4.05 × 10−3
EMA_beta	7/9/9.7	0.4876–0.4886	1.98 × 10−2–1.98 × 10−2	4.05 × 10−3–4.05 × 10−3

presents the sensitivity analysis results of the key hyperparameters.

It can be observed that BOST-GRPO maintains stable training and effective voltage regulation under different hyperparameter settings, and no voltage violation occurs in all test cases. The actor learning rate, exploration-noise scaling factor, and learning-rate decay base have a certain influence on the final performance, but the overall fluctuation remains limited. In contrast, the UCB adjustment coefficient and EMA smoothing factor have only minor effects on the final test cost, voltage RMSE, and steady-state error, indicating that the BOST outer-loop selection mechanism is not sensitive to its own parameters. These results demonstrate that BOST-GRPO has good robustness and engineering reproducibility within a reasonable hyperparameter range.

5.3. Comparison of Controller Dynamic Performance

To evaluate the dynamic response and steady-state performance of different control strategies during operation, a typical undervoltage disturbance scenario is selected for comparison. In this scenario, the initial average system voltage is approximately 0.93 p.u., and 22 buses initially violate the undervoltage limit. Figure 3 and Figure 4 show the dynamic response curves of the average nodal voltage and the average reactive power action magnitude under different control algorithms, respectively. In Figure 3, the red dashed horizontal lines indicate the allowable voltage operating limits of 0.95 p.u. and 1.05 p.u.

From the time-domain response of the average voltage, all four methods can restore the system average voltage to the safe operating range within the first control step. This indicates that all methods have effective undervoltage correction capability under action constraints. Combined with the average action response curves in Figure 4, this rapid initial correction is mainly attributed to the large reactive power compensation commands rapidly generated by each algorithm after the disturbance occurs.

However, significant differences exist among the strategies in terms of voltage recovery magnitude and subsequent tracking accuracy. At the second time step, the average voltages under Reinforce, standard GRPO, THO-GRPO, and BOST-GRPO recover to 0.9617, 0.9667, 0.9772, and 0.9810 p.u., respectively. As indicated by Figure 4, BOST-GRPO and THO-GRPO adopt more aggressive reactive power regulation actions to achieve better voltage recovery, thereby producing a higher initial voltage rise.

Overall, as the voltage gradually approaches the safe margin, the reactive power regulation actions of all algorithms show a smooth convergence trend. However, BOST-GRPO maintains the highest voltage recovery trajectory over the entire time horizon. In the subsequent convergence stage, it effectively overcomes the steady-state residual error caused by the conservative actions of Reinforce and standard GRPO through more refined and continuous reactive power coordination. As a result, its steady-state voltage is closer to the nominal reference value, demonstrating a better overall trade-off between voltage control quality and action regulation intensity.

5.3.1. Steady-State Control Accuracy

Table 4. Comparison of key performance metrics of four control strategies.

Metric	Reinforce	GRPO	THO-GRPO	BOST-GRPO
Final test cost	0.7956	0.6853	0.5590	0.4872
Mean steady-state error	8.93 × 10−3	7.37 × 10−3	4.73 × 10−3	4.05 × 10−3
Voltage RMSE	0.0241	0.0228	0.0205	0.01980
Voltage MAE	0.0171	0.0152	0.0118	0.01060
Overshoot	0.0155	0.0148	0.0171	0.01740
Number of voltage violations after the first control step	0	0	0	0
Average absolute action value	0.0026	0.0029	0.0039	0.0042
Training time for 300 episodes	24.25 s	29.35 s	30.49 s	32.33 s

summarizes the overall error and action characteristics during the dynamic process. According to Table 4, the four strategies show significant differences in steady-state control accuracy. First, the mean steady-state voltage error of Reinforce is 8.93 × 10⁻³, indicating relatively limited control accuracy. Second, after adopting the standard GRPO algorithm, this error decreases to 7.37 × 10⁻³, although a certain residual error remains. Furthermore, the THO-GRPO algorithm with conventional hyperparameter optimization effectively reduces the error to 4.73 × 10⁻³. Finally, the proposed BOST-GRPO algorithm reduces the steady-state error to a very low level of 4.05 × 10⁻³, representing reductions of 45.0% and 14.4% compared with standard GRPO and THO-GRPO, respectively. This result indicates that BOST-GRPO can achieve more refined reactive power coordination and further suppress residual voltage deviations.

5.3.2. Dynamic Regulation Characteristics

In terms of dynamic regulation trajectories, BOST-GRPO exhibits smoother and more accurate regulation characteristics. Its voltage root mean square error (RMSE) over the entire control period is 0.0198, outperforming THO-GRPO, standard GRPO, and Reinforce, whose RMSE values are 0.0205, 0.0228, and 0.0241, respectively. This indicates that BOST-GRPO provides better voltage quality over the entire time horizon. The corresponding voltage mean absolute error is also the lowest, at 0.01060, demonstrating stronger deviation suppression during the dynamic stage.

Regarding the equipment action cost, the average absolute magnitude of the reactive power control actions output by BOST-GRPO is approximately 4.22 × 10⁻³. This indicates that the algorithm adopts more active regulation actions to achieve a higher voltage recovery level. Nevertheless, no physical constraint violations are triggered, and the overshoot remains within a controllable range. Overall, BOST-GRPO achieves a higher voltage recovery level under smaller dynamic errors, reflecting a better trade-off between voltage quality and action intensity.

5.4. Analysis of Physical Constraints and Safety

In distribution network volt–var control, action feasibility and safety are prerequisites for algorithm deployment. In this paper, a hyperbolic tangent function-based action compression and amplitude scaling mechanism is integrated at the output end of the policy network. As a result, all output actions are effectively restricted within the physical capacity range of [−0.05, 0.05] p.u., thereby ensuring the output safety of control devices by design. Simulation statistics show that the constraint violation rate of all four algorithms remains strictly at 0.0% during both the high-exploration stage at the beginning of training and the steady-state testing stage. This indicates that the policy outputs always remain within the feasible region, and no actions go out of bounds.

In terms of dynamic recovery speed, the settling time of all four methods is one simulation step, indicating that they can all achieve rapid correction after an undervoltage disturbance. Under the same safety constraints, different methods show significant differences in recovery accuracy and overall cost. BOST-GRPO further reduces dynamic error and control cost while achieving the fastest safe recovery, demonstrating strong disturbance recovery capability.

5.5. Robustness Analysis Under Model Mismatch and Operating-State Variations

To further evaluate the adaptability of BOST-GRPO under model mismatch and operating-state variations, two robustness tests are conducted without retraining the controllers. The Reinforce, GRPO, THO-GRPO, and BOST-GRPO policies trained in the IEEE 33-bus case are directly used, while only the test environment or the initial voltage condition is changed.

First, the uncertainty of the voltage-reactive power sensitivity matrix is considered. During testing, the actual sensitivity matrix is set as X_test = (1 + δ)X_nominal, where δ takes the values 0, ±5%, ±10%, ±15%, and ±20%. For each mismatch level, 20 Monte Carlo simulations are performed, and the same initial-state samples are used for the four controllers. Since the controller parameters are not updated during testing, this experiment reflects the robustness of the trained policies to finite sensitivity-matrix mismatch. Figure 5 and Figure 6 show the voltage RMSE and average action magnitude under different reactance perturbation ratios, respectively.

As shown in

Table 5. Average performance comparison of different controllers under sensitivity-matrix mismatch.

Controller	Total Cost	Voltage RMSE	Steady-State Error
Reinforce	1.1615	3.7369 × 10−3	9.9682 × 10−4
GRPO	1.0024	3.4983 × 10−3	8.5313 × 10−4
THO-GRPO	0.8031	3.2484 × 10−3	6.6920 × 10−4
BOST-GRPO	0.7088	3.1300 × 10−3	5.6937 × 10−4

, BOST-GRPO achieves the lowest total cost, voltage RMSE, and steady-state error among all controllers. Compared with standard GRPO, BOST-GRPO reduces the average total cost by 29.28%, the average voltage RMSE by 10.53%, and the average steady-state error by 33.26%, indicating better robustness under sensitivity-matrix mismatch. Under the strongest mismatch cases, BOST-GRPO still reduces the total cost by 30.33% when X is decreased by 20% and by 26.60% when X is increased by 20%.

Second, the initial voltage disturbance intensity is varied while the nominal sensitivity matrix is kept unchanged. The initial average voltage is set to 0.93, 0.95, 0.97, 1.00, 1.03, 1.05, and 1.07 p.u., and 20 Monte Carlo simulations are conducted for each level. This test covers both undervoltage and overvoltage operating conditions, and the voltage range remains within the practical region where inverter-based reactive power control is meaningful. Figure 7 and Figure 8 show the voltage RMSE and average action magnitude under different initial voltage levels, respectively. In Figure 7 and Figure 8, the red dashed vertical lines indicate the lower and upper allowable voltage limits of 0.95 p.u. and 1.05 p.u.

As shown in

Table 6. Average performance comparison of different controllers under initial voltage disturbance tests.

Controller	Total Cost	Voltage RMSE	Steady-State Error	Average Number of Voltage Violations
Reinforce	9.1580	2.1587 × 10−2	9.7880 × 10−3	47.99
GRPO	8.5873	2.0874 × 10−2	9.0456 × 10−3	45.06
THO-GRPO	7.6111	1.9725 × 10−2	7.9302 × 10−3	41.70
BOST-GRPO	6.7704	1.8656 × 10−2	6.7641 × 10−3	38.14

, BOST-GRPO maintains the lowest total cost, voltage RMSE, steady-state error, and average number of voltage violations among all controllers. Compared with standard GRPO, BOST-GRPO reduces the average total cost by 21.16%, the average voltage RMSE by 10.63%, the average steady-state error by 25.22%, and the average number of voltage violations by 15.37%. In the 0.93 p.u. undervoltage case, the improvement is the most pronounced: BOST-GRPO reduces the total cost by 30.54%, the voltage RMSE by 14.70%, the steady-state error by 40.84%, and the number of voltage violations by 23.88% compared with standard GRPO. In the 1.07 p.u. overvoltage case, BOST-GRPO still maintains a performance advantage, reducing the four metrics by 11.44%, 5.73%, 12.07%, and 9.28%, respectively. These results indicate that BOST-GRPO improves voltage recovery performance under both undervoltage and overvoltage disturbances, with a more evident advantage in the undervoltage case.

The action characteristics provide further insight into this robustness advantage. In both robustness tests, BOST-GRPO produces larger average action magnitudes and action variations than the other algorithms, indicating that it adopts more active and dynamically adjusted reactive power regulation. This behavior is particularly beneficial under severe undervoltage disturbances, where stronger reactive power support is required for voltage recovery. Meanwhile, the contraction-mapping-based stability constraint embedded in the policy output layer restricts the control commands within a safe convergence region, and the action-capacity mapping ensures that all reactive power commands satisfy the physical device limits. Therefore, BOST-GRPO can exploit the available control margin more effectively without inducing voltage instability, which supports its robustness advantage in practical deployment.

5.6. Scalability Analysis on the IEEE 69-Bus Distribution System

To further examine the scalability of the proposed method, an extension experiment is conducted on the IEEE 69-bus radial distribution system. The dataset is generated from the MATPOWER case69/Das 69-bus system, and the system contains 68 controllable buses. Compared with the IEEE 33-bus system, the IEEE 69-bus system has more than twice the number of buses, a larger reactance matrix, more controllable nodes, and a more complex power-flow coupling structure. Therefore, it provides a more challenging testbed for evaluating whether the decentralized control architecture and online hyperparameter tuning mechanism of BOST-GRPO remain effective as the system scale increases. All controllers are trained for 400 episodes. Except for the network topology, action-capacity data, and number of training episodes, the algorithmic implementation and evaluation protocol are kept consistent with the main IEEE 33-bus experiment. The dynamic simulation uses the same undervoltage disturbance scenario, with an initial average voltage of 0.93 p.u. The reactive power capacity limits are determined from the system data, and the stability gain bound k_max is recomputed using the reactance matrix of the IEEE 69-bus system. To compare the training convergence behavior of different algorithms after the system scale is increased, Figure 9 shows the training cost curves of the four controllers on the IEEE 69-bus system.

The IEEE 69-bus results show a performance trend consistent with the main IEEE 33-bus case. BOST-GRPO achieves the lowest final test cost of 5.0223, compared with 6.1514 for Reinforce, 5.8290 for GRPO, and 5.4584 for THO-GRPO. Compared with standard GRPO, BOST-GRPO reduces the final test cost by 13.84%, the voltage RMSE by 5.12%, the steady-state error by 12.83%, and the number of voltage violations by 7.97%. The settling time is also reduced from five control steps to four control steps. Compared with THO-GRPO, BOST-GRPO further reduces the final test cost by 7.99%, while achieving a slightly lower voltage RMSE, fewer voltage violations, the same settling time, and a comparable steady-state error. These results indicate that BOST-GRPO still maintains favorable overall control performance on the larger feeder.

The training curves further show that THO-GRPO decreases rapidly during the early training stage, but its test cost increases again after reaching a relatively low value around the first 100 episodes. This indicates that the fixed hyperparameters obtained by offline tuning can accelerate early convergence, but may not remain optimal throughout the entire training process. In contrast, BOST-GRPO exhibits a more sustained decreasing trend and achieves both a lower minimum test cost and a lower final test cost. This suggests that the online self-tuning mechanism helps maintain convergence quality in the later training stage. Nevertheless, this observation should be interpreted as an auxiliary finding from the current IEEE 69-bus experiment, rather than as evidence that THO-GRPO is generally unstable. To further analyze the closed-loop dynamic response under the IEEE 69-bus undervoltage disturbance scenario, Figure 10 and Figure 11 show the average voltage trajectories and average reactive power action curves of the four controllers, respectively.

The dynamic voltage trajectories further confirm that BOST-GRPO can achieve faster and higher-level voltage recovery on the larger feeder. Under BOST-GRPO, the average voltage increases from 0.9300 p.u. to 0.9628 p.u. within the first four control updates and finally reaches 0.9904 p.u.; in comparison, the final average voltage under GRPO is 0.9876 p.u. Therefore, BOST-GRPO can still maintain the physical feasibility of reactive power actions and effective voltage recovery when coordinating a larger number of decentralized local controllers.

This result is consistent with the decentralized design of the proposed method. During execution, each nodal controller generates reactive power commands based only on its local voltage measurement, without requiring global state aggregation or centralized online coordination. Meanwhile, the stability constraint is constructed through the local policy-gain bound derived from the voltage-reactive power sensitivity model. Therefore, the IEEE 69-bus experiment provides supportive evidence that the proposed architecture can be extended to a larger radial feeder without introducing centralized control or communication-assisted coordination.

It should also be noted that the performance margin of BOST-GRPO on the IEEE 69-bus system is smaller than that in the main IEEE 33-bus experiment. This result should not be interpreted as a failure of scalability, because the qualitative ranking is preserved and BOST-GRPO still achieves the lowest final test cost, with a 13.84% reduction compared with GRPO. One possible explanation is that the larger feeder introduces more controllable nodes and stronger voltage–reactive power coupling, which makes the control task more challenging and may reduce the marginal benefit of online hyperparameter tuning. Therefore, the IEEE 69-bus experiment provides supportive evidence of scalability within the tested radial-feeder setting, rather than a complete validation across all distribution-network configurations.

6. Conclusions

To address voltage violations and stochastic voltage fluctuations caused by the high penetration of distributed photovoltaic generation in distribution networks, this paper proposes a stability-constrained voltage–var self-optimizing control method based on the BOST-GRPO algorithm. The proposed method establishes a communication-free, decentralized, and locally observable reinforcement learning control framework, enabling each node to independently generate reactive power regulation commands using only local voltage measurements. Meanwhile, a contraction-mapping-based stability constraint and a tanh-based action-capacity mapping are embedded into the policy output process, thereby incorporating closed-loop stability requirements and device physical constraints into the learned control policy and improving its safety and deployability.

Simulation results on the IEEE 33-bus test system show that BOST-GRPO achieves better voltage regulation performance than Reinforce, standard GRPO, and THO-GRPO in terms of final test cost, voltage deviation suppression, steady-state error, and regulation speed. In the main IEEE 33-bus experiment, BOST-GRPO obtains the lowest final test cost of 0.4872, compared with 0.7956 for Reinforce, 0.6853 for GRPO, and 0.5590 for THO-GRPO. It also reduces the mean steady-state error to 4.05 × 10⁻³ and the voltage RMSE to 0.0198. Under sensitivity-matrix mismatch, BOST-GRPO reduces the average total cost, voltage RMSE, and steady-state error by 29.28%, 10.53%, and 33.26%, respectively, compared with standard GRPO. Under different initial voltage disturbances, it reduces the average total cost, voltage RMSE, steady-state error, and voltage violations by 21.16%, 10.63%, 25.22%, and 15.37%, respectively. In the IEEE 69-bus scalability test, BOST-GRPO achieves the lowest final test cost of 5.0223 and reduces the final test cost by 13.84% compared with GRPO. These results indicate that the proposed method maintains favorable robustness and scalability within the tested systems and operating scenarios.

Future work will further validate the proposed method under more engineering-oriented scenarios, including hardware-in-the-loop experiments and field deployment tests, closed-loop stability verification under full AC power-flow models, and generalization evaluation on more typical radial feeders, strongly coupled feeders, and systems of different scales. In addition, the deployment performance of the proposed control policy on practical edge hardware, such as microcontrollers and edge gateways, will be further tested.