Dynamic Leader Election and Model-Free Reinforcement Learning for Coordinated Voltage and Reactive Power Containment Control in Offshore Island AC Microgrids

Abstract

Island microgrids are essential for the exploitation and utilization of offshore renewable energy resources. However, voltage regulation and accurate reactive power sharing remain significant technical challenges that need to be addressed. To tackle these issues, this paper proposes an algorithm that integrates a dynamic leader election (DLE) mechanism and model-free reinforcement learning (RL). The algorithm aims to address the issue of fixed leaders restricting reactive power flow between buses during heavy load variations in island microgrids, while also overcoming the challenge of obtaining model parameters such as resistance and inductance in practical microgrids. First, we establish a voltage containment control and reactive power error model for island alternating current (AC) microgrids and construct a corresponding value function based on this error model. Second, a dynamic leader election algorithm is designed to address the issue of fixed leaders restricting reactive power flow between buses due to preset voltage limits under unknown or heavy load conditions. The algorithm adaptively selects leaders based on bus load, allowing the voltage limits to adjust accordingly and regulating reactive power flow. Then, to address the difficulty of accurately acquiring parameters such as resistance and inductance in microgrid lines, a model-free reinforcement learning method is introduced. This method relies on real-time measurements of voltage and reactive power data, without requiring specific model parameters. Ultimately, simulation experiments on offshore island microgrids are conducted to validate the effectiveness of the proposed algorithm.

1. Introduction

Renewable generation is playing an increasingly important role in the development and utilization of marine resources [1,2]. Specifically, the proportion of renewable generation units in offshore island microgrids is higher than that in land-based microgrids. These renewable generation units are integrated into the grid via inverters that convert direct current (DC) into AC [3]. Voltage serves as a crucial parameter in island AC microgrid operation [4]. Proper reactive power allocation affects not only power balance but also voltage regulation and line loss optimization. Since island microgrids are not connected to the main grid, voltage and reactive power regulation depend entirely on internal coordinated control [5]. Additionally, island microgrids operate under harsh marine climatic conditions, which pose significant challenges to their operational security. Given their important role in promoting the development of marine resources, voltage regulation and reactive power sharing have become key issues that require extensive research [6].

Voltage regulation and reactive power sharing have been extensively investigated in previous research. The average voltage regulation method in [7] ensures accurate reactive power sharing but may violate voltage safety constraints under load fluctuations. The weighted coefficient method in [8] maintains voltage within safe limits but relies on empirical tuning, limiting scalability. The optimization-based approach in [9] achieves precise control but is computationally intensive and hard to implement in real time. These limitations hinder the practical deployment of these methods in island microgrids.

To address the conflict between voltage regulation and reactive power sharing in island AC microgrids, containment control has gradually attracted increasing attention. In [10], the use of containment control was first proposed to balance voltage regulation and reactive power sharing in microgrids, laying a theoretical foundation for subsequent research in this field. The application of containment control was further extended in [11] to address voltage regulation issues among multiple interconnected microgrids, which significantly improved the performance and stability of coordinated multi-microgrid operation. In [12], the discussion focuses on a containment control-based strategy for voltage regulation in microgrids facing communication and sensor failures. However, in the above methods, the leaders for containment control are typically predetermined. When the bus associated with the upper-bound leader experiences a heavy load, its voltage may drop significantly, which contradicts the assumption that the upper leader should always maintain the highest voltage. This situation leads to ineffective reactive power transfer among buses. The fixed leader configuration lacks flexibility and reduces the adaptability of the island microgrid under dynamic operating conditions.

In island microgrids that have already been put into operation, it is difficult to accurately obtain model parameters such as resistance and inductance due to the limited precision of measurement devices and the influence of operating conditions (such as temperature variations). In contrast, data-driven approaches that do not rely on explicit model parameters can directly utilize operational data from the microgrid for controller design. In [13], a deep learning-based secondary controller for microgrids is designed using historical data. In [14], Koopman’s operator theory enables voltage control based on input–output data. In [15], a data-driven distributed predictive control method achieves voltage restoration and current sharing via an incremental linear model. In [16], least squares and Gaussian process regression are used to learn system sensitivity and estimate modeling errors, ensuring optimal and safe microgrid control. However, the method in [13] requires processing a large quantity of offline data, while the approach in [15] has high computational complexity and slow convergence of control performance, making it difficult to satisfy the real-time requirements of microgrid control. The methods in [14,16] rely heavily on prior knowledge and historical data. These issues limit the widespread application of data-driven methods in practical island microgrids.

Based on the above discussion, this paper proposes an island microgrid voltage regulation and reactive-power-sharing control strategy that combines a dynamic leader election algorithm with a model-free reinforcement learning algorithm. The proposed strategy utilizes a leader election algorithm to dynamically adjust the leader roles according to the load conditions of each bus in the island microgrid: the distributed generation (DG) corresponding to the bus with a higher load is set as the lower-bound leader, while the DG associated with the bus with a lighter load is set as the upper-bound leader. In this way, flexible reactive power flow among buses is promoted, enabling precise reactive power sharing. Meanwhile, by designing value functions for voltage and reactive power errors and employing a model-free reinforcement learning algorithm, the controller is designed based solely on island microgrid operational data without requiring any model information. Furthermore, this paper theoretically proves the convergence of the leader election algorithm and the optimality of the policy iteration algorithm in model-free reinforcement learning. Lastly, the proposed strategy is validated through a series of simulation experiments. In the experiments, the effectiveness of the proposed method is validated in three distinct case studies, which confirm that it restores the voltage of island AC microgrids to the reference range set by containment control and accomplishes accurate reactive power sharing. The main contributions of this research are summarized below:

• To address the limitations of containment control methods based on fixed leaders, as proposed in [10,11], which struggle with complex scenarios like sudden large load changes, this paper introduces a novel DLE algorithm. Unlike the static nature of fixed-leader approaches, our DLE mechanism is based on bus voltage estimation, allowing each DG to dynamically select the leader according to the relative magnitude of the estimated voltages. This adaptive capability enables accurate reactive power sharing even under sudden load changes or large load fluctuations, significantly enhancing the microgrid’s flexibility.
• To overcome the challenges of model-based controller design, as highlighted in [17,18], where obtaining practical parameters like resistance and inductance is difficult, this paper proposes a data-driven online reinforcement learning approach. In contrast to model-based methods that are sensitive to parameter uncertainties and measurement errors, our algorithm does not require extensive offline data processing. The control policy is iteratively optimized online by minimizing a value function, enabling accurate reactive power sharing and effective voltage control in the microgrid without relying on a precise system model.

The structure of this paper is as follows: The necessary background and preliminaries are outlined in Section 2. Section 3 proves the convergence of policy iteration, proposes a dynamic leader election algorithm, and designs a model-free reinforcement learning algorithm. Section 4 validates the effectiveness of the proposed methods through numerical experiments. Finally, Section 5 summarizes the main contributions of this paper.

2. Preliminaries and Problem Formulation

This section presents the graph theory, island microgrid modeling, analysis of reactive power and voltage coupling, and the formulation of performance indices for containment control.

2.1. Graph Theory

This paper considers an island AC microgrid modeled as a multi-agent system (MAS), comprising N follower agents and M virtual leader agents. The MAS is represented by an undirected graph $\Psi =(V,\Im )$, where $V=\left\{v_0,v_1,\dots ,v_N\right\}$ denotes the node set, while $\Im \subseteq V\times V$ corresponds to the collection of edges. Two nodes $v_i$ and $v_j$ are considered neighbors if $\left(v_i,v_j\right)\in \Im$. For the graph $\Psi$, $A=\left[a_{ij}\right]\in {\mathbb{R}}^{N\times N}$ is the adjacency matrix, where $a_{ij}=1$ if $\left(v_i,v_j\right)\in \Im$; otherwise, $a_{ij}=0$. The degree matrix is $D=diag\left\{d_i\right\}$, where $d_i={\sum }_{j\in N_i}{a}_{ij}$ is the degree of the ith node in the graph, and $N_i$ represents the set of neighbors of node $v_i$. The Laplacian matrix L is then defined by $L=D-A$. Matrix $S_r=diag\left\{s_1^r,\dots ,s_i^r,\dots ,s_N^r\right\}\in {\mathbb{R}}^{N\times N}$ ($i=1,\dots ,N,r=1,\dots ,M$) is the pinning gain matrix associated with the rth virtual leader, where $s_i^r=1$ if the rth virtual leader can communicate with the ith follower; otherwise, $s_i^r=0$.

2.2. Model Descriptions of Island AC Microgrids

Assume that all DGs are interfaced with the island microgrid through voltage source inverters (VSIs), each equipped with an output LC filter, as illustrated in Figure 1. By applying feedback linearization [17], the following relationship is established:

$${\ddot{\tilde{\mathit{z}}}}_i=L_{F_i}^2{h}_i+L_{g_i}{L}_{F_i}\left({\tilde{\wp }}_i\right)h_i{\tilde{V}}_i+\ddot{v}={\upsilon }_i+\ddot{v},$$

(1)

where $F_i\left({\tilde{\wp }}_i\right)=f_i\left({\tilde{\wp }}_i\right)+k_i\left({\tilde{\wp }}_i\right)D_i$, and $L_{F_i}^2{h}_i=L_{F_i}\left(L_{F_i}{h}_i\right)={\displaystyle \frac{\partial L_{F_i}{h}_i}{\partial {\tilde{\wp }}_i}}{F}_i·v$. For the sake of brevity, while the complete derivation is detailed in [11], the key terms are briefly defined here. ${\tilde{\wp }}_i$ is the state vector of the ith DG, which includes filter currents and capacitor voltages in the dq-frame. $h_i$ is the output function, defined as the bus voltage $v_{bdi}$. $f_i$ and $g_i$ represent the drift and input vector fields of the nonlinear system, respectively. $L_{F_i}{h}_i$ and $L_{F_i}^2{h}_i$ are the first- and second-order Lie derivatives of the output function with respect to the system dynamics. ${\upsilon }_i$ is the new, linearized control input. According to [11], ${\mathsf{\Xi }}_i$ is used to denote the bus voltage $v_{bdi}$. Since the control of microgrids is typically implemented within a digital control framework, it is necessary to discretize the aforementioned continuous-time equations. After discretization, the variable ${\mathsf{\Xi }}_i$ in Equation (1) can be reformulated as follows:

$${\mathsf{\Xi }}_i(k+1)=A_d{\mathsf{\Xi }}_i\left(k\right)+B_d{u}_i\left(k\right),i=1,\dots ,N$$

(2)

where the new control input is $u_i={\upsilon }_i+\ddot{v}$. The state variable is defined by ${\mathsf{\Xi }}_i={\left[v_{bdi},{\dot{v}}_{bdi}\right]}^T$. The discrete-time system matrices $A_d$ and $B_d$ are derived from A and B (as described in [11]) using the zero-order hold method with sampling interval $T_s$, where $A_d=e^{A{T}_s}$ and $B_d={\int }_0^{T_s}{e}^{A\tau }Bd\tau$ [19]. Thus, $A_d\approx \left(\begin{array}{cc}1& T_s\\ 0& 1\end{array}\right)$ and $B_d\approx \left(\begin{array}{c}{T}_s^2/2\\ T_s\end{array}\right)$.

The linearized system uses the state vector ${\mathsf{\Xi }}_i$ to describe the voltage dynamics, where $v_{bdi}$ and its derivative ${\dot{v}}_{bdi}$ are captured.

2.3. Reactive Power Sharing and Voltage Regulation

Under islanded conditions, each DG applies the conventional $Q\text{-}V$ droop control [20], i.e., $E_i={\tilde{V}}_i-n_i{Q}_i$, where $E_i$ and ${\tilde{V}}_i$ are the voltage and voltage magnitude references, $n_i$ is the droop coefficient, and $Q_i$ is the reactive power output of the ith DG. The objective of reactive power sharing is [21]: $n_1{Q}_1=n_2{Q}_2=\dots =n_i{Q}_i,i=1,2,\dots ,N$.

Following the conventional Kron reduction based on steady-state parameters [22], the reduced bus admittance matrix $\mathbf{Y}$ is obtained. The reactive power at bus i is given by [9]:

$$Q_i=v_{bdi}\sum _{j\in N_i}{v}_{bdj}\left(G_{ij}sin{\theta }_{ij}-{\mathcal{B}}_{ij}cos{\theta }_{ij}\right)$$

(3)

where $G_{ij}$ and ${\mathcal{B}}_{ij}$ are the real and imaginary parts of $\mathbf{Y}$ between buses i and j ($\mathbf{Y}=\mathbf{G}+j\mathcal{B}$), ${\theta }_{ij}={\theta }_i-{\theta }_j$ is the phase angle difference, and $N_i$ is the set of buses connected to i (including i itself). Assuming small power angles ($sin{\theta }_{ij}\approx {\theta }_{ij}$, $cos{\theta }_{ij}\approx 1$) [23] and predominantly inductive feeder impedance ($R/X\ll 1$) [24], (3) simplifies to $Q_i=-{\sum }_{j=1}^n{\mathcal{B}}_{ij}{v}_{bdj}$. Letting $\tilde{Q}=[Q,\dot{Q}]$ and combining with (2), the discrete-time dynamics of ${\tilde{Q}}_i$ are:

$${\tilde{Q}}_i(k+1)=A_d{\tilde{Q}}_i\left(k\right)-B_d\left(\sum _{j=1}^n{\mathcal{B}}_{ij}{u}_j\right)$$

(4)

Accurate reactive power sharing is difficult due to the coupling between voltage regulation and reactive power [10]. Tight voltage control limits reactive power exchange and leads to sharing imbalance. To overcome this, a containment control strategy is used to maintain voltage within set bounds. The boundary dynamics are given by:

$${\mathsf{\Xi }}_0^r(k+1)=A_d{\mathsf{\Xi }}_0^r\left(k\right),r=1,2.$$

(5)

where ${\mathbf{\omega }}_0^r=\left(\begin{array}{c}{V}_r\\ 0\end{array}\right)$, $r=1,2$. Here, $V_1$ and $V_2$ denote the upper and lower voltage reference limits. According to MAS theory, the neighborhood containment error ${\varrho }_i^v\left(k\right)$ is given by:

$${\varrho }_i^v\left(k\right)=\sum _{j\in N_i}{a}_{ij}\left({\mathsf{\Xi }}_i\left(k\right)-{\mathsf{\Xi }}_j\left(k\right)\right)+\sum _{r=1}^2{s}_i^r\left({\mathsf{\Xi }}_0^r\left(k\right)-{\mathsf{\Xi }}_i\left(k\right)\right),i=1,\dots ,N.$$

(6)

Based on (2), (5), and (6), the dynamic equation satisfied by the containment voltage error can be derived as follows:

$${\varrho }_i^v(k+1)=A_d{\varrho }_i^v\left(k\right)-\left(d_i+\sum _{r=1}^2{s}_i^r\right)B_d{u}_i\left(k\right)+\sum _{j\in N_i}{a}_{ij}{B}_d{u}_j\left(k\right)$$

(7)

The reactive power sharing error ${\varrho }_i^q\left(k\right)$ quantifies deviations in the reactive power contribution of DGs. It is defined by:

$${\varrho }_i^q\left(k\right)=\sum _{j\in N_i}{a}_{ij}\left(n_i{\tilde{Q}}_i\left(k\right)-n_j{\tilde{Q}}_j\left(k\right)\right)$$

(8)

According to (4), the dynamic equation of the reactive-power-sharing error is given by:

$${\varrho }_i^q(k+1)=A_d{\varrho }_i^q\left(k\right)-B_d\sum _{j\in N_i}{a}_{ij}\left[n_i\sum _{m=1}^n{\mathcal{B}}_{im}{u}_m-n_j\sum _{w=1}^n{\mathcal{B}}_{jw}{u}_w\right]$$

(9)

By minimizing these errors, the proposed control strategy achieves both containment-based voltage regulation and accurate reactive power sharing.

Assumption A1.

For any virtual leader, one or more paths exist that connect its dynamic behavior to every follower DG in the network.

2.4. Optimal Performance Metrics

Each DG i optimizes its cost via a game, using a local performance index as in [25] to ensure proper power sharing, low energy consumption, and voltage security:

$$J_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)=\sum _{t=k}^{\infty }{\beta }^{t-k}{M}_i\left({\varrho }_i^v\left(t\right),u_i\left(t\right),{\varrho }_i^q\left(t\right)\right)$$

(10)

where $M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)={\varrho }_i^{vT}\left(k\right){\mathsf{\Theta }}_{1ii}{\varrho }_i^v\left(k\right)+u_i^T\left(k\right){\mathsf{\Theta }}_{2ii}{u}_i\left(k\right)+{\varrho }_i^{qT}\left(k\right){\mathsf{\Theta }}_{3ii}{\varrho }_i^q\left(k\right)$, and ${\mathsf{\Theta }}_{1ii}>0$, ${\mathsf{\Theta }}_{2ii}>0$, ${\mathsf{\Theta }}_{3ii}>0$ are all positive weighting matrices. $\beta$ denotes the discount factor, satisfying $0<\beta \le 1$. Each DG optimizes its control strategy locally through communication with neighbors to achieve system objectives.

Definition 1

([25]). The control action $u_i\left(k\right)$ is considered admissible if it stabilizes Equations (7) and (9) and ensures that $J_i$ remains bounded.

For any admissible control policy $u_i\left(k\right)$, the local performance function $V_i\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ of the ith DG can be expressed as $V_i({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))=M_i({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right))+\beta V_i({\varrho }_i^v(k+1),{\varrho }_i^q(k+1))$ by applying the Bellman optimality principle. Specifically, the optimal local performance function is given by:

$$V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=\underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^{*}\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}$$

(11)

where $V_i^{*}$ denotes the optimal value function, subject to the boundary condition $V_i^{*}(0,0)=0$. Equation (11) represents the HJB equation. Accordingly, $u_i^{*}\left(k\right)$ represents the local containment control input that achieves optimality for the microgrid, and its derivation is provided below:

$$u_i^{*}\left(k\right)=-{\displaystyle \frac{\beta }{2}}{\mathsf{\Theta }}_{2ii}^{-1}\left[{\displaystyle \frac{\partial V_i^{*}}{\partial {\varrho }_i^v(k+1)}}\left(-\left(d_i+\sum _{r=1}^2{s}_i^r\right)B_d\right)+{\displaystyle \frac{\partial V_i^{*}}{\partial {\varrho }_i^q(k+1)}}\left(-B_d\sum _{j\in N_i}{a}_{ij}(n_i{\mathcal{B}}_{ii}-n_j{\mathcal{B}}_{ji})\right)\right]$$

(12)

Remark 1.

While conventional model-based methods can theoretically compute the optimal control $u_i^{*}\left(k\right)$ by solving the HJB Equation (12), their practical application is hindered by a significant limitation: the reliance on precise microgrid parameters that are often unavailable or uncertain in real-world scenarios. To address this fundamental challenge, this paper proposes a model-free reinforcement learning approach. Instead of requiring an explicit system model, this method approximates the HJB solution and derives the optimal policy directly from input–output data using an actor–critic framework. This data-driven nature represents a key advantage, enhancing the controller’s robustness and practical value compared to conventional methods that depend on an idealized and often inaccurate system model.

3. Coordinated Voltage and Reactive Power Control Scheme Design Based on DLE and RL Algorithms

This section describes the coordinated control scheme for voltage and reactive power based on DLE and RL algorithms. Figure 2 shows the overall control procedure.

3.1. Convergence Analysis of Policy Iteration

The iterative learning algorithm is applied to the containment controller as an optimization method using historical data. Each DG exchanges voltage and reactive power information with others via the communication network. A time sequence $t_1,t_2,\dots$ with interval $s=t_{k+1}-t_k$ is defined. In policy iteration, the performance function is evaluated for a feasible policy and, as s increases, both performance function $V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ and control policy $u_i^s\left(k\right)$ are iteratively updated.

Step 1: Initialize $u_i^0\left(k\right)$ and $V_i^0\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ = 0;

Step 2: Update the performance function $V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$;

Step 3: Update the control actions $u_i^{s+1}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$;

Step 4: The algorithm terminates, while $∥V_i^{s+1}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)-V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)∥\le \iota$, where $\iota$ is a predefined positive constant. The iteration index s is updated to $s+1$, and the process returns to Step 2 for further iteration.

The objective is to guarantee convergence of both the control strategy and the local performance function to their respective optimal values. To establish $u_i^s\left(k\right)\to u_i^{*}\left(k\right)$ and $V_i^s\left({\xi }_i\left(k\right)\right)\to V_i^{*}({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ as $s\to \infty$, an essential lemma is presented below.

Lemma 1

([26]). Startingfrom any initial admissible control policies $u_i^0\left(k\right)$, $V_i^s({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ and $u_i^s\left(k\right)$ are updated iteratively via Steps 2 and 3. It can be shown that $V_i^s({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ is monotonically nonincreasing, i.e., $V_i^{s+1}({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))\le V_i^s({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$.

Theorem 1.

Let $V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ and $u_i^s\left(k\right)$ be generated by Step 2 and Step 3. As $s\to \infty$, $V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ converges to the optimal value $V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$, and $u_i^s\left(k\right)$ converges to the optimum $u_i^{*}\left(k\right)$, i.e., ${lim}_{s\to \infty }{V}_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right),{lim}_{s\to \infty }{u}_i^s\left(k\right)=u_i^{*}\left(k\right).$

Proof. 

Let $V_i^s({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ denote the value function at iteration s, and define its pointwise limit as $V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)={lim}_{s\to \infty }{V}_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right).$ By Step 2 and Step 3, for all s, the following recursion holds:

$$V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=\underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^s\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}$$

(13)

First, we note that for any $ϵ>0$, there exists an integer $s_0$ such that for all $s\ge s_0$,

$$\left|V_i^s\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)-V_i^{\infty }\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right|<ϵ.$$

(14)

For any admissible $u_i\left(k\right)$, as $s\to \infty$, it follows that

$$V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)\le \underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^{\infty }\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}+2ϵ$$

(15)

For $s\to \infty$, we have

$$V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)\ge \underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^{\infty }\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}-ϵ$$

(16)

Combining inequalities (15) and (16), for any $ϵ>0$, we have

$$\begin{array}{cc}\hfill & \underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^{\infty }\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}-ϵ\hfill \\ \hfill & \le V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)\le \underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^{\infty }\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}+2ϵ\hfill \end{array}$$

(17)

Since $ϵ>0$ is arbitrary, by letting $ϵ\to 0$, we conclude that

$$V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=\underset{u_i\left(k\right)}{min}\left\{M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta V_i^{\infty }\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)\right\}$$

(18)

For any admissible control $u_i\left(k\right)$, a new performance index can be used to equivalently express the problem:

$${\mathsf{\Psi }}_i\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=M_i\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)+\beta {\mathsf{\Psi }}_i\left({\varrho }_i^v(k+1),{\varrho }_i^q(k+1)\right)$$

(19)

Furthermore, assume there exists a state $({\overline{\varrho }}_i^v,{\overline{\varrho }}_i^q)$ such that ${\mathsf{\Psi }}_i({\overline{\varrho }}_i^v,{\overline{\varrho }}_i^q)<V_i^{\infty }({\overline{\varrho }}_i^v,{\overline{\varrho }}_i^q)$. By recursively unfolding the definition of ${\mathsf{\Psi }}_i$, for a finite horizon N (and noting that the terminal cost vanishes as $({\varrho }_i^v\left(N\right),{\varrho }_i^q\left(N\right))\to 0$), we obtain

$${\mathsf{\Psi }}_i\left({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right)\right)=\sum _{t=0}^{N-1}{\beta }^t{M}_i\left({\varrho }_i^v\left(t\right),u_i\left(t\right),{\varrho }_i^q\left(t\right)\right).$$

(20)

According to the definition of $V_i^{\infty }\left({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right)\right)$, we have

$$V_i^{\infty }\left({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right)\right)=\underset{u_i\left(0\right),\dots ,u_i(N-1)}{min}\sum _{t=0}^{N-1}{\beta }^t{M}_i\left({\varrho }_i^v\left(t\right),u_i\left(t\right),{\varrho }_i^q\left(t\right)\right).$$

(21)

By the principle of optimality, $V_i^{\infty }\left({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right)\right)$ is the minimal cost. Therefore,

$$V_i^{\infty }\left({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right)\right)\le \sum _{t=0}^{N-1}{\beta }^t{M}_i\left({\varrho }_i^v\left(t\right),u_i\left(t\right),{\varrho }_i^q\left(t\right)\right)={\mathsf{\Psi }}_i\left({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right)\right)$$

(22)

This contradicts our previous assumption. Thus, it must hold that ${\mathsf{\Psi }}_i({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))\ge V_i^{\infty }({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ for all k. $V_i^{\infty }({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ serves as a global lower bound on the cost for any admissible policy, with equality attained under the optimal policy, i.e.,

$$V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)={\mathsf{\Psi }}_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right).$$

(23)

Similarly, it holds that $V_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)\ge V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ for any iteration s. Taking the limit as $s\to \infty$, we have $V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)\ge V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$. On the other hand, by the definition of $V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ as the minimal cost achievable by any admissible policy, $V_i^{\infty }\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$ cannot be smaller than $V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$. Therefore, the following equality holds: ${lim}_{s\to \infty }{V}_i^s\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)=V_i^{*}\left({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right)$.

This completes the proof. □

This algorithm ensures voltage convergence to the optimal values under containment control and achieves accurate reactive power sharing.

3.2. Stability Analysis of Coordinated Voltage and Reactive Power Control

Section 3.1 proved that the policy iteration algorithm converged to the optimal control policy $u_i^{*}\left(k\right)$. This section demonstrates that the application of this optimal policy ensures the asymptotic stability of the closed-loop system. Specifically, we prove that the containment voltage error ${\varrho }_i^v\left(k\right)$ and the reactive-power-sharing error ${\varrho }_i^q\left(k\right)$ converge to zero.

To analyze the stability, we employ a Lyapunov-based approach. The optimal value function $V_i^{*}({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ derived from the Bellman equation serves as a natural candidate for a Lyapunov function for the closed-loop error dynamics of the ith DG.

Theorem 2.

For the error dynamics described by (7) and (9), if the control policy $u_i^{*}\left(k\right)$ is obtained from the converged policy iteration algorithm as described in Section 3.1, then the closed-loop system is asymptotically stable at the origin, i.e., ${lim}_{k\to \infty }{\varrho }_i^v\left(k\right)=\mathbf{0}$ and ${lim}_{k\to \infty }{\varrho }_i^q\left(k\right)=\mathbf{0}$.

Proof. 

The optimal value function $V_i^{*}({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))$ satisfies the Bellman optimality equation for the optimal policy $u_i^{*}\left(k\right)$:

$$V_i^{*}({\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right))=M_i({\varrho }_i^v\left(k\right),u_i^{*}\left(k\right),{\varrho }_i^q\left(k\right))+\beta V_i^{*}({\varrho }_i^v(k+1),{\varrho }_i^q(k+1))$$

(24)

where $M_i({\varrho }_i^v\left(k\right),u_i^{*}\left(k\right),{\varrho }_i^q\left(k\right))={\varrho }_i^{vT}\left(k\right){\mathsf{\Theta }}_{1ii}{\varrho }_i^v\left(k\right)+u_i^{*T}\left(k\right){\mathsf{\Theta }}_{2ii}{u}_i^{*}\left(k\right)+{\varrho }_i^{qT}\left(k\right){\mathsf{\Theta }}_{3ii}{\varrho }_i^q\left(k\right)$.

According to Definition 1, for any admissible control policy, the performance index $J_i$ must be bounded. The optimal policy $u_i^{*}\left(k\right)$ is, by definition, an admissible policy. Therefore, the optimal value function $V_i^{*}({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right))$, which is the minimum possible value of $J_i$, must be finite.

$$V_i^{*}({\varrho }_i^v\left(0\right),{\varrho }_i^q\left(0\right))=J_i^{*}=\sum _{t=0}^{\infty }{\beta }^t{M}_i({\varrho }_i^v\left(t\right),u_i^{*}\left(t\right),{\varrho }_i^q\left(t\right))<\infty$$

(25)

From the definition of $M_i$, since the weighting matrices ${\mathsf{\Theta }}_{1ii}$, ${\mathsf{\Theta }}_{2ii}$, and ${\mathsf{\Theta }}_{3ii}$ are all positive definite, $M_i({\varrho }_i^v\left(k\right),u_i^{*}\left(k\right),{\varrho }_i^q\left(k\right))\ge 0$ for all k. The equality $M_i=0$ holds if and only if ${\varrho }_i^v\left(k\right)=\mathbf{0}$, ${\varrho }_i^q\left(k\right)=\mathbf{0}$, and $u_i^{*}\left(k\right)=\mathbf{0}$.

For the infinite series in (25) to converge to a finite value with a discount factor $0<\beta \le 1$, it is a necessary condition that the terms of the series approach zero, that is:

$$\underset{k\to \infty }{lim}{\beta }^k{M}_i({\varrho }_i^v\left(k\right),u_i^{*}\left(k\right),{\varrho }_i^q\left(k\right))=0$$

Since $\beta$ is a constant, this implies:

$$\underset{k\to \infty }{lim}{M}_i({\varrho }_i^v\left(k\right),u_i^{*}\left(k\right),{\varrho }_i^q\left(k\right))=0$$

(26)

Given that $M_i$ is a sum of non-negative terms, for their sum to be zero, each individual term must be zero. Therefore, we must have:

$$\begin{array}{cc}\hfill \underset{k\to \infty }{lim}{\varrho }_i^{vT}\left(k\right){\mathsf{\Theta }}_{1ii}{\varrho }_i^v\left(k\right)& =0\hfill \\ \hfill \underset{k\to \infty }{lim}{u}_i^{*T}\left(k\right){\mathsf{\Theta }}_{2ii}{u}_i^{*}\left(k\right)& =0\hfill \\ \hfill \underset{k\to \infty }{lim}{\varrho }_i^{qT}\left(k\right){\mathsf{\Theta }}_{3ii}{\varrho }_i^q\left(k\right)& =0\hfill \end{array}$$

Since ${\mathsf{\Theta }}_{1ii}$ and ${\mathsf{\Theta }}_{3ii}$ are positive definite matrices, this directly leads to the conclusion that the error states converge to zero:

$$\begin{array}{cc}\hfill \underset{k\to \infty }{lim}{\varrho }_i^v\left(k\right)& =\mathbf{0}\hfill \\ \hfill \underset{k\to \infty }{lim}{\varrho }_i^q\left(k\right)& =\mathbf{0}\hfill \end{array}$$

This demonstrates that the origin of the error system is asymptotically stable under the optimal control policy $u_i^{*}\left(k\right)$. □

3.3. Dynamic Leader Election Algorithm

Containment control maintains voltage safety in microgrids by enforcing upper and lower bounds. However, conventional approaches usually predefine the upper-bound leader. If this leader experiences a heavy load, its voltage may decrease, violating the highest-voltage assumption and impairing effective reactive power sharing.

To dynamically adjust the containment control leader based on bus voltage, each DG must access the voltage of all DG-connected buses. However, due to the distributed communication architecture in microgrids, non-adjacent DGs cannot directly share information. Thus, a bus voltage estimation algorithm is required to enable indirect acquisition of voltage data among non-adjacent buses.

Let ${\widehat{\mathbf{\chi }}}_i={\left[{\widehat{\chi }}_{i1},{\widehat{\chi }}_{i2},\dots ,{\widehat{\chi }}_{iN}\right]}^T\in {\mathbb{R}}^N$ denote the vector of bus voltage estimates by the ith DG, where ${\widehat{\chi }}_{ij}$ represents the ith DG’s estimate of the voltage at the bus to which the jth DG is connected; furthermore, ${\chi }_i={\mathsf{\Xi }}_{i,1}$. Then, the update rule for the estimated value ${\widehat{\chi }}_{ij},\forall i,j\in N$ takes the form

$${\dot{\widehat{\chi }}}_{ij}=-\left(\sum _{k=1}^N{a}_{ik}\left({\widehat{\chi }}_{ij}-{\widehat{\chi }}_{kj}\right)+a_{ij}\left({\widehat{\chi }}_{ij}-{\chi }_j\right)\right)$$

(27)

where $a_{ij}$ specifies the $(i,j)$ entry in the adjacency matrix. The first term, $a_{ik}({\widehat{\chi }}_{ij}-{\widehat{\chi }}_{kj})$, represents the difference between the ith DG’s estimate and its neighboring kth DG’s estimate of the bus voltage at the jth DG. The second term, $a_{ij}({\widehat{\chi }}_{ij}-{\chi }_j)$, captures the error between the ith DG’s estimate and the actual bus voltage at the jth DG.

During each iteration of the microgrid controller, each DG estimates the voltages at all buses according to (27). Based on the estimated vector ${\widehat{\mathbf{\chi }}}_i={\left[{\widehat{\chi }}_{i1},{\widehat{\chi }}_{i2},\dots ,{\widehat{\chi }}_{iN}\right]}^T$, the ith DG determines whether it is elected as an upper- or lower-bound leader. The specific rules are given as follows:

• If ${\widehat{\chi }}_{ii}$ is the maximum value in ${\widehat{\mathbf{\chi }}}_i$, then node i is selected as the upper-bound leader.
• If ${\widehat{\chi }}_{ii}$ is the minimum value in ${\widehat{\mathbf{\chi }}}_i$, then node i is selected as the lower-bound leader.
• If ${\widehat{\chi }}_{ii}$ is neither the maximum nor the minimum value in ${\widehat{\mathbf{\chi }}}_i$, node i is not selected as a leader.
• If there are multiple nodes whose values are equal to ${\widehat{\chi }}_{ii}$ and these values are the maximum or minimum in ${\widehat{\mathbf{\chi }}}_i$, node i is elected as the upper- or lower-bound leader only if its index i is the smallest; otherwise, it is not selected as a leader.

Other nodes follow the same procedure to determine their leadership status. In the following, we prove the convergence of the bus voltage estimation algorithm.

Theorem 3.

Consider the estimator dynamic for bus voltage in a microgrid, given by (27). Under Assumption 1, it holds that ${lim}_{t\to \infty }{\widehat{\chi }}_{ij}\left(t\right)={\chi }_j$ for all $i,j=1,\dots ,N$.

Proof. 

Introduce the error variable ${\varpi }_{ij}\left(t\right)$, defined by ${\varpi }_{ij}\left(t\right)={\widehat{\chi }}_{ij}\left(t\right)-{\chi }_j,$ where ${\varpi }_{ij}\left(t\right)$ represents the estimation error of the bus voltage at jth DG, as estimated by the ith DG. Since ${\chi }_j$ is constant or varies very slowly in steady state, it follows that ${\dot{\varpi }}_{ij}\left(t\right)={\dot{\widehat{\chi }}}_{ij}\left(t\right).$

Substituting the system dynamics yields:

$${\dot{\varpi }}_{ij}=-\left(\sum _{k=1}^N{a}_{ik}\left({\widehat{\chi }}_{ij}-{\widehat{\chi }}_{kj}\right)+a_{ij}\left({\widehat{\chi }}_{ij}-{\chi }_j\right)\right).$$

(28)

By substituting the relation ${\varpi }_{kj}={\widehat{\chi }}_{kj}-{\chi }_j$ into the above equation, the error dynamics can be expressed as

$${\dot{\varpi }}_{ij}=-\sum _{k=1}^N{a}_{ik}\left({\varpi }_{ij}-{\varpi }_{kj}\right)-a_{ij}{\varpi }_{ij}=-\left(\sum _{k=1}^N{a}_{ik}+a_{ij}\right){\varpi }_{ij}+\sum _{k=1}^N{a}_{ik}{\varpi }_{kj}$$

(29)

Next, define the Lyapunov function as

$$V\left(t\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^N\sum _{j=1}^N{\left({\varpi }_{ij}\left(t\right)\right)}^2,$$

(30)

which satisfies $V\left(t\right)\ge 0$ for all t, and $V\left(t\right)$ equals zero only when ${\varpi }_{ij}\left(t\right)$ is zero for all $i,j$.

By differentiating (30) with respect to time, we obtain

$$\dot{V}\left(t\right)=\sum _{i=1}^N\sum _{j=1}^N{\varpi }_{ij}{\dot{\varpi }}_{ij}=\sum _{i=1}^N\sum _{j=1}^N{\varpi }_{ij}\left[-\sum _{k=1}^N{a}_{ik}\left({\varpi }_{ij}-{\varpi }_{kj}\right)-a_{ij}{\varpi }_{ij}\right]$$

(31)

To facilitate further analysis, we separate the right-hand side of (31) into two components and define

$$V_1\left(t\right)=\sum _{i=1}^N\sum _{j=1}^N\sum _{k=1}^N{a}_{ik}{\varpi }_{ij}\left({\varpi }_{ij}-{\varpi }_{kj}\right)\mathrm{and}{V}_2\left(t\right)=\sum _{i=1}^N\sum _{j=1}^N{a}_{ij}{\varpi }_{ij}^2.$$

Noting that $a_{ik}=a_{ki}$, we exchange the indices i and k in $V_1\left(t\right)$ to obtain an equivalent form. Summing the two expressions yields

$$2V_1\left(t\right)=\sum _{i=1}^N\sum _{j=1}^N\sum _{k=1}^N{a}_{ik}{({\varpi }_{ij}-{\varpi }_{kj})}^2,$$

(32)

which implies that

$$V_1\left(t\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^N\sum _{j=1}^N\sum _{k=1}^N{a}_{ik}{({\varpi }_{ij}-{\varpi }_{kj})}^2\ge 0.$$

(33)

Since every term ${({\varpi }_{ij}-{\varpi }_{kj})}^2$ is nonnegative and $a_{ik}\ge 0$, we also have $V_2\left(t\right)\ge 0$. Therefore, $\dot{V}\left(t\right)\le 0$, which shows that ${\varpi }_{ij}\to 0$ as $t\to \infty$, and consequently, ${\widehat{\chi }}_{ij}\left(t\right)\to {\chi }_j$. Therefore, the ith DG’s estimation error for the jth DG’s bus voltage gradually converges to zero.

This completes the proof. □

Remark 2.

In direct contrast to conventional containment control methods based on fixed leaders, as proposed in [10,11], this work addresses their well-known limitation in handling complex scenarios like sudden large load changes. The fixed-leader approach often fails to ensure accurate reactive power sharing under such conditions. To overcome this specific flaw, this paper introduces a novel DLE algorithm. This mechanism, based on bus voltage estimation, allows each DG to dynamically select the leader according to real-time operating conditions. By doing so, it enables accurate reactive power sharing precisely where the conventional method falters, providing a clear, practical demonstration of its superiority over the static, fixed-leader approach.

3.4. RL-Based Containment Control Implementation

To ensure voltage containment and accurate reactive power sharing, this section proposes a control method based on actor–critic reinforcement learning. The actor network generates the control policy, while the critic network evaluates and guides its optimization. Through online iteration, the algorithm converges to the optimal control. The implementation structure is shown in Figure 2.

3.4.1. Critic Network

The critic network is designed to approximate the optimal value function, expressed as $V_i^s\left(k\right)=V_i^s\left({\varrho }_i^v\left(k\right),u_i\left(k\right),{\varrho }_i^q\left(k\right)\right)$. This network adopts a three-layer back-propagation neural architecture. Define the input vector of the critic as ${\upsilon }_i\left(k\right)={\left[{\varrho }_i^v\left(k\right),{\varrho }_i^q\left(k\right)\right]}^{\mathrm{T}}$, where $N_c$ denotes the quantity of neurons within the hidden layer, and ${\omega }_{c1}$ and ${\omega }_{c2}$ represent the weight matrices for the hidden and output layers, respectively. Accordingly, the hidden layer is supplied with ${\varsigma }_{c1}\left(k\right)={\omega }_{c1}{\left[{\upsilon }_i\left(k\right),u_i\left(k\right)\right]}^{\mathrm{T}}$ as its input. A hyperbolic tangent activation function, $\psi$, is employed in the hidden layer to capture smooth nonlinear relationships, with $\psi \left({\varsigma }_{c1}\left(k\right)\right)={\displaystyle \frac{1-exp\left(-{\varsigma }_{c1}\left(k\right)\right)}{1+exp\left(-{\varsigma }_{c1}\left(k\right)\right)}}$. The corresponding hidden layer output is ${\mathsf{\Lambda }}_c\left(k\right)=\psi \left({\varsigma }_{c1}\left(k\right)\right)$. Ultimately, the output of the critic network at time k is represented by $V_i^s\left(k\right)={\omega }_{c2}{\mathsf{\Lambda }}_c\left(k\right)$.

The error term for the critic network is defined by $e_c\left(k\right)=V_i^{s+1}\left(k\right)-\left[M_i\left(k\right)+\beta V_i^s(k+1)\right]$. To train the critic network, gradient descent is employed to minimize the error $e_c\left(k\right)$, resulting in the following objective function: ${\displaystyle \underset{{\omega }_{c1},{\omega }_{c2}}{min}{E}_c\left(k\right)=\underset{{\omega }_{c1},{\omega }_{c2}}{min}\frac{1}{2}{\left[e_c\left(k\right)\right]}^2}$.

The iterative update laws for the weights ${\omega }_{c1}$ and ${\omega }_{c2}$ are given by

$${\omega }_{c1}^{l+1}\left(k\right)={\omega }_{c1}^l\left(k\right)+\Delta {\omega }_{c1}^l\left(k\right),\Delta {\omega }_{c1}^l\left(k\right)=-\tau \left[{\displaystyle \frac{\partial E_c^l\left(k\right)}{\partial {\omega }_{c1}^l\left(k\right)}}\right]$$

(34)

$${\displaystyle \frac{\partial E_c^l\left(k\right)}{\partial {\omega }_{c1}^l\left(k\right)}}={\displaystyle \frac{\partial E_c^l\left(k\right)}{\partial e_c^l\left(k\right)}}{\displaystyle \frac{\partial e_c^l\left(k\right)}{\partial V_{i,l}^{s+1}\left(k\right)}}{\displaystyle \frac{\partial V_{i,l}^{s+1}\left(k\right)}{\partial {\mathsf{\Lambda }}_c\left(k\right)}}{\displaystyle \frac{\partial {\mathsf{\Lambda }}_c\left(k\right)}{\partial {\varsigma }_{c1}\left(k\right)}}{\displaystyle \frac{\partial {\varsigma }_{c1}\left(k\right)}{\partial {\omega }_{c1}\left(k\right)}}.$$

(35)

where l is the neural network iteration index, and $\tau$ represents the learning rate. For the output layer weights:

$${\omega }_{c2}^{l+1}\left(k\right)={\omega }_{c2}^l\left(k\right)+\Delta {\omega }_{c2}^l\left(k\right),\Delta {\omega }_{c2}^l\left(k\right)=-\tau \left[{\displaystyle \frac{\partial E_c^l\left(k\right)}{\partial {\omega }_{c2}^l\left(k\right)}}\right]$$

(36)

$${\displaystyle \frac{\partial E_c^l\left(k\right)}{\partial {\omega }_{c2}^l\left(k\right)}}={\displaystyle \frac{\partial E_c^l\left(k\right)}{\partial e_c^l\left(k\right)}}{\displaystyle \frac{\partial e_c^l\left(k\right)}{\partial V_{i,l}^{s+1}\left(k\right)}}{\displaystyle \frac{\partial V_{i,l}^{s+1}\left(k\right)}{\partial {\omega }_{c2}^l\left(k\right)}}$$

(37)

3.4.2. Actor Network

The actor network is constructed to approximate the optimal control policy $u_i^{*}$. It is implemented as a three-layer neural network, where the hidden layer consists of $N_a$ neurons and employs the hyperbolic tangent activation function. Denote ${\omega }_{a1}$ and ${\omega }_{a2}$ as the weight matrices for the hidden and output layers, respectively. The hidden layer output can be expressed as ${\mathsf{\Lambda }}_a\left(k\right)=\psi \left({\varsigma }_{a1}\left(k\right)\right)={\displaystyle \frac{1-{exp}^{-{\varsigma }_{a1}\left(k\right)}}{1+{exp}^{-{\varsigma }_{a1}\left(k\right)}}}$, where ${\varsigma }_{a1}\left(k\right)={\omega }_{a1}{\upsilon }_i\left(k\right)$. The final output of the network is given by $u_i^s\left(k\right)={\displaystyle \frac{1-{exp}^{-{\vartheta }_i^a\left(k\right)}}{1+{exp}^{-{\vartheta }_i^a\left(k\right)}}}$, with ${\vartheta }_i^a\left(k\right)={\omega }_{a2}{\mathsf{\Lambda }}_a\left(k\right)$.

By continuously adjusting the parameters ${\omega }_{a1}$ and ${\omega }_{a2}$, the network aims to derive the optimal control input based on ${\upsilon }_i\left(k\right)$. The parameter update is guided by minimizing the objective function ${\displaystyle \underset{{\omega }_{a1},{\omega }_{a2}}{min}{E}_a\left(k\right)=\underset{{\omega }_{a1},{\omega }_{a2}}{min}\frac{1}{2}{\left[e_a\left(k\right)\right]}^2}$, where the error term is defined by $e_a\left(k\right)=\eta V_i^{s+1}(k+1)+\gamma {\left(u_i^{s+1}\left(k\right)\right)}^2$.

The purpose of the optimization problem is to ensure that the actor network produces an optimal control action that minimizes the value function. Similar to the critic network, the weights ${\omega }_{a1}$ and ${\omega }_{a2}$ are updated using gradient descent:

$${\omega }_{a1}^l\left(k\right)={\omega }_{a1}^l\left(k\right)+\Delta {\omega }_{a1}^l\left(k\right),\Delta {\omega }_{a1}^l\left(k\right)=-\tau \left[{\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial {\omega }_{a1}^l\left(k\right)}}\right]$$

(38)

$${\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial {\omega }_{a1}^l\left(k\right)}}={\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial e_a^l\left(k\right)}}{\displaystyle \frac{\partial e_a^l\left(k\right)}{\partial u_i^{s+1}\left(k\right)}}{\displaystyle \frac{\partial u_i^{s+1}\left(k\right)}{\partial {\omega }_{a1}^l\left(k\right)}}+{\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial e_a^l\left(k\right)}}{\displaystyle \frac{\partial e_a^l\left(k\right)}{\partial V_{i,l}^{s+1}(k+1)}}{\displaystyle \frac{\partial V_{i,l}^{s+1}(k+1)}{\partial u_i^{s+1}(k+1)}}{\displaystyle \frac{\partial u_i(k+1)}{\partial {\omega }_{a1}^l\left(k\right)}}$$

(39)

Similarly, the weights of the output layer are updated by

$${\omega }_{a2}^{l+1}\left(k\right)={\omega }_{a2}^l\left(k\right)+\Delta {\omega }_{a2}^l\left(k\right),\Delta {\omega }_{a2}^l\left(k\right)=-\tau \left[{\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial {\omega }_{a2}^l\left(k\right)}}\right]$$

(40)

$${\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial {\omega }_{a2}^l\left(k\right)}}={\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial e_a^l\left(k\right)}}{\displaystyle \frac{\partial e_a^l\left(k\right)}{\partial u_i^{s+1}\left(k\right)}}{\displaystyle \frac{\partial u_i^{s+1}\left(k\right)}{\partial {\omega }_{a2}^l\left(k\right)}}+{\displaystyle \frac{\partial E_a^l\left(k\right)}{\partial e_a^l\left(k\right)}}{\displaystyle \frac{\partial e_a^l\left(k\right)}{\partial V_{i,l}^{s+1}(k+1)}}{\displaystyle \frac{\partial V_{i,l}^{s+1}(k+1)}{\partial {\omega }_{a2}^l\left(k\right)}}$$

(41)

Remark 3.

As a typical nonlinear system, island microgrids present challenges for model-based controller design due to difficulties in obtaining practical parameters such as resistance and inductance [17,18]. These methods are also sensitive to measurement errors, further complicating controller design. In contrast to [18], this paper proposes a data-driven online reinforcement learning approach that does not require extensive offline data processing. The control policy is iteratively optimized by minimizing the value function, enabling accurate reactive power sharing and effective voltage control in the microgrid.

Remark 4.

From a practical standpoint, the proposed DLE and model-free RL framework is designed to address key operational challenges in real-world offshore microgrids. First, the DLE algorithm provides crucial operational flexibility and resilience. It enables the microgrid to autonomously adapt to the harsh and dynamic conditions of offshore environments (e.g., sudden load changes, volatile renewables), overcoming the rigidity of fixed-leader schemes to ensure stability without manual intervention. Second, the model-free RL controller eliminates the reliance on an accurate system model, which is a significant practical challenge because obtaining line parameters is often both difficult and costly. By learning directly from measurement data, our approach simplifies deployment, reduces commissioning costs, and enhances robustness against parameter uncertainties and system aging. Collectively, these features make the proposed framework not only technically effective but also practical, cost-efficient, and resilient for real-world offshore applications.

4. Simulation Results

As shown in Figure 3, the offshore island AC microgrid system consisted of four renewable generation units. The algorithm proposed in this paper was validated on a simulation model built on the Simulink platform. The validation strategy across the following cases was deliberately designed to highlight the practical value of the proposed model-free approach. Instead of a direct numerical comparison against a model-based controller, which can be misleading (as its performance is entirely dependent on an idealized, perfectly accurate model that is unavailable in reality), our validation focused on two key aspects. First, in Case 4.1, we conducted a head-to-head comparison with a conventional fixed-leader method [10] to demonstrate that our DLE algorithm solved a fundamental operational flaw. Second, in Cases 4.2 and 4.3, we verified that our model-free controller robustly achieved all control objectives under challenging conditions (load changes and plug-and-play), thereby proving its effectiveness and practical viability on its own terms. Following the approach in [10], the allowable voltage deviation was set to ±1%, and the rated bus voltage $V_{\mathrm{bus}}$ was selected as 311 V, which served as the system design objective. Specific simulation parameters are provided in

Table 1. System parameters.

Symbol	Parameter	Value	Symbol	Parameter	Value
$V_{bus}$	Rated bus voltage	311 V	$S_{rate,1}$	Rated power DG1	25 kW, 20 kVar
$Z_{line,1}$	Line impedance of DG1	$0.03\mathsf{\Omega }+0.56\mathrm{mH}$	$S_{rate,2}$	Rated power DG2	20 kW, 10 kVar
$Z_{line,2}$	Line impedance of DG2	$0.06\mathsf{\Omega }+0.8\mathrm{mH}$	$S_{rate,3}$	Rated power DG3	25 kW, 20 kVar
$Z_{line,3}$	Line impedance of DG3	$0.03\mathsf{\Omega }+0.56\mathrm{mH}$	$S_{rate,4}$	Rated power DG4	20 kW, 10 kVar
$Z_{line,4}$	Line impedance of DG4	$0.06\mathsf{\Omega }+0.8\mathrm{mH}$	$L_f$	DG filter inductance	1 mH
$Z_{line,12}$	Line impedance of DG1,2	$0.6\mathsf{\Omega }+3.2\mathrm{mH}$	$C_f$	DG filter capacitor	100 μF
$Z_{line,23}$	Line impedance of DG2,3	$0.4\mathsf{\Omega }+2.4\mathrm{mH}$	$load1$	Capacity of load1	12.5 kW, 8 kVar
$Z_{line,34}$	Line impedance of DG3,4	$0.5\mathsf{\Omega }+2.8\mathrm{mH}$	$load2$	Capacity of load2	15.8 kW, 10 kVar

, and other related parameters were as follows: $\beta$ was selected to be $0.98$ as the discount factor. The performance index employed the following weighting parameters: both ${\mathsf{\Theta }}_{1ii}$ and ${\mathsf{\Theta }}_{3ii}$ were diagonal matrices with diagonal elements equal to 1, and ${\mathsf{\Theta }}_{2ii}=0.1$. Both the actor and critic networks employed five hidden neurons.

4.1. Dynamic Leader Election

This case was designed to validate the effectiveness of the proposed dynamic leader election (DLE) algorithm. To achieve this, we first established a benchmark scenario from $t=0\mathrm{s}$ to $t=10\mathrm{s}$ by implementing the fixed-leader containment control described in [10]. The purpose of this benchmark was to replicate a well-known limitation of conventional methods. As shown in Figure 4, during this benchmark period, while the system successfully maintained voltage containment (i.e., all bus voltages remained within the safe range), the reactive-power-sharing ratio failed to achieve the desired $2:1:2:1$. This outcome was an expected consequence of the fixed-leader topology: since DG1 was designated as the upper-limit leader, its bus voltage was consistently maintained at the highest level, which inherently restricted reactive power flow and prevented equitable sharing among the DGs. This scenario effectively highlighted the specific problem that the data-driven model-free approach was designed to overcome without relying on pre-configured roles or precise system parameters. Subsequently, at $t=10\mathrm{s}$, the load distribution was changed by transferring the load on Bus 3 to Bus 2. With the leaders still fixed as DG1 and DG4, it can be observed from Figure 4 that although the microgrid remained effective in voltage containment control, reactive power sharing was still not fully achieved.

At $t=20\mathrm{s}$, the proposed dynamic leader election algorithm was enabled. As shown in Figure 4, the dynamic leader election algorithm allowed the upper-limit leader to be automatically elected as DG4 and the lower-limit leader to be automatically elected as DG1. Under the effect of the dynamic leader election algorithm, the microgrid not only achieved voltage containment control but also brought the reactive-power-sharing ratio to $2:1:2:1$, successfully achieving precise reactive power sharing. At $t=30\mathrm{s}$, the load on Bus 2 was transferred back to Bus 3. From Figure 4, it can be observed that after a brief transient process, the microgrid once again achieved voltage containment control, and the reactive-power-sharing ratio was restored to $2:1:2:1$, ensuring precise reactive power sharing. The results confirm that the proposed algorithm adaptively selects leaders based on bus voltage magnitude.

4.2. Load Variation

Through simulation and comparative experiments, this study verified the effectiveness of the proposed dynamic leader election algorithm and model-free reinforcement learning algorithm. The proposed approach achieved both voltage recovery and accurate reactive power sharing. First, from $t=0\mathrm{s}$ to $t=5\mathrm{s}$, the microgrid employed only the conventional PI control strategy described in [7], as shown in Figure 5. During that phase, the load on Bus 1 of the microgrid was $8\mathrm{kW}$, and the load on Bus 3 was $10\mathrm{kW}$. It can be seen that the microgrid voltage was not restored to the safe level, nor was the reactive-power-sharing ratio precisely maintained at $2:1:2:1$.

At $t=5\mathrm{s}$, the proposed dynamic leader election and model-free reinforcement learning algorithms were enabled. As shown in Figure 5, the voltage quickly recovered to within the safe constraint range, and the reactive-power-sharing ratio also reached $2:1:2:1$, achieving accurate reactive power sharing. To further validate the robustness of the algorithms under load variation, at $t=10\mathrm{s}$, the load on Bus 3 (DG3) was increased by $3\mathrm{kW}$. From Figure 5, it can be observed that after experiencing a brief transient process, the microgrid voltage returned to the steady state and remained within the safe range. Simultaneously, the reactive-power-sharing ratio again reached $2:1:2:1$, achieving precise reactive power sharing. Figure 6 shows the evolution of actor–critic neural network weights for DG1 during the simulation. All weights converged to stable values, as illustrated.

4.3. Plug-And-Play Capability

The plug-and-play capability of microgrids enables the rapid integration or removal of DGs, allowing the system to adapt to load changes and equipment failures, thereby improving overall flexibility and scalability. To comprehensively and realistically validate the plug-and-play performance of the proposed algorithm, this section designs an experiment that includes a “plug-out” event and a “plug-in” process that mimics real-world engineering scenarios.

The simulation results are shown in Figure 7. During the time period from $t=0\mathrm{s}$ to $t=10\mathrm{s}$, the microgrid operated stably with all four DGs. The proposed algorithms achieved voltage containment control and accurate reactive power sharing with a ratio of $2:1:2:1$. At $t=10\mathrm{s}$, DG4 was disconnected (plug-out) to simulate its removal from operation. The simulation results show that after DG4 was removed, the power deficit was automatically compensated by the remaining DGs. The system voltage, after a brief transient, quickly stabilized and remained within the safe constraint range. Meanwhile, reactive power was redistributed among the three remaining DGs, reaching a new stable sharing ratio of $2:1:2$. To simulate the reconnection process of a DG, at $t=20\mathrm{s}$, DG4 initiated the synchronization process with the microgrid. During that phase, DG4 adjusted its output voltage frequency, phase, and amplitude to match the microgrid’s parameters in preparation for grid connection. At $t=25\mathrm{s}$, upon successful synchronization, DG4 was physically connected to the microgrid, and its controller was activated. As observed in Figure 7, the system seamlessly reintegrated DG4. The voltage remained stable, and the reactive-power-sharing ratio, after a short dynamic adjustment, was accurately restored to the initial $2:1:2:1$ state.

This complete test, encompassing both a plug-out event and a realistic plug-in process, robustly demonstrates that the proposed dynamic leader election and model-free reinforcement learning algorithms provide the microgrid with plug-and-play capability, ensuring safe and stable operation under dynamic topological changes.

5. Conclusions

This paper developed a secondary control method for offshore island microgrids based on a model-free reinforcement learning algorithm and a dynamic leader election mechanism. First, by combining the microgrid’s voltage containment error and reactive power sharing error, a value function for policy iteration was constructed. Then, a dynamic leader election algorithm was designed, enabling different DGs to be dynamically elected as leaders to facilitate accurate reactive power allocation. Subsequently, a model-free reinforcement learning algorithm was developed, which relied solely on real-time measurements of voltage and reactive power without requiring a complex system model.

However, it is important to acknowledge that this study was conducted under the assumption of an ideal island microgrid model, where factors such as communication delays, external disturbances, and potential cyber-attacks were not considered. Communication delays, which are inherent in distributed control systems, could introduce time lags in the information exchange among DGs. This might affect the timeliness of the dynamic leader election process and degrade the performance of the reinforcement learning algorithm, potentially leading to oscillations or even instability. Similarly, other disturbances, such as measurement noise and unmodeled dynamics, could impact the accuracy of the data-driven RL algorithm, which is highly dependent on the quality of measurement data. Addressing these practical challenges is crucial for real-world implementation. Therefore, these aspects will be the focus of our future work. We plan to investigate and develop more robust control strategies that can tolerate communication delays and are resilient to various disturbances. This may involve integrating predictive control mechanisms or designing delay-compensation techniques within the RL framework. To validate the effectiveness and robustness of the enhanced methods, we intend to conduct more comprehensive hardware-in-the-loop simulations or tests on a physical experimental platform.