A Transferable DRL-Based Intelligent Secondary Frequency Control for Islanded Microgrids

Abstract

Frequency instability poses a significant challenge to the overall stability of islanded microgrid systems. Deep reinforcement learning (DRL)-based intelligent control strategies are drawing considerable attention for their ability to operate without the need for previous system dynamics information and the capacity for autonomous learning. This paper proposes an intelligent frequency secondary compensation solution that divides the traditional secondary frequency control into two layers. The first layer is based on a PID controller and the second layer is an intelligent controller based on DRL. To address the typically extensive training durations associated with DRL controllers, this paper integrates transfer learning, which significantly expedites the training process. This scheme improves control accuracy and reduces computational redundancy. Simulation tests are executed on an islanded microgrid with four distributed generators and an IEEE 13-bus system is utilized for further validation. Finally, the proposed method is validated on the OPAL-RT real-time test platform. The results demonstrate the superior performance of the proposed method.

1. Introduction

Microgrids offer accessible operational solutions for utilizing distributed generators (DGs) powered by clean energy sources like photovoltaic and wind power. This also reduces the users’ reliance on the main grid [1,2,3]. Moreover, effectively controlling and optimizing power flows while operating multiple DGs is a critical issue that must be addressed to ensure an efficient performance of the microgrid. To tackle such problems, hierarchical control schemes including primary, secondary, and tertiary control have been proposed [4]. The objective of a primary control is to achieve stable voltage and frequency output while realizing accurate power distribution. The secondary control compensates for the voltage and frequency output of the primary control to return them to their nominal values. Tertiary control is typically used to achieve economic dispatch and optimized power flow [5].

Droop control allows for distributed control with plug-and-play features, and is thus extensively used in primary controls [6]. However, sudden changes in load can cause the system frequency to deviate from the rated value. Maintaining frequency stability is an essential requirement for microgrid operations, as the frequency deviation of the system can lead to equipment failure, clock synchronization failure and other stability issues [7,8]. As a result, the secondary frequency compensation control has been widely pointed out as an area of concern by scholars and is also the focus of this paper. Secondary control can be categorized into centralized control and distributed control. The distributed control scheme eliminates the need for a centralized control center, as agents only need to exchange information with adjacent units. This reduces the communication overhead and offers greater flexibility, widening its potential application scope [9]. Various control schemes such as proportional integral controller (PID), model predictive control (MPC), and consensus algorithms are widely applied in the secondary control of microgrids [10]. In [11], the authors utilized PID controllers for frequency compensation. Building on this research, ref. [12] introduced a PSO-PID controller, a solution that employs the particle swarm optimization (PSO) algorithm for tuning the parameters of the PID controller. A two-stage fractional order scheme based on an imperialistic competitive algorithm for PID parameter design was proposed in [13]. However, the PID controller still has the drawbacks of an overshooting problem and poor dynamic characteristics. In [14], a secondary frequency compensation based on MPC controller was proposed and the operating cost was taken as a constraint. However, MPC has drawbacks such as sensitivity to parameter variations and complex calculations, potentially leading to reduced robustness. In [15], the authors proposed a secondary control based on small AC signal injection, which injects an AC signal into each DG output power supply to achieve frequency compensation while maintaining active power distribution. A finite time frequency secondary compensation was proposed in [16], but its algorithm convergence is affected by the initial operating conditions of the microgrid, making it challenging to guarantee convergence.

The research in machine learning (ML) used for control systems has been a new focus in recent years, and ML techniques with powerful data processing and computational capabilities have the potential to realize optimal control of complex operating conditions containing multiple DGs [17]. In particular, reinforcement learning (RL), which is an important branch of ML, has become an alternative to traditional control due to its improved dynamic performance and robustness [18]. In [19], the authors used Q-learning to achieve frequency compensation with plug-and-play functionality, but the accuracy of Q-learning compensation is relatively low, and its accuracy depends heavily on the initialized Q-table. In [20], a secondary frequency control based on actor–critic-based RL was proposed, which requires no a priori data and takes into account the disturbances of uncertain line parameters. But the compensation accuracy under multi-DG operating conditions remains to be enhanced. In contrast to Q-learning, deep reinforcement learning combines deep neural networks (DNN) with RL, offering a more sophisticated control solution. A DRL frequency compensation scheme based on continuous action domain was proposed in [21], which could improve the compensation efficiency. In [22], a multi-agent DRL frequency compensation scheme based on deep deterministic policy gradient (DDPG) was proposed to improve the compensation accuracy by adjusting each agent parameter on the basis of setting up a globally optimal control objective. In [23], a multi-agent quantum DRL controller was proposed for frequency compensation. It utilized a quantum neural network instead of the traditional NN, which enhanced compensation accuracy. However, the computational intensity was higher and the process was more complex. While employing a DRL-based control agent improves the compensation accuracy at the result level, achieving complete accuracy during agent training remains a challenge. The probability of insufficiently accurate frequency compensation is higher under complex operating conditions caused by large sudden load changes [18]. Furthermore, all agents in [19,20,21,22,23] adopt the strategy of offline training followed by online application to achieve frequency compensation. Nevertheless, training or optimizing the parameters of the neural networks in all agents requires great computational effort and is inefficient. Transfer learning (TL) provides a solution to simplify the computational process described above. Transfer learning aims to use previous learning experiences to help to improve the computational efficiency of the target [24,25]. In [26], the authors adopted a TL-based fault detection protocol to improve the computational efficiency. The use of a TL technique in [27] significantly reduced the computational effort of DNN-based secondary control.

To overcome the above challenge, this paper proposes an intelligent distributed frequency secondary control scheme combining PID controller and DRL for AC islanded microgrids. The method is free of a priori data and does not require system modeling. The traditional frequency secondary compensation control is divided into two layers, the first layer is compensated by the traditional PID controller, and the second layer is compensated by the model-free DRL controller based on the Deep Q-learning (DQN) algorithm to re-compensate the output result after the first layer output. Compared to the Q-learning algorithm, the DQN algorithm including DNN has better state and action space and function approximation capabilities [28], also it has more accurate compensation capability. In the second layer of DRL-based frequency compensation, a method that entails training the agent offline and applying it online is used. During the offline training, the training dataset is derived from the deviations that still exist between the frequency compensation based on the PID and the rated values. In the online application, the agent has the capability to follow the frequency output for compensation. To better illustrate the research gap,

Table 1. Comparison of typical secondary control methods for microgrids.

Method	Model	Accuracy	Training Type	Advantages	Disadvantages
PID	No	Low	Offline	Simple, easy to implement	Poor dynamic performance, overshoot
PSO-PID	No	Medium	Offline	Optimized tuning	Limited adaptability under dynamic conditions
FOPID-ICA	No	Medium	Offline	Improved robustness	High tuning complexity
MPC	Yes	High	Model-based	High accuracy under ideal model	Sensitive to modeling errors, complex
Q-learning	No	Low	Online	Plug-and-play, no model required	Slow learning, Q-table dependent
AC-RL	No	Medium	Online	Handles uncertainties	Limited accuracy under multi-DG settings
DDPG	No	High	Online	Accurate continuous control	High training cost per agent
Quantum DRL	No	High	Online	Enhanced accuracy via quantum design	Very high computation cost
Proposed	No	High	Offline	Accurate and efficient with TL	Offline training still required

provides a comparative analysis of representative frequency secondary control strategies in microgrids. The main contributions are as follows:

• An intelligent two-layer distributed secondary frequency compensation structure is proposed, which preserves the PID controller differently from other ML-based secondary control. Building on the first layer of PID compensation, a second layer of DRL-based intelligent frequency controller is proposed to achieve precise frequency compensation.
• The DRL-based frequency controller employs the DQN algorithm to realize the improved accuracy of frequency compensation. Further, the fine-tuning model TL is used to reduce the DQN training time under multi-agent conditions.

The rest of the paper is organized as follows. The primary control principle is presented in Section 2. In Section 3, the proposed secondary control principle is illustrated and discussed. In Section 4, case study results are provided. Finally, Section 5 states the conclusions of the research.

2. Primary Control of AC Islanded Microgrids

Voltage source inverters (VSIs) are suitable for achieving voltage–frequency stabilized output in autonomous operation [11]. It is assumed that all DGs are integrated into an islanded microgrid through interfacing VSI, equipped with voltage-current control loops as well as a droop controller [8]. Further, virtual impedance is used to solve the line impedance inconsistency problem. The voltage-current loops use PI-based controllers to ensure stable voltage and current operation. Droop control is adopted as a primary control scheme to achieve active power, reactive power distribution and synchronized control of all VSIs. Specifically, the droop control mechanism operates as follows:

$${\omega }_i={\omega }^{\ast }-K_{\omega i}(P_i^m-P_i^{\ast })$$

(1)

$$V_i=v^{\ast }-K_{Vi}(Q_i^m-Q_i^{\ast })$$

(2)

where ${\omega }_i$ and $V_i$ are the frequency and voltage of the ${\mathrm{DG}}_i$ output. ${\omega }^{\ast }$ and $v^{\ast }$ represent the rated value. $K_{\omega i}$ and $K_{V_i}$ are the droop coefficients, the value selection needs to consider the system stability requirements, frequency response rate and load variation. $P_i^m$ and $Q_i^m$ are the active and reactive power at the output of ${\mathrm{DG}}_i$ after low-pass filtering, as shown in (3) and (4), while $P_i^{\ast }$ and $Q_i^{\ast }$ represent the rated value.

$$P_i^m={\displaystyle \frac{{\omega }_c}{{\omega }_c+s}}\ast \left(v_{o\alpha i}{i}_{o\alpha i}+v_{o\beta i}{i}_{o\beta i}\right)$$

(3)

$$Q_i^m={\displaystyle \frac{{\omega }_c}{{\omega }_c+s}}\ast \left(v_{o\beta i}{i}_{o\alpha i}+v_{o\alpha i}{i}_{o\beta i}\right)$$

(4)

where $v_{o{\alpha }_i}=i_{o{\alpha }_i}{R}_{vi}-i_{o{\beta }_i}{\omega }_i{L}_{vi}$, $v_{o{\beta }_i}=i_{o{\beta }_i}{R}_{vi}+i_{o{\beta }_i}{\omega }_i{L}_{vi}$, $R_{vi}$, $L_{vi}$ is the virtual impedance. The instantaneous output voltages and currents of ${\mathrm{DG}}_i$ are demonstrated in the $\alpha -\beta$ reference frame. ${\omega }_c$ is the cutoff frequency of the low-pass filter. Specifically, conventional droop control lacks the capability to constrain frequency deviations under abrupt load variations, resulting in a steady-state error from the nominal frequency. To address this limitation, secondary frequency control is required to restore the system frequency to its reference value, which constitutes the core focus of this study. The secondary compensation component is added to (1), as given in (5). ${\mathrm{\Omega }}_i$ is the output variable of secondary control.

$${\omega }_i={\omega }^{\ast }-K_{\omega i}(P_i^m-P_i^{\ast })+{\mathrm{\Omega }}_i$$

(5)

3. Proposed Intelligent Secondary Control

3.1. Frequency Control Framework

In this research endeavor, droop control is adopted as a primary control. An intelligent distributed frequency secondary control system with a dual-layer architecture is proposed to compensate for the frequency deviation generated when the load active power changes, as depicted in Figure 1. $z_v$ is virtual impedance. $Load$ is active load. The first layer incorporates a conventional PID controller designed for compensation, whose output compensation value is $\delta f_1$. Its input is the difference between the DG output frequency $f_i$ and the rated value $f^{\ast }$. The proportional $k_p$ and integral $k_i$ coefficients are depicted in Figure 1, and the compensation principle is elucidated in (6).

$$\delta f_1=k_p(f_i-f^{\ast })+k_i\int (f_i-f^{\ast })\mathrm{d}t$$

(6)

The secondary layer integrates a DRL scheme based on the DQN algorithm. Notably, the DRL scheme operates to further compensate atop the output generated by the first layer. The DRL-based frequency compensation in the second layer unfolds in two phases: offline training and on-line application. During the offline training, the agent will interact with the environment (corresponding to the DG), as shown in Figure 2. The agent strategically explores actions contingent upon the frequency output of the corresponding DG. Simultaneously, the DG dynamically regulates its output frequency in response to the frequency action values provided by the agent. Following this interaction, a reward function is employed, motivating the agent to adapt its behavior. The overarching objective is the minimization of frequency deviation. In addition, this paper combines the offline training process with fine-tuning transfer learning that can effectively reduce the training time for other agents.

In the online application phase, the state quantity means the input deviation quantity, namely the frequency deviation that persists after the first layer of PID compensation, denoted as $d{f}^{″}$, as shown in (7). The trained agent selects the Q value based on this input state. Subsequently, the agent determines the action value in accordance with the chosen Q value. The second layer of compensation is $\delta f_2$. The frequency control for each DG is constructed upon this two-layer control framework, featuring an autonomous agent—an intelligent frequency controller—engineered to facilitate precise frequency restoration.

$$d{f}^{″}=f^{\ast }-f_i+\delta f_1$$

(7)

Generalization of the above the secondary control quantities in (5) is as follows:

$${\mathrm{\Omega }}_i=\delta f_1+\delta f_2$$

(8)

3.2. Application Progress of Deep Q-Learning

3.2.1. Deep Q-Learning Principle

DQN is an innovative algorithm that fuses the principles of deep learning with reinforcement learning, aiming to tackle decision-making challenges within complex input spaces. Unlike traditional Q-learning, which relies on a Q-table for discrete state–action pairs, DQN leverages a DNN to handle more intricate inputs efficiently. This algorithm learns to discern optimal decisions by approximating the action–value function (Q-function) through a neural network. A pivotal strategy in DQN is the use of experience replay, where a repository of past interactions, including states, actions, rewards, and subsequent states, is maintained. Training the network on randomly sampled experiences from this collection significantly reduces the temporal correlations, thereby improving the stability of the training program [29].

Additionally, it incorporates a dual DNN structure—comprising a main network and a target network—to further stabilize training by decoupling the fluctuations in the learning targets. These innovations empower DQN to excel in domains like control and optimization, laying a solid groundwork for future developments in deep reinforcement learning. The DQN training process is shown in Figure 3, which illustrates the DQN architecture. The environment provides the current state s and the agent selects an action a via the prediction network. After executing a, the environment returns a reward r and the next state $s^{\prime }$. The transition tuple $(s,a,r,s^{\prime })$ is stored in the memory buffer. The prediction network estimates $Q_P(s,a)$, while the target network computes $Q_T(s^{\prime },a^{\prime })$ for loss calculation. The loss is used to update the prediction network, and the target network is periodically synchronized. More principles and specific details will be given in later paragraphs. In Figure 1, once the training of the agent is completed, the DNN will give the optimal frequency compensation value directly based on the Q-value of the input.

3.2.2. Application Process

Regarding the proposed second layer compensation agent input state quantity, in accordance with IEEE1547 [30]. requirements and to avoid redundant calculations, the control objective of this section is to limit the frequency deviation to be within 0.01 Hz. The $\mathit{s}$ representation is crucial for determining how the DQN senses its environment. To product the input vector, $\mathit{s}$ is represented by a vector encoding the frequency deviation. For the above-mentioned purposes, $\mathit{s}$ is decomposed into three components: ${\mathit{s}}_1$, ${\mathit{s}}_2$, ${\mathit{s}}_3$, representing units, tenths, and hundredths, as shown as shown in Equation (9).

$$\mathit{s}\left(t\right)={\mathit{s}}_1\left(t\right)+0.1·{\mathit{s}}_2\left(t\right)+0.01·{\mathit{s}}_3\left(t\right)$$

(9)

Then, the input vector $\mathbf{x}$ is formed as a column vector. Here, the matrix $\mathbf{x}\left(t\right)$ is used to represent the input variable instead of the original $s\left(t\right)$.

$$\mathbf{x}\left(t\right)={[{\mathit{s}}_1\left(t\right),{\mathit{s}}_2\left(t\right),{\mathit{s}}_3\left(t\right)]}^T$$

(10)

To minimize the frequency deviation, define the objective as maximizing the expected cumulative reward over time, as shown in Equation (11).

$${\pi }^{\ast }=arg\underset{\pi }{max}\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(\mathbf{x},a)\right]$$

(11)

where a is compensation action, $\gamma$ is discount factor for future rewards, $r(\mathbf{x},a)$ is immediate reward for taking action a in state $\mathbf{x}$, and t is step size. The Q-learning algorithm updates the Q-values, which represent the expected future rewards for state–action pairs. The update rule is as shown in Equation (12)

$$Q_{\mathrm{new}}(\mathbf{x},a)=(1-\alpha )·Q(\mathbf{x},a)+\alpha ·\left(r(\mathbf{x},a)+\gamma ·\underset{a^{\prime }}{max}Q({\mathbf{x}}^{\prime },a^{\prime })\right)$$

(12)

where $\alpha$ is the learning rate. ${\mathbf{x}}^{\prime }$ is the next input. $a^{\prime }$ is the next action. $Q(\mathbf{x},a)$ is the vector of Q-values for all possible actions a in state x.

DQN utilizes DNN to approximate a Q-function that estimates the quality of taking an action in a given state. $\theta$ are the parameters of DNN, ${\theta }_i=[{\mathit{W}}^i,{\mathit{b}}^i]$. The DNN architecture is given by Equations (13) and (14).

$${\mathbf{z}}^{\left(l\right)}={\mathbf{W}}^{\left(l\right)}{\mathbf{x}}^{(l-1)}+{\mathbf{b}}^{\left(l\right)}$$

(13)

$${\mathbf{a}}^{\left(l\right)}=\sigma \left({\mathbf{z}}^{\left(l\right)}\right)$$

(14)

where ${\mathbf{W}}^{\left(l\right)}$ and ${\mathbf{b}}^{\left(l\right)}$ are the weights and biases of the $l-th$ layer of the DNN, respectively. ${\mathbf{a}}^{\left(0\right)}=\mathbf{x}$ and $\sigma$ is the activation function. In order to direct the agent to improve its decision-making process by realizing the Q-value update, the reward function is set as follows, where K is a positive constant that scales the penalty for deviations greater than 0.01. A reward threshold of 0.01 Hz is adopted as a practical boundary for acceptable frequency deviation, and additional segmentation for larger errors is omitted to simplify the reward design [31,32].

$$r=\left\{\begin{array}{cc}-K·\left(\right|s+a|-0.01),\hfill & \mathrm{if}|s+a|>0.01\hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(15)

To break the correlation between consecutive samples, DQN uses a replay buffer to store transitions and sample from it randomly. Transition information $(\mathbf{x},a,r,{\mathbf{x}}^{\prime },done)$ will be stored in replay buffer D, $done$ is a symbol of whether an episode is completed or not, it is “True” if it is completed, or “False” if it fails. The DNN parameters are updated mainly with data obtained by random sampling from D. In addition, the DQN uses two networks: a main network (for the current Q-value prediction, $Q_p$) and a target network (for the Q-value prediction for the next state, $Q_t$). The target network is introduced to increase the stability of the learning process and reduce the oscillations and divergence problems that could arise from using the same network to estimate current and future Q values. For a batch of N samples, the loss function is defined as:

$$L\left(\theta \right)={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(Q_p(x_i,a_i)-Q_t(x_i,a_i)\right)}^2$$

(16)

The gradient for $\theta$ of each layer is systematically calculated using the chain rule, as follows:

$${\displaystyle \frac{\partial L}{\partial \mathit{W}}}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}2\left(Q_p(x_i,a_i)-Q_t(x_i,a_i)\right)·{\displaystyle \frac{\partial Q_p(x_i,a_i)}{\partial \mathit{W}}}$$

(17)

$${\displaystyle \frac{\partial L}{\partial \mathit{b}}}={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}2\left(Q_p(x_i,a_i)-Q_t(x_i,a_i)\right)·{\displaystyle \frac{\partial Q_p(x_i,a_i)}{\partial \mathit{b}}}$$

(18)

The updated weights and biases are shown below:

$$\mathit{W}=\mathit{W}-\alpha {\displaystyle \frac{\partial L}{\partial \mathit{W}}}$$

(19)

$$\mathit{b}=\mathit{b}-\alpha {\displaystyle \frac{\partial L}{\partial \mathit{b}}}$$

(20)

By employing gradient descent to minimize the loss function, the DNN weights and biases are iteratively adjusted. These updates are made at each training step, and as training progresses, the main network learns to more accurately estimate the value of taking different actions in different states, thus improving the strategy.

To promote a more efficient exploration process for DQNs, an exponentially decaying $ϵ$-greedy approach is utilized for optimization. This method involves selecting random actions with a probability of ${ϵ}_t$ and actions that maximize the Q-value with a probability of $1-{ϵ}_t$. ${ϵ}_t$ is given in (21), defining the decay rate as $\delta$. The agent training is finally accomplished based on (12), (16)–(21) to achieve optimal frequency compensation.

$${ϵ}_t={ϵ}_{\mathrm{start}}\times {\delta }^t$$

(21)

3.3. Application Progress of Fine-Tuning Transfer Learning

Transfer learning is a method that exploits a model trained on one task (source task) and applies it to a different, yet related, task (target task). This approach is particularly useful when addressing challenges in new domains (target domains), enabling rapid updates to the model using the experience gained from an existing domain (source domain). It finds extensive application in control fields [25].

In the proposed control scheme, each DG in the islanded microgrid contains DQN control, and the operating conditions of each DG may be varied, which means that the agents of each DG need to be trained during offline training, which will result in a large computational redundancy. For training different agents, the target task and the source task are the same, i.e., the frequency compensation is realized, and the target domain is slightly different from the source domain, because each DG will have different operating conditions. As shown in Figure 2, to improve the computational efficiency, this paper adopts fine-tuning TL. By completing the training of one agent, it then inherits part of the experience that has already been trained to the rest of the DGs. This approach significantly reduces computational demands and boosts efficiency.

The DNN of the agent consists of one input layer, two hidden layers and one output layer. The input layer and the first hidden layer will be retained, transferred from the trained agenti (resource) to the rest of the agentj (target), as shown in Figure 4. This transfer includes retaining the parameters $[W^1,b^1]$. During the training of agentj, the DNN’s training procedure is divided into two main phases: feed-forward propagation and backward updating.

In forward propagation, only the weights and biases $[W^2,b^2]$ and $[W^3,b^3]$ of the second hidden layer and the output layer are updated during the training process, while the weights and biases of the first hidden layer and the input layer remain unchanged. Following the obtained output values with the predicted values their mean square error loss is calculated as follows.

$$\zeta ={\displaystyle \frac{1}{m}}\sum _{m=1}^m{({\widehat{y}}^{\left(m\right)}-y^{\left(m\right)})}^2$$

(22)

where m is the number of neurons in the i-$th$ layer, y is the true output, and $\widehat{y}$ is the predicted output. The gradient of the loss function with respect to each parameter is computed in the backward update according to (16)–(20) and the parameters are updated using the gradient descent method. The rest of the agent’s training process is the same as described in the previous section.

4. Case Study

The effectiveness of the proposed intelligent frequency secondary control is demonstrated in this section. The construction of the islanded microgrid system is accomplished in MATLAB/Simulink 2023, and the DQN-based compensation and fine-tuning TL are implemented via Python 3.7 and Pytorch libraries. These computational processes are carried out on a laptop equipped with an i7-1165G7 CPU, operating at 2.8 GHz, and 32 GB of RAM.

4.1. Offline Training Process

In the first step of offline training, we constructed an AC islanded microgrid model with four DGs, as depicted in Figure 5. This model incorporates two variable loads within its configuration. Each DG incorporates primary control through droop control mechanisms and traditional secondary control using PID regulation, with the specific control parameters detailed in

Table 2. Simulation parameters.

Parameter	Value
Rated frequency	50 Hz
Rated voltage	311 V
Droop gain (P-w)	2 $\times {10}^{-4}$
Droop gain (Q-v)	2 $\times {10}^{-4}$
Sampling time	1 $\times {10}^{-6}$ s
DC voltage	700 V
Capacitance of LC filter	150 $\mathrm{\mu }$F
Impedance of LC filter	3 mH
Line impedance 1	0.16 $\mathrm{\Omega }$, 0.1 mH
Line impedance 2	0.32 $\mathrm{\Omega }$, 0.1 mH
Line impedance 3	0.23 $\mathrm{\Omega }$, 0.1 mH
Line impedance 4	0.25 $\mathrm{\Omega }$, 0.23 mH
Line impedance 5	0.642 $\mathrm{\Omega }$, 0.2 mH

. The load demands within the system are defined at specific time intervals: at t = 0 s, Load 1 is 20 kW and Load 2 is 15 kW; at t = 1 s, Load 1 increases to 30 kW and Load 2 to 20 kW; and at t = 2 s, they revert to 20 kW and 15 kW, respectively. The simulation time is set to 3 s. All four DGs have a rated active power of 25 kW. Following the establishment of the microgrid model, the next phase involved initializing the agent’s architecture. For this purpose, a DNN comprising four layers, including the input layer, was integrated into the DQN, with the nuanced parameters of this network outlined in

Table 3. Parameter of DNN used in DQN.

Layer	Input Size	Output Size	Bias Quantity	Activation Function
Input	300	300	0	None
Hidden 1	300	32	32	ReLU
Hidden 2	32	32	32	ReLU
Output	32	300	300	ReLU

. The learning rate $\alpha$ was set to 0.009, with a buffer size D of 10,000, batch = 32, i.e., the size of the data extracted from D, and a discount factor $\lambda$ of 0.99. ${ϵ}_{\mathrm{start}}$ is 0.7, ${\delta }^t$ is 0.995, and training episodes are 3000.

Based on the property that DRL does not require a large amount of a priori data, in the third phase, the study utilized the frequency output signal following secondary control PID compensation from the first step, quantifying the deviation $df$ from this signal to the nominal value to compile a dataset for agent training. The frequency-compensated action of the agent output during training is $\delta f_2^i$, $\delta f_2^j$. After successfully training the agent associated with DGi, fine-tuning TL is applied to streamline and improve the training process for the following agents. Figure 6 illustrates the whole process of offline training of the system, with i = 1 indicating the initially trained agent and j representing agents corresponding to DGs in the range [2, 4].

Following the successful training of the agent for DG1, the study moved forward to evaluate the performance of the agents for DG2, DG3, and DG4. This evaluation was conducted using two distinct methodologies: the traditional training approach, which treats each agent’s training process independently from scratch, and a fine-tuning TL strategy, which leverages the pre-trained model of DG1 to accelerate and potentially enhance the learning process for the subsequent agents. Figure 7 succinctly illustrates the progression of loss function over the training period, from which it is apparent that the fine-tuning TL methodology achieves a more rapid convergence and yields a lower loss in comparison to the original scheme. A detailed comparison of the duration of training under the two methodologies is presented in

Table 4. Transfer learning efficiency and accuracy comparison.

DGj	DQN	Fine-Tuning   TL DQN	Time Saved   Proportion	Accuracy	Accuracy   (5% Noise)
DG2	20.34 s	14.48 s	28.81%	99.21%	98.5%
DG3	19.99 s	14.21 s	28.91%	99.34%	98.5%
DG4	22.34 s	15.34 s	31.33%	99.52%	98.6%

, where the compensation accuracy is also given. These results underscore the profound influence of fine-tuning TL on the training process. By employing this approach, the improved method successfully reduced the training duration for datasets DG2, DG3, and DG4 by 28.81%, 28.91%, and 31.33%, respectively. Notably, this time efficiency was achieved without compromising the precision of the original compensation accuracy.

To evaluate the robustness of the DRL-based controller, we conducted additional training experiments in which 5% zero-mean Gaussian noise was added to the source data (specifically, frequency deviation signals used as agent inputs). The results show that the trained agents are able to maintain stable learning and control performance despite the injected noise. The final frequency compensation accuracy remains consistent, with less than 1% performance degradation in steady-state error compared to the noise-free case as shown in Table 4. These results demonstrate the robustness and generalization capability of the proposed method.

4.2. Simulation Results

To assess the effectiveness of the training outcomes, simulations were conducted in Matlab/Simulink for various islanded microgrid scenarios. These simulations aimed to validate the trained agent by comparing the proposed control strategy not only with the conventional PID control but also against a Q-learning based approach detailed in [33]. In the latter, the learning rate and discount factor were set to 0.1 and 0.9, respectively. In the simulation scenario shown in Figure 5, only DG1, DG2, and load 1 are running, with the simulation duration set to 4 s. Initially, Load 1 consumed 40 kW, which increased to 50 kW at t = 2 s. Further simulation parameters are detailed in Table 2.

Figure 8 shows the performance of DG1 under three different control schemes.

Table 5. Comparison of frequency compensation results.

Method	PID	PID + Q-Learning		Proposed Method
Deviation	Value (Hz)	Value (Hz)	Improve	Value (Hz)	Improve
DG1	0.62	0.41	33.87%	0.11	82.26%
DG2	0.64	0.42	31.25%	0.12	81.25%

details the maximum frequency deviation of each DG operating under different control schemes. This comparison highlights that the proposed strategy not only exhibits a dynamic performance comparable to that of the Q-learning approach but also surpasses the traditional PID control in terms of frequency compensation and dynamic response. Moreover, the proposed control scheme significantly improves the compensation accuracy over the PID + Q-learning method. In Figure 9, it can be seen that both DG1 and DG2 output active power is stabilized at 19.17 kW before 2 s and 23.91 kW after 2 s. This indicates effective load management and distribution.

In a subsequent validation step, simulations are conducted based on the configuration outlined in Figure 5. Different droop control parameters are provided, with the K value set to 2 $\times {10}^{-4}$ for DG1 and DG3, and 1.6 $\times {10}^{-4}$ for DG2 and DG4. The rest of the simulation parameters are consistent with those listed in Table 2. All components operating normally over a total simulation time of 6 s. When t = 0 s, Load 1 and Load 2 are 40 kW, which are adjusted at 2 s to 60 kW for Load 1 and 44 kW for Load 2, before returning to 40 kW for both loads at 4 s. The results of these simulations for the islanded AC microgrid under normal operating conditions are depicted in Figure 10. Figure 10 shows a performance comparison of DG1 under three different control strategies. A quantitative analysis of the DG operation results is detailed in

Table 6. Comparison of frequency compensation results.

Method	PID	PID + Q-Learning		PID + SAC		Proposed Method
Deviation	Value	Value	Improve	Value	Improve	Value	Improve
DG1	0.99	0.59	40.4%	0.28	71.72%	0.21	78.79%
DG2	0.48	0.29	39.58%	0.16	66.67%	0.09	81.25%
DG3	0.92	0.56	39.13%	0.27	76.51%	0.18	89.13%
DG4	0.49	0.26	46.94%	0.21	57.14%	0.08	83.67%

. The results indicate that the PID + Q learning method and the proposed scheme have better dynamic performance than PID control. Additionally, a comparison was made with the PID + Soft Actor-Critic (SAC) algorithm, where the SAC network structure was set to two layers, each containing 128 neural units, with a learning rate of 3 $\times {10}^{-4}$ and a discount factor of 0.9. Based on the results in Table 6, the proposed scheme demonstrates excellent compensation accuracy, with the highest improvement in compensation for DG1 reaching 78.79%. The proposed scheme outperforms the PID + SAC scheme in compensation because DQN is more suitable for discrete controllers, while the continuous action space of SAC is prone to error accumulation. Figure 11 shows the system’s active power output when using the proposed scheme. Before the load changes, DG1 and DG3 output 21.13 kW, while DG2 and DG4 output 17.05 kW. After the load changes, DG1 and DG3 increase to 27.26 kW, and DG2 and DG4 increase to 22.31 kW, demonstrating stable active power output proportional to the droop coefficient.

Furthermore, the IEEE 13-bus system is utilized to assess the effectiveness of a newly proposed control strategy. The configuration of the control system, as detailed in Figure 12, incorporates four DGs. The offline training methodology for each DG adheres to previously described protocols, employing identically configured DNNs. Each DG has an active power rating of 10 kW. The load distribution initially assigns Load 1: 4 kW and Load 3: 2 kW, Load 2, 4, 5, 6, and 7 are 8 kW. The simulation time is 18 s. At the 6-s mark, Load 1 is adjusted to 12 kW, and at the 12-s mark, it is back to 4 kW. Frequency training datasets are generated using an IEEE 13-bus system framework based on PID compensation. This study compares the PID + Q learning method and PID + SAC with the proposed control strategy. Figure 13 shows the frequency compensation results for DG1. The maximum deviation of the proposed scheme is 0.089 Hz, a reduction of 59.56%, while PID + SAC is 0.12 Hz and the PID + Q learning method is 0.22 Hz. Correspondingly, the frequency deviations of DG2, DG3, and DG4 based on the proposed scheme are reduced by 57.23%, 58.78%, and 58.91%, respectively. Figure 14 shows the active power output of the system based on the proposed control scheme, which still achieves the active power sharing equally.

4.3. Real-Time Test Result

An OPAL-RT platform is used for real-time simulation to verify the performance. The 4DG test structure in Figure 5 is constructed in Matlab/Simulink, and the DRL controller is trained offline and preloaded in the control system. When t = 0 s, Load 1 and Load 2 are 40 kW, at t = 0.82 s Load 1 changes to 60 kW and Load 2 to 44 kW. the total simulation time is 2 s, and the rest of the parameters are the same as in Table 2. The real-time simulation platform is shown in Figure 15. Figure 16 and Figure 17 show the results of frequency output based on PID + Q-learning compensation and the proposed scheme respectively, both schemes can realize the accurate performance of compensation frequency when load fluctuation is generated. It is worth noting that, compared with the PID + Q-learning method, the frequency deviation of DG1 was reduced from 0.9 to 0.65, DG2 from 1.05 to 0.72, DG3 from 1.05 to 0.72, and DG4 from 1.27 to 0.85. The improvements are 27.78%, 31.43%, 31.43%, and 33.07%. In addition, the active power output based on the proposed scheme is given in Figure 18, which demonstrates stability of the active power output.

5. Conclusions

A distributed intelligent frequency secondary control scheme was proposed to improve the frequency compensation performance by combining a data-driven DRL-based control scheme with a traditional solution in an AC islanded microgrid. Unlike other ML-based compensation solutions, this paper did not completely abandon or replace the PID controller. The secondary control was divided into two layers, the first layer was based on PID control, and the second layer was a DQN-based intelligent control. The method of offline training and online application was adopted, and the fine-tuning TL scheme was used to inherit the previous learning experience during offline training to reduce the offline training time and the computational redundancy. In the validation stage, the proposed control strategy was proven to have higher compensation accuracy in Matlab/Simulink and compared to other schemes. According to the verification results, the proposed scheme achieves an average improvement of 83.21% in compensation accuracy compared to the traditional PI scheme, and, through TL, it can save an average of 29.68% of training time, significantly enhancing performance. It was also validated in an IEEE standard 13-bus system. Further, real-time test results also demonstrated the effectiveness of the proposed control strategy. Future work will focus on applying the proposed scheme to more complex network structures and addressing challenges related to communication links and data quality.