A Novel Reactive Power Sharing Control Strategy for Shipboard Microgrids Based on Deep Reinforcement Learning

Abstract

Reactive power sharing in distributed generators (DGs) is one of the key issues in the control technologies of greenship microgrids. Reactive power imbalance in ship microgrids can cause instability and potential equipment damage. In order to improve the poor performance of the traditional adaptive droop control methods used in microgrids under high-load conditions and influenced by virtual impedance parameters, this paper proposes a novel strategy based on the deep reinforcement learning DQN-VI, in which a deep Q network (DQN) is combined with the virtual impedance (VI) method. Unlike traditional methods which may use static or heuristically adjusted VI parameters, the DQN-VI strategy employs deep reinforcement learning to dynamically optimize these parameters, enhancing the microgrid’s performance under varying conditions. The proposed DQN-VI strategy considers the current situation in greenships, wherein microgrids are generally equipped with cables of different lengths and measuring the impedance of each cable is challenging due to the lack of space. By modeling the control process as a Markov decision process, the observation space, action space, and reward function are designed. In addition, a deep neural network is used to estimate the Q function that describes the relationship between the state and the action. During the training of the DQN agent, the process is optimized step-by-step by observing the state and rewards of the system, thereby effectively improving the performance of the microgrids. The comparative simulation experiments verify the effectiveness and superiority of the proposed strategy.

1. Introduction

Addressing climate change has emerged as a universally acknowledged imperative within the global community. The International Maritime Organization (IMO) continues to strengthen regulatory measures targeting carbon emissions from ships [1]. Against this background, accelerating the development of greenships and achieving the green transformation of existing ships is a practical and feasible implementation plan for achieving energy conservation and emission reduction in the shipping industry [2]. Greenships meet the requirements of sustainable development by using various technologies that reduce atmospheric emissions, fuel consumption, and the production of environmentally harmful substances.

With the development of greenships, many new types of clean energy, such as photovoltaic and fuel cells, are gradually being used in conjunction with power battery packs. Although these new types of energy are clean and environmentally friendly, their random and intermittent nature poses new challenges for energy management [3]. For this reason, ship microgrid technology has emerged, coordinating various DGs to efficiently use new energy and becoming a key technology for greenships [4]. However, since traditional centralized control is difficult to deal with the complex ship microgrid, distributed control technology has become the research focus. Droop control achieves autonomous power distribution among distributed power sources by adjusting the voltage and frequency without a complex communication network. Droop control is particularly suitable for marine microgrid environments.

The droop-controlled inverter does not need to obtain information from other inverters connected in parallel to the ship’s power grid [5]. It only needs to detect the active and reactive power of its own output and then adjust the voltage amplitude and frequency of its own output to achieve stable operation of the entire ship’s microgrid system [6]. However, ship power grids often use medium and low voltage, and the line impedance mainly exhibits its resistive characteristics, which is unsuitable for the droop relationship obtained by treating the line impedance as approximately purely inductive, as in traditional droop control [7]. Existing solutions mostly use the virtual impedance method, which introduces a virtual inductive reactance at the control level to cancel the coupling of active and reactive power in the line impedance so that the line impedance of the DGs appears to be inductive [8,9].

Virtual impedance effectively solves the problem of droop control in non-purely inductive circuits. However, due to the space constraints of ships, the lengths of the cables connecting the various micro-power supplies in their power grids vary greatly, which can result in large differences in cable impedance. In this case, the reactive impedance is corrected using virtual impedance. Although the reactive part can be added by adding a large “inductor”, the resistance characteristics cannot be accurately compensated due to the different lengths of the cables connecting the micro-power supplies, resulting in difficulties in the distribution of reactive power, reactive circulation between inverters, and instability of the ship’s power grid [10]. Currently, there is no reliable method to estimate virtual impedance, which restricts the system’s ability to adapt effectively to changes in the distribution of DGs and intensifies reactive power circulation. The resulting excessive circulation can damage the DG units and shorten their service life.

Various strategies have been explored to enhance reactive power distribution and minimize circulating currents among DGs. In [11], a method combining virtual impedance and coupled compensation was proposed. While this approach demonstrably improved reactive power distribution and reduced circulating currents, it induced a voltage drop in the grid, thereby limiting its practical applicability. In [12], a harmonic virtual impedance design was introduced to optimize reactive power distribution and enhance power quality by managing harmonic distortion between DGs. While effective in addressing harmonics, this method primarily targeted power quality improvements related to harmonic content. Moreover, recent advancements in control strategies such as model-predictive control and fractional-order controllers [13,14,15] have highlighted their effectiveness in specific applications, yet they may face challenges in adapting to dynamic microgrid environments or achieving robust performance across diverse configurations. Furthermore, a reinforcement Q learning algorithm was utilized to address the issue of uneven reactive power distribution in microgrids with a high penetration of photovoltaic generation [16]. Although promising, this approach primarily focuses on photovoltaic-dominated systems, and its broader applicability to diverse microgrid configurations warrants further investigation. The reinforcement learning algorithm is used to learn the reactive power deviation between each DG and adjust the droop coefficient of each inverter to achieve the effect of equal reactive power distribution. Although it effectively reduces the reactive power deviation, the effect of power equalization achieved by the method proposed in this paper can be further improved [8]. Although there are many studies in the literature on using virtual impedance methods to solve the problem of reactive power distribution of DGs, further research is needed on how to assign the optimal value of the corresponding virtual impedance for each inverter. With the development of artificial intelligence, researchers have begun to apply reinforcement learning to control reactive power distribution of DGs in ship AC microgrids.

This paper extends prior research on reactive power distribution within ship microgrids by introducing an adaptive virtual impedance droop control strategy utilizing deep Q learning (DQN). The proposed method employs a neural network to learn the relationship between system states and control actions in complex microgrid environments, thus addressing the constraints of conventional Q tables. A real-time interaction is established between an agent and the ship’s microgrid, enabling the collection of reactive power distribution data from each distributed generator (DG). The virtual impedance is then dynamically adjusted based on the discrepancy between actual and desired reactive power levels. Simulation results confirm that this strategy substantially improves reactive power sharing among parallel-connected DGs. Therefore, the key contributions of this research are the following:

1.

Integration of Deep Learning and Reinforcement Learning: Combining these techniques facilitates adaptive control within intricate microgrid settings.

2.

Development of an Adaptive Virtual Impedance Droop Control Strategy: This strategy dynamically modifies the virtual impedance according to reactive power deviations, leading to enhanced reactive power distribution.

3.

Comprehensive Simulation Validation: The simulations demonstrate the proposed strategy’s superior performance in improving reactive power sharing among DGs.

This article’s structure is as follows: Section 2 provides a review of conventional droop control strategies for ship AC microgrids, examining both the basic droop control principle and the adaptive approach based on virtual impedance (VI). Section 3 outlines the proposed intelligent control strategy, which is based on deep Q learning (DQN). This section details the general DQN framework for ship microgrids, the design of the droop control strategy (DQN-VI), and the training and optimization process for the DQN agent. Section 4 presents the simulation analysis, and Section 5 concludes this paper.

2. Conventional Droop Control Strategy for Ship AC Microgrids

2.1. Conventional Droop Control Principle

For a ship AC microgrid with two DG inverters connected to the ship’s grid, the equivalent circuit of the inverter single-phase output is shown in Figure 1 [17].

Figure 1. Equivalent circuit of inverter single-phase output.

The equivalent circuit of the inverter single-phase output consists of two inverter units equivalent to voltage sources, line impedance, and a load [18], where $E_{\mathrm{i}}\angle {\delta }_{\mathrm{i}}$, $i=1,2$ denotes the single-phase output voltage of the inverter $i$, whose output impedance is $Z_{\mathrm{i}}\angle {\theta }_{\mathrm{i}}$, and $I_{\mathrm{oi}}$ is the output current phasor, $I_{\mathrm{o}}$ is the current phasor of the load, the impedance of the load is $Z_{\mathrm{L}}$, and the voltage phasor across the load is $U_{\mathrm{L}}$ [19]. Real power and reactive power of the output can be given by the following:

$$\left\{\begin{array}{c}{P}_{\mathrm{i}}=E_{\mathrm{i}}\cdot i_{\mathrm{oi}}\cdot \cos \delta \\ Q_{\mathrm{i}}=E_{\mathrm{i}}\cdot i_{\mathrm{oi}}\cdot \sin \delta \end{array}\right.$$

(1)

Then, $I_{\mathrm{oi}}$ can be given by the following:

$$I_{\mathrm{oi}}={\displaystyle \frac{E_{\mathrm{i}}\angle {\delta }_{\mathrm{i}}-U_{\mathrm{L}}\angle 0^{°}}{Z_{\mathrm{i}}\angle {\theta }_{\mathrm{i}}}}={\displaystyle \frac{E_{\mathrm{i}}}{Z_{\mathrm{i}}}}{\mathrm{e}}^{\mathrm{j}\left({\delta }_{\mathrm{i}}-{\theta }_{\mathrm{i}}\right)}-{\displaystyle \frac{U_{\mathrm{L}}}{Z_{\mathrm{i}}}}{\mathrm{e}}^{-\mathrm{j}{\theta }_{\mathrm{i}}}$$

(2)

Substituting the above into Equation (1), we obtain the following:

$$\left\{\begin{array}{c}{P}_{\mathrm{i}}=\left({\displaystyle \frac{E_{\mathrm{i}}{U}_{\mathrm{L}}}{Z_{\mathrm{i}}}}\cos {\delta }_{\mathrm{i}}-{\displaystyle \frac{U_{\mathrm{L}}^2}{Z_{\mathrm{i}}}}\right)\cos {\theta }_{\mathrm{i}}+{\displaystyle \frac{E_{\mathrm{i}}{U}_{\mathrm{L}}}{Z_{\mathrm{i}}}}\sin {\delta }_{\mathrm{i}}\sin {\theta }_{\mathrm{i}}\\ Q_{\mathrm{i}}=\left({\displaystyle \frac{E_{\mathrm{i}}{U}_{\mathrm{L}}}{Z_{\mathrm{i}}}}\cos {\delta }_{\mathrm{i}}-{\displaystyle \frac{U_{\mathrm{L}}^2}{Z_{\mathrm{i}}}}\right)\sin {\theta }_{\mathrm{i}}-{\displaystyle \frac{E_{\mathrm{i}}{U}_{\mathrm{L}}}{Z_{\mathrm{i}}}}\sin {\delta }_{\mathrm{i}}\cos {\theta }_{\mathrm{i}}\end{array}\right.$$

(3)

When the line impedance is inductive, the following holds true [11]:

$$\left\{\begin{array}{c}{\delta }_{\mathrm{i}}\approx 0^{+°}\\ {\theta }_{\mathrm{i}}\approx {90}^{°}\end{array}\right.$$

(4)

If the inverter uses the droop characteristics of $P-\omega$ and $Q-E$, Equation (3) can be simplified as follows:

$$\left\{\begin{array}{c}{P}_{\mathrm{i}}={\displaystyle \frac{U_{\mathrm{L}}}{X_{\mathrm{i}}}}{E}_{\mathrm{i}}{\delta }_{\mathrm{i}} \\ Q_{\mathrm{i}}={\displaystyle \frac{U_{\mathrm{L}}}{X_{\mathrm{i}}}}\left(E_{\mathrm{i}}-U_{\mathrm{L}}\right)\end{array}\right.$$

(5)

The corresponding expressions for the droop characteristics of $P-\omega$ and $Q-E$ can be expressed as follows:

$$\left\{\begin{array}{c}{\omega }_{\mathrm{i}}={\omega }^{*}-m_{\mathrm{i}}P\\ E_{\mathrm{i}}=E^{*}-n_{\mathrm{i}}Q\end{array}\right.$$

(6)

where ${\omega }^{*}$ is the rated frequency when the DG is unloaded, $E^{*}$ is the voltage when the DG is unloaded, $m_{\mathrm{i}}$ is the frequency droop coefficient, $n_{\mathrm{i}}$ is the voltage droop coefficient, and ${\omega }_{\mathrm{i}}$ is the actual frequency [20].

In Figure 2, $I_{\mathrm{oabc}i}$ is the line impedance current of the inverter $i$, and $U_{\mathrm{oabc}i}$ is the voltage of its output filter capacitor. The reference voltage generation loop obtains the reference voltage reference signals ${E^{*}}_{\mathrm{dref}}$ and ${E^{*}}_{\mathrm{qref}}$, while the voltage control loop obtains the drive signal that controls the inverter switching device.

Figure 2. Block diagram of the ship microgrid control structure based on conventional droop control strategy.

2.2. Adaptive Droop Control Strategy Based on Virtual Impedance (VI)

Within ship microgrids, frequency behaves as a global variable [21,22]. Consequently, using the active droop coefficient from the traditional droop control method (Equation (6)) [23] allows for a proportional allocation of active power. Voltage, however, acts as a local variable. Variations in cable length between each distributed generator (DG) and the point of common coupling (PCC) lead to differing line impedances. This discrepancy ultimately causes an uneven distribution of reactive power among the DGs.

In the circuit shown in Figure 1, two DG inverters in parallel operation have the following relationship:

$$E_{\mathrm{i}}=U_{\mathrm{PCC}}+\left(Z_{\mathrm{line}\mathrm{i}}/E^{*}\right)Q_{\mathrm{i}}$$

(7)

where $U_{\mathrm{PCC}}$ represents the voltage of the PCC, and $Z_{\mathrm{line}\mathrm{i}}$ represents the impedance of the $i^{\mathrm{th}}$ line connecting the DGs to the microgrid. Assuming that the line impedance of ${\mathrm{DG}}_1$ is less than that of ${\mathrm{DG}}_2$, $Z_{\mathrm{line}2}>Z_{\mathrm{line}1}$. Then, the output terminal voltage of ${\mathrm{DG}}_2$ with high line impedance is larger than another; then, $Q_2<Q_1$. It is assumed that

$$Z_{\mathrm{line}2}=Z_{\mathrm{line}1}+\mathrm{\Delta }Z \left(\mathrm{\Delta }Z>0\right)$$

(8)

where $\mathrm{\Delta }Z$ is the additional line impedance of ${\mathrm{DG}}_2$ with respect to ${\mathrm{DG}}_1$. Therefore, the voltage drop across the virtual impedance must be equal to the voltage drop across $\mathrm{\Delta }Z$ [24,25,26], which is related by the following equation:

$$E_{\mathrm{\Delta }\mathrm{Z}}+E_{Z_{\mathrm{v}}}=0$$

(9)

where $E_{\mathrm{\Delta }\mathrm{Z}}$ is the voltage drop across the resistive part of the DG output power through the $\mathrm{\Delta }Z$, and $E_{Z_{\mathrm{v}}}$ is the voltage drop across the $Z_{\mathrm{v}}$ of the DGs. The voltage drop across the virtual impedance caused by DGs must be equal to the voltage drop across the additional line impedance $\mathrm{\Delta }Z$. The process is as follows:

$${\displaystyle \frac{X_{\mathrm{V}}{Q}_2+R_{\mathrm{v}}{P}_2}{E^{*}}}+{\displaystyle \frac{\mathrm{\Delta }X{Q}_2+\mathrm{\Delta }R{P}_2}{E^{*}}}=0$$

(10)

where the resistive part of the virtual impedance $Z_{\mathrm{v}}$ is represented by $X_{\mathrm{V}}$ and its reactive part by $R_{\mathrm{v}}$; the reactive part of the additional line impedance $\mathrm{\Delta }Z$ is represented by $\mathrm{\Delta }X$ and its resistive part by $\mathrm{\Delta }R$. The line impedance of the substitution into Equation (6) is compensated for the ${\mathrm{DG}}_2$, and the single-phase output voltage of the inverter $i$ can be given by the following:

$$\left\{\begin{array}{l}{E}_1=U_{\mathrm{PCC}}^{\prime }+\left(Z_{\mathrm{line}1}/E^{*}\right)Q_1\\ E_2=U_{\mathrm{PCC}}^{\prime }+\left[\left(Z_{\mathrm{line}2}-Z_{\mathrm{V}}\right)/E^{*}\right]Q_2\end{array}\right.$$

(11)

where the voltage at the PCC point is denoted as $U_{\mathrm{PCC}}^{\prime }$, which is reduced due to the decrease in the voltage on the virtual impedance [27].

Therefore, the overall impedance characteristic curve of the two DG lines is shifted downward as a whole. The original output of more ${\mathrm{DG}}_1$ reactive power maintains the same slope (still maintaining $k=Z_{\mathrm{line}1}/E^{*}$), while ${\mathrm{DG}}_2$ introduces virtual impedance to reduce its slope ($k=\left(Z_{\mathrm{line}2}-Z_{\mathrm{V}}\right)/E^{*}$).

As can be seen from Figure 3, $\alpha$ is the angle between the overall reactive power distribution curves of the two DGs before the addition of the virtual impedance curve $①$ and curve $②$ [28]; $\mathrm{\Delta }Q$ is the difference in their reactive power; ${\alpha }^{\prime }$ is the new angle between the new reactive power distribution curves $④$ and curve $⑤$ of the two DGs after the addition of the virtual impedance; finally, ${\mathrm{\Delta }Q}^{\prime }$ is the difference in their reactive power [29,30]. A certain virtual impedance voltage drop is included, which causes the change of curve $①$ to curve $④$ $\left(k_{①}=k_{④}\right)$ and the change of curve $②$ to curve $⑤$ $\left(k_{②}>k_{⑤}\right)$. Therefore, according to Figure 3, the following relationship can be given:

$$\left\{\begin{array}{c}\alpha >{\alpha }^{\prime }>0\\ \mathrm{\Delta }Q>\mathrm{\Delta }{Q}^{\prime }>0\end{array}\right.$$

(12)

Figure 3. Reactive power distribution using adaptive droop control with virtual impedance (VI).

The control system repeatedly executes the above adjustment process, gradually reducing the angle between the reactive power distribution curves of the two DGs and continuously reducing the reactive power deviation until achieving complete balance.

Droop control is widely used in microgrid control because it is simple to distribute power effectively. However, due to the different cable lengths of each DG connected to the power grid and the large difference in the line impedance of the electrical system, it is difficult to evenly distribute the reactive power of each DG, resulting in circulation between DGs and affecting their service life. This problem is more prominent in the ship environment with limited space. When a ship is navigating in the high seas, far from land, ensuring the safety of the ship’s power distribution system is essential. Therefore, by introducing virtual impedance, the problem caused by impedance differences can be solved while retaining the advantages of the droop control, and the accurate reactive power distribution of DG on ship microgrids can be realized.

3. Intelligent Control Strategy for Ship Microgrids Based on DQN

3.1. General Framework of Intelligent DQN-Based Control of Ship Microgrids

Due to the limited space of the ship’s power grid, the length of the cable varies greatly, resulting in significant differences in the line impedance of each access to the ship’s distribution network, thus affecting the sharing of reactive power. The adaptive droop control strategy based on virtual impedance proposed in Section 2.2 improves the distribution of reactive power by introducing virtual impedance to compensate for line impedance differences. However, this method still has limitations when applied to ship power grids characterized by a high load density. These limitations stem primarily from the inability to automatically adjust control parameters in real time based on actual conditions, thereby hindering effective responses to changing operating conditions.

In order to further improve the flexibility and accuracy of adaptive droop control, this paper introduces an improved strategy of ship microgrid intelligent control based on DQN. Through the addition of DQN agents, the control strategy can adjust the control parameters in real time to better cope with load changes and achieve more accurate reactive power distribution. Next, this framework and its working principle will be introduced in detail, as shown in Figure 4.

Figure 4. Schematic diagram of the droop control strategy using the deep Q network-integrated virtual impedance (DQN-VI).

Figure 4 illustrates a modular design. The green section represents the conventional droop control component, where the relationship between reference voltage setpoints ${E^{*}}_{\mathrm{dref}}$ and ${E^{*}}_{\mathrm{qref}}$ is derived from droop tracking. The blue section depicts the adaptive droop control approach, which utilizes virtual impedance. By incorporating a specified virtual impedance value ${R_{\mathrm{v}}}^{*}$, a virtual impedance voltage drop of $E_{\mathrm{d}{R}_{\mathrm{V}}}$ and $E_{\mathrm{q}{R}_{\mathrm{V}}}$ is generated. This compensates for the voltage drop caused by the additional line impedance [31,32].

The red module highlights an adaptive droop control strategy employing DQN-VI. This agent monitors the reactive power deviation $\mathrm{\Delta }Q$ of the DG. Real-time correction of the virtual impedance is achieved by dynamically adjusting the virtual impedance gain $k_{\mathrm{gain}}$. This module enhances the reactive power distribution within the ship’s microgrid.

3.2. Design of Droop Control Strategy Based on DQN-VI

In order to cope with the ship’s power grid with a high load density, this paper proposes the DQN-VI droop control strategy. This strategy dynamically adjusts the virtual impedance by monitoring the operating status of each DG on the ship’s electrical network in real time and calculating the power deviation, thereby achieving the precise distribution of reactive power. The design idea of the proposed control strategy is shown in Figure 5.

Figure 5. The principle block diagram of the deep Q network-integrated virtual impedance (DQN-VI) strategy.

First, the output voltage $U_{\mathrm{cabc}}$ and output current $I_{\mathrm{oabc}}$ of the real-time monitoring DGs are used to calculate the active power $P$ and reactive power $Q$ according to Equation (1). The calculated actual reactive power $Q$ is compared with the given reactive power $Q*$ to obtain the reactive power deviation $\mathrm{\Delta }Q$. This is used as an observation value by the agent, and the output action of the agent is the gain value $k_{\mathrm{gain}}$ of the virtual impedance. $k_{\mathrm{gain}}$ is applied to $Z\mathrm{v}$ to correct it. With the corrected $X\mathrm{v}$, $R\mathrm{v}$ then passes through the virtual impedance link in Figure 6.

Figure 6. Virtual impedance element structure diagram.

In this way, each time the agent performs an action, i.e., adjusts the virtual impedance gain $k_{\mathrm{gain}}$, it will be rewarded according to the observed $\mathrm{\Delta }Q$ after the adjustment. The reward function $r$ is thus designed based on the reduction in the reactive power deviation $\mathrm{\Delta }Q$. When $\mathrm{\Delta }Q$ decreases, the agent receives a positive reward, and vice versa. The learning goal of the agent is to maximize the long-term cumulative reward, so as to learn the optimal control strategy. Therefore, when the agent adjusts the action, it does not always use the current optimal action, but in some cases, the agent will randomly select different action values to explore the observation effect.

After training, the agent uses the learned Q function to select the best virtual impedance gain $k_{\mathrm{gain}}$ based on the observed $\mathrm{\Delta }Q$ to obtain the best virtual impedance amplitude. The optimal virtual impedance amplitude is added to the control loop to compensate ${E^{*}}_{\mathrm{dref}},{E^{*}}_{\mathrm{qref}}$ to obtain a better control of the inverter output voltage $E_{\mathrm{dref}},E_{\mathrm{qref}}$. The DQN-VI strategy continuously feeds back and adjusts the output of the DQN agent according to the real-time output $\mathrm{\Delta }Q$ of the inverter. Through this strategy, the ship microgrid with a high load rate can ensure more accurate reactive power distribution among DGs.

3.3. Training and Optimization Process of the DQN Agent

The DQN agent needs to be trained in advance before it can be used in the control strategy in Figure 5. Therefore, this section details the training and optimization process of the DQN agent, as shown in Algorithm 1.

Algorithm 1: DQN-based VI droop control strategy

First, two neural networks with identical structures are initialized: the $Q(s,a;\theta )$ network and the target network $Q^{\prime }(s,a;{\theta }^{\prime })$. Here, $Q(s,a;\theta )$ is the Q network used for learning, and $Q^{\prime }(s,a;{\theta }^{\prime })$ is the target network, with $\theta$ and ${\theta }^{\prime }$ representing the weight parameters of the two networks, respectively. $s$ indicates the agent’s current state, i.e., the reactive power deviation $\mathrm{\Delta }Q$ currently output by DGs; $a$ indicates the current output of the agent; the virtual impedance gain is $k_{\mathrm{gain}}$. The Q network is used to estimate the value of each possible action. In contrast, the target network stabilizes the training process and reduces the volatility in the Q estimation. Here, the network parameters are initialized at random, and an experience replay pool is also created to store the experience data generated when the agent interacts with the environment.

During training, the DQN agent first obtains the reactive power deviation information $\mathrm{\Delta }Q$ processed by the real-time monitor of the ship’s power grid as the observation state $s$ of the DQN agent. After the agent performs $a$, the reward value is calculated based on the changed feedback state. The reward function is established as the agent observation and reward mechanism. The reward function is as follows:

$$r=\mu {\displaystyle \int max(0,min(1,\gamma \left(\alpha \times \mathrm{\Delta }Q+\beta \right)))}+\lambda$$

(13)

where $r$ is the reward value; μ, β, λ, and γ are auxiliary parameters used to adjust the response speed, sensitivity, and reward interval, respectively. The reward function $r$ is designed to guide the agent in the direction of reducing the power deviation.

The agent uses the $\mathrm{\epsilon }\text{-greedy}$ strategy to select actions at each time step. Most of the time, the agent will select the action that it currently estimates to be optimal, but it will also occasionally select actions at random in order to explore potentially better strategies. After performing the selected action, the ship microgrid environment will give feedback on the new state and instant reward. This experience (including state, action, reward, and new state) will be stored in the experience replay pool for subsequent training. Then,

$$y_{\mathrm{t}}=r_{\mathrm{t}}+\gamma {\max }_{{\mathrm{a}}^{\prime }}Q(s_{\mathrm{t}+1},a^{\prime };{\theta }^{-})$$

(14)

where $y_{\mathrm{t}}$ is the target for the current time step, $r_{\mathrm{t}}$ is the reward received at time step t, $\gamma$ is the discount factor, $s_{\mathrm{t}+1}$ is the new state, and ${\max }_{{\mathrm{a}}^{\prime }}Q(s_{\mathrm{t}+1},a^{\prime };{\theta }^{-})$ is the maximum of the Q value of all possible actions in the target network for the next state. The weight parameters of the Q network are gradually adjusted by calculating the loss, so that it more accurately approximates the optimal Q function.

Finally, in order to ensure that the agent can converge within a reasonable amount of time and avoid overtraining, there are two termination conditions:

$$\left\{\begin{array}{c}{n}_{\mathrm{ep}}^{\mathrm{c}}=n_{\mathrm{ep}}\\ \mathrm{or}\\ R_{\mathrm{total}}⩾R_{\mathrm{target}}\end{array}\right.$$

(15)

where $n_{\mathrm{ep}}^{\mathrm{c}}$ is the number of training rounds, with a maximum value limited to $n_{\mathrm{ep}}$, $R_{\mathrm{total}}$ is the accumulated reward value, and $R_{\mathrm{target}}$ is the preset accumulated reward threshold. The training automatically terminates when the preset maximum number of training rounds, $n_{\mathrm{ep}}$, is reached. Alternatively, the training can be terminated early when the cumulative reward exceeds a preset threshold $R_{\mathrm{target}}$, as the agent is considered to have learned an optimal control strategy. The setting of these two conditions enables the DQN algorithm proposed in this paper to effectively control the training time, improve training efficiency, and avoid overfitting while ensuring the performance of the control strategy.

Upon completion of training, the DQN agent employs the acquired Q function to determine the best action for each observed state, maximizing the cumulative reward. The virtual impedance-based droop control strategy is then adjusted according to the learned optimal virtual impedance gain. Consequently, the output voltage of the DG inverters is modified, leading to a precise distribution of reactive power among the DGs within the ship’s microgrid.

4. Simulation Analysis

4.1. DQN Agent Training Simulation Parameter Settings

Table 1 gives the hyperparameters of the DQN agent. During training, the agent randomly sampled four states at each step. After 16 iterations, the minimum batch size reached 64, which triggered the start of the training process. This sampling and training process repeated until convergence.

Table 1. Hyperparameters used in DQN agent training.

Hyperparameter	Value
Learning rate	$1\times {10}^{-5}$
L2 regularization factor	$1\times {10}^{-4}$
Replay buffer size	$1\times {10}^5$
Target smooth factor	$5\times {10}^{-2}$
Target update frequency	5
Discount factor	1.0
Epsilon min	$1\times {10}^{-9}$
Epsilon final	1
Epsilon decay rate	1/30
Maximum iteration episode	2000
Stop training value	$1\times {10}^{10}$

Figure 7 shows the reward curve during training. The training process reached the maximum saturation reward after the 25th round and continued to train for 74 sets for a total of about 21 min. All training was performed on a laptop equipped with an Intel 12th-generation Core i7-12700H CPU with 14 cores and 32 GB of memory.

Figure 7. Convergence analysis of DQN agent training with different learning rates: (a) learning rate $\alpha =1\times {10}^{-5}$ and (b) learning rate $\alpha =1\times {10}^{-4}$.

To investigate the effect of the learning rate on the convergence behavior of our DQN-VI strategy, we compared two different values, $\alpha =1\times {10}^{-5}$ and $\alpha =1\times {10}^{-4}$. As illustrated in Figure 7, the episode reward and average reward curves highlighted that a lower learning rate could result in slower initial convergence but more stable performance, whereas a higher learning rate sped up convergence but exhibited larger reward oscillations. This analysis underscored the importance of selecting an appropriate learning rate to balance training stability and convergence speed.

4.2. Evaluation Index of System Performance

In a ship’s power grid, due to frequent changes in load, the ship’s microgrid system requires accurate reactive power distribution from each DG to ensure power stability. In order to accurately evaluate the system’s performance in this environment, this section designs two main performance indicators to evaluate the accuracy and error of reactive power distribution: average error ($A_{\mathrm{E}}$) and weighted mean square error ($W_{\mathrm{MSE}}$). $A_{\mathrm{E}}$ is utilized to assess the reactive power distribution accuracy in the ship’s microgrid system by calculating the relative error between the actual and required reactive power at each time step. Its calculation formula is the following:

$$A_{\mathrm{E}}={\displaystyle \frac{1}{{\mathrm{T}}_{\mathrm{n}}}}{\displaystyle \sum _{t=0}^{\mathrm{T}}\left|\frac{Q_i(t)-{Q^{*}}_i(t)}{{Q^{*}}_i(t)}\right|}\times 100\%$$

(16)

where $Q_i(t)$ represents the reactive power distribution of ${\mathrm{DG}}_i$ in the ship microgrid at time $t$, ${Q^{*}}_i(t)$ represents the desired reactive power of ${\mathrm{DG}}_i$ at that moment, and ${\mathrm{T}}_{\mathrm{n}}$ represents the time points. This index can reflect the accuracy of the overall reactive power distribution of the ship microgrid system.

$W_{\mathrm{MSE}}$ is specifically designed to address the distribution accuracy during critical periods. It highlights the reactive power distribution error during the operation of the ship’s microgrid through time weighting, reflecting the response capability of the ship’s microgrid system during critical periods. Its calculation formula is as follows:

$$W_{\mathrm{MSE}}={\displaystyle \frac{{\displaystyle \sum _{t=1}^{\mathrm{T}}w(t)}\cdot {\left(Q_i(t)-{Q^{*}}_i(t)\right)}^2}{{\displaystyle \sum _{t=1}^{T}w(t)}}}$$

(17)

where $w(t)$ is the weight at time $t$, which is the sum of two Gaussian distributions, emphasizing the error at a specific time period, defined as

$$w(t)=e^{-{\displaystyle \frac{{\left(t-{\mu }_1\right)}^2}{2{\sigma }_1^2}}}+e^{-{\displaystyle \frac{{\left(t-{\mu }_2\right)}^2}{2{\sigma }_2^2}}}$$

(18)

where ${\mu }_1$ and ${\mu }_2$ denote the centers of the two Gaussian distributions, while ${\sigma }_1$ and ${\sigma }_2$ are the standard deviations which govern their width.

4.3. Simulation Analysis

In order to validate the effectiveness and superiority of the proposed strategy applied in the greenship microgrid, this study used Matlab/Simulink to build a ship microgrid simulation model. The established ship microgrid simulation model consisted of two DGs with the same parameters in parallel to the ship microgrid, except for the inconsistent line impedance. The specific simulation parameters of the input side of the ship’s microgrid are shown in Table 2.

Table 2. Simulation parameters of the input side of the ship’s microgrid.

	Parameter	Value
DGs	Filter inductor $L/(\mathrm{H})$	3 × 10−3
	Filter capacitor $C/(\mathrm{F})$	30 × 10−6
	Filter inductor equivalent resistance $R/(\mathrm{\Omega })$	0.015
line	${\mathrm{DG}}_1$ line resistance $R_1/(\mathrm{\Omega })$	1 × 10−2
	${\mathrm{DG}}_1$ line reactance $X_1/(\mathrm{\Omega })$	1 × 10−2
	${\mathrm{DG}}_2$ line resistance $R_2/(\mathrm{\Omega })$	2 × 10−2
	${\mathrm{DG}}_2$ Line reactance $X_2/(\mathrm{\Omega })$.	2 × 10−2

The ship microgrid simulation model adopted linear and non-linear loads, respectively. The adaptive VI droop control method, fuzzy adaptive VI droop control method, and DQN-VI droop control strategy were compared under each load type. Furthermore, we conducted harmonic distortion (THD) assessment of the output current of inverter #1 (Ioabc1) and the total output current (Iout) of all inverters, as depicted in Figure 8. This analysis supplemented our investigation into the effectiveness of the DQN-VI strategy in our shipboard microgrid system.

Figure 8. Harmonic distortion assessment: (a) inverter #1 output current ($\mathit{I}\mathrm{oabc}1$) and (b) total output current ($\mathit{I}\mathrm{out}$).

4.3.1. Simulation Case 1

This study conducted simulation experiments on the operation of an inverter parallel network system with linear loads. The load of the parallel system consisted of three parts: ${\mathrm{Load}}_1$ was connected at 0 s and disconnected at 1.0 s; ${\mathrm{Load}}_2$ and ${\mathrm{Load}}_3$ subsequently took turns working for 1 s. Table 3 gives the load sizes and their corresponding states used in this simulation.

Table 3. Linear load size of shiploads.

Linear Load	Active Power	Reactive Power
${\mathrm{Load}}_1$	$20 \mathrm{kW}$	$10 \mathrm{kVar}$
${\mathrm{Load}}_2$	$30 \mathrm{kW}$	$15 \mathrm{kVar}$
${\mathrm{Load}}_3$	$12 \mathrm{kW}$	$8 \mathrm{kVar}$

Figure 9 displays the active and reactive power curves over time, with clear shifts and noticeable spikes. There are sudden dips, particularly at 1.1–1.3 s and 2.1–2.2 s, suggesting that the system struggled with responsiveness. This aligns with the idea that traditional control has limitations, such as poor reaction to changes during high load and linear load switching, leading to imbalances in power distribution.

Figure 9. Power distribution of each DG in the microgrid using the traditional adaptive virtual impedance droop control method facing linear loads of the ship.

Figure 10 demonstrates that the DG’s power distribution was managed using the fuzzy adaptive droop control method incorporating virtual impedance. On the other hand, Figure 10 shows active power with less pronounced changes compared to Figure 9, and reactive power variations appear better controlled. While both figures have fluctuations, Figure 10 seems to manage these more smoothly, likely due to the fuzzy logic adjusting virtual impedance dynamically. This suggests improved adaptability, especially during challenging load conditions.

Figure 10. Power distribution of each DG in the microgrid using the fuzzy adaptive virtual impedance (VI) droop control method facing linear loads of the ship.

Comparing Figure 10 and Figure 11, it can be seen that, using the droop control strategy of DQN-VI, the DQN agent was added to estimate a function that described the relationship between $\mathrm{\Delta }Q$ of the two DGs, and the existing virtual impedance was corrected in real time.

Figure 11. Power distribution of each DG in the microgrid using the DQN-VI strategy facing linear loads of the ship.

The DQN-VI strategy greatly reduced the impact of line impedance differences on the ship microgrid system and significantly optimized the response performance of DG in the face of instantaneous changes in the linear load of the ship. In the face of 0 state response of ${\mathrm{Load}}_1$, the response speed of the ship microgrid to DG reactive power distribution increased from 0.6 s to 0.5 s. When the linear load was switched from ${\mathrm{Load}}_1$ to ${\mathrm{Load}}_2$ and from ${\mathrm{Load}}_2$ to ${\mathrm{Load}}_3$, the distribution pulsation of DG was greatly reduced.

4.3.2. Simulation Case 2

To further evaluate the proposed control strategy, a frequency converter and a propulsion motor were used as a non-linear load. The speed of the propulsion motor was regulated by the frequency converter, thereby modifying its operating condition. Table 4 shows the load size and corresponding state used in this simulation case.

Table 4. Non-linear load size of shiploads.

Operating Condition	Active Power	Reactive Power
Rated	$44 \mathrm{kW}$	$39 \mathrm{kVar}$
Overload	$49 \mathrm{kW}$	$43 \mathrm{kVar}$
Light load	$29 \mathrm{kW}$	$28 \mathrm{kVar}$

The ship microgrid model was initially in a rated state. At 1.0 s, the motor was adjusted to the overload state by frequency conversion drive, and at 2.0 s, it was switched to a light-load state. Figure 12 exhibits significant power fluctuations and sudden drops, indicating its poor response to non-linear loads and imbalanced power allocation.

Figure 12. Power distribution of each DG in the microgrid using the traditional adaptive virtual impedance (VI) droop control method facing non-linear loads on the ship.

Figure 13 shows reduced power fluctuations, with active power variations which are not pronounced and reactive power fluctuations which are better managed. This suggests that it is more capable of adapting to non-linear loads compared to the traditional adaptive virtual impedance droop control method.

Figure 13. Power distribution of each DG in the microgrid using the fuzzy adaptive virtual impedance (VI) droop control method facing non-linear loads on the ship.

Comparing Figure 13 and Figure 14, we can see that the droop control strategy using DQN-VI optimized the dynamic performance when facing the switching of ship non-linear load conditions. When the motor was accelerated from the rated state to the full-load state and from the full-load state to the light-load state, the response pulsation of the ship’s microgrid to the DG reactive power distribution was greatly reduced, and the response time was shortened from 0.2 s to 0.05 s.

Figure 14. Power distribution of each DG in the microgrid using the DQN-VI strategy facing non-linear loads of the ship.

The two evaluation indexes established in Section 4.2 were used to evaluate the reactive power deviation of the ship microgrid model in the steady state, as shown in Table 5.

Table 5. The effect of different control strategies on the power distribution of shiploads.

Load Type	Control Strategy	${\mathit{A}}_{\mathbf{E}}$	${\mathit{W}}_{\mathbf{MSE}}$
Linear load	Adaptive VI	3.44%	0.85
	Fuzzy adaptive VI	2.48%	0.89
	DQN-VI	1.66%	0.67
Non-linear load	Adaptive VI	2.83%	0.107
	Fuzzy adaptive VI	1.54%	0.118
	DQN-VI	1.27%	0.076

When the shipload changed, the DQN-VI droop control strategy optimized the reactive power deviation. Comparing fuzzy adaptive VI and DQN-VI, we can see that, under the linear load, $A_{\mathrm{E}}$ was reduced by 33%, and $W_{\mathrm{MSE}}$ was reduced by 24%; under the non-linear load, $A_{\mathrm{E}}$ was reduced by 17%, and $W_{\mathrm{MSE}}$ was reduced by 35%. Through the independent simulation of these two load types, this study analyzed the performance of the improved DQN-VI droop control strategy compared to the VI adaptive droop control strategy under different load characteristics. The simulation results showed that the control strategy better coped with the changes in linear and non-linear loads and achieved more accurate distribution of the reactive power of DGs.

5. Conclusions

This paper mainly solves the problem of reactive power sharing in greenship microgrids. The traditional adaptive droop control method employed in microgrids has limitations under high-load conditions, and its performance significantly depends on virtual impedance parameters. To address these issues, this paper proposes the DQN-VI strategy using deep reinforcement learning. The strategy models the control process as a Markov decision process, which includes defining an observation space, an action space, and a reward function. The proposed method uses a deep neural network to estimate the Q function, which relates the system state to the corresponding actions. After training, the agent automatically adjusts the virtual impedance gain, reducing reactive power deviations and enhancing the performance of greenship microgrids. Comparative simulation experiments were conducted to validate the effectiveness and superiority of the proposed strategy in greenship microgrids. The simulation results demonstrate that the proposed DQN-VI droop control strategy optimizes reactive power deviation, achieving significant reductions in error metrics compared to fuzzy adaptive VI. Quantitatively, under linear loads, $A_{\mathrm{E}}$ was reduced by 33%; under a non-linear load, $W_{\mathrm{MSE}}$ was reduced by 35%.

Although this study demonstrates the effectiveness of the DQN-VI strategy through simulation, future work will include experimental validation on hardware platforms to confirm its practical performance. Moreover, extending this strategy to hybrid AC/DC microgrids provides an opportunity to explore its versatility in more complex power system configurations.