Microgrid Operation Optimization Strategy Based on CMDP-D3QN-MSRM Algorithm

Abstract

This paper addresses the microgrid operation optimization challenges arising from the variability in and uncertainty and complex power flow constraints of distributed power sources. A novel method is proposed, based on an improved Dual-Competitive Deep Q-Network (D3QN) algorithm, which is enhanced by a multi-stage reward mechanism (MSRM) and formulated within a Constrained Markov Decision Process (CMDP) framework. First, the reward mechanism of the D3QN algorithm is optimized by introducing a redesigned MSRM, enhancing the training efficiency and the optimality of trained agents. Second, the microgrid operation optimization problem is modeled as a CMDP, thereby enhancing the algorithm’s capacity for handling complex constraints. Finally, numerical experiments demonstrate that our method reduces operating costs by 16.5%, achieves a better convergence performance, and curtails bus voltage fluctuations by over 40%, significantly improving the economic efficiency and operational stability of microgrids.

1. Introduction

As renewable energy becomes more widely utilized, the energy structures of power systems are gradually transforming into cleaner and more sustainable forms [1]. Since its inception in 2003 [2], numerous pilot programs have been implemented to supply electrical (and sometimes thermal) energy to diverse users, include rural households, public institutions, industrial facilities, military bases, and remote or island communities connected to the main grid. However, distributed renewable energy sources, such as wind photovoltaic power, are subject to natural conditions, exhibiting high volatility and intermittency. These uncertainties pose a significant challenge to day-ahead scheduling in microgrids. Moreover, the complex voltage fluctuations introduced by distributed renewable energy integration create stability challenges for microgrid operation. Therefore, exploring new optimization algorithms to adapt to the volatility and intermittency of renewable energy sources has become an important direction of microgrid operation optimization research.

For over a decade, the real-time scheduling of microgrids has been a focus of attention in the research community. Many approaches have been attempted for the real-time scheduling of microgrids, such as traditional classical optimization algorithms [3], particle swarm optimization [4], stochastic programming algorithms [5], and robust optimization algorithms [6]. Classical optimization and particle swarm algorithms perform well for deterministic linear and nonlinear problems associated with microgrid operation. However, real-world microgrids involve uncertainties, such as renewable energy generation and load demands. Stochastic programming and robust optimization are primarily employed to address these uncertainties. However, both approaches heavily rely on domain expertise to construct accurate microgrid models and parameters, resulting in high development costs and limited portability and scalability.

In recent years, with the remarkable performance of deep reinforcement learning in various fields, such as gaming [7], Go [8], and robotics [9], researchers in the power system domain have begun exploring the application of this approach to the real-time scheduling of microgrids. Ref. [10] proposes a model-free backstepping control strategy based on deep reinforcement learning for controlling an advanced nine-level rectifier, which reduces the strategy’s dependence on mathematical models. Ref. [11] proposes the utilization of deep Q-networks (DQNs) to learn from forecasted loads, renewable outputs, and real-time pricing for operational strategy formulation. Ref. [12] optimizes the action selection of DQNs using a multi-parameter action mechanism to improve the solution ability. Ref. [13] showcases the application of a PSO-MPPT strategy, effectively boosting the efficiency and power output of PV systems in AC microgrids under varying atmospheric conditions. Ref. [14] presents a hybrid optimization technique that integrates an artificial neural network with Q-learning for scheduling in-home energy management. Ref. [15] proposes a Q-learning-based energy management algorithm for microgrid building clusters to reduce costs. Ref. [16] uses multi-agent reinforcement learning to solve the distributed energy management problem in microgrids. Ref. [17] proposes a microgrid energy optimization strategy based on the state of charge (SOC), according to the function of the energy storage system in different operation modes. Although effective against uncertainty, these DQN-based approaches inherently suffer from reward sparsity due to discrete action spaces. Ref. [18] mitigates this by replacing traditional baseline reward functions (BRFs) with a multi-stage reward mechanism (MSRM), improving the training efficiency and agent optimality. Although this method effectively addresses the issue of sparse rewards, its application model is oversimplified. It fails to account for self-adjustable generation units, such as fuel-based generators or electric boilers, and does not consider complex power flow variations, which results in significant limitations in practical applications. Furthermore, the design of its multi-stage reward mechanism does not comprehensively consider various data scenarios, which could lead to erroneous outcomes.

In recent years, many researchers have turned to reinforcement learning to address the challenges of reward function formulation in microgrid scheduling, which arise due to complex power flow constraints. Ref. [19] proposes a Q-learning-based energy management strategy for microgrids to cope with the efficiency and reliability of energy storage. Ref. [20] proposes a multi-agent Q-learning algorithm for load scheduling but encounters the curse of dimensionality. Ref. [21] employed a fully distributed PPO algorithm to maximize the photovoltaic active power output and maintain voltage stability in an unbalanced distribution network. Ref. [22] incorporates safety constraints—such as power flow—as gradient information to ensure that the optimal control strategy yields safe and feasible decisions. Ref. [23] proposes a DQN algorithm for unbalanced distribution systems to minimize power losses and regulate voltage. Ref. [24] discretizes battery actions via neural networks, solves remaining variables with nonlinear programming, computes immediate rewards, and derives optimal policies through Q-learning. Ref. [25] designs a convolutional neural network structure to learn optimal scheduling policies, bypassing nonlinear optimization and reducing forecast dependency. Although these methods partially resolve high-dimensional state challenges, difficulties persist in the reward function design under complex power constraints. This results in the inability of the optimization strategies generated by these methods to fully meet the requirements of microgrid operation. To overcome these limitations, this paper proposes the CMDP-D3QN-MSRM algorithm with three key innovations:

(1)

Refinement of the functional design of the multi-stage reward mechanism to prevent erroneous outcomes caused by exceptional or edge-case data.

(2)

Replacement of the original BRF in the D3QN algorithm with the improved MSRM to mitigate poor convergence resulting from reward sparsity, thereby enhancing the algorithm’s optimality-seeking capability.

(3)

Utilization of the constructed CMDP to strengthen the algorithm’s ability to handle complex power flow scenarios and improve the operational stability of microgrids.

2. Microgrid Optimization Model

2.1. Microgrid Equipment Modeling

2.1.1. Energy Storage System

The energy storage system comprises battery units that implement peak-shaving and valley-filling functionalities. Its operation coordinates with renewable energy sources to enhance microgrid reliability and economic efficiency. As a short-term reserve resource, the storage system must supply at least two hours of rated load energy. The SOC during operation is determined by the rated capacity (E) and charge/discharge power, as mathematically expressed in the following equation:

$$SO{C}_t=SO{C}_{t-1}+{\displaystyle \frac{{\eta }_{ch}{P}_t^{ch}\Delta t}{E}}-{\displaystyle \frac{P_t^{dis}\Delta t}{{\eta }_{dis}E}}$$

(1)

where $P_t^{ch}$ is the power on the busbar after passing through the converter when the energy storage system is discharging at time period t; $P_t^{dis}$ is the power on the busbar before passing through the converter when the energy storage system is charging at time period t; ${\eta }_{dis}$ and ${\eta }_{ch}$ are the converter efficiencies during discharging and charging, respectively.

2.1.2. Fuel Units

Fuel units provide an adjustable power supply to the microgrid by burning fuel oil, effectively reducing the dependence of the microgrid on the external grid. Their operational cost follows the quadratic function expressed in Equation (2):

$$C_t^g=a_g{(P_t^g)}^2+b_g{P}_t^g+c_g$$

(2)

where $C_t^g$ is the generation cost of the fuel unit at moment t; $P_t^g$ is the generation power of the fuel unit at moment t; and $a_g,b_g,c_g$ are the cost coefficients of the fuel unit.

2.1.3. Power Line Losses

Line losses in an AC microgrid are not negligible. We employ a branch current-based method for calculating power losses. The total system line loss ($P_t^{loss}$) can be expressed as follows:

$$\begin{array}{l}{P}_t^{loss}={{\displaystyle {\sum }_{k=1}^N{R}_k·\left|I_t^k\right|}}^2\\ R_k=\rho ·l\end{array}$$

(3)

where $R_k$ is the resistance of branch k, and $\left|I_t^k\right|$ is the magnitude of the current in branch k at time t. $\rho$ is the resistivity of the line, and l is the length.

2.1.4. Microgrid Bus

The microgrid bus is connected to various components, including distributed power sources (e.g., photovoltaic and wind power systems), energy storage systems, fuel units, and the main power grid. The power balance at the bus is given by the following equation:

$$P_t^b=P_t^w·{\eta }_w+P_t^{pv}·{\eta }_{pv}+P_t^g-P_t^{ch}+P_t^{grid}-P_t^{load}+P_t^{dis}-P_t^{loss}$$

(4)

where $P_t^b$ is the net power balance; the system is balanced when $P_t^b=0$. $P_t^w$ is the active power output from wind power generation in time period t. $P_t^{pv}$ is the active power output from photovoltaic power generation in time period t. $P_t^g$ is the active power output from the power generation from fuel units. $P_t^{grid}$ is the power of energy exchange between the microgrid and the main grid; power is sold to the main grid in the case of $P_t^{grid}<0$, and power is purchased from the main grid in the case of $P_t^{grid}>0$. $P_t^{load}$ is the magnitude of active loads in time period t. ${\eta }_w$ and ${\eta }_{pv}$ represent the converter efficiencies of wind power generation and photovoltaic power generation, respectively.

2.2. Constraints

2.2.1. AC Trend Constraints

$$V_{\min }\le V_t^i\le V_{\max },0\le I_t^i\le I_{\max }$$

(5)

where $V_t^i$ is the voltage at bus i at time t; $V_{\max }=1.05·V$ and $V_{\min }=0.95·V$ are its upper and lower constraints; and V is the standard value of the bus voltage. $I_t^i$ is the current at bus i at time t; $I_{\max }=1.05·I$ is its upper constraints; and I is the standard value of the bus current.

2.2.2. Fuel Generator Operating Constraints

$$0\le P_t^g\le P_{\max }^g$$

(6)

where $P_{\max }^g$ and $P_{\min }^g$ are the upper and lower constraints for fuel generators.

2.2.3. Battery Operation Constraints

$$SO{C}_{\min }\le SO{C}_t\le SO{C}_{\max }$$

(7)

where $SO{C}_{\max }$ and $SO{C}_{\min }$ are the maximum and minimum values of the battery charge state.

2.2.4. Microgrid and the Main Grid Interaction Constraints

$$-P_{\max }^{grid}\le P_t^{grid}\le P_{\max }^{grid}$$

(8)

where $P_{\max }^{grid}$ is the maximum power exchanged between the microgrid and the main grid.

3. Optimization Model Based on Constrained Markov Strategy Process

This paper models the microgrid operation optimization process as a Markov Decision Process (MDP). The MDP framework incorporates stochastic renewable generation, real-time electricity pricing, and the energy storage system SOC to define system states and determine optimal energy dispatch decisions. Formally, the MDP is defined by the tuple <S,A,P,R>.

3.1. State Space

The state space, denoted by S, contains all the environmental information required for the microgrid operation optimization process. To simplify the model, only the time-varying information is included in the state. The state space S is defined by the following equation, where $p_{price}$ is the real-time electricity price of the main grid:

$$S=[t,P_t^{load},P_t^{pv},P_t^w,SO{C}_t,p_{price}]$$

(9)

3.2. Action Space

In the microgrid, the controllable actions include the output power of the fuel unit and the charge/discharge power of the energy storage system. The action space A is defined as follows:

$$A=[P_t^g,P_t^{dis},P_t^{ch},P_t^{grid}]$$

(10)

3.3. Transfer Probability

Since the state transition probability is affected by renewable energy uncertainties and is difficult to model accurately, a deep reinforcement learning algorithm is employed in this study to extract the transition model from historical data:

$$p_{ss’}^a=p[S_{t+1}=s’|S_t=s,A_t=a]$$

(11)

3.4. Reward Function

The reward function consists of the self-balancing rate ($R_t^s$), reliability rate ($R_t^r$), microgrid operating cost ($C_t$), and penalty function ($f_t$), as shown in (12):

$$R(s_t,a_t)=f(R_t^s,R_t^r,C_t,f_t)$$

(12)

3.4.1. Self-Balancing Rate

The self-balancing rate is defined as the percentage of a microgrid’s load that is supplied by its internal distributed energy resources over a defined period. Achieving a higher self-balancing rate contributes to reduced dependence on the main grid, increased energy self-sufficiency, lower power procurement costs, and diminished carbon emissions, as shown in Equation (13):

$$R_t^s=1-{\displaystyle \frac{P_t^g+P_t^{grid}}{P_t^{laod}+P_t^{ch}}}$$

(13)

In this study, the threshold for the self-balancing rate is set at 0.9. This signifies that 90% of the energy demand is met by internal resources. This approach encourages the microgrid to reduce its reliance on the main grid, decreases carbon emissions, and avoids the excessive costs associated with pursuing 100% self-sufficiency.

3.4.2. Reliability Rate

Reliability is defined as the ability of a microgrid to deliver the required power to its stakeholders continuously and stably under specified operational conditions. This ability is directly quantified by the reliability rate, which calculates the ratio of the total generation to the total demand. A value consistently maintained within the [0.95, 1.05] range indicates a highly reliable system that avoids both power shortages (below 0.95) and significant power surpluses (above 1.05) that could lead to stability issues:

$$R_t^r={\displaystyle \frac{P_t^w+P_t^{pv}+P_t^g-P_t^{ch}+P_t^{grid}+P_t^{dis}-P_t^{loss}}{P_t^{load}}}$$

(14)

3.4.3. Operating Costs

The microgrid operating costs mainly include the cost of purchasing power purchases from the main grid and the cost of generating power from the fuel units, as shown in (15):

$$\begin{array}{l}{C}_t=C_t^g+C_t^{grid}\\ C_t^{grid}=P_t^{grid}\ast p_{price}\ast \Delta t\end{array}$$

(15)

3.4.4. Penalty Functions

Within the traditional MDP framework, penalty functions can be employed to handle complex constraints by incorporating them into the objective function, as shown in (16), where k is the penalty coefficient and m represents the number of violated constraints:

$$f=km$$

(16)

Since m solely represents the number of violated constraints and does not reflect their severity, the setting of the penalty coefficient (k) significantly impacts the algorithm’s performance. This necessitates extensive trial-and-error adjustments during the design phase, increasing implementation difficulty and reducing the algorithm’s ability to handle complex constraints. Therefore, this paper employs the CMDP to formulate the penalty function for the microgrid constraints, as shown below:

$$\begin{array}{l}{f}_t=\beta \left(\max \left[\max \left(0,V_t^i-V_{\max }\right),V_{\min }-V_t^i\right]/V+\right.\max \left[\max \left(0,SO{C}_t-SO{C}_{\max }\right),SO{C}_{\min }-SO{C}_t\right]+\\ \max \left(0,P_t^g-P_{\max }^g\right)/P_{\max }^g+\left.\max \left[\max \left(0,P_t^{grid}-P_{\max }^{grid}\right),-P_{\max }^{grid}-P_t^{grid}\right]/P_{\max }^{grid}\right)\end{array}$$

(17)

where the first term calculates the degree of overloading of each bus voltage. The second term is the degree of constraint violation of the energy storage system’s state of charge. The third term is the degree of constraint violation of the fuel unit. The fourth term is the degree of constraint violation of the exchanged power of the main grid.

4. Microgrid Operation Optimization Based on CMDP-D3QN-MSRM Algorithm

4.1. D3QN Algorithm

The D3QN algorithm is a deep reinforcement learning algorithm that combines the Double DQN and Dueling DQN [26]. It leverages the Dueling DQN architecture to construct a Q-network that decomposes the Q-value estimation into a state value function and an advantage function. This decomposition facilitates more effective action selection. Concurrently, D3QN employs the Double DQN mechanism to mitigate the Q-value overestimation problem, thereby enhancing the algorithm’s ability to converge towards near-optimal solutions. The network structure, illustrated in Figure 1, consists of two fully connected layers. The first fully connected layer extracts salient features from the input data. These features are then fed into the second fully connected layer, which implements the Dueling DQN structure by branching into separate streams: one estimating the state value function and the other estimating the advantage function. This decoupling enables better policy generalization by independently learning the values of states and the relative importance of actions.

The valuation of the estimation network (Q) is defined as shown in Equation (17):

$$Q(s_t,a_t;\theta ,{\theta }_v,{\theta }_A)=V(s_t;\theta ,{\theta }_v)+A(s_t,a_t;\theta ,{\theta }_A))-{\displaystyle \frac{1}{N_a}}A(s_t,a’;\theta ,{\theta }_A)$$

(18)

where $s_t,a_t$ are the state and action at time t, respectively; ${\theta }_v,{\theta }_A$ is the parameter of the state value function network; $N_a$ is the dimension value of the agent’s action; and $a’$ represents all possible actions.

The D3QN algorithm utilizes the $\epsilon -greedy$ strategy to select the action of the agent; that is, in each selection of the action, the agent will have a probability of $1-\epsilon$ to select the action with the largest Q-value and randomly select the action with a probability of $\epsilon$, and the value of $\epsilon$ will be gradually reduced to make the algorithm result converge. The target value of the Q-network in this process is shown in Equation (18):

$$y_t=r_t+\gamma Q’(s_{t+1},\arg \underset{a}{\max }Q(s_{t+1},a_t;\theta ,{\theta }_v,{\theta }_A);\theta ’,{\theta }_v’,{\theta }_A’)$$

(19)

where $\gamma$ is the discount factor and $\theta ’,{\theta }_v’,{\theta }_A’$ is the network parameter of $Q’$. The training objective of the D3QN is to minimize the loss function of the following equation.

$$L(\theta )={\rm E}[{(y_t-Q(s_t,a_t;\theta ,{\theta }_v,{\theta }_A))}^2]$$

(20)

4.2. Multi-Stage Reward Evaluation Mechanism

In conventional deep reinforcement learning approaches for microgrid operation optimization, BRFs are typically employed to simultaneously address operational cost minimization and constraint satisfaction. This function generally takes the following mathematical form:

$$\begin{array}{l}{r}_t=-(C_t+D_t^r+D_t^s-f_t)\\ D_t^r=\left|P_t^w+P_t^{pv}+P_t^g-P_t^{ch}+P_t^{grid}-P_t^{load}+P_t^{dis}\right|\\ D_t^s=\left|P_t^w+P_t^{pv}-P_t^{ch}-P_t^{dis}-P_t^{load}\right|\\ f=km\end{array}$$

(21)

where $D_t^r$ in (20) is the active power balance equation, and $D_t^s$ is the constraint for achieving self-sufficiency using distributed energy resources. However, such a simple design of the reward function leads to the problem of sparse rewards.

To address the above issues, this paper introduces a multi-stage reward mechanism. This mechanism divides the reward function into three stages: the step reward, the final reward, and external expert intervention, thereby enhancing the training efficiency and solution quality. The corresponding pseudocode is provided below (Algorithm 1).

Algorithm 1. Multi-Stage Reward Mechanism

1: Set penalty reward coefficient ${\zeta }_1,{\zeta }_2,{\zeta }_3$ 2: for episode in 1 to M do 3:  Initialize final reward $r_{final}$ and step reward $r_t^{step}$ 4:  for t in l to T do 5:     Observe state St and action At 6:     if External expert intervention is met then 7:       $P_t^E$ is given by exert inputs 8:     else 9:       $P_t^E$ is given by DRL 10:    end if 11:    Calculate step reward at every t: $r_t=r_t^{step}$ 12:    Calculate final reward at every t:                $r_{final}=r_{final}+r_{t1}+r_{t2}+r_{t3}$ 13:    if episode terminates t = T then 14:      $r_t=r_{final}$ 15:    else 16:       $r_t=r_t^{step}$ 17:    end if

4.3. Step Rewards

Step rewards are immediate rewards provided to the agent after each action, representing an assessment of the environmental state. These rewards are calculated based on defined metrics, as shown below:

$$r_t^{step}={\delta }_1·\left[{\displaystyle \sum _{i=1}^t{R}_t^s}/t-{\displaystyle \sum _{i=1}^t{R}_t^r}/t-\right.\left.\left({\displaystyle \sum _{i=1}^t\frac{C_t}{(P_t^{load}-P_t^w-P_t^{pv})\ast p_{price}}}\right)/t-2\right]$$

(22)

$$r_t^{step}={\displaystyle \frac{{\zeta }_1·{\displaystyle \sum _{i=1}^t{R}_t^s}-{\zeta }_2·{\displaystyle \sum _{i=1}^t\left|R_t^r-1\right|}-{\zeta }_3·{\displaystyle \sum _{i=1}^t{C}_t}-{\displaystyle \sum _{i=1}^t{f}_t}}{t}}$$

(23)

where Equation (21) represents the step reward adopted from Ref. [12], and ${\delta }_1$ is the reward factor. The first term in (21) reflects the self-balancing rate of the microgrid operation from time step 1 to t. The second term represents the reliability rate of the microgrid operation from time step 1 to t. The third term indicates the degree of cost savings achieved during the microgrid operation over the same period.

However, the second term in (21) only encourages the agent to select actions where the total power output from all components exceeds the load demand. This formulation fails to ensure actual power balance and has therefore been revised as the second term in (22). The third term in (21) was originally formulated as a ratio to evaluate cost savings. However, a critical numerical instability arises when the load demand approaches the renewable generation output: the denominator (representing the operating cost) tends toward zero, causing the reward value to diverge toward infinity. This exploding reward does not reflect the actual performance and could mislead the agent toward suboptimal solutions. To avoid this singularity, we replace the fractional form with the denominator itself—the operating cost—as the third term in the revised reward function (22). This reformulation preserves the incentive to minimize costs while ensuring numerical stability across the entire operational state space.

Furthermore, Equation (21) neglects the operational challenges posed by power flow constraints. Consequently, a fourth penalty term is added in (22). Due to the differing orders of magnitude among these revised reward components, a single reward coefficient (as used in (21)) is inadequate. Separate coefficients are therefore assigned to normalize their magnitudes. To determine appropriate scaling coefficients, a global sensitivity analysis framework based on Bayesian optimization was employed. This study utilized the variation range of the three components in Equation (23) to preliminarily determine the baseline values for the scaling coefficients. Subsequently, a sensitivity analysis was conducted based on these baseline values, ultimately determining that the values of ${\zeta }_1,{\zeta }_2,{\zeta }_3$ are 10, 0.2, and 100.

4.4. Final Reward

The final reward is provided to the agent at the end of each training episode. It offers a comprehensive evaluation of the agent’s behavior throughout the episode, thereby encouraging the pursuit of long-term benefits. Similar to the step reward, the final reward also comprises four components, as shown in (23):

$$r_t^{final}=({\displaystyle \sum _{t=1}^T{r}_{t1}+r_{t2}+r_{t3}})-f_t$$

(24)

The final reward in this study differs from that in Ref. [12] primarily through the incorporation of an additional penalty function. The first component assesses the agent’s contribution to the microgrid’s self-balancing rate. An additional reward is granted if the self-balancing rate is greater than or equal to 0.95; otherwise, the reward is calculated based on the ratio of the battery discharge capacity to the total load demand. It is defined as follows:

$$r_{t1}=\left\{\begin{array}{l}\left(1+{\displaystyle \frac{P_t^{dis}}{P_t^{load}+P_t^{ch}}}\right)·{\zeta }_1,R_t^s\ge 0.95\\ \left({\displaystyle \frac{P_t^{dis}}{P_t^{load}+P_t^{ch}}}\right)·{\zeta }_1,R_t^s<0.95\end{array}\right.]$$

(25)

The second component evaluates the agent’s effectiveness at maintaining the power supply reliability throughout the episode. A fixed reward is given if the reliability rate remains between 0.95 and 1.05; otherwise, no reward is granted. It is defined as follows:

$$r_{t2}=\left\{\begin{array}{l}0,\left|R_t^r-1\right|>0.05\\ {\zeta }_2·\left|R_t^r-1\right|,\left|R_t^r-1\right|\le 0.05\end{array}\right.$$

(26)

The third component quantifies the cost savings achieved by the agent through optimized energy management. It incentivizes actions that reduce electricity procurement costs from the main grid, thereby enhancing the microgrid’s economic efficiency. It is defined below:

$$r_{t3}=-C_t·{\zeta }_3$$

(27)

The fourth component is a penalty that assesses whether the agent’s actions violate operational constraints. Penalties are assigned according to the severity of the violation, guiding the agent to develop optimal strategies that enhance operational security without breaching constraints. Its definition is consistent with (16).

4.5. Expert Intervention

During agent training, the external expert intervention module continuously monitors environmental states. Based on predefined operational thresholds, it generates explicit control signals by incorporating domain-specific a priori knowledge derived from classical energy management rules or optimization techniques. This intervention mechanism prevents convergence to suboptimal policies. Specifically, when system states violate predefined safety boundaries, supplementary rule-based controls override the agent’s learning process:

$$\begin{array}{l}{P}_t^{ess}=\left\{\begin{array}{l}{P}_t^{dis},SO{C}_{t-1}>0.85\cap R_{t-1}^s>0.9\\ P_t^{ch},SO{C}_{t-1}<0.25\cap R_{t-1}^s<0.9\\ P_t^{DRL},other\end{array}\right.\\ P_t^{dis}=P^L-P^{PV}-P^{WT}；P_t^{ch}=U_t^E\ast I_{ch}^E；I_{ch}^E=r_{ch}\ast B_E\end{array}$$

(28)

where $P_t^{DRL}$ denotes the charging or discharging strategy formulated by the agent, $U_t^E$ denotes the terminal voltage of the energy storage system, and $I_{ch}^E$ denotes the charging current (calculated from the charging rate ($r_{ch}$) and capacity ($B_E$)).

The energy storage system is mandated to discharge when it is nearly saturated ($SO{C}_{t-1}>0.85$ at the previous time step) and renewable energy generation is sufficient ($R_{t-1}^s>0.9$). This prevents overcharging. Conversely, the system is required to charge when its available energy is low ($SO{C}_{t-1}<0.25$ at the previous time step) and renewable energy generation is insufficient ($R_{t-1}^s<0.9$). This prevents excessive discharging. The terminology used in this paper is summarized in Appendix A.

5. Example Analysis

5.1. Simulation Environment Setup

The experimental data—including the PV generation, wind power output, electrical load, and real-time electricity prices—originated from actual operating records of the CAISO system [27] in California. We utilized 2021–2022 data as the training set for neural network development and reserved the 2023 data for testing.

As shown in Figure 2, the microgrid configuration comprises three 350 kVA diesel generators, two 2000 kWh battery storage units, three 300 kW photovoltaic arrays, two 300 kW wind turbines, and multiple load centers. Key operational parameters include a 1000 kVA power exchange limit with the main grid and battery state-of-charge constraints (10–90% operating range).

This study implements three algorithms sharing identical agent training frameworks: D3QN-BRF(DB), D3QN-MSRM(DM), and CMDP-D3QN-MSRM(CDM). Their neural network architecture consists of two feature extraction modules with fully connected layers sized 5 × 512 and 512 × 512; a value function stream comprising layers 512 × 256 and 256 × 1; and an advantage function stream containing layers 512 × 256 and 512 × 3. Here, dimension 5 corresponds to the state space size, while 3 denotes the action space dimension. All hidden layers employ ReLU activation functions. The implementation, developed in Python3.9, utilizes TensorFlow2.0 for neural network training. The values of the parameters used in the algorithm are shown in

Table 1. Parameters of the D3QN-based microgrid operation optimization method.

Parameter	Value
Scheduling duration (T)	24
Batch size	128
Learning rate (L)	0.001
Initial, final, decay rates of $\mathsf{\epsilon }$	1, 0.01, 0.995
Discount factor ($\mathsf{\gamma }$)	0.97
Target network update frequency	480
Experience replay pool	24,000
Converter efficiencies (${\eta }_{ch},{\eta }_{dis},{\eta }_{pv},{\eta }_w$)	0.85, 0.9, 0.9, 0.9
Penalty coefficient (β)	1000
Resistivity ($\rho$)	$0.15\Omega /\mathrm{km}$
Length between each node (l)	500 m
Standard voltage (V)	10 kV
Standard current (I)	580 A
Fuel cost factors ($a_g,b_g,c_g$)	0.00015、1.057、0.4712

. The experimental setup comprised an 18-core AMD EPYC 9754 CPU, an NVIDIA GeForce RTX 3090 GPU, and 24 GB of RAM. The total training duration was 8 h.

5.2. Analysis of Training Results of Algorithms

The uncertainty and stochasticity inherent in distributed power systems can lead to reward sparsity problems for DRL algorithms, hindering their convergence to the optimal solution. This section evaluates the performance of the MSRM in improving the convergence properties of the reward function by comparing the training reward curves of the algorithms.

As shown in the training reward curves of Figure 3, all three algorithms exhibit convergence behavior around 2000 rounds. The DB algorithm displays substantial reward fluctuations during its early exploration phase. Although converging around 2000 rounds, its reward values subsequently exhibit a decreasing trend. In contrast, both the DM and CDM algorithms demonstrate a gradual and steady increase in their reward values, maintaining a slow upward trajectory throughout. These observations indicate that the BRF-based DRL algorithm, despite convergence, tends to settle toward a suboptimal solution following exploratory fluctuations. Conversely, the MSRM-based DRL algorithm provides more precise guidance, enabling it to achieve a final reward closer to the optimal solution, and yields more stable and reliable learning outcomes.

To evaluate the stability of the algorithms, five independent training runs were conducted for each algorithm using different random seeds. To reduce variance, the final performance after convergence for each run was evaluated by taking the average reward over the last 100 episodes. The results are presented in

Table 2. Statistical analysis of each algorithm’s performance metrics.

Algorithm	Average Rewards	Standard Deviation	95% Confidence Interval
DB	−1731.4	7.36	(−1738.6, −1724.3)
DM	4904.6	9.75	(4894.5, 4914.8)
CDM	1012.5	5.9	(1005.1, 1019.9)

.

Table 2 presents the performance statistics of the three algorithms—DB, DM, and CDM—including the average reward, variance, and confidence interval upon convergence over five independent runs. The results clearly demonstrate that the convergence values of all three algorithms exhibit minimal fluctuation across multiple runs. The coefficients of variation (CV) for the DB, DM, and CDM algorithms are 0.00425, 0.00199, and 0.00583, respectively, all below 1%. Therefore, it can be concluded that the DB, DM, and CDM algorithms all demonstrate highly stable and reproducible convergence behavior.

5.3. Analysis of Optimization Results of Algorithms

5.3.1. Constraints of the Running Strategy

Complex power flow dynamics impose high-dimensional, multi-period nonlinear constraints on operational optimization, posing challenges for convergence to feasible solutions. This section evaluates the CMDP’s efficacy at enhancing the algorithmic handling of these constraints through a comparative analysis of constraint satisfaction metrics across optimization strategies.

Figure 4 displays the real-time voltage fluctuations across all buses following operational optimization using the CDM algorithm. It can be observed that only Bus 11 experienced a voltage limit violation at 16:00, while all other buses maintained voltages well within the permissible constraints throughout the operational period. The bus voltage conditions for the strategies formulated by the DB and DM algorithms are compared and analyzed in

Table 3. Balance constraints and voltage constraints for strategies formulated by each algorithm.

Comparison Value		Power Balance	Bus 1	Bus 2	Bus 3	Bus 4	Bus 5	Bus 6	Bus 7	Bus 8	Bus 9	Bus 10	Bus 11
Standard deviation	DB	0.159	0.006	0.011	0.003	0.031	0.011	0.051	0.009	0.013	0.046	0.042	0.069
	DM	0.073	0.006	0.009	0.002	0.03	0.01	0.048	0.008	0.013	0.043	0.039	0.066
	CDM	0.061	0.006	0.009	0.001	0.029	0.012	0.035	0.005	0.003	0.026	0.020	0.025
Maximum value	DB	1.619	1.008	1.016	1.002	1.031	1.02	1.085	0.996	1.013	1.067	1.062	1.097
	DM	1.141	1.007	1.017	1.001	1.033	1.018	1.011	1.015	0.995	1.052	1.067	1.097
	CDM	1.194	1.002	1.016	1.001	1.035	1.017	1.018	1.001	1.001	1.033	1.047	1.053
Minimum value	DB	0.906	0.994	0.981	0.991	0.96	0.992	0.953	0.981	0.978	1.016	1.012	1.038
	DM	0.91	0.991	0.986	0.996	0.954	0.99	0.93	0.989	0.981	0.942	1.011	1.013
	CDM	0.941	0.991	0.991	0.997	0.966	0.985	0.932	0.99	0.994	0.967	0.995	0.998

.

First, regarding the power setpoints for fuel units and energy storage systems during interactions with the main grid, none of the strategies generated by the three algorithms exceeded the prescribed power limits. Moreover, none of the energy storage systems experienced overcharging or over-discharging conditions.

As shown in Table 2, in terms of power balance constraints, the strategy utilizing the MSRM method significantly outperforms the BRF-based strategy across all metrics—the maximum value, minimum value, and standard deviation. The CMDP framework also shows an improved power balance performance.

Under complex power flow conditions, voltage fluctuations at buses 1–5, 7, and 8 are highly consistent across all three algorithms. In contrast, at buses 6 and 9–11, the CDM algorithm reduced voltage deviations by approximately 40% compared to the other two algorithms, and more importantly, resulted in only a single voltage limit violation, whereas the other two algorithms exhibited multiple violations.

These results demonstrate that the CMDP-based DRL algorithm effectively addresses the high-dimensional challenges arising from complex power flow conditions.

5.3.2. Analysis of Operational Strategies Developed by Each Algorithm

The three pre-trained neural networks were deployed separately to optimize the operation of the test system, each yielding a distinct strategy with its associated cost. The strategy produced by the DB algorithm (Figure 5a) incurred a cost of USD 285.06. In comparison, the DM algorithm (Figure 5b) achieved a lower cost of USD 253.57, while the CDM algorithm (Figure 5c) resulted in the lowest cost of USD 238.15. The real-time electricity tariff used in formulating these strategies is shown in Figure 6.

Analysis of the optimization strategies in Figure 5 and the electricity price profile in Figure 6 reveals the following operational patterns: the fuel units and the energy storage system primarily respond to the main grid’s behavior by balancing power. Since the fuel units’ generation cost is constant, they are dispatched when their production cost is lower than purchasing electricity from the main grid. This economic dispatch principle leads to increased fuel unit output during high-electricity-price periods.

All three algorithms optimize grid interaction by purchasing power from the main grid during low-price periods (0–8 h and 17–23 h) to supply loads or charge the battery and then sell power back to the grid during high-price periods (9–16 h) to reduce operating costs. However, the DB algorithm deviated from this optimal pattern by performing several counterproductive actions (e.g., at hours 3, 6, and 11) during both buying and selling windows, highlighting its convergence to a suboptimal solution. While the DM and CDM algorithms exhibited highly similar strategies from 0–16 h, the CDM algorithm achieved further cost reductions during 17–23 h. This is attributed to its superior management of the energy storage system, resulting in lower total operating costs compared to the DM algorithm.

Collectively, these results demonstrate that the CDM algorithm effectively addresses (1) the reward sparsity problem caused by the stochastic and uncertain nature of distributed power sources, and (2) the high-dimensional optimization challenges stemming from complex power flow constraints. As a result, the CDM algorithm enhances both the economic efficiency and operational reliability of scheduling strategies.

6. Conclusions and Future Perspectives

This paper addresses the operational optimization challenges posed by the uncertainty, stochasticity, and complex power constraints of distributed power sources in microgrids. To tackle these challenges, we propose a CMDP-D3QN-MSRM algorithm-based optimization strategy. The algorithm’s feasibility and effectiveness were validated through comparative experiments, leading to the following conclusions:

(1)

The CMDP-D3QN-MSRM strategy proposed in this study provides an efficient data-driven tool for addressing the trade-off between “economic efficiency” and “security” in practical microgrid operations. Compared to the D3QN algorithm, this method resolves the issue of sparse rewards, significantly reducing the microgrid’s operating costs (by 16.5%). Simultaneously, it mitigates operational risks arising from complex power flow (with voltage fluctuations reduced by over 40%). This offers a feasible technical pathway for the autonomous and intelligent operation of microgrids.

(2)

Although this study validated the proposed algorithm on a standard test microgrid, it demonstrates strong potential for scalability. For larger microgrid networks, the core framework of the algorithm requires no fundamental changes; only the state space dimensionality and constraint conditions need to be adjusted accordingly.

While this study has achieved its intended outcomes, several limitations remain, which illuminate pathways for future research:

(1)

The training in this study was based on a fixed historical data environment. Future work will explore online learning mechanisms, enabling the agent to adaptively adjust its strategies through continuous interaction with the real microgrid environment to cope with unknown load variations or equipment aging, thereby ultimately achieving real-time deployment.

(2)

To further enhance the global optimality of the scheduling scheme, we plan to investigate a hybrid framework integrating DRL with meta-heuristic algorithms For instance, DRL could be employed for rapid real-time decision making, while meta-heuristic algorithms could be utilized for refined cost optimization over longer timescales, with the two working in concert. We also plan to further refine the algorithm by comparing it with more methods (such as those in references [10,13,17], etc.).

(3)

Although this study employs heuristically designed penalty functions, it lacks formal proof of CMDP constraint satisfaction and relies on manual coefficient adjustments, which reduces its versatility. Therefore, subsequent research will utilize the Lagrangian relaxation method to address the constraint satisfaction issues and eliminate the reliance on manual parameter tuning.