A Low-Carbon Economic Scheduling Strategy for Multi-Microgrids with Communication Mechanism-Enabled Multi-Agent Deep Reinforcement Learning

Abstract

To facilitate power system decarbonization, optimizing clean energy integration has emerged as a critical pathway for establishing sustainable power infrastructure. This study addresses the multi-timescale operational challenges inherent in power networks with high renewable penetration, proposing a novel stochastic dynamic programming framework that synergizes intraday microgrid dispatch with a multi-phase carbon cost calculation mechanism. A probabilistic carbon flux quantification model is developed, incorporating source–load carbon flow tracing and nonconvex carbon pricing dynamics to enhance environmental–economic co-optimization constraints. The spatiotemporally coupled multi-microgrid (MMG) coordination paradigm is reformulated as a continuous state-action Markov game process governed by stochastic differential Stackelberg game principles. A communication mechanism-enabled multi-agent twin-delayed deep deterministic policy gradient (CMMA-TD3) algorithm is implemented to achieve Pareto-optimal solutions through cyber–physical collaboration. Results of the measurements in the MMG containing three microgrids show that the proposed approach reduces operation costs by 61.59% and carbon emissions by 27.95% compared to the least effective benchmark solution.

1. Introduction

Driven by sustained global economic growth and technological advancement, global fossil energy consumption has exhibited an exponential growth trajectory. Greenhouse gases from fossil energy combustion contribute to global warming and the frequency of extreme weather events. To address this planetary-scale challenge, the international community is accelerating energy transition initiatives, with the large-scale deployment of renewable energy, primarily wind turbine (WT) and photovoltaic (PV) systems, positioned as the cornerstone strategy for achieving carbon neutrality objectives [1,2].

China introduced its dual carbon goals of achieving carbon peaking by 2030 and carbon neutrality by 2060 in 2021, aiming to accelerate the transition of its power system by prioritizing renewable energy deployment (e.g., WT and PV) to reduce fossil fuel dependency [3]. However, the inherent variability and intermittency of renewable energy sources have hindered their effective integration into the microgrid (MG) that integrates distributed power sources, energy storage devices, and loads, limiting capacity expansion. A significant penetration of renewable energy into the microgrid may lead to challenges such as abrupt voltage fluctuations [4,5]. This necessitates innovative energy management strategies that reconcile rising consumption with decarbonization imperatives. Within the power sector, fossil-based generation continues to account for the largest share of electricity production, inevitably generating CO₂ emissions during operation [6]. The electricity generation percentage of China and worldwide from various energy sources in 2024 is shown in Figure 1. Given the direct linkage between power generation and carbon footprints, extensive research has focused on optimizing emission reduction pathways from the supply side, including carbon capture retrofitting, fuel-switching technologies, and efficiency-driven dispatch models.

Recent research demonstrates that advanced multi-energy conversion technologies, including micro-combined heat and power systems and electrically driven heat pumps, enable systemic energy efficiency enhancements and decarbonization through synergistic integration with multiple energy vectors (electrical, thermal, and cooling) [7]. In particular, the ability to regulate the dynamics of carbon flows, as a key medium linking energy conversion and carbon emissions, directly affects the level of decarbonized operation of the system. A blank planning model for energy–carbon coupled multi-energy flow synergistic optimization is established to implement system-wide collaborative design under the constraints of multi-energy–carbon flow [8]. In order to enhance the capacity of microgrids to mobilize and consume renewable energy and thereby reduce the carbon emissions of the system [9], power-to-gas (P2G) technology and carbon capture and storage (CCS) have been introduced into the integrated energy system (IES) [10]. A low-carbon optimization framework integrating carbon capture technology (CCT) and a stepped carbon trading mechanism are proposed. Based on a coupled carbon cost–benefit analysis, a system operation model with the objective of economic optimization and carbon constraints is established to realize the benefit assessment of CCT plants and multi-dimensional carbon emission co-regulation [11]. Based on the refined carbon emission flow model, a framework for the low-carbon optimal operation of flexible distribution grids is constructed, and an energy–carbon joint optimization method oriented to carbon flow density sensitivity analysis is proposed through the multi-energy synergistic mechanism by coupling distributed power sources and intelligent soft switches [12]. Energy storage systems (ESSs) have become one of the most important devices for reducing PV curtailment [13]. Here, a low-carbon dispatch architecture for a hydrogen–electric complementary ESS, including reconfigured energy storage (RES) and hydrogen energy storage (HES), is proposed. On this basis, a full life cycle carbon emission flow tracking model for IES based on energy consumption–carbon emission coupling and adopting a model predictive control method is constructed [14]. A power system carbon tracking model integrating a real-time carbon intensity model and a power flow coupling mechanism is established, which achieves carbon flow resolution through the Newton–Raphson algorithm and constructs the unit–load carbon mapping relationship [15].

In the framework of energy system integration, the typical architecture of a multi-energy MG (MMG) usually consists of a distributed generation unit, a multi-energy flow coupling device, an area baseload system, and an ESS [16]. In the context of the low-carbon transition of the power system, an electric–hydrogen co-trading strategy incorporating energy–carbon coupling mechanisms for hydrogen–energy integrated MMGs has been proposed, aiming at energy self-balancing and low-carbon emission co-optimization of MMG clusters [17]. Empirical evidence indicates that modern MGs have emerged as innovative supply architectures that demonstrate dual merits in economic viability and decarbonization. This is achieved through coordinated multi-energy dispatch frameworks coupled with enhanced RES penetration strategies [18,19]. However, constrained by spatial–temporal resource distribution patterns, these localized energy systems exhibit inherent technical constraints in two critical aspects: insufficient flexibility in multi-energy coupling capacity margins and limited inertial response for mitigating RES generation intermittency, i.e., such systems demonstrate suboptimal dynamic response characteristics under rapidly fluctuating electrical load conditions due to extended transient intervals required for convergence to steady-state operational setpoints. These challenges require dynamic compensation via advanced optimization strategies, potentially involving either interconnected MMG configurations or hybrid energy storage systems with multi-timescale response characteristics [20].

Effective optimization methods are essential for the safe operation of the system [21,22]. A low-carbon-driven energy sharing pricing mechanism based on supply–demand synergy is introduced, which realizes the synergistic optimization of economic and carbon emissions of MG clusters by constructing a non-cooperative game framework to interact with dynamic price signals from virtual energy-sharing centers [23]. A two-stage robust optimization model incorporating source–load uncertainty has been developed [24], and a solution algorithm based on the alternating direction multiplier method has been designed to verify the dual advantages of the mechanism in terms of multi-energy complementary efficiency improvement and carbon emission reduction target achievement. A low-carbon cooperative scheduling method for grid-connected MMGs based on credible chance-constrained optimization is proposed [25]. The conservatism of distributed energy uncertainty modeling is reduced by constructing a composite fuzzy set, and a linear decision rule is employed to achieve the coordinated optimization of multi-energy flow and carbon emission control. An MMG synergistic optimal dispatch model with integrated CCS is proposed by integrating demand response (IDR) and waste heat recycling mechanisms [26]. With the gradual enlargement of the system scale, the gradual complexity of the system structure, and the diversification of the control variables, the above-mentioned methods are incapable of obtaining the optimal low-carbon economic strategy quickly and effectively.

With the rapid development of AI technology, deep reinforcement learning (DRL) methods with memory capabilities are widely used in the fields of power system voltage control [27,28] and energy management [29,30]. The concept of spatio-temporal carbon intensity has been proposed [31], which can dynamically reflect the differences in carbon emissions in different regions and time periods. Based on this indicator, a model for the fair distribution of carbon emission responsibility and real-time low-carbon scheduling is constructed by combining non-cooperative game strategy and multi-agent deep reinforcement learning (MADRL). A hierarchical MADRL framework, which decomposes collaborative planning and action control tasks hierarchically, is proposed [32]. An MMG-optimized scheduling strategy based on the heterogeneous MADRL algorithm, including electricity and carbon allowance trading, is proposed to address privacy leakage issues, which involves mutual access to user data across multiple microgrids, leading to the leakage of user power usage information, multi-energy coupling complexity, and cold-start challenges in interconnected MMG co-scheduling [33]. To address the shortcomings of existing residential MG cluster optimization methods in terms of local observation adaptation, privacy preservation, and scalability, a decentralized optimization model based on MADRL is proposed [34].

To enhance energy manageability, we propose an MMG energy management model based on a communication mechanism-enabled multi-agent twin delayed deep deterministic policy gradient (CMMA-TD3), which aims to minimize carbon emissions and operation costs while satisfying the safe operation of the system. The contributions of this paper are specifically summarized as follows:

(1)

In order to be able to realistically reflect the carbon emissions of a generating unit running for a long period of time, a dynamic carbon emission calculation model is constructed.

(2)

A new low-carbon optimized scheduling model based on a communication mechanism-enabled MADRL is proposed for MMG systems to achieve the efficient integration of local observation information of MGs, where each MG acts as a separate agent. The optimal scheduling decision is finally obtained by observing local information and interacting with other agents in an information game.

The remainder of this article is summarized as follows. Section 2 expounds the carbon emission theory and constructs the dynamic carbon emission calculation model and the multi-phase carbon cost calculation model. Section 3 elaborates on the multi-microgrid model principle and the computational framework of the proposed approach. In order to verify the effectiveness of the proposed approach, Section 4 analyzes and discusses the results of the proposed approach and other benchmark approaches. Finally, Section 5 summarizes this article.

2. Theoretical Studies on Carbon Emission Flows

2.1. Overview of the Carbon Emissions Theory

When fossil energy sources such as coal are utilized for electricity generation, significant CO₂ emissions are released during combustion, contributing to carbon emissions. Contrary to traditional assumptions where these emissions are directly discharged into the atmosphere at power plants, the proposed framework posits that such emissions propagate along with electricity transmission until reaching end-users. Consequently, electricity consumers share the cost of additional carbon credit purchases, shifting the burden from exclusive reliance on power producers to distributed bearing. This mechanism is formalized through the carbon emission flow (CEF) methodology illustrated in Figure 2, which quantifies the interdependency between emissions generated at the power supply side and downstream consumption patterns.

As shown in Figure 2, each generator set (e.g., wind farms or coal plants) and aggregated electricity demand area (typically representing industrial parks or urban districts) are modeled as nodes in the network. The transmission lines connecting these nodes constitute branches. Nodes 1 to 5 act as power load centers to consume electrical energy, while nodes 1 and 5 serve as connection points to the external microgrid. G1 is connected to node 2 in order to efficiently transmit the generated power to the rest of the system for regulating and controlling the distribution of power in the system. The ground (GND) represents the load balance and carbon emission balance of all power nodes in the system. As electrical energy is transmitted in the system, carbon flow is also transmitted along with electrical energy. Each branch consumes a certain amount of electrical energy when transmitting electrical energy due to the line resistance. Node carbon emission intensity and branch carbon emission intensity jointly construct the carbon emission flow analysis system of the power system to achieve full-chain tracking of the carbon footprint and energy–carbon collaborative optimization.

The CEF framework mandates that carbon responsibility allocation should originate from user-side analysis rather than conventional generator-centric approaches. By integrating topological power flow models with emission tracing algorithms, the CEF theory [15] enables precise allocation of power generation-side carbon emission obligations to end-users, thereby establishing a causality chain from energy consumption to emission accountability.

2.2. Definitions Related to Carbon Flow Theory

2.2.1. Carbon Emission Flow Rate

The CEF rate $B_{\mathrm{CEF}}$ represents the flow of CO₂ through a node of the system per unit of time. Its unit is expressed as tCO₂/h:

$$B_{\mathrm{CEF}}={\displaystyle \frac{\mathrm{d}{E}_{\mathrm{CEF}}}{\mathrm{d}t}},$$

(1)

where $E_{\mathrm{CEF}}$ represents flows of carbon emissions, and $t$ denotes time.

2.2.2. Branch Carbon Emission Intensity

The core mechanism of the carbon flow theory is rooted in the distributional characteristics of the active current in the power network. At the system operation level, a unit increment of electrical energy transmitted by any branch will lead to a corresponding increment of CO₂ on the generation side. Branch carbon emission intensity (CEI) [35], whose unit is tCO₂/(KW·h), is calculated as follows:

$$J_{\mathrm{BCE}m}={\displaystyle \frac{B_{\mathrm{BCEF}}}{P_m}},$$

(2)

where $J_{\mathrm{BCE}m}$ represents branch CEI of branch m, and $P_m$ denotes the active power of branch m.

2.2.3. Node Carbon Emission Intensity

Node CEI $J_{\mathrm{NCE}j}$ mainly describes the amount of CO₂ generated on the power generation side due to node consumption per unit of electricity, which can be calculated as follows:

$$J_{\mathrm{NCE}j}={\displaystyle \frac{{\displaystyle \sum _{m\in N^{\prime }}{P}_m{J}_{\mathrm{BCE}m}}}{{\displaystyle \sum _{m\in N^{\prime }}{P}_m}}}={\displaystyle \frac{{\displaystyle \sum _{m\in N^{\prime }}{B}_m}}{{\displaystyle \sum _{m\in N^{\prime }}{P}_m}}},$$

(3)

where $N^{\prime }$ represents all nodes connected to node j.

2.3. Carbon Emission Flow Calculation Method

According to the principle of proportional sharing [36], in any outgoing node branch current, there exists a component of the inflow to that node branch current. The carbon flow rate of the outgoing branch active current shall be the sum of the carbon emission flows from all branches in the inflow node to branches flowing into that node. Taking node 2 in Figure 1 as an example, the node CEI of node 2 is calculated as follows [35]:

$$\begin{array}{cc}\hfill J_{\mathrm{NCE}2}& ={\displaystyle \frac{{\displaystyle \sum _{m\in N^{\prime }}{P}_m{J}_{\mathrm{BCE}m}+P_2{J}_{\mathrm{NCE}{g}_2}}}{{\displaystyle \sum _{m\in N^{\prime }}{P}_m+P_2}}}\hfill \\ & ={\displaystyle \frac{P_{m_2}{J}_{\mathrm{BCE}{m}_2}+P_2{J}_{\mathrm{NCE}{g}_2}}{P_{m_2}+P_2}}\hfill \end{array},$$

(4)

where $P_{m_2}$ indicates the active power of branch m at node 2; $P_2$ indicates the active power of node 2; $J_{\mathrm{BCE}m2}$ represents branch carbon emission intensity of branch m at node 2; $J_{\mathrm{NCE}{g}_2}$ represents the CEF of the generator at node 2.

Under the theoretical framework of CEF, when the electric energy is transmitted along the network topology, the CEI of any power branch is directly determined by the CEI of its upstream head node.

$$J_{\mathrm{BCE}{m}_2}=J_{\mathrm{NCE}2}.$$

(5)

According to the CEI of each generator at each node and the load capacity of each node of the system, the carbon emissions of each node can be solved.

$$M_{\mathrm{em}i}=J_{\mathrm{NCE}i}·E_i·h,$$

(6)

where $M_{\mathrm{em}i}$ indicates the carbon emission of node i; $J_{\mathrm{NCE}i}$ represents the CEI of loading node i; $E_i$ is the load power of node i; $h$ represents the hours.

2.4. Dynamic Carbon Emission Calculation Model

In conventional carbon emission accounting methodologies, generator units are typically ascribed a constant emission coefficient that establishes a linear correlation between power generation and carbon emissions. This static quantification approach demonstrates diminishing accuracy over prolonged operational cycles due to its inability to precisely capture the inherent dynamic emission characteristics of thermal power generation systems under load-varying conditions. To address this limitation, an advanced piecewise–linear emission-intensity model is proposed, implementing load-dependent segmentation of the generator’s operational output. Through progressive refinement of load intervals under increasing generation capacity, the model exhibits stepwise attenuation characteristics in emission-intensity growth rates across operational regimes, as shown in Figure 3. Specifically, the nonlinear emission intensity curves are approximated by a multistage linearization method with interval division. As the generator approaches its rated power, the incremental marginal emission factor per unit of power output gradually decreases, thus improving the accuracy of the emission quantification under transient operating conditions. The carbon emissions $M(g,t)$ are shaded in light green. The dynamic CEI is shown as a logarithmic function, which is the real curve to the approximate. Constant CEI is shown as the purple curve in the figure and does not change with increasing power. By constantly adjusting the number of intervals, the carbon emissions gradually approach the true curve.

The specific calculation method of the dynamic carbon emission calculation model is as follows:

$$M(g,t)=\left\{\begin{array}{ll}{\omega }_1(P(g,t)-P_0)& if \min (P_{\min ,g},P_0)\le P(g,t)<P_0+\upsilon \\ {\omega }_1(P(g,t)-P_0)+{\omega }_1(P(g,t)-P_0-\upsilon ){\psi }_1& if P_0+p\le P(g,t)<P_0+2\upsilon \\ {\omega }_1(P(g,t)-P_0)+{\omega }_1(P(g,t)-P_0-\upsilon ){\psi }_1+{\omega }_1(P(g,t)-P_0-2\upsilon ){\psi }_2& if P_0+2p\le P(g,t)<P_0+3\upsilon \\ & \hfill ⋮\hfill \\ {\omega }_1(P(g,t)-P_0)+\cdots +{\omega }_1(P(g,t)-P_0-\upsilon ){\psi }_n& if P_0+(n-1)\upsilon \le P(g,t)<\min (P_0+n\upsilon ,P_{\max ,g})\end{array}\right.,$$

(7)

where $M(g,t)$ indicates the total carbon emission of unit g at time t; $P(g,t)$ represents the power output of unit g at time t; $\upsilon$ is the interval length; $P_{\min ,g}$ and $P_{\max ,g}$ represent the minimum and maximum output power interval length of unit q, respectively; $P_0$ indicates the lowest output power of the unit; ${\omega }_1$ represents the baseline value of CEI; ${\psi }_n$ represents the growth coefficient of carbon emission intensity, which gradually decreases with the gradual increase in carbon emission.

2.5. Multi-Phase Carbon Cost Calculation Model

The multi-phase carbon cost calculation mechanism is a segmented carbon accounting framework that leverages economic incentives to drive emission reductions. Its operational logic establishes multiple tiers based on user/enterprise carbon emission levels, with each tier assigned differential unit pricing coefficients. This structure exhibits a progressive pricing characteristic aligned with the principle that higher emission volumes correspond to elevated per-unit carbon costs. Its calculation model can be expressed as follows:

$$C_t^{g,\mathrm{CE}}=\left\{\begin{array}{ll}{\omega }_2(M(g,t)-M_c)& if M(g,t)<M_c+m\\ {\omega }_2(M(g,t)-M_c)+{\omega }_2(M(g,t)-M_c-\upsilon )\eta & if M_c+m\le M(g,t)<M_c+2m\\ {\omega }_2(M(g,t)-M_c)+{\omega }_2(M(g,t)-M_c-\upsilon )\eta +{\omega }_2(M(g,t)-M_c-2\upsilon )\eta & if M_c+2m\le M(g,t)<M_c+3m\\ ⋮& \end{array}\right.,$$

(8)

where $C_t^{g,\mathrm{CE}}$ indicates the total carbon emission cost of unit g at time t; ${\omega }_2$ represents the carbon emission cost coefficient; $m$ is the interval length; $M_c$ represents free carbon credits for generators, respectively; $\eta$ indicates the price increase in carbon cost between ranges, which is small.

3. Multi-Microgrid Optimal Scheduling

The MMG system composition contains $g$ ($g=1,2,\cdots ,G$) microgrid with autonomous regulation capabilities. The MMG modeling framework is shown in Figure 4. Each microgrid includes a diesel generator (DG), PV, WT, ESS, and electrical loads. Among the microgrid units, DG, PV, and WT act as active power generation units with the ability to directly inject electric power into the microgrid topology. ESS acts as a bi-directional energy buffer unit, which can either inject active power through the discharging mode or absorb excess power of the system through the charging mode. Its dynamic regulation characteristics effectively support the power balance and energy space–time transfer requirements of the microgrid. The system architecture is interconnected by power lines to form a cooperative operation network topology with N-1 redundancy characteristics, in which the MG realizes power interaction and voltage support through common connection points. The rest of the sections are summarized as follows: Section 3.1 presents the optimization objective function and constraints for the multi-microgrid model. Section 3.2 presents a MADRL-based approach to computational procedures.

3.1. Optimal Scheduling Model

3.1.1. MMG Scheduling Objective Function

From a system operator’s perspective, the fundamental objective of online energy management resides in developing real-time adaptive control strategies to address the stochastic uncertainties arising from renewable generation intermittency and load–demand volatility. This operational paradigm necessitates tri-objective optimization through (1) carbon emission abatement via multi-energy complementarity coordination, (2) operational stability assurance through dynamic security-constrained optimal power flow, and (3) cost minimization via convex dispatch scheduling. Based on the consideration of the above factors, the following low-carbon economic optimization scheduling objectives are introduced:

$$\min {\displaystyle \sum _{t=1}^T(C_t^{\mathrm{EP}}+{\displaystyle \sum _{g=1}^G(C_t^{g,\mathrm{DG}}+C_t^{g,\mathrm{CE}}})-{\displaystyle \sum _{i=1}^N{C}_{i,t}^{\mathrm{Voltage}})}},$$

(9)

$$C_t^{\mathrm{EP}}=P_t·E{P}_t·\Delta t,$$

(10)

$$C_t^{g,\mathrm{DG}}=C_t^{g,\mathrm{Fuel}}+C_t^{g,\mathrm{Operation}},$$

(11)

$$C_t^{g,\mathrm{Fuel}}={\displaystyle \sum _{i=1}^N{s}_{i,t}^{g,\mathrm{DG}}(b·{(P_{i,t}^{g,\mathrm{DG}})}^2+c·P_{i,t}^{g,\mathrm{DG}}+d)·\Delta t},$$

(12)

$$C_t^{g,\mathrm{Operation}}={\displaystyle \sum _{i=1}^N{C}_i^{g,\mathrm{start}/\mathrm{stop}}·\left|s_{i,t}^{g,\mathrm{DG}}-s_{i,t-1}^{g,\mathrm{DG}}\right|},$$

(13)

$$C_{i,t}^{Voltage}=\left\{\begin{array}{cc}7.5-150·\left|1-v_{i,t}^g\right|& v_{i,t}^g\in [0.95-1.05]\\ -150·\left|1-v_{i,t}^g\right|& v_{i,t}^g\in [0.85-0.95]\cup v_{i,t}^g\in [1.05-1.25]\\ -1500& \mathrm{Otherwise}\end{array}\right.,$$

(14)

where T denotes the number of timesteps; $C_t^{\mathrm{EP}}$ indicates the cost incurred in purchasing electricity from the main grid; $C_t^{g,\mathrm{DG}}$ stands for the cost incurred by the operation of the DG in the g-th microgrid; $C_{i,t}^{\mathrm{Voltage}}$ represents the cost of penalizing node voltage crossings. Discontinuities in function at certain points do not cause numerical problems in the subsequent algorithmic solution. $P_t$ denotes the power interaction between the main network and the MMG system; $E{P}_t$ denotes the price of electricity at time t; $\Delta t$ indicates a timestep; $C_t^{g,\mathrm{Fuel}}$ and $C_t^{g,\mathrm{Operation}}$ denote the fuel cost and operating cost of DG for the g-th microgrid system, respectively; $s_{i,t}^{g,\mathrm{DG}}$ denotes the working condition of DG; b, c, and d indicate the DG fuel cost factor; $P_{i,t}^{g,\mathrm{DG}}$ denotes the active power of the g-th DG at time t; $C_i^{g,\mathrm{start}/\mathrm{stop}}$ indicates the cost of a change in the operating status of the g-th DG; $v_{i,t}^g$ denotes the voltage of the i-th node in the g-th microgrid at time t.

3.1.2. Constraints

The current dynamics across interconnected microgrids exhibit analogous mathematical formulations to those observed within individual microgrid subsystems. The power flow model framework primarily comprises four fundamental governing equations: branch voltage balance equations, active and reactive power balance equations at nodal injection points, and branch power transmission constraints.

$$V_j^2=V_i^2+(r_{ij}^2+x_{ij}^2)H_{ij}^2-2(r_{ij}{P}_{ij}+x_{ij}{Q}_{ij}),\forall (i,j)\in E,i\in N,j\in W,$$

(15)

$${\displaystyle \sum _{(j,k)\in E}{P}_{jk}}-{\displaystyle \sum _{(i,j)\in E}(P_{ij}-r_{ij}{H}_{ij}^2)=P_j},\forall i\in N,j\in W,$$

(16)

$${\displaystyle \sum _{(j,k)\in E}{Q}_{jk}}-{\displaystyle \sum _{(i,j)\in E}(Q_{ij}-x_{ij}{H}_{ij}^2)=Q_j},\forall i\in N,j\in W,$$

(17)

$$P_{ij}^2+Q_{ij}^2=H_{ij}^2·V_{ij}^2,\forall (i,j)\in E,i\in N,$$

(18)

where $V_i$ denotes the voltage magnitude at node i; $r_{ij}$ indicates the resistance from node i to node j; $x_{ij}$ stands for the reactance from node i to node j; $P_{ij}$ represents the active power from node i to node j; $Q_{ij}$ denotes the reactive power from node i to node j; E represents the set of all the branches; N represents the set of all nodes; W represents all nodes connected to node i; $H_{ij}$ denotes the magnitude of the current flowing from node i to node j.

In order to achieve power quality standards and ensure a reliable power supply, the nodal voltage should be kept within ±5% of the permissible range [4,5]. The upper and lower limits of nodal voltage are denoted as $V_{\min }$ and $V_{\max }$, respectively, and the nodal voltage values are limited to the upper and lower limits:

$$V_{\min }\le V_{i,t}\le V_{\max }.$$

(19)

For maintaining grid stability, generation-capable nodes must maintain operational boundaries within rigorously defined active–reactive power domains that satisfy both static and dynamic security constraints.

The PV active power range is defined as follows:

$$0\le P_{i,t}^{\mathrm{PV}}\le P_{\max }^{\mathrm{PV}},$$

(20)

where $P_{i,t}^{\mathrm{PV}}$ denotes the active power injected into node i by the PV at time t, and $P_{\max }^{\mathrm{PV}}$ indicates the maximum active power of the PV.

The WT active power range is defined as follows:

$$0\le P_{i,t}^{\mathrm{WT}}\le P_{\max }^{\mathrm{WT}},$$

(21)

where $P_{i,t}^{\mathrm{WT}}$ denotes the active power injected into node i by the WT at time t, and $P_{\max }^{\mathrm{WT}}$ indicates the maximum active power of the WT.

The ESS active power range is defined as follows:

$$-P_{\max }^{\mathrm{ESS}}\le P_{i,t}^{\mathrm{ESS}}\le P_{\max }^{\mathrm{ESS}},$$

(22)

where $P_{i,t}^{\mathrm{ESS}}$ denotes the active power injected into node i by the ESS at time t, and $P_{\max }^{\mathrm{ESS}}$ indicates the maximum active power of the ESS.

The upper and lower bounds on the state of charge ($SOC$) of the ESS located at node i at time t are shown below:

$$SO{C}_{\min }^{\mathrm{ESS}}\le SO{C}_{i,t}^{\mathrm{ESS}}\le SO{C}_{\max }^{\mathrm{ESS}},$$

(23)

where $SO{C}_{\min }^{\mathrm{ESS}}$ and $SO{C}_{\max }^{\mathrm{ESS}}$ indicate the maximum and minimum of the SOC separately.

The climbing restraints of the DG, i.e., the rate at which the DG increases or decreases the output power is subject to certain constraints, are determined as follows:

$$0\le P_{i,t}^{g,\mathrm{DG}}\le P_{\max }^{g,\mathrm{DG}},$$

(24)

$$P_{i,t}^{g,\mathrm{DG}}-P_{i,t-\Delta t}^{g,\mathrm{DG}}\le Y_{\mathrm{up}}^{g,\mathrm{DG}}·\Delta t·S_{t-\Delta t}^{g,\mathrm{DG}}+P_{\max }^{g,\mathrm{DG}}·(1-S_t^{g,\mathrm{DG}}),$$

(25)

$$P_{i,t}^{g,\mathrm{DG}}-P_{i,t-\Delta t}^{g,\mathrm{DG}}\le Y_{\mathrm{down}}^{g,\mathrm{DG}}·\Delta t·S_{t-\Delta t}^{g,\mathrm{DG}}+P_{\max }^{g,\mathrm{DG}}·(1-S_{t-\Delta t}^{g,\mathrm{DG}}),$$

(26)

where $P_{\max }^{g,\mathrm{DG}}$ indicates the maximum power of the g-th DG; $P_{i,t}^{g,\mathrm{DG}}$ denotes the active power produced by the g-th DG at node i at time t; $Y_{\mathrm{up}}^{g,\mathrm{DG}}$ and $Y_{\mathrm{down}}^{g,\mathrm{DG}}$ denote the rate of the climb for DG rising and descending, respectively; $S_t^{g,\mathrm{DG}}$ denotes the working condition of the g-th DG at time t, which is a binary variable (1: on; 0: off).

The relationship between the active power, reactive power, and capacity of the above equipment is shown below:

$${(P_{i,t}^{\mathrm{PV}})}^2+{(Q_{i,t}^{\mathrm{PV}})}^2\le {(S_{\max }^{\mathrm{PV}})}^2,$$

(27)

$${(P_{i,t}^{\mathrm{WT}})}^2+{(Q_{i,t}^{\mathrm{WT}})}^2\le {(S_{\max }^{\mathrm{WT}})}^2,$$

(28)

$${(P_{i,t}^{\mathrm{DG}})}^2+{(Q_{i,t}^{\mathrm{DG}})}^2\le {(S_{\max }^{\mathrm{DG}})}^2,$$

(29)

$${(P_{i,t}^{\mathrm{ESS}})}^2+{(Q_{i,t}^{\mathrm{ESS}})}^2\le {(S_{\max }^{\mathrm{ESS}})}^2,$$

(30)

where $S_{\max }^I$ indicates the maximum capacity of the device, and $I=\left\{\mathrm{PV},\mathrm{WT},\mathrm{DG},\mathrm{ESS}\right\}$.

3.2. MADRL-Based Optimal Scheduling Approach

3.2.1. Markov Game Process

DRL is mathematically modeled through a Markov decision process, which contains three core elements: state, action, and reward. The learning mechanism is expressed as a continuous interaction process between the agent and the environment: at each discrete time step, the agent first perceives the current environment state and then chooses to perform an action based on the established policy. The action will trigger a change in the environment state through the state transfer function, and simultaneously, the environment will feed back a scalar reward as the behavioral evaluation. This ‘observation–decision–feedback’ loop mechanism enables the agent to gradually optimize its decision-making strategy through trial-and-error learning. For MMGs composed of multiple independent MGs (including G agents), it is necessary to construct a new scheduling framework based on stochastic game theory. Global optimization is achieved through strategic interaction among agents while taking into account both individual interests and system stability. This complex multi-objective coordination challenge can be rigorously formulated as a partially observable stochastic Markov game process [37].

• Agent: Each microgrid comprises an agent;
• Environment: All the information on the microgrid;
• State: The state information required by each agent g at each time t is organized as follows:

  $$\begin{array}{cc}\hfill s_t^g=& \{t,E{P}_{t-24},E{P}_{t-23},\cdots ,E{P}_{t-1},\hfill \\ & Loa{d}_{t-24},Loa{d}_{t-23},\cdots ,Loa{d}_{t-1}\hfill \\ & P_{i,t-24}^{g,\mathrm{PV}},P_{i,t-23}^{g,\mathrm{PV}},\cdots ,P_{i,t-1}^{g,\mathrm{PV}}\hfill \\ & P_{i,t-24}^{g,\mathrm{WT}},P_{i,t-23}^{g,\mathrm{WT}},\cdots ,P_{i,t-1}^{g,\mathrm{WT}}\hfill \\ & SO{C}_{i,t}^g,P_{i,t}^{g,\mathrm{DG}},V_t^g\}\hfill \end{array},$$

  (31)

  where $E{P}_{t-24},E{P}_{t-23},\cdots ,E{P}_{t-1}$ indicate the information of the time series tariff in the past 24 h of the current time t; $Loa{d}_{t-24},Loa{d}_{t-23},\cdots ,Loa{d}_{t-1}$ denote the total load information after summing the node loads for the past 24 h at the current time t; $P_{i,t-24}^{g,\mathrm{PV}},P_{i,t-23}^{g,\mathrm{PV}},\cdots ,P_{i,t-1}^{g,\mathrm{PV}}$ are the output timing data of the PV at node i; $P_{i,t-24}^{g,\mathrm{WT}},P_{i,t-23}^{g,\mathrm{WT}},\cdots ,P_{i,t-1}^{g,\mathrm{WT}}$ denote the output timing data of the WT at node i.
• Action: The agent action variables for the g-th microgrid are as follows:

  $$a_t^g=\left\{P_{i,t}^{g,\mathrm{PV}},P_{i,t}^{g,\mathrm{WT}},P_{i,t}^{g,\mathrm{ESS}},P_{i,t}^{g,\mathrm{DG}},Q_{i,t}^{g,\mathrm{PV}},Q_{i,t}^{g,\mathrm{WT}},Q_{i,t}^{g,\mathrm{ESS}},Q_{i,t}^{g,\mathrm{DG}}\right\}.$$

  (32)
• Reward: The optimization objective of the proposed control approach is to maximize the reward value. The reward for each agent is calculated as shown below:

  $$r_t^g=-\left({\displaystyle \sum _{t=1}^T(C_t^{\mathrm{EP}}+C_t^{g,\mathrm{CE}}-{\displaystyle \sum _{i=1}^N{C}_{i,t}^{\mathrm{Voltage}}})}\right)/G+C_t^{g,\mathrm{DG}}.$$

  (33)
• State transition function: The transfer function $s_{t+1}^g=\Upsilon (s_t^g,a_t^g,\aleph )$ obtains the next state $s_{t+1}^g$ based on the currently acquired state $s_t^g$, actions $a_t^g$, and environment random variables $\aleph$.

3.2.2. Proposed Approach Based on MADRL

Traditional reinforcement learning methods (e.g., Q-learning [38]) exhibit computational inefficiency due to their dependence on an exhaustive Q-table traversal in discrete action spaces. To overcome this limitation, DRL employs neural networks to achieve continuous action space mapping. The twin-delayed deep deterministic policy gradient (TD3) algorithm implemented in this study extends the deep deterministic policy gradient (DDPG) framework, effectively mitigating the Q-value overestimation bias through its dual-network architecture with delayed updates [39]. Key algorithmic enhancements comprise three technical components: an alternating update protocol between independent critic networks, a delayed synchronization mechanism for target networks, and optimized policy update intervals. In the communication mechanism-enabled DRL called CommNet, each agent can only obtain its own state individually and then transmit it to the communication platform. Each agent can obtain the common average state (CAS) by accessing the propagated communication mechanism. The CAS is treated as the input to the subsequent layer, while the actions of each agent are generated by the final output layer [40]. The extended structure of CommNet is presented in Figure 5. Each row typically represents a different layer within the network. Each neuron in a layer is connected to every neuron in the subsequent layer, and the columns reflect these connections. These layers include input layers, hidden layers, and an output layer. The input layer receives the initial data (e.g., individual agent states). The hidden layers perform intermediate calculations and transformations, including the communication steps $T^1$ and $T^2$, where agents share information. The output layer generates the final actions for each agent. The operations in the right-hand box of Figure 5 are performed based on Equation (34). Specifically, the operation is to add $L^k{l}_g^k$ and $B^k{b}_g^k$ and pass the result to the tanh activation function. $l_g^{k+1}$ is the result represented by the black line, which is processed by the linear transformation and the tanh activation function to be used for the next decision or strategy update. The blue line indicates the output $l_g^{k+1}$ of $f_g^k$ transmitted to the mean function. The red line indicates the output value after processing by the mean function.

The detailed calculations are shown below [41]:

$$l_g^{k+1}=f_g^k(l_g^k,b_g^k)=\tanh (L^k{l}_g^k+B^k{b}_g^k),$$

(34)

$$b_g^{k+1}={\displaystyle \frac{1}{G-1}}{\displaystyle \sum _{g^{\prime }\ne g}{l}_{g^{\prime }}^{k+1}},$$

(35)

where k represents the layers in the CommNet; $\tanh (·)$ is an activation function; $l_g^k$ represents the g-th agent for the k-th hidden layer vector; $b_g^k$ denotes the g-th agent for the k-th communication vector; $l_g^{k+1}$ is the g-th agent for the k + 1-th hidden layer vector; $L^k$ and $B^k$ indicate the coefficient matrices. Equation (34) can be further rewritten as follows:

$$l_g^{k+1}=\tanh (T^k{l}^k),$$

(36)

$$l^k={\left[l_1^k,\cdots ,l_g^k\right]}^T,$$

(37)

$$T^k=\left(\begin{array}{cccc}{L}^k& {\overline{B}}^k& \cdots & {\overline{B}}^k\\ {\overline{B}}^k& L^k& \cdots & {\overline{B}}^k\\ ⋮& ⋮& \ddots & ⋮\\ {\overline{B}}^k& {\overline{B}}^k& \cdots & L^k\end{array}\right),$$

(38)

$${\overline{B}}^k={\displaystyle \raisebox{1ex}{$B^k$}\!\left/ \!\raisebox{-1ex}{$(G-1)$}\right.},$$

(39)

where $T^k$ represents the coefficient matrices of k-th communication step.

3.2.3. Critic Network

The critic network of the CMMA-TD3 algorithm consists of two critic functions that compute the Q-value. Suppose $w_t$ is the result of the objective critic function. It is presented as follows:

$$w_t^g=r_t^g+\zeta \underset{j=1,2}{\min }{Q^{\prime }}_j^g(S_{t+1},a_{t+1}^g|{\theta }^{g,j^{\prime }}),$$

(40)

where $r_t^g$ is the g-th agent value of the reward at moment t; $\zeta$ denotes the discount factor; $Q^{\prime }(·)$ is the target critic function; $S_{t+1}$ is the all-agents state at time t + 1; $a_{t+1}^g$ represents the g-th agent action at time t + 1; ${{\theta }^{\prime }}^g$ denotes the g-th agent parameters of the target critic function.

The critic network loss function is denoted as follows:

$$L({\theta }^{g,j})=\mathbf{E}[{(w_t^g-Q_j^g(S_t,a_t^g|{\theta }^{g,j}))}^2],j=1,2,$$

(41)

where $\mathbf{E}(·)$ represents the mathematical expectation function.

The parameter gradients of the critic network are derived via gradient-based optimization techniques, mathematically formulated as follows:

$${\nabla }_{{\theta }^{g,j}}J({\theta }^{g,j})=L({\theta }^{g,j})·{\nabla }_{{\theta }^{g,j}}{Q}_j^g(S_t,a_t^g|{\theta }^{g,j})],j=1,2,$$

(42)

$${\theta }_{t+1}^{g,j}={\theta }_t^{g,j}-\sigma {\nabla }_{{\theta }^{g,j}}J({\theta }^{g,j}),j=1,2,$$

(43)

where ${\nabla }_{{\theta }^{g,j}}J({\theta }^{g,j})$ is the g-th agent decreasing gradient function of the critic network, and $\sigma$ denotes the learning rate of the critic network.

3.2.4. Actor Network

The parametric formulation governing the policy function of the actor network can be mathematically represented as follows:

$$a_t^g={\pi }_{{\vartheta }^g}(S_{t+1}|{\vartheta }^g),$$

(44)

where ${\vartheta }^g$ denotes g-th agent parameters of the actor network.

The aforementioned mechanism consequently induces the mathematical derivation of the policy gradient for the actor network, formally expressed as follows:

$${\nabla }_{{\vartheta }^g}J({\vartheta }^g)=\mathrm{E}[{\nabla }_{\sigma }{Q}^g(S_t,a_t^g|{\theta }^g)·{\nabla }_{{\vartheta }^g}({\pi }_{{\vartheta }^g}(S_t)+\tau )],$$

(45)

$${\vartheta }_{t+1}^g\leftarrow {\vartheta }_t^g-{\varpi }_a{\nabla }_{{\vartheta }^g}L({\vartheta }^g),$$

(46)

where $\tau$ denotes the added noise in the target action function. This regularization mechanism mitigates premature convergence in the agent’s decision-making process by strategically injecting controlled Gaussian noise into both the policy parameterization and state-action value estimation, thereby circumventing suboptimal policy collapse through entropy-regularized optimization. ${\varpi }_a$ represents the learning rate of the actor network.

The expression for the target actor function is shown below:

$${a^{\prime }}^g={\pi }_{{{\vartheta }^{\prime }}^g}(S_{t+1}|{{\vartheta }^{\prime }}^g)+\tau ,$$

(47)

$$\tau \sim clip(N(0,\beta ),-f,f),f>0,$$

(48)

where $\tau$ denotes a noise in a normal distribution, $\beta$ represents variance, and $f$ is the value truncation position.

The temporal difference-driven optimization framework in actor–critic architectures operates through differentiable guidance from the critic network to the policy network. The significant divergence between the estimated state-action value and the target value function induces two critical failure modes: (1) policy gradient dispersion in the actor network due to high-variance advantage estimates and (2) value function approximation instability caused by non-stationary target distributions. To mitigate these coupled instabilities, delayed target network synchronization is implemented via the following:

$${{\theta }^{\prime }}_{t+1}^g=\eta {\theta }_t^g-(1-\eta ){{\theta }^{\prime }}_t^g,$$

(49)

$${{\vartheta }^{\prime }}_{t+1}^g\leftarrow \eta {\vartheta }_{t+1}^g+(1-\eta ){{\vartheta }^{\prime }}_t^g,$$

(50)

where $\eta$ denotes the soft update factor, which can be constrained by $0<\eta \le 1$ to ensure stable temporal difference learning through quasi-static target propagation.

The specific computational flow of the proposed approach is illustrated in Figure 6.

4. Case Study

4.1. Dataset

In order to effectively verify the effectiveness of the proposed method, CMMA-TD3, we conducted training analysis in the MMG shown in Figure 7, which consists of three micromesh networks in total. The circuit parameters in the MMG system are shown in

Table 1. The line parameters of the MMG system.

Line	From Node	To Node	Resistance (Ω)	Reactance (Ω)
1	1	2	0.0125	0.005
2	2	3	0.0375	0.015
3	2	4	0.0300	0.012
4	2	5	0.0225	0.009
5	2	6	0.0150	0.006
6	2	7	0.7500	0.300
7	2	12	0.7500	0.300
8	1	7	0.0125	0.005
9	7	8	0.0250	0.010
10	7	9	0.0200	0.008
11	7	10	0.0225	0.009
12	7	11	0.0275	0.011
13	7	12	0.7500	0.300
14	1	12	0.0125	0.005
15	12	13	0.0175	0.007
16	12	14	0.0200	0.008
17	12	15	0.0250	0.010
18	12	16	0.0250	0.010

[16]. The interconnected MGs contain three MGs, each of which contains ESS, WT, PV, and DG. Detailed calculation parameters of Section 3 in the MMG system are shown in

Table 2. The parameters of the system.

Symbol	Value	Unit	Symbol	Value	Unit
${\omega }_1$	0.8	tCO2/MW·h	$\upsilon$	10	-
${\omega }_2$	8	$/tCO2	${\psi }_1$	0–0.05	-
$c$	30.4	$/MW·h	$\eta$	0.05	-
$b$	1040	$/(MW)2·h	$m$	0.5	-
$d$	1.3	$	$C^{g,\mathrm{start}/\mathrm{stop}}$	5	$

and

Table 3. The parameters of the MMG system.

MG	${\mathit{P}}_{\mathbf{max}}^{\mathit{g},\mathbf{PV}}$ [MW]	${\mathit{P}}_{\mathbf{max}}^{\mathit{g},\mathbf{WT}}$ [MW]	${\mathit{P}}_{\mathbf{max}}^{\mathit{g},\mathbf{DG}}$ [MW]	${\mathit{P}}_{\mathbf{min}}^{\mathit{g},\mathbf{DG}}$ [MW]	${\mathit{P}}_{\mathbf{max}}^{\mathit{g},\mathbf{ESS}}$ [MW]
MG1	0.20	0.25	0.05	0.02	0.050
MG2	0.15	0.20	0.05	0.02	0.045
MG3	0.10	0.15	0.05	0.02	0.040
MG	${\mathit{E}}_{\mathbf{max}}^{\mathit{i}}$ [MW]	$\mathit{S}\mathit{O}{\mathit{C}}_{\mathbf{max}}^{\mathit{g},\mathbf{ESS}}$	$\mathit{S}\mathit{O}{\mathit{C}}_{\mathbf{min}}^{\mathit{g},\mathbf{ESS}}$	${\mathit{Y}}_{\mathbf{up}}^{\mathit{g},\mathbf{DG}}$ [MW/h]	${\mathit{Y}}_{\mathbf{down}}^{\mathit{g},\mathbf{DG}}$ [MW/h]
MG1	0.20	1	0.05	0.025	0.025
MG2	0.18	1	0.05	0.025	0.025
MG3	0.16	1	0.05	0.025	0.025

[16]. The parameter settings of the proposed scheme are shown in

Table 4. Parameters of the proposed approach.

Parameters	Value
Train epochs	104
Batch size	256
Reward discount factor	0.95
Memory capacity	106
Learning rate of actor	10−4
Learning rate of critic	10−4

. In total, 200 days of training data and 30 days of test data were selected for the scheme proposed in this paper. The electricity load, WT, and PV-predicted power under typical scenarios are shown in Figure 8. MG1 serves as the primary load center with substantial renewable energy integration, characterized by elevated power consumption and a significant renewable energy generation capacity. MG2 exhibits reduced power consumption compared to MG1, coupled with a lower renewable energy generation capacity. MG3 demonstrates the highest renewable energy penetration among the three MGs while maintaining relatively modest electricity demand levels. The DG is the core device to ensure the power balance of the system, and its power output depends on the system’s demand. ESS, as a rechargeable and dischargeable device, mainly realizes the energy space–time transfer to improve the consumption of renewable energy.

4.2. Baselines

4.2.1. Benchmark Approaches

We evaluate the superiority of the presented methodology by employing multiple benchmarking methods for comparative analyses:

(1)

Particle swarm optimization algorithm (PSO) [42]: The particle constantly adjusts its speed and position according to its own historical optimal position and the global optimal position of the group, moves in the direction of the better solution, and finally, makes the group converge to the approximate optimal solution.

(2)

TD3: To enhance the performance and stability of DRL algorithms in continuous-action spaces, TD3 employs two critic networks to mitigate value-estimation bias and adopts a delayed-update strategy to stabilize the training process.

(3)

Multi-agent TD3 (MATD3) [43]: It is extended based on single-agent TD3. The core objective is to address the training instability problem in multi-agent body environments due to environmental non-stationarity and strategy coordination difficulties.

4.2.2. Performance in Training

As shown in Figure 9, this study compares the normalized cumulative reward values obtained during the training process of multiple reinforcement learning-based control methods. The convergence of the PSO training process is not demonstrated because the results of the PSO training process are quite different from the DRL-based method. It should be particularly noted that the agent does not have an a priori knowledge reserve during the early stage of decision-making, which forces the system to accumulate operational experience through the mechanism of action space exploration. With the advancement of training rounds, the agent gradually masters the energy management optimization strategy of the interconnected MMG system, which achieves the continuous gain of the reward function. It is worth noting that all the training curves show a monotonous upward trend until complete convergence, which indicates the termination of the training process.

During the training process, the TD3 algorithm exhibits the lowest benchmark value, and the results oscillate significantly. This demonstrates that when faced with the multi-dimensional co-optimization of MMG systems, its lack of a co-optimization mechanism results in an undesirable management strategy. MATD3 further improves the information interaction capability between MGs by introducing the multi-agent body framework, which leads to a more optimal scheduling strategy. The proposed approach further enhances the communication interaction between MGs by incorporating an information mechanism in the actor network to obtain the best performance.

4.2.3. Performance in Testing

Table 5. The average cost and carbon emissions of various methods.

Approach	Average Cost (USD/Day)	Carbon Emissions (tCO2/Day)
PSO	3244.55	10.95
TD3	2130.45	9.46
MATD3	1641.67	8.55
Proposed approach	1246.33	7.89

demonstrates the average cost and carbon emissions of the various methods over the 30 days of the test set. Compared with PSO, TD3, and MATD3, the operating cost of the proposed approach in the MMG system is reduced by 61.59%, 41.48%, and 24.08%, respectively, and the carbon emission of the proposed approach in the MMG system is reduced by 27.95%, 16.6%, and 7.72%, respectively. Due to the impossibility of PSO to accommodate all the constraints and the lack of synergy among MGs in the face of multi-objective optimization, its optimization effect shows the difficulty of traditional methods in obtaining excellent results in the face of complex power systems. TD3 further enhances the information interaction capability through the introduction of an agent, which is still insufficient to address the scheduling requirements of the MMG system. MATD3 further validates the necessity of multi-agents in MMG systems by introducing multi-agents. Compared to MATD3, the proposed approach demonstrates the optimal scheduling capability of MMG systems by enhancing the communication capability between agents.

4.3. Performance on the Test Set over a Day

To further validate the effectiveness and generalization ability of the proposed approach, Figure 10 shows the scheduling strategy for a typical day. Figure 10a shows the real tariff for a typical day [44]. From Figure 8, it can be seen that MG1 has the lowest RES share and the lowest capability against voltage control. Therefore, the ESS1 exhibits a different scheduling strategy from the other two MGs, as shown in Figure 10b. MG1 exhibits characteristic voltage sag during the diurnal load ramping phase, necessitating its ESS to provide dynamic reactive power compensation. This countermeasure effectively mitigates voltage deviation propagation through controlled active power injection. In contrast, MG3′s configuration with a high renewable penetration ratio enables inherent voltage stabilization via RES inertia emulation. Such an operational paradigm permits ESS reallocation for energy arbitrage optimization, implementing time-of-use tariff-guided charging during off-peak periods and scheduled discharging at peak pricing intervals. The ESS2 is also able to profit by regulating the output under certain circumstances, but its behavior is more conservative compared to that of MG. As shown in Figure 10c, the SOC of the ESS3 at 18 h is smaller than that of the ESS2 at 24 h. Since the RES capacity of MG2 is smaller than that of MG3, the ESS needs more energy to stabilize the voltage in order to ensure the safety and stability of the MG2 system.

The operation of DGs is shown in Figure 10d. The DG of MG1 hardly operates in the first six hours, and as the electricity load increases, the DG begins to operate nearly at full capacity. The first six hours almost maintain the stable operation of the MG1 system due to the small electrical load and energy replenishment from the ESS. As the load increases, it is difficult for MG1 to maintain the stability of the system, and DG is required to access the system to maintain the voltage. MG2 and MG3 do not require DG access as a way to reduce the operating cost after the strategy by scheduling optimization due to sufficient RES.

5. Conclusions

Aiming at the global low-carbon economy optimization problem of MMG systems, this research constructs a distributed coordination mechanism based on the CMMA-TD3 algorithm. Distinguishing from the traditional centralized scheduling’s reliance on global information sharing, this framework guides heterogeneous microgrid subjects to form a Pareto-optimal energy interaction pattern under the protection of data sovereignty by designing a marginal utility-driven incentive compatibility strategy. Specifically, the multi-energy co-optimization problem is modeled as a stochastic game process with incomplete information, in which each agent implements a distributed consensus protocol based on locally observed state-space mapping.

The analysis results in MMG show that (1) compared with the benchmark algorithms, PSO, TD3, and MATD3, the proposed method reduces the operation cost by 61.59%, 41.48%, and 24.08%, respectively, and reduces carbon emissions by 27.95%, 16.6%, and 7.72%, respectively. (2) Under the validation in the test set of 30 days, the proposed dispatch optimization method can develop optimal training results for the MMG system. (3) Under test set scenarios, the proposed method is more stable and achieves the best training results. The proposed scheduling optimization method is able to formulate the optimal control strategy for the MMG system. (4) The operation of the equipment within each microgrid further demonstrates that the optimization of the MMG internal coordination mechanism can effectively reduce the overall operation cost, which verifies the effectiveness of the proposed method.