A Cooperative Game-Based Low-Carbon Optimal Operation Strategy for Multi-Microgrids Based on Multi-Agent Deep Reinforcement Learning

Abstract

Distributed integrated energy microgrids support the low-carbon transition of regional energy systems. However, multi-agent trading among microgrids still faces insufficient cross-market coordination, weak low-carbon incentives, and difficulties in fair benefit allocation. To address these issues, this paper proposes a cooperative-game-based low-carbon optimal operation strategy for multi-microgrids using multi-agent deep reinforcement learning. First, an energy-carbon-green certificate peer-to-peer coordinated trading mechanism and a green-carbon offsetting-based dual-incentive model are developed to link energy exchange, carbon quota adjustment, and green certificate circulation. Second, a Nash bargaining-based cooperative game model is formulated for multi-commodity P2P trading to maximize coalition benefits and ensure a fair allocation of surplus. Finally, the cooperative game is transformed into a Markov decision process, and a centralized training and decentralized execution framework with homogeneous agents is constructed based on the multi-agent soft actor-critic algorithm. Case studies using data from the Yangtze River Delta region of China show that the proposed method achieves a 1.25% optimality gap compared with the centralized MILP benchmark and reduces the coalition operating cost by 8.19% relative to independent operation. The carbon trading costs of the three microgrids are reduced by 37.27%, 40.13%, and 33.82%, respectively, verifying the economic applicability of the proposed method.

1. Introduction

Driven by national carbon peaking and carbon neutrality targets, energy systems are shifting toward clean, low-carbon, secure, and efficient operation. Microgrids serve as key carriers for integrating distributed renewable energy and coordinating source, grid, load, and storage resources. They therefore form an important part of new power systems. Conventional microgrids usually have simple structures and limited energy supply forms. Thus, they cannot effectively cope with the high randomness of wind and photovoltaic power generation or the variable load demand. To address these issues, distributed integrated energy microgrids (DIEMs) integrate multiple energy conversion technologies and carbon-resource response capabilities. DIEMs support the coordinated coupling, flexible conversion, and cascade utilization of different energy forms. Therefore, DIEMs provide a key technical approach for improving the flexibility, economic performance, and low-carbon performance of energy systems [1].

The operation mode of microgrids shifts from independent operation toward multi-agent interconnection and mutual support. By exploiting the complementary potential of multi-microgrids through cooperation and peer-to-peer interactions, this operation mode provides an important approach to efficient regional energy allocation and low-carbon operation [2,3]. In this context, multi-microgrids engage in direct energy exchange through peer-to-peer (P2P) trading. Trading quantities and prices are negotiated independently based on the supply–demand surplus or deficit for each microgrid. In this way, surplus renewable energy is locally consumed, and complementary load support among microgrids is achieved [4,5,6]. However, most existing studies focus primarily on energy interactions. The integrated design of green certificate consumption obligations and surplus–deficit adjustment of carbon emission quotas remains insufficient. This limitation limits the potential for low-carbon, cost-effective operation of multi-microgrid systems [7,8]. Meanwhile, P2P trading among multi-microgrids features differentiated interests, autonomous decision-making, and interactive game relationships. Each participant pursues individual benefit maximization, while the overall optimum is achieved through coalition cooperation. Therefore, cooperative game theory serves as a core theoretical tool for describing multi-agent collaborative decision-making, maximizing coalition benefits, and achieving fair benefit allocation [9,10]. In Ref. [11], cooperative game theory is combined with risk-averse stochastic programming to improve the collaborative benefits of multi-microgrids under carbon-reduction constraints. In Refs. [12,13,14,15], multi-microgrid trading optimization models are constructed based on Nash bargaining theory to achieve the fair allocation of multi-agent cooperative surplus. However, existing studies still have three limitations. First, most P2P trading studies focus primarily on electricity or multi-energy exchange, while the coordinated interactions among energy trading, carbon quota adjustment, and green certificate circulation have not been sufficiently modeled. Second, although Nash bargaining has been used for cooperative surplus allocation, its integration with energy-carbon-green certificate coordinated trading remains limited. Third, existing DRL-based methods primarily focus on energy scheduling performance, while their integration with cooperative-game payoff allocation and multi-commodity P2P settlement remains insufficient.

In terms of model solving, traditional methods mainly include optimization methods based on Karush–Kuhn–Tucker (KKT) conditions [16] and heuristic algorithms [17]. Although these methods are theoretically sound, they have several limitations in cooperative game problems with nonconvexity, nonlinearity, and strong uncertainty. These limitations include complex modeling, low solution efficiency, and insufficient robustness. These limitations make them difficult to adapt to dynamic multi-agent interactions and real-time decision-making requirements. Multi-agent deep reinforcement learning provides an efficient solution for complex cooperative game problems because it does not require an accurate model. It also adapts to environmental dynamics and supports high-dimensional continuous decision-making [18,19]. In Refs. [20,21], improved deep deterministic policy gradient (DDPG) algorithms show superior performance in power balance and energy scheduling. In Ref. [22], a multi-agent deep reinforcement learning algorithm is adopted to improve the energy utilization efficiency of multi-energy microgrids. A novel model-free, data-driven algorithm based on centralized training and decentralized execution is also developed. However, Ref. [22] mainly focuses on MASAC-based energy management and does not couple the learning framework with Nash bargaining-based surplus allocation or carbon-green certificate P2P settlement. Unlike Ref. [23], which emphasizes green-carbon offsetting in a Stackelberg game, this study integrates green-carbon offsetting into a cooperative multi-microgrid P2P trading framework and solves it using CTDE-MASAC. Notably, multi-microgrid cooperative games often exhibit nonconvexity and non-stationarity. Recent studies have applied MASAC and its improved variants to multi-microgrid scheduling and control problems, showing their applicability to multi-agent coordination under renewable energy uncertainty and dynamic operating conditions [24,25]. Compared with deterministic policy methods such as MADDPG, MASAC introduces a maximum-entropy objective and stochastic policy exploration, making it suitable for continuous action scheduling problems with multiple coupled agents [25]. Compared with on-policy methods such as MAPPO, its off-policy learning mechanism can reuse historical experiences and improve sample efficiency. Therefore, MASAC is adopted in this study as the solution framework, and its convergence and robustness are further evaluated through comparative experiments in Section 5.2. To further clarify the research gaps,

Table 1. Comparison of related studies on multi-microgrid trading and low-carbon optimization.

Reference	Market Type	Game Model	Algorithm	Uncertainty Handling	Carbon/Green Certificate Mechanism
[11]	P2P energy trading with carbon constraints	Asymmetric Nash bargaining	Stochastic programming	Scenario-based uncertainty	Carbon constraints only
[12,13,14,15]	Multi-microgrid energy trading	Nash bargaining	Mathematical optimization	Limited consideration	Limited carbon coupling; no GCT
[20,21]	Microgrid energy scheduling	Non-cooperative scheduling	DDPG/automated RL	Data-driven forecasting	Not considered
[22]	Networked multi-energy microgrid management	Multi-agent coordination	MASAC	Source-load uncertainty learning	Low-carbon operation; no GCT
[23]	Integrated energy system with green-carbon interaction	Stackelberg game	Mathematical optimization	Deterministic scenario	Green-carbon offsetting
This study	Energy-carbon-GCT P2P coordinated trading	Nash bargaining cooperative game	CTDE-MASAC	WT/PV forecast bounds	CET-GCT coupling with P2P trading

compares representative studies from five aspects: market type and trading mechanism, game model, solution method, uncertainty handling, and carbon/GCT mechanism. The comparison shows that existing studies typically address P2P energy trading, cooperative benefit allocation, and DRL-based scheduling separately, while the integration of these approaches with energy-carbon-green certificate coordination remains insufficient.

To address these gaps, this paper proposes a cooperative-game-based low-carbon optimal operation strategy for multi-microgrids that considers coordinated trading of energy-carbon-green certificates. The main contributions are summarized as follows:

(1) An energy-carbon-green certificate P2P coordinated trading mechanism is developed for multi-microgrids. By coupling energy exchange, carbon quota adjustment, and green certificate circulation, the proposed mechanism establishes a value transmission pathway among the energy, CET, and GCT markets. Furthermore, a dual-incentive pricing model based on green-carbon offsetting is introduced to convert surplus green certificates into carbon-offset value, thereby strengthening the incentives for renewable energy consumption and carbon emission reduction.

(2) A Nash bargaining-based cooperative game model is formulated for multi-microgrid P2P trading. The model considers multi-commodity transactions involving electricity, heat, hydrogen, carbon quotas, and green certificates. It maximizes coalition benefits while allocating cooperative surplus fairly among DIEMs, thereby improving both system-level efficiency and individual rationality.

(3) A CTDE-MASAC solution framework with homogeneous agents is designed for the proposed cooperative game. The optimization problem is formulated as a Markov decision process, and decentralized actors with a centralized critic are used to enable adaptive decision-making in continuous action spaces. This framework improves the applicability of multi-agent deep reinforcement learning to the coordinated, low-carbon operation of multiple microgrids.

This paper is organized as follows: Section 2 develops the energy-carbon-green certificate P2P coordinated trading mechanism and the green-carbon offsetting-based dual-incentive model. Section 3 formulates the cooperative-game-based optimal operation model and the Nash bargaining allocation mechanism. Section 4 presents the CTDE-MASAC solution framework. Section 5 reports the case studies and comparative results. Section 6 concludes the paper and discusses future research directions.

2. Bidirectional Interaction Mechanism for Multi-Microgrids Considering P2P Energy-Carbon-Green Certificate Coordinated Market Trading

2.1. Architectural Design of the Distributed Integrated Energy Microgrid

The overall architecture of the distributed integrated energy microgrid (DIEM) is shown in Figure 1. The system comprises an energy supply side, an energy conversion and coupling side, an energy demand side, and a green certificate-carbon market trading side. Compared with conventional microgrids, DIEM offers three key technical advantages: (1) enhanced on-site renewable energy consumption through electricity-heat-hydrogen multi-energy coupling; (2) a closed-loop carbon pathway enabled by integrated carbon capture, utilization, and storage (CCUS) and power-to-gas (P2G) technologies; and (3) coordinated participation in the electricity market, carbon trading market, and green certificate trading market, achieving joint optimization of economic performance and low-carbon operation.

Specifically, the DIEM architecture is described as follows. (1) Energy supply side. Wind power, photovoltaics (PV), the main grid, and natural gas serve as the core supply inputs, ensuring the microgrid’s basic energy supply while enhancing system resilience through a diversified energy portfolio. (2) Energy conversion and coupling side. As the core hub of the system, this side integrates combined heat and power (CHP) units, gas boilers (GB), power-to-gas (P2G) equipment, carbon capture, utilization, and storage (CCUS) devices, hydrogen fuel cells (HFC), and energy storage systems (ESS), enabling deep coupling and cascaded utilization of natural gas, electricity, heat, hydrogen, and carbon flows. Among these, the operational characteristics of key devices are as follows. The CHP unit consumes natural gas to simultaneously produce electricity and heat, with its power output $P_t^{\mathrm{CHP}}$ and heat output $H_t^{\mathrm{CHP}}$ satisfying the heat-to-power ratio constraint $H_t^{\mathrm{CHP}}={\eta }_{\mathrm{CHP}}^{\mathrm{h}/\mathrm{e}}{P}_t^{\mathrm{CHP}}$, where ${\eta }_{\mathrm{CHP}}^{\mathrm{h}/\mathrm{e}}$ is the heat-to-power ratio coefficient. The GB converts the chemical energy of natural gas into heat with a thermal efficiency of approximately 85–95%. The P2G device produces hydrogen via water electrolysis, and its hydrogen output $P_t^{{\mathrm{P}2\mathrm{G},\mathrm{H}}_2}$ and electrical input $P_t^{\mathrm{P}2\mathrm{G}}$ satisfy the linear relationship $P_t^{{\mathrm{P}2\mathrm{G},\mathrm{H}}_2}={\eta }_{\mathrm{P}2\mathrm{G}}{P}_t^{\mathrm{P}2\mathrm{G}}$, where ${\eta }_{\mathrm{P}2\mathrm{G}}$ is the power-to-hydrogen efficiency. The CCUS unit captures CO₂ emitted from the system, with its captured amount $E_t^{\mathrm{CCUS}}$ being a quadratic function of its energy consumption $E_t^{\mathrm{CCUS}}=\sigma (a_{\mathrm{CCUS}}{(P_t^{\mathrm{CCUS}})}^2+b_{\mathrm{CCUS}}{P}_t^{\mathrm{CCUS}})$. The HFC converts hydrogen into electricity and heat, with electrical and thermal efficiencies of approximately 50–60% and 40–45%, respectively. The ESS enables multi-period shifting of electricity, heat, and hydrogen, with its state of charge $SO{C}_t$ governed by $SO{C}_{t+1}=SO{C}_t+{\eta }^{\mathrm{cha}}{P}_t^{\mathrm{cha}}-P_t^{\mathrm{dis}}/{\eta }^{\mathrm{dis}}$, subject to charging/discharging power limits and capacity constraints. The mathematical operation models of these components are detailed in Ref. [23]. (3) Energy demand side. Energy is precisely supplied to electrical, thermal, and hydrogen loads through conversion and storage devices, thereby meeting diverse user demands and achieving efficient supply–demand matching. (4) “Green certificate-carbon market” trading side. Through bidirectional interfaces with the carbon trading market and the green certificate trading market, carbon emission costs and the environmental benefits of renewable energy are explicitly quantified, providing market-based support for low-carbon operation. In summary, the DIEM constructs an integrated generation-conversion-storage-trading operating framework through physical-layer multi-energy coupling and market-layer value linkage, supporting the low-carbon and efficient operation of microgrids in new power systems.

2.2. Design of the Bidirectional Interaction Mechanism for Multi-Microgrids

Building on the low-carbon operational architecture of the DIEM described above, a peer-to-peer (P2P) coordinated trading mechanism covering energy, carbon quotas, and green certificates is developed for multi-microgrid systems. With microgrid autonomy and mutual benefit as the core principles, each microgrid acts as an independent trading entity and conducts direct peer-to-peer transactions at the execution level. The distribution network operator, shown as a dashed box in Figure 2, does not participate in P2P price negotiation or in transaction volume determination. Its role is limited to market supervision, transaction compliance review, and distribution-network security verification. Therefore, the proposed mechanism preserves decentralized P2P trading among DIEMs while ensuring operational security and transaction compliance.

As shown in Figure 2, multi-microgrids form a three-dimensional coordinated trading system through the energy, carbon, and green certificate markets. Through autonomous negotiation and peer trading, energy consumption, carbon responsibility sharing, and green certificate circulation are accomplished simultaneously. This mechanism enhances on-site renewable energy utilization and operational flexibility while reducing microgrid electricity costs and grid dependence, thereby providing a direct trading scenario and market foundation for cooperative games among multi-microgrids. The coupling of energy, carbon, green certificate, and transaction flows forms a decentralized, efficient, and autonomous coordinated trading mode, further improving the economy, low-carbon performance, and reliability of the multi-microgrid system.

2.3. Dual-Incentive Mechanism Model for Green Certificate-Carbon Trading Based on Green-Carbon Offsetting

To incentivize distributed integrated energy microgrids to increase their renewable energy output share and reduce carbon emissions, a green certificate-carbon trading dual-incentive mechanism based on “green-carbon” offsetting is developed, as illustrated in Figure 3. The core of this mechanism is to exploit the low-carbon attribute value of green certificates and establish a bidirectional linkage between the green certificate trading (GCT) market and the carbon emission trading (CET) market.

A green certificate is an official certification of renewable energy generation fed into the grid. Each green certificate corresponds to an equivalent amount of carbon emission reduction from renewable energy generation, which can be quantified through carbon footprint tracking and carbon emission flow methods, and represents the low-carbon benefit relative to conventional coal-fired power [26]. Based on this quantified value, a bidirectional interaction channel is established between the green certificate market and the carbon market, with green certificate acquisition and carbon quota trading serving as the exchange media. Under this mechanism, a DIEM can convert surplus green certificates into carbon quotas to offset part of its own carbon emissions. These carbon quotas obtained through green certificate conversion are defined as green carbon. Through the “green-carbon” offsetting mechanism, a DIEM can simultaneously enhance renewable energy consumption and obtain additional carbon quota revenue, forming a positive incentive loop between green electricity production and carbon emission compliance.

2.3.1. Modeling of the Dynamic Reward-and-Penalty-Based Tiered Carbon Trading Mechanism

China’s carbon market is currently undergoing deepening development, and pilot projects have provided a foundation for distributed entities to participate in carbon trading. To clarify the low-carbon operational space for microgrids under regional carbon-emission constraints [27] and to promote deep coupling between carbon costs and operational strategies, a carbon-market participation model for DIEMs is established based on the consumption-side carbon-emission accounting principle, following the “polluter pays” rule. The carbon emission responsibility boundaries are defined as follows: (1) as end-use energy prosumers, DIEMs must bear direct and indirect carbon emission responsibilities throughout their operational lifecycle; (2) indirect carbon emissions from purchased electricity are uniformly quantified using the regional grid benchmark emission factors published by the national authority; (3) devices such as ESS and HFC, which only participate in energy conversion and storage processes without directly consuming fossil fuels, are excluded from the DIEM’s carbon emission accounting boundary. It should be clarified that the upstream carbon emissions associated with the electricity stored in the ESS are already accounted for in the “purchased electricity” term $E_{\mathrm{e}}^{\mathrm{buy}}$ in Equation (2); similarly, the carbon footprint of hydrogen consumed by the HFC is already accounted for at the production stage (e.g., P2G hydrogen production or purchased hydrogen). Double-counting these emissions at the DIEM consumption side would overestimate actual carbon emissions and distort the incentive signals of the carbon trading mechanism. Therefore, in line with the “no double-counting” principle of carbon emissions accounting, this study excludes ESS and HFC from the terminal carbon emissions boundary.

To strengthen the incentive and constraint functions of the carbon market, a dynamic reward-and-penalty-based tiered carbon trading mechanism is introduced. This mechanism comprises three components: initial carbon quota allocation, actual carbon emission calculation, and tiered carbon trading cost accounting [28]. The main symbols and parameters used in the mathematical models are summarized in Appendix A,

Table A1. Description of mathematical symbols and parameters.

Symbols	Meaning
$E_0,E$	Total carbon quota and actual carbon emission of a DIEM, respectively
$E_t,E_t^{\mathrm{in}},E_{\mathrm{green}}$	Carbon trading volume, carbon trading volume after green-carbon offsetting, and green-carbon offset volume, respectively
$C_{\mathrm{C}},C_{\mathrm{C}}^{in}$	Carbon trading cost and carbon trading cost after green-carbon offsetting, respectively
${\chi }_{\mathrm{e}},{\chi }_{\mathrm{h}}$	Carbon quota coefficients for electricity generated by coal-fired units and natural gas consumed by gas-fired units, respectively
$l,\alpha ,\epsilon ,\nu$	Interval length, price coefficient, price compensation coefficient, and price growth coefficient in the tiered carbon trading model, respectively
$G_{\mathrm{r}},G_{\mathrm{e}},G_{\mathrm{GCT}}$	Green certificates obtained from renewable energy generation, green certificate quota requirement, and green certificate trading volume, respectively
$G_{\mathrm{green}},G_{\mathrm{CET}}$	Green certificate volume used for green-carbon offsetting and tiered green certificate quota reward-penalty volume, respectively
${\lambda }_t^{\mathrm{GCT}},{\lambda }_0^{\mathrm{GCT}}$	Green certificate price at time slot $t$ and benchmark green certificate price, respectively
$k_1,k_2,\zeta$	Inverse price function parameters and price ratio coefficient in the GCT market
$\theta ,{\eta }_{\mathrm{green}},{\omega }_{\mathrm{green}}$	Renewable energy emission reduction efficiency, renewable energy output share, and ratio of actual carbon emissions to carbon quota, respectively
$\kappa$	Green-carbon correction coefficient
$\gamma$	Discount factor in the reinforcement learning model
$\tau$	Target-network soft-update coefficient in the MASAC algorithm
$\rho ,\xi$	Weighting coefficients in the green-carbon offsetting model
${\omega }_1,{\omega }_2,{\omega }_3$	Green certificate quota reward-penalty coefficients
$P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}},P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}},P_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}$	P2P electricity, heat, and hydrogen trading volumes between DIEM $i$ and DIEM $j$, respectively
$E_t^{\mathrm{P}2\mathrm{P},i},G_t^{\mathrm{P}2\mathrm{P},i}$	Total P2P carbon quota and green certificate trading volumes of DIEM $i$, respectively
$\begin{array}{l}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}\\ {\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{C}},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{G}}\end{array}$	P2P trading prices of electricity, heat, hydrogen, carbon quotas, and green certificates between DIEM $i$ and DIEM $j$, respectively
${\lambda }_t^{i,\mathrm{e}}$	Marginal electricity value of DIEM $i$ at time slot $t$ extracted from the critic network
$C_i^{\mathrm{DIEM}},C_{\mathrm{total}}^{\mathrm{DIEM}}$	Total operating cost of DIEM $i$ and total operating cost of the multi-microgrid coalition, respectively
$\begin{array}{l}{C}_{\mathrm{utility},i},C_{\mathrm{units},i},\\ C_{\mathrm{cut},i},C_{\mathrm{ESS},i}\end{array}$	External interaction cost, unit operation and maintenance cost, wind and solar curtailment cost, and energy storage operation and maintenance cost of DIEM $i$, respectively
$C_{C,i},C_{\mathrm{GCT},i},C_{\mathrm{total},i}^{\mathrm{P}2\mathrm{P}}$	Carbon trading cost, green certificate trading cost, and comprehensive P2P trading cost of DIEM $i$, respectively
$C_i^{\mathrm{P}2\mathrm{P},\mathrm{energy}},C_i^{\mathrm{P}2\mathrm{P},\mathrm{C}},C_i^{\mathrm{P}2\mathrm{P},\mathrm{G}}$	P2P energy trading cost, P2P carbon quota trading cost, and P2P green certificate trading cost of DIEM $i$ respectively
${\lambda }_t^{{\mathrm{CH}}_4},N_t^{{\mathrm{CH}}_4,i}$	Natural gas purchase price and natural gas purchase volume of DIEM $i$ at time slot $t$, respectively
${\lambda }_t^{\mathrm{Gb}},{\lambda }_t^{\mathrm{Gs}}$	Time-of-use electricity price and feed-in tariff, respectively
$P_t^{\mathrm{sell},i},P_t^{\mathrm{P}2\mathrm{G},i},P_t^{{\mathrm{HFC},\mathrm{H}}_2,i}$	Electricity sold to the main grid, P2G electrical input power, and hydrogen consumed by HFC in DIEM $i$, respectively
${\alpha }_1\sim {\alpha }_6,{\beta }_1\sim {\beta }_2$	Operating cost coefficients of energy conversion devices
${\lambda }_{\mathrm{cur}},P_t^{\mathrm{WT},\mathrm{cur},i},P_t^{\mathrm{PV},\mathrm{cur},i}$	Wind and solar curtailment cost coefficient, wind curtailment volume, and PV curtailment volume of DIEM $i$, respectively
$\varsigma$	Degradation cost per unit of charged or discharged energy
$\begin{array}{l}{P}_t^{\mathrm{ESS},\mathrm{e},\mathrm{cha},i},P_t^{\mathrm{ESS},\mathrm{h},\mathrm{cha},i},\\ P_t^{{\mathrm{ESS},\mathrm{H}}_2,\mathrm{cha},i}\end{array}$	Charging powers of electrical, heat, and hydrogen energy storage systems in DIEM $i$
$\begin{array}{l}{P}_t^{\mathrm{ESS},\mathrm{e},\mathrm{dis},i},P_t^{\mathrm{ESS},\mathrm{h},\mathrm{dis},i},\\ P_t^{{\mathrm{ESS},\mathrm{H}}_2,\mathrm{dis},i}\end{array}$	Discharging powers of electrical, heat, and hydrogen energy storage systems in DIEM $i$
${\widehat{G}}_i^{\mathrm{GCT}}$	Total green certificate trading volume of DIEM $i$
${\widehat{P}}_t^{\mathrm{PV},i},{\widehat{P}}_t^{\mathrm{WT},i},{\delta }^{\mathrm{PV}},{\delta }^{\mathrm{WT}}$	Forecast PV and wind outputs of DIEM $i$ and their maximum allowable fluctuation coefficients, respectively
$P_t^{\mathrm{EL},i},L_t^{\mathrm{h},i},L_t^{{\mathrm{H}}_2,i}$	Electrical input power of the electrolyzer, heat load, and hydrogen load of DIEM $i$, respectively
$H_t^{\mathrm{HFC},\mathrm{h},i},P_t^{{\mathrm{P}2\mathrm{G},\mathrm{H}}_2,i},P_t^{{\mathrm{MR},\mathrm{H}}_2,i}$	Heat output of HFC, hydrogen output of P2G, and hydrogen output of the methane reactor in DIEM $i$, respectively
$\begin{array}{l}{N}_{\max }^{{\mathrm{CH}}_4,i},N_{\min }^{{\mathrm{CH}}_4,i},N_t^{{\mathrm{MR},\mathrm{CH}}_4,i},\\ N_t^{\mathrm{CHP},i},N_t^{\mathrm{GB},i}\end{array}$	Upper and lower limits of natural gas purchase, natural gas generated by MR, and natural gas consumption of CHP and GB in DIEM $i$, respectively
$\begin{array}{l}{E}_t^{\mathrm{CHP},\mathrm{C},i},E_t^{\mathrm{GB},\mathrm{C},i},E_t^{\mathrm{CCUS},\mathrm{C},i},\\ E_t^{\mathrm{CCUS},\mathrm{s},i},E_t^{\mathrm{P}2\mathrm{G},\mathrm{C},i},E_t^{\mathrm{out},i}\end{array}$	Carbon emissions from CHP and GB, CO2 captured by CCUS, CO2 stored by CCUS, CO2 used by P2G, and carbon emissions released into the atmosphere in DIEM $i$, respectively
$u_i,d_i,w_i$	Payoff, disagreement point, and bargaining power weight of DIEM $i$, respectively
$u_i^{*}$	Optimal operational payoff of DIEM $i$ obtained in Subproblem 1
$\begin{array}{l}{P}_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e},*},P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h},*},\\ P_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2,*},E_t^{\mathrm{P}2\mathrm{P},ij,*},\\ G_t^{\mathrm{P}2\mathrm{P},ij,*}\end{array}$	Optimal P2P electricity, heat, hydrogen, carbon quota, and green certificate trading quantities between DIEM $i$ and DIEM $j$, respectively
$S,A,P,R$	State space, action space, transition function, and reward function in the Markov decision process
$s_t^i,a_t^i$	Local state and action of agent $i$ at time slot $t$, respectively
$R_t^i,r_t$	Immediate reward of agent $i$ and global reward of all agents at time slot $t$, respectively
${\pi }_i$	Policy of agent $i$
$Q(s_t,a_t),V(s_t)$	Soft action-value function and soft state-value function, respectively
${\theta }_j,{\varphi }_i$	Parameter sets of the $j$-th critic network and the actor network of agent $i,$ respectively
$\mathcal{D}$	Experience replay buffer
$\mathcal{H}(\cdot )$	Policy entropy
$M$	Training hyperparameter listed in Table 3

.

(1) Initial carbon quota model

To incentivize energy conservation and emission reduction, carbon quotas are allocated to DIEMs at no cost. Based on the carbon emission accounting boundary, DIEM carbon emissions mainly originate from two sources: (i) indirect carbon emissions from purchased electricity, which are accounted for on the consumption side; and (ii) direct carbon emissions from natural gas consumed by CHP and GB units. The initial carbon quota model is formulated as Equation (1):

$$\left\{\begin{array}{l}{E}_0=E_{\mathrm{e},0}^{\mathrm{buy}}+E_0^{\mathrm{CHP}}+E_0^{\mathrm{GB}}\hfill \\ E_{\mathrm{e},0}^{\mathrm{buy}}={\chi }_{\mathrm{e}}{\displaystyle \sum _{t=1}^T{P}_t^{\mathrm{buy}}}\hfill \\ E_0^{\mathrm{CHP}}={\chi }_{\mathrm{h}}{\displaystyle \sum _{t=1}^T(P_t^{\mathrm{CHP}}+H_t^{\mathrm{CHP}})}\hfill \\ E_0^{\mathrm{GB}}={\chi }_{\mathrm{h}}{\displaystyle \sum _{t=1}^T{H}_t^{\mathrm{GB}}}\hfill \end{array}\right.$$

(1)

where $E_{\mathrm{e},0}^{\mathrm{buy}}$ is the carbon quota for purchased electricity; $E_0^{\mathrm{CHP}}$ and $E_0^{\mathrm{GB}}$ are the carbon quotas for CHP and GB, respectively; $P_t^{\mathrm{buy}}$ is the electricity purchased from the main grid at time slot $t$; $P_t^{\mathrm{CHP}}$ is the CHP output power; $H_t^{\mathrm{CHP}}$ and $H_t^{\mathrm{GB}}$ are the heat outputs of CHP and GB at time $t$; and $T$ denotes the 24 h daily optimization horizon.

(2) Actual carbon emission model

Given that electricity purchased from the main grid is primarily supplied by coal-fired units, and that the system is equipped with CCUS devices capable of capturing CO₂ and converting it into methane via P2G, the actual carbon emission model is formulated as Equation (2) based on the aforementioned accounting boundary:

$$\left\{\begin{array}{l}\begin{array}{l}E=E_{\mathrm{e}}^{\mathrm{buy}}+E_{\mathrm{total}}^{\mathrm{CHP},\mathrm{GB}}-E^{\mathrm{CCUS}}\hfill \\ E_{\mathrm{e}}^{\mathrm{buy}}={\displaystyle \sum _{t=1}^T\left(a_1+b_1{P}_t^{\mathrm{buy}}+c_1{\left(P_t^{\mathrm{buy}}\right)}^2\right)}\hfill \\ E_{\mathrm{total}}^{\mathrm{CHP},\mathrm{GB}}={\displaystyle \sum _{t=1}^T\left(\begin{array}{l}{a}_2+b_2\left(P_t^{\mathrm{CHP}}+H_t^{\mathrm{CHP}}+H_t^{\mathrm{GB}}\right)\\ +c_2{\left(P_t^{\mathrm{CHP}}+H_t^{\mathrm{CHP}}+H_t^{\mathrm{GB}}\right)}^2\end{array}\right)}\hfill \end{array}\\ E^{\mathrm{CCUS}}={\displaystyle \sum _{t=1}^T\varpi }{P}_t^{\mathrm{CCUS}}\end{array}\right.$$

(2)

where $E_{\mathrm{e}}^{\mathrm{buy}}$ is the indirect carbon emission from purchased electricity; $E_{\mathrm{total}}^{\mathrm{CHP},\mathrm{GB}}$ is the direct carbon emission from CHP and GB; $E^{\mathrm{CCUS}}$ is the CO₂ captured by CCUS; $\varpi$ is the capture parameter; $P_t^{\mathrm{CCUS}}$ is the total power consumption of CCUS at time $t$; and $a_1$, $b_1$, $c_1$ and $a_2$, $b_2$, $c_2$ are the carbon emission calculation parameters for coal-fired and gas-fired units, respectively.

(3) Dynamic reward-and-penalty-based tiered carbon trading model

The carbon trading volume for the DIEM is given by Equation (3):

$$E_t=E-E_0$$

(3)

The tiered carbon trading cost model for the DIEM is expressed as [29]:

$$C_{\mathrm{C}}=\left\{\begin{array}{l}-{\alpha }_0^{\mathrm{C}}(1+\epsilon )(E_t-l)-{\alpha }_0^{\mathrm{C}}l,-2l\le E_t\le -l\hfill \\ -{\alpha }_0^{\mathrm{C}}{E}_t,-l\le E_t\le 0\hfill \\ {\alpha }_0^{\mathrm{C}}{E}_t,0\le E_t\le l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+\nu )(E_t-l)+{\alpha }_0^{\mathrm{C}}l,l\le E_t\le 2l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+2\nu )(E_t-2l)+{\alpha }_0^{\mathrm{C}}(2+\nu )l,2l\le E_t\le 3l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+3\nu )(E_t-3l)+{\alpha }_0^{\mathrm{C}}(3+3\nu )l,3l\le E_t\le 4l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+4\nu )(E_t-4l)+{\alpha }_0^{\mathrm{C}}(4+6\nu )l,4l\le E_t\le 5l\hfill \end{array}\right.$$

(4)

When $E_t<0$, the DIEM’s actual carbon emissions are below its allocated allowance, and the surplus carbon quota can be sold in the market. As the surplus increases, the selling price increases progressively in a tiered manner (each interval of length, with a price compensation coefficient $\epsilon$), rewarding deep emission reductions. When $E_t>0$, the emissions exceed the allowance, and the DIEM needs to purchase additional quotas; as the purchase volume increases, the buying price increases progressively (with price growth coefficient $\nu$), penalizing high emissions. The first two piecewise segments in Equation (4) correspond to the “surplus selling” scenario ($-2l\le E_t\le 0$), while the last five segments correspond to the “deficit purchasing” scenario ($0\le E_t\le 5l$). This tiered mechanism, through its nonlinear price gradients, provides positive incentives for low-carbon behavior and strong disincentives for high-carbon behavior.

2.3.2. Modeling of the Green Certificate-Carbon Trading Mechanism Based on “Green-Carbon” Offsetting

(1) Green Certificate Trading Mechanism

To quantify the environmental value of renewable energy generation in the DIEM and establish green certificate circulation rules aligned with compliance requirements, a basic green certificate trading model is developed. The model directly links renewable energy output to green certificate volume and defines the green certificate quota obligations for the DIEM, from which the green certificate trading volume is derived. A quantity-based Cournot model is then introduced to capture the price formation mechanism in the GCT market and dynamically reflect the market value of green certificates through supply–demand interactions, as expressed in Equations (5)–(7):

$$G_{\mathrm{r}}={\displaystyle \frac{\alpha \left(P_t^{\mathrm{PV}}+P_t^{\mathrm{WT}}\right)}{1000}}$$

(5)

$$G_{\mathrm{e}}={\displaystyle \frac{\beta L_t^{\mathrm{e}}}{1000}}$$

(6)

$$G_{\mathrm{GCT}}=G_{\mathrm{r}}-G_{\mathrm{e}}$$

(7)

where $P_t^{\mathrm{PV}}$ and $P_t^{\mathrm{WT}}$ are the actual PV and wind power outputs at time slot $t$, respectively; $L_t^{\mathrm{e}}$ is the electrical load at time slot $t$. Since the scheduling interval is 1 h, dividing by 1000 converts kWh into MWh. One MWh of renewable electricity generation corresponds to one green certificate. $\alpha$ is the conversion coefficient from renewable electricity generation to green certificates, and $\beta$ is the quota coefficient for green certificate demand.

The GCT price based on the quantity-competition Cournot model is given by Equations (8)–(10):

$${\lambda }_t^{\mathrm{GCT}}=k_1-k_2{G}_{\mathrm{GCT}}$$

(8)

$$k_1={\lambda }_0^{\mathrm{GCT}}$$

(9)

$$k_2=\left(1-\zeta \right){\lambda }_0^{\mathrm{GCT}}/G_{\mathrm{e}}$$

(10)

(2) Green-Carbon Offsetting Mechanism

To fully leverage the low-carbon attributes of green certificates and realize value linkage between green certificates and carbon quotas, the green-carbon offset volume is quantified along three dimensions: renewable energy emission-reduction efficiency, renewable energy output share, and the ratio of actual carbon emissions to allocated carbon quotas, as given by Equations (11)–(13). A tiered green certificate quota reward-penalty function is also designed, linking the carbon trading volume after green-carbon offsetting to reward-and-penalty intensity through graduated coefficient adjustments to enhance emission reduction incentives.

$$E_{\mathrm{green}}=\kappa G_{\mathrm{green}}\left(\theta +\rho {\eta }_{\mathrm{green}}-\xi {\omega }_{\mathrm{green}}\right)$$

(11)

$${\eta }_{\mathrm{green}}={\displaystyle \frac{P_t^{\mathrm{PV}}+P_t^{\mathrm{WT}}}{P_t^{\mathrm{PV}}+P_t^{\mathrm{WT}}+P_t^{\mathrm{buy}}+P_t^{\mathrm{HFC},\mathrm{e}}+P_t^{\mathrm{CHP}}+P_t^{\mathrm{ESS},\mathrm{e},\mathrm{dis}}}}$$

(12)

$${\omega }_{\mathrm{green}}={\displaystyle \frac{E}{E_0}}$$

(13)

All three are normalized dimensionless indices (ranging from 0 to 1), quantifying the carbon reduction effect of renewable generation, the share of clean energy in the overall energy mix, and the contribution of carbon allowance surplus/deficit to the demand for green-carbon offsetting. The weighting coefficients $\rho$ and $\xi$ are determined by the analytic hierarchy process (AHP) combined with expert experience, with values set to $\rho =0.4$ and $\xi =0.3$; the fact that $\rho >\xi$ reflects the relatively dominant contribution of renewable energy emission reduction efficiency to the green-carbon offset volume. The correction coefficient $\kappa$ calibrates the value equivalence between the green certificate market and the carbon market; based on monthly trading data from the Yangtze River Delta region in 2023, it is fitted as $\kappa =1.05$. $P_t^{\mathrm{HFC},\mathrm{e}}$ is the electrical power generated by HFC at time $t$; and $P_t^{\mathrm{ESS},\mathrm{e},\mathrm{dis}}$ is the discharge power of the electrical energy storage system.

The tiered green certificate quota reward-penalty function is expressed as Equations (14) and (15):

$$E_t^{\mathrm{in}}=E-E_0-E_{\mathrm{green}}$$

(14)

$$G_{\mathrm{CET}}=\left\{\begin{array}{l}-\left({\omega }_1+{\omega }_2\right){\displaystyle \frac{l}{\theta }}+{\omega }_3{\displaystyle \frac{E_t^{\mathrm{in}}+2l}{\theta }},E_t^{\mathrm{in}}\le -2l\hfill \\ -{\omega }_1{\displaystyle \frac{l}{\theta }}+{\omega }_2{\displaystyle \frac{E_t^{\mathrm{in}}+l}{\theta }},-2l\le E_t^{\mathrm{in}}\le -l\hfill \\ {\omega }_1{\displaystyle \frac{E_t^{\mathrm{in}}}{\theta }},-l\le E_t^{\mathrm{in}}\le l\hfill \\ {\omega }_1{\displaystyle \frac{l}{\theta }}+{\omega }_2{\displaystyle \frac{E_t^{\mathrm{in}}-l}{\theta }},l\le E_t^{\mathrm{in}}\le 2l\hfill \\ \left({\omega }_1+{\omega }_2\right){\displaystyle \frac{l}{\theta }}+{\omega }_3{\displaystyle \frac{E_t^{\mathrm{in}}-2l}{\theta }},E_t^{\mathrm{in}}\ge 2l\hfill \end{array}\right.$$

(15)

where ${\omega }_1$, ${\omega }_2$, ${\omega }_3$ are the green certificate quota reward-penalty coefficients, and they satisfy ${\omega }_1<{\omega }_2<{\omega }_3$.

The tiered green certificate quota reward-penalty volume model divides the system’s carbon quota trading volume into multiple intervals. When the system needs to purchase carbon quotas in the carbon market, additional green certificate quota constraints are imposed; conversely, when the system holds surplus carbon quotas available for sale, corresponding green certificate quota reductions are granted. Through this graduated quota adjustment, compliance pressure and incentive signals from the CET market are transmitted to the GCT market, forming bidirectional constraints and coordinated incentives between the two markets.

(3) Participation in Carbon Market Trading

On the basis of Equation (4), substituting Equation (14) into the carbon trading volume yields the tiered carbon trading model as Equation (16):

$$C_{\mathrm{C}}=\left\{\begin{array}{l}-{\alpha }_0^{\mathrm{C}}(1+\epsilon )(E_t^{\mathrm{in}}-l)-{\alpha }_0^{\mathrm{C}}l,-2l\le E_t^{\mathrm{in}}\le -l\hfill \\ -{\alpha }_0^{\mathrm{C}}{E}_t^{\mathrm{in}},-l\le E_t^{\mathrm{in}}\le 0\hfill \\ {\alpha }_0^{\mathrm{C}}{E}_t^{\mathrm{in}},0\le E_t^{\mathrm{in}}\le l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+\nu )(E_t^{\mathrm{in}}-l)+{\alpha }_0^{\mathrm{C}}l,l\le E_t^{\mathrm{in}}\le 2l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+2\nu )(E_t^{\mathrm{in}}-2l)+{\alpha }_0^{\mathrm{C}}(2+\nu )l,2l\le E_t^{\mathrm{in}}\le 3l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+3\nu )(E_t^{\mathrm{in}}-3l)+{\alpha }_0^{\mathrm{C}}(3+3\nu )l,3l\le E_t^{\mathrm{in}}\le 4l\hfill \\ {\alpha }_0^{\mathrm{C}}(1+4\nu )(E_t^{\mathrm{in}}-4l)+{\alpha }_0^{\mathrm{C}}(4+6\nu )l,4l\le E_t^{\mathrm{in}}\le 5l\hfill \end{array}\right.$$

(16)

Under the green certificate-carbon trading bidirectional interaction framework, an increase in renewable energy generation directly drives growth in green certificate holdings and correspondingly raises the green-carbon offset volume in the tiered CET mechanism, thereby incentivizing renewable energy production. Meanwhile, when the carbon quota trading volume enters a higher interval, the CET market price rises in a tiered manner, and the green certificate quota reward-penalty intensity also increases accordingly. If the system needs to purchase carbon quotas to fulfill compliance obligations, it will face green certificate quota penalties; otherwise, it will receive green certificate quota reductions. Through the dual transmission pathways of carbon market price signals and green certificate quota adjustments, this mechanism strengthens the coordinated linkage between the GCT and CET markets and effectively increases trading activity in both markets.

3. Cooperative-Game-Based Optimal Operation Model for Multi-Microgrid P2P Trading Considering Nash Bargaining

3.1. P2P Trading Model for Energy, Carbon Quotas, and Green Certificates Among Multi-Microgrids

3.1.1. Energy P2P Trading

The total P2P electricity trading volume of the $i$-th DIEM at time slot $t$ is the sum of its P2P trading volumes with all other DIEMs, reflecting the net P2P electricity trading volume at that time slot, as expressed in Equation (17):

$$P_t^{\mathrm{P}2\mathrm{P},i,\mathrm{e}}={\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}}$$

(17)

where $P_t^{\mathrm{P}2\mathrm{P},i,\mathrm{e}}$ is the total P2P electricity trading volume of the $i$-th DIEM at time slot $t$; and ${\varphi }_i$ is the set of other DIEMs available for trading with $\mathrm{DIEM} i$. The trading direction is defined as follows: when $P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}>0$, $\mathrm{DIEM} i$ is the buyer and $\mathrm{DIEM}j$ is the seller; when $P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}<0$, $\mathrm{DIEM} i$ is the seller and $\mathrm{DIEM}j$ is the buyer.

The relationship between the sign of P2P trading volume and trading direction is defined as follows: for any $\mathrm{DIEM} i$, $P_t^{\mathrm{P}2\mathrm{P},i,\mathrm{e}}>0$ indicates that $\mathrm{DIEM} i$ is a net buyer (purchasing electricity), while $P_t^{\mathrm{P}2\mathrm{P},i,\mathrm{e}}<0$ indicates that $\mathrm{DIEM} i$ is a net seller (selling electricity). The sign conventions for carbon allowance trading $E_t^{\mathrm{P}2\mathrm{P},i}$ and green certificate trading $G_t^{\mathrm{P}2\mathrm{P},i}$ follow the same rule.

For the buyer and seller of the same P2P electricity transaction, a bilateral consistency constraint applies:

$$P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}+P_t^{\mathrm{P}2\mathrm{P},ji,\mathrm{e}}=0$$

(18)

For $\mathrm{DIEM} i$, the total P2P electricity trading cost is the sum of trading costs across all time slots, as in Equation (19):

$$C_{\mathrm{e}}^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{t\in T}{\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}}$$

(19)

where $C_{\mathrm{e}}^{\mathrm{P}2\mathrm{P},i}$ is the P2P electricity trading cost.

Similarly, the P2P trading costs for heat and hydrogen are obtained as Equations (20) and (21):

$$C_{\mathrm{h}}^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{t\in T}{\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}}}$$

(20)

$$C_{{\mathrm{H}}_2}^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{t\in T}{\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}}$$

(21)

where $C_{\mathrm{h}}^{\mathrm{P}2\mathrm{P},i}$ and $C_{{\mathrm{H}}_2}^{\mathrm{P}2\mathrm{P},i}$ are the P2P trading costs for heat and hydrogen energy, respectively.

$$C_{\mathrm{energy}}^{\mathrm{P}2\mathrm{P},i}=C_{\mathrm{e}}^{\mathrm{P}2\mathrm{P},i}+C_{\mathrm{h}}^{\mathrm{P}2\mathrm{P},i}+C_{{\mathrm{H}}_2}^{\mathrm{P}2\mathrm{P},i}$$

(22)

3.1.2. Carbon Quota P2P Trading

P2P carbon quota trading is a key pathway for optimizing carbon resource allocation across multiple microgrids. Its trading logic is consistent with P2P electricity trading and adjusts for surplus–deficit in carbon quotas through peer interactions. For the $i$-th DIEM, its total P2P carbon quota trading volume at time slot $t$ is the sum of carbon quota trades with all other DIEMs, as in Equation (23):

$$E_t^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{j\in {\varphi }_i}{E}_t^{\mathrm{P}2\mathrm{P},ij}}$$

(23)

where $E_t^{\mathrm{P}2\mathrm{P},i}$ is the total P2P carbon quota trading volume of the $i$-th DIEM at time slot $t$. The trading direction convention for $E_t^{\mathrm{P}2\mathrm{P},ij}$ is identical to that of P2P electricity trading, following the direction definition in Equation (17).

To ensure transaction consistency and fairness, the buyer and seller of the same P2P carbon quota transaction must satisfy:

$$E_t^{\mathrm{P}2\mathrm{P},ij}+E_t^{\mathrm{P}2\mathrm{P},ji}=0$$

(24)

For $\mathrm{DIEM} i$, the total P2P carbon quota trading cost is the sum of trading costs across all time slots:

$$C_{\mathrm{C}}^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{t\in T}{\displaystyle \sum _{j\in {\varphi }_i}{E}_t^{\mathrm{P}2\mathrm{P},ij}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{C}}}}$$

(25)

where $C_{\mathrm{C}}^{\mathrm{P}2\mathrm{P},i}$ is the P2P carbon quota trading cost.

3.1.3. Green Certificate P2P Trading

P2P green certificate trading is a key mechanism for promoting renewable energy consumption and transmitting green value signals. Its trading logic is also consistent with P2P electricity trading. For the $i$-th DIEM, its total P2P green certificate trading volume at time slot $t$ is the sum of green certificate trades with all other DIEMs, as in Equation (26):

$$G_t^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{j\in {\varphi }_i}{G}_t^{\mathrm{P}2\mathrm{P},ij}}$$

(26)

where $G_t^{\mathrm{P}2\mathrm{P},i}$ is the total P2P green certificate trading volume of the $i$-th DIEM at time slot $t$. The trading direction convention is identical to that of P2P electricity trading. Refer to the definition of the trading direction in Equation (17).

For the buyer and seller of the same P2P green certificate transaction, a bilateral consistency constraint applies:

$$G_t^{\mathrm{P}2\mathrm{P},ij}+G_t^{\mathrm{P}2\mathrm{P},ji}=0$$

(27)

For $\mathrm{DIEM} i$, the total P2P green certificate trading cost is the sum of trading costs across all time slots:

$$C_{\mathrm{G}}^{\mathrm{P}2\mathrm{P},i}={\displaystyle \sum _{t\in T}{\displaystyle \sum _{j\in {\varphi }_i}{G}_t^{\mathrm{P}2\mathrm{P},ij}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{G}}}}$$

(28)

where $C_{\mathrm{G}}^{\mathrm{P}2\mathrm{P},i}$ is the P2P green certificate trading cost.

3.2. Cooperative-Game-Based Optimal Operation Model for Multi-Microgrids Based on Nash Bargaining

3.2.1. Cooperative Operating Cost Model for Multi-Microgrids

(1) Objective Function

Each $\mathrm{DIEM} i$ minimizes its total operating cost, which includes the interaction cost with the external grid, microgrid operation and maintenance costs, wind and solar curtailment costs, energy storage operation and maintenance costs, carbon trading costs, green certificate trading costs, and comprehensive P2P trading costs, as expressed in Equation (29):

$$C_i^{\mathrm{DIEM}}=\left(C_{\mathrm{utility},i}+C_{\mathrm{units},i}+C_{\mathrm{cut},i}+C_{\mathrm{ESS},i}+C_{\mathrm{C},i}+C_{\mathrm{GCT},i}+C_{\mathrm{total},i}^{\mathrm{P}2\mathrm{P}}\right)$$

(29)

1) External interaction cost:

$$C_{\mathrm{utility},i}={\displaystyle \sum _{t=1}^T}\left[{\lambda }_t^{{\mathrm{CH}}_4}{N}_t^{{\mathrm{CH}}_4,i}+{\lambda }_t^{\mathrm{Gb}}{P}_t^{\mathrm{buy},i}-{\lambda }_t^{\mathrm{Gs}}{P}_t^{\mathrm{sell},i}\right]$$

(30)

2) Unit operation and maintenance cost:

$$C_{\mathrm{units},i}={\displaystyle \sum _{t=1}^T}\left[\left(\begin{array}{l}{\alpha }_1{P}_t^{\mathrm{CHP},i}+{\beta }_1{\left(P_t^{\mathrm{CHP},i}\right)}^2+{\alpha }_2{H}_t^{\mathrm{GB},i}\\ +{\beta }_2{\left(H_t^{\mathrm{GB},i}\right)}^2+{\alpha }_3{P}_t^{\mathrm{P}2\mathrm{G},i}+{\alpha }_4{P}_t^{{\mathrm{HFC},\mathrm{H}}_2,i}\\ +{\alpha }_5{P}_t^{\mathrm{CCUS},i}+{\alpha }_6\end{array}\right)\right]$$

(31)

3) Wind and solar curtailment cost:

$$C_{\mathrm{cut},i}={\displaystyle \sum _{\mathrm{t}=1}^T{\lambda }_{\mathrm{cur}}\left(P_t^{\mathrm{WT},\mathrm{cur},i}+P_t^{\mathrm{PV},\mathrm{cur},i}\right)}$$

(32)

4) Energy storage operation and maintenance cost:

$$C_{\mathrm{ESS},i}={\displaystyle \sum _{t=1}^T\varsigma \left(\begin{array}{l}{P}_t^{\mathrm{ESS},\mathrm{e},\mathrm{cha},i}+P_t^{\mathrm{ESS},\mathrm{h},\mathrm{cha},i}+P_t^{{\mathrm{ESS},\mathrm{H}}_2,\mathrm{cha},i}\\ +P_t^{\mathrm{ESS},\mathrm{e},\mathrm{dis},i}+P_t^{\mathrm{ESS},\mathrm{h},\mathrm{dis},i}+P_t^{{\mathrm{ESS},\mathrm{H}}_2,\mathrm{dis},i}\end{array}\right)}$$

(33)

5) Green certificate trading cost:

$${\widehat{G}}_{\mathrm{GCT},i}=G_{\mathrm{r},i}-G_{\mathrm{e},i}-G_{\mathrm{CET},i}-G_{\mathrm{green},i}$$

(34)

$$C_{\mathrm{GCT},i}={\displaystyle \sum _{t=1}^T{\widehat{G}}_{\mathrm{GCT},i}{\lambda }_t^{\mathrm{GCT}}}$$

(35)

6) P2P comprehensive trading cost:

$$C_{\mathrm{total},i}^{\mathrm{P}2\mathrm{P}}=C_{\mathrm{energy}}^{\mathrm{P}2\mathrm{P},i}+C_{\mathrm{C}}^{\mathrm{P}2\mathrm{P},i}+C_{\mathrm{G}}^{\mathrm{P}2\mathrm{P},i}$$

(36)

(2) Constraints

1) Energy supply–demand balance constraints:

$$\begin{array}{l}{P}_t^{\mathrm{PV},i}+P_t^{\mathrm{WT},i}+P_t^{\mathrm{buy},i}+P_t^{\mathrm{HFC},\mathrm{e},i}+P_t^{\mathrm{CHP},i}+P_t^{\mathrm{ESS},\mathrm{e},\mathrm{dis},i}\\ -P_t^{\mathrm{EL},i}-P_t^{\mathrm{CCUS},i}+{\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}}=P_t^{\mathrm{sell},i}+P_t^{\mathrm{ESS},\mathrm{e},\mathrm{cha},i}+L_t^{\mathrm{e},i}\end{array}$$

(37)

$$H_t^{\mathrm{CHP},i}+H_t^{\mathrm{GB},i}+H_t^{\mathrm{HFC},\mathrm{h},i}+P_t^{\mathrm{ESS},\mathrm{h},\mathrm{dis},i}+{\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}}}=P_t^{\mathrm{ESS},\mathrm{h},\mathrm{cha},i}+L_t^{\mathrm{h},i}$$

(38)

$$P_t^{P{2\mathrm{G},\mathrm{H}}_2,i}+P_t^{{\mathrm{ESS},\mathrm{H}}_2,\mathrm{dis},i}+{\displaystyle \sum _{j\in {\varphi }_i}{P}_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}}=P_t^{\mathrm{MR},{\mathrm{H}}_2,i}+P_t^{{\mathrm{HFC},\mathrm{H}}_2,i}+P_t^{{\mathrm{ESS},\mathrm{H}}_2,\mathrm{cha},i}+L_t^{{\mathrm{H}}_2,i}$$

(39)

$${\widehat{P}}_t^{\mathrm{PV},i}-P_t^{\mathrm{PV},i}=P_t^{\mathrm{PV},\mathrm{cur},i}$$

(40)

$${\widehat{P}}_t^{\mathrm{WT},i}-P_t^{\mathrm{WT},i}=P_t^{\mathrm{WT},\mathrm{cur},i}$$

(41)

$${\widehat{P}}_t^{\mathrm{PV},i}\left(1-{\delta }^{\mathrm{PV}}\right)\le P_t^{\mathrm{PV},i}\le {\widehat{P}}_t^{\mathrm{PV},i}\left(1+{\delta }^{\mathrm{PV}}\right)$$

(42)

$${\widehat{P}}_t^{\mathrm{WT},i}\left(1-{\delta }^{\mathrm{WT}}\right)\le P_t^{\mathrm{WT},i}\le {\widehat{P}}_t^{\mathrm{WT},i}\left(1+{\delta }^{\mathrm{WT}}\right)$$

(43)

2) Natural gas purchase constraint:

$$N_{\min }^{{\mathrm{CH}}_4,i}\le \left|N_t^{{\mathrm{CH}}_4,i}\right|\le N_{\max }^{{\mathrm{CH}}_4,i}$$

(44)

3) Natural gas flow balance constraint:

$$N_t^{{\mathrm{CH}}_4,i}+N_t^{{\mathrm{MR},\mathrm{CH}}_4,i}=N_t^{\mathrm{CHP},i}+N_t^{\mathrm{GB},i}$$

(45)

4) Carbon emission flow balance constraints:

$$E_t^{\mathrm{CHP},\mathrm{C},i}+E_t^{\mathrm{GB},\mathrm{C},i}-E_t^{\mathrm{CCUS},\mathrm{C},i}+{\displaystyle \sum _{j\in {\varphi }_i}{E}_t^{\mathrm{P}2\mathrm{P},ij}}=E_t^{\mathrm{out},i}$$

(46)

$$E_t^{\mathrm{CHP},\mathrm{C},i}+E_t^{\mathrm{GB},\mathrm{C},i}-E_t^{\mathrm{P}2\mathrm{G},\mathrm{C},i}-E_t^{\mathrm{CCUS},\mathrm{s},i}+{\displaystyle \sum _{j\in {\varphi }_i}{E}_t^{\mathrm{P}2\mathrm{P},ij}}=E_t^{\mathrm{out},i}$$

(47)

When $N$ DIEMs form a cooperative coalition, the total operational cost of the multi-microgrid cooperative system is given by Equation (48):

$$C_{\mathrm{total}}^{\mathrm{DIEM}}={\displaystyle \sum _{i=1}^N{C}_i^{\mathrm{DIEM}}}$$

(48)

where $C_{\mathrm{total}}^{\mathrm{DIEM}}$ is the total operational cost of the multi-microgrid cooperative coalition.

3.2.2. Multi-Microgrid Cooperative Game Model Based on Nash Bargaining

In multi-microgrid cooperative operation, participants are concerned not only with their individual benefit gains but also with the fair and reasonable distribution of cooperative profits. Moreover, under the energy-carbon-green certificate coordinated trading mechanism, achieving supply–demand balance while reducing carbon emissions through cooperative operations is a key challenge. As an important branch of cooperative game theory, Nash bargaining theory is centered on collective rationality and social optimization, effectively characterizing multi-party cooperative interactions and providing a systematic analytical framework for balancing efficiency improvement with fair distribution. Unlike traditional optimization methods that focus solely on minimizing total cost, the Nash bargaining model introduces a disagreement point (i.e., the cost under non-cooperation) and a cooperative surplus-distribution mechanism, achieving Pareto optimality while satisfying individual rationality constraints.

The standard form of the Nash bargaining problem can be stated as follows: given N participants, each participant $i$ has a disagreement point $d_i$ and a feasible payoff set $\mathcal{U}\subseteq {ℝ}^N$, the Nash bargaining solution is the solution to the optimization problem in Equation (49):

$$\underset{u\in \mathcal{U},u_i\ge d_i}{\max }{\displaystyle \prod _{i=1}^N{\left(u_i-d_i\right)}^{w_i}}$$

(49)

where $u_i$ is the payoff of $\mathrm{DIEM} i$ under cooperation (the negative of operational cost, i.e., $u_i=-C_i^{\mathrm{DIEM}}$); $d_i$ is the payoff of $\mathrm{DIEM} i$ without cooperation, i.e., the Nash disagreement point. Specifically, $d_i$ is obtained by solving an independent operation optimization problem for $\mathrm{DIEM} i$: setting all P2P trading variables to zero ($P_t^{\mathrm{P}2\mathrm{P},i,*}=0,E_t^{\mathrm{P}2\mathrm{P},i}=0,G_t^{\mathrm{P}2\mathrm{P},i}=0$) in the objective function and constraints of Equations (29)–(47), retaining only each DIEM’s interactions with the external grid and natural gas network, and solving for the optimal individual operational cost $C_i^{\mathrm{DIEM},\mathrm{non}}$. Then $d_i=-C_i^{\mathrm{DIEM},\mathrm{non}}$. The term $u_i-d_i$ represents the incremental payoff (cooperative surplus) obtained by $\mathrm{DIEM} i$ through cooperative operation. $w_i$ is the bargaining power weight of participant $i$, reflecting its contribution to the coalition.

In this study, equal bargaining weights are adopted, i.e., $w_1=w_2=w_3=1/3$. The rationale is threefold: (1) the three DIEMs in the case study have similar installed capacities and load scales, belonging to homogeneous agents without significant differences in bargaining power; (2) the equal-weight setting reduces Equation (49) to the standard Nash bargaining solution, facilitating comparison with existing Nash bargaining studies [12,13,14,15] to validate the effectiveness of the model and algorithm; (3) in practical applications where significant differences exist in installed capacity, carbon emissions, or renewable penetration among DIEMs, the weights can be set proportionally to these indicators.

The Nash bargaining model above is a complex optimization problem with multiple variables, strong coupling, non-convexity, and nonlinearity. It involves multi-commodity trading decisions for electricity, heat, hydrogen, carbon quotas, and green certificates, as well as operational coupling constraints across DIEMs, making direct solution via traditional optimization methods infeasible. The original problem is therefore equivalently decomposed into two tractable subproblems: a cooperative coalition total-profit maximization subproblem and a multi-commodity P2P trading payment-negotiation subproblem.

Subproblem 1: cooperative profit maximization

$$u_i=-C_i^{\mathrm{DIEM}}=-\left(\begin{array}{l}{C}_{\mathrm{utility},i}+C_{\mathrm{units},i}+C_{\mathrm{cut},i}+C_{\mathrm{ESS},i}\\ +C_{\mathrm{C},i}+C_{\mathrm{GCT},i}+C_{\mathrm{total},i}^{\mathrm{P}2\mathrm{P}}\end{array}\right)$$

(50)

$$\max {\displaystyle \sum _{i=1}^N{u}_i}=-C_{\mathrm{total}}^{\mathrm{DIEM}}$$

(51)

$$\mathrm{s.t.} \mathrm{Equations} \left(37\right)–\left(47\right)$$

(52)

Subproblem 2: multi-commodity P2P trading payment negotiation

$$\max {\displaystyle \prod _{i=1}^N{\left(u_i-d_i\right)}^{w_i}}$$

(53)

To facilitate a solution, taking the logarithm of Equation (53) converts the product form into a summation form, as given by Equations (54)–(56):

$$\max {\displaystyle \sum _{i=1}^N{w}_i\ln }\left(u_i-d_i\right)$$

(54)

$$\max {\displaystyle \sum _{i=1}^N{w}_i\ln }\left(u_i^{*}-{\displaystyle \sum _{j=1,j\ne i}^N\left(\begin{array}{l}{P}_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{e}}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}+P_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{h}}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}}\\ +P_t^{\mathrm{P}2\mathrm{P},ij,{{\mathrm{H}}_2}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}+E_t^{\mathrm{P}2\mathrm{P},i{j}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{C}}\\ +G_t^{\mathrm{P}2\mathrm{P},i{j}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{G}}\end{array}\right)-d_i}\right)$$

(55)

$$u_i^{*}-{\displaystyle \sum _{j=1,j\ne i}^N\left(\begin{array}{l}{P}_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{e}}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}+P_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{h}}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}}\\ +P_t^{\mathrm{P}2\mathrm{P},ij,{{\mathrm{H}}_2}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}+E_t^{\mathrm{P}2\mathrm{P},i{j}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{C}}\\ +G_t^{\mathrm{P}2\mathrm{P},i{j}^{*}}{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{G}}\end{array}\right)\ge d_i}$$

(56)

In the proposed implementation, the Nash bargaining allocation is realized endogenously through the negotiated settlement prices for electricity, heat, hydrogen, carbon quotas, and green certificates. The resulting bilateral settlement payments are included in the comprehensive P2P trading cost $C_{\mathrm{total},i}^{\mathrm{P}2\mathrm{P}}$ defined in Equation (36) and are therefore already reflected in the reported operating cost of each DIEM. No additional ex-post lump-sum side-payment variable is introduced in the proposed framework; therefore, separate side-payment values are not reported.

3.3. Nash Bargaining Equilibrium Conditions

The Nash bargaining solution is the unique solution satisfying a set of axiomatic conditions in cooperative game theory. Starting from the axiomatic theory of Nash bargaining, this section proves that the solution to the multi-microgrid cooperative game optimization model in Equation (53) satisfies the Nash bargaining equilibrium conditions.

For a cooperative game with N participants, given a disagreement point $d_i=(d_1,d_2,\dots ,d_N)$ and a feasible payoff set $\mathcal{U}\subseteq {ℝ}^N$, Nash proved that there exists a unique bargaining solution $u_i^{*}=(u_1^{*},u_2^{*},\dots ,u_N^{*})$ satisfying the following axioms:

(1) Individual Rationality:

$u_i^{*}\ge d_i,\forall i\in N$, each participant obtains a payoff no lower than the disagreement point. Since the payoff is defined as the negative of the operating cost, i.e., $u_i=-C_i^{\mathrm{DIEM}}$ and $d_i=-C_i^{\mathrm{DIEM},\mathrm{non}}$, the individual rationality condition $u_i\ge d_i$ is equivalently expressed in the cost domain as $C_i^{\mathrm{DIEM}}\le C_i^{\mathrm{DIEM},\mathrm{non}}$.

(2) Pareto Optimality:

By contradiction, suppose another feasible solution $u^{\prime }\in \mathcal{U}$ exists such that $u_i^{\prime }\ge u_i^{*}$ for all $i$, with at least one strict inequality. Since the objective function ${\displaystyle \prod _{i=1}^N{\left(u_i-d_i\right)}^{w_i}}$ is strictly increasing in each component, then ${\displaystyle {\displaystyle \prod _{i=1}^N}{\left(u_i^{\prime }-d_i\right)}^{w_i}}>{\displaystyle {\displaystyle \prod _{i=1}^N}{\left(u_i^{*}-d_i\right)}^{w_i}}$, contradicting the fact that $u_i^{*}$ is the optimal solution of the maximization problem in Equation (53). Therefore, $u_i^{*}$ must be Pareto optimal.

(3) Symmetry:

If two DIEMs $i$ and $j$ have identical cost functions, disagreement points $d_i=d_j$, and bargaining weights $w_i=w_j$, the model objective function is symmetric with respect to $u_i$ and $u_j$. If the optimal solution satisfies $u_i^{*}\ne u_j^{*}$, swapping the values of $i$ and $j$ yields another optimal solution, contradicting the uniqueness of the solution. Therefore, $u_i^{*}=u_j^{*}$ must hold.

(4) Invariance to Affine Transformations:

For any affine transformation $u_i^{\prime }={\alpha }_i{u}_i+{\beta }_i ({\alpha }_i>0)$ applied to the payoff of each $\mathrm{DIEM} i$, the disagreement point correspondingly becomes $d_i^{\prime }={\alpha }_i{d}_i+{\beta }_i$. The resulting objective function is expressed as Equation (57):

$${\displaystyle {\displaystyle \prod _{i=1}^N}{(u_i^{\prime }-d_i^{\prime })}^{w_i}}={\displaystyle {\displaystyle \prod _{i=1}^N}{({\alpha }_i{u}_i+{\beta }_i-{\alpha }_i{d}_i-{\beta }_i)}^{w_i}}={\displaystyle {\displaystyle \prod _{i=1}^N}{\alpha }_i^{w_i}}{(u_i-d_i)}^{w_i}=\left({\displaystyle \prod _{i=1}^N{\alpha }_i^{w_i}}\right)\cdot {\displaystyle {\displaystyle \prod _{i=1}^N}{(u_i-d_i)}^{w_i}}$$

(57)

Since ${\displaystyle \prod _{i=1}^N{\alpha }_i^{w_i}}$ is a positive constant, the solution to the optimization problem is unchanged, i.e., $u_i^{{*}^{\prime }}={\alpha }_i{u}_i^{*}+{\beta }_i$.

In summary, the solution of the above model satisfies the axiomatic conditions of Nash bargaining. This completes the proof.

4. Homogeneous Multi-Agent Deep Reinforcement Learning Solution Framework for Multi-Microgrids

To address the optimization challenges of the Nash bargaining cooperative game model under complex constraints, this paper proposes a multi-agent deep reinforcement learning method based on centralized training and decentralized execution. By modeling the cooperative game as a Markov decision process, this study designs a multi-agent deep reinforcement learning framework using the MASAC algorithm to establish collaborative interactions among multiple DIEMs. The framework maps the strategic interactions in the Nash bargaining game into a dynamic state-action-reward iterative optimization process. A centralized critic network approximates the weighted Nash product objective, while decentralized actor networks enable autonomous decision-making for each DIEM. This approach effectively overcomes the limitations of traditional methods in non-convex, non-linear, and multi-commodity trading scenarios, and it provides an end-to-end adaptive solution for the cooperative operation of multi-microgrids in a coordinated energy-carbon-green certificate market.

4.1. Markov Decision Process Modeling

The optimal operation problem of the cooperative multi-microgrid game is formulated as a Markov decision process, defined by the 5-tuple $\mathrm{MDP}=\left(S,A,P,R,\gamma \right)$. By defining the state space $S$, action space $A$, state transition function $P$, reward function $R$, and discount factor $\gamma$, cooperative game strategies are mapped to sequential decisions. As homogeneous agents, the DIEMs share identical structures for their state space, action space, and reward functions, but are coupled through peer-to-peer transactions. Agent interactions are governed by a centralized training and decentralized execution framework. During training, a centralized critic network observes global information to evaluate the joint policy; during execution, each DIEM makes independent decisions based solely on local observations.

Let $\mathcal{N}=\{1,2,\dots ,N\}$ denote the set of agents, where each agent $i$ corresponds to a DIEM and has a homogeneous state space, action space, and reward function structure.

4.1.1. State Space $S$

The local state $s_t^i$ of agent $i$ at time step $t$ contains information reflecting its operating status and the external market environment:

$$s_t^i=\left\{L_t^{\mathrm{e},i},L_t^{\mathrm{h},i},L_t^{{\mathrm{H}}_2,i},P_t^{\mathrm{WT},i},P_t^{\mathrm{PV},i},{\mathrm{SOC}}_t^{\mathrm{ESS},\mathrm{e},i},{\mathrm{SOC}}_t^{\mathrm{ESS},\mathrm{h},i},{\mathrm{SOC}}_t^{{\mathrm{ESS},\mathrm{H}}_2,i},P_t^{\mathrm{sell},i},P_t^{\mathrm{buy},i},{\lambda }_t^{\mathrm{GCT},i},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij}\right\}$$

(58)

$${\lambda }_t^{\mathrm{P}2\mathrm{P},ij}=\left[{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij{,\mathrm{H}}_2},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{C}},{\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{G}}\right]$$

(59)

$$s_t=\left[s_t^1,s_t^2,\cdots \cdots s_t^N\right]$$

(60)

where ${\lambda }_t^{\mathrm{P}2\mathrm{P},ij}$ is the set of P2P transaction prices for agent $i$ at time slot $t$, and $s_t$ is the joint state set of all agents at time slot $t$.

4.1.2. Action Space $A$

The continuous action $a_t^i$ of agent $i$ at time slot $t$ comprises microgrid outputs, energy storage charging and discharging, and multi-commodity P2P trading volumes. The equipment in DIEM is relatively complex and can be categorized into energy conversion devices and energy storage systems.

$$a_t^i=\left\{P_{i,w,t},B_{i,w,t},P_t^{\mathrm{P}2\mathrm{P},ij},E_t^{\mathrm{P}2\mathrm{P},ij},G_t^{\mathrm{P}2\mathrm{P},ij}\right\}$$

(61)

$$P_t^{\mathrm{P}2\mathrm{P},ij}=\left[P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}},P_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{h}},P_t^{\mathrm{P}2\mathrm{P},ij,{\mathrm{H}}_2}\right]$$

(62)

$$a_t=\left[a_t^1,a_t^2,\cdots \cdots a_t^N\right]$$

(63)

where $P_{i,w,t}$ is the power output of the energy conversion device $w$ for agent $i$ at time slot $t$, $B_{i,w,t}$ is the energy output or input of the energy storage device $w$ for agent $i$ at time slot $t$, $P_t^{\mathrm{P}2\mathrm{P},ij}$ is the set of P2P electricity, heat, and hydrogen trading volumes for agent $i$ at time slot $t$; and $a_t$ is the joint action set of all agents at time slot $t$.

To ensure the feasibility of continuous actions generated by decentralized agents, constraint-handling and post-processing mechanisms are applied to the action outputs. First, local equality constraints, including the energy, gas, and carbon flow balance constraints in Equations (37)–(39) and (45)–(47), are handled using a penalty-based method during training. Any violation of these balance constraints introduces a large negative penalty into the reward function, thereby guiding the actor networks to learn feasible operating policies. In addition, a projection layer is applied to keep equipment outputs, storage charging and discharging powers, and trading quantities within their physical upper and lower bounds. For bilateral P2P transactions, the consistency condition $P_{ij,t}^{\mathrm{P}2\mathrm{P}}+P_{ji,t}^{\mathrm{P}2\mathrm{P}}=0$ should be satisfied. To meet this condition in decentralized execution, the agents’ raw continuous outputs are treated as trading bids. Before physical dispatch, a lightweight bilateral matching rule is applied to determine the actually executed trading volume, where $P_{ij,t}^{\mathrm{P}2\mathrm{P},\mathrm{actual}}=(P_{ij,t}^{\mathrm{P}2\mathrm{P},\mathrm{bid}}-P_{ji,t}^{\mathrm{P}2\mathrm{P},\mathrm{bid}})/2$ and $P_{ji,t}^{\mathrm{P}2\mathrm{P},\mathrm{actual}}=-P_{ij,t}^{\mathrm{P}2\mathrm{P},\mathrm{actual}}$. This post-processing rule ensures consistency in bilateral trading while preserving the decentralized execution structure.

4.1.3. State Transition Function $P$

The system state transitions are driven by the dynamic characteristics of the DIEM and source-load uncertainties, formulated as $s_{t+1}~P(s_{t+1}|s_t,a_t)$, where $a_t=[a_t^1,a_t^2,\cdots \cdots ,a_t^N]$ is the joint action. The state transition function is jointly determined by the DIEM’s physical dynamics and the evolution of external market prices.

4.1.4. Reward Function $R$

The weighted Nash bargaining model is integrated into the multi-agent reinforcement learning training objective. The immediate reward $r_t^i$ for agent $i$ at time slot $t$ is defined as the negative of its individual operational cost $C_{i,t}^{\mathrm{DIEM}}$, consistent with the individual cost objective rather than the coalition total. The detailed calculation is provided in Section 3.2.1:

$$r_t^i=-C_{i,t}^{\mathrm{DIEM}}$$

(64)

To achieve the Nash bargaining objective of the cooperative game, centralized training employs a global reward function that approximates the Nash product. During the early stages of training, exploratory and suboptimal policies may cause an agent’s payoff to fall below the disagreement point, resulting in a non-positive argument ($R_i^t-d_i\le 0$), which would make the logarithm undefined. To handle this, a clipping mechanism with a small positive constant $\epsilon$ and a penalty term is introduced. The global reward function is formulated as follows:

$$r_t^{\mathrm{global}}=\sum _{i\in \mathcal{N}}{w}_i\ln \left(\max \left(R_i^t-d_i,\epsilon \right)\right)-\sum _{i\in \mathcal{N}}{P}_i^t$$

(65)

$$R_i^t={\displaystyle \sum _{\mathsf{\ell }=1}^t}{r}_{i,\mathsf{\ell }}$$

(66)

where $r_t^{\mathrm{global}}$ is the global reward for all agents at time slot $t$; $R_i^t$ represents the accumulated payoff of agent $i$ up to time $t$; $d_i$ is the disagreement point, namely the payoff under non-cooperation; and $\epsilon$ is a small positive constant, e.g., 10⁻³, used to keep the logarithm well defined. $P_i^t$ is a large penalty applied when $R_i^t-d_i\le 0$, discouraging policies that violate individual rationality. This formulation computes the global reward from individual costs and remains consistent with the decentralized execution architecture and the Nash bargaining model. Although $R_i^t$ is accumulated progressively within one daily scheduling episode, while $d_i$ is obtained from the full-day independent-operation benchmark, this comparison is used as a reward-shaping mechanism during training. The individual rationality condition is finally evaluated at the terminal time $t=T$, where $R_i^T$ and $d_i$ are defined over the same full-day horizon.

4.2. Principles and Framework of the Deep Reinforcement Learning Algorithm for Multi-Microgrids Based on CTDE-MASAC

To address non-convexity and non-stationarity in cooperative multi-microgrid games, deep reinforcement learning offers an effective solution, as it adapts well to complex, dynamic environments. For continuous action-space optimization, SAC, an off-policy actor-critic algorithm based on the maximum entropy framework, introduces a policy entropy term into the conventional cumulative reward objective. This constructs a joint optimization objective that maximizes both the expected cumulative reward and the policy entropy, effectively balancing decision-making performance with exploration. To this end, the SAC algorithm is extended to a multi-agent variant, namely MASAC, to meet the requirements of continuous action space optimization in cooperative multi-microgrid games. The effective operation of MASAC relies on a suitable multi-agent deep reinforcement learning framework, within which the centralized training and decentralized execution strategy is highly compatible.

The core principle of the CTDE framework is that agents are trained centrally but execute independently in a decentralized manner. During training, the centralized critic network accesses the states and actions of all agents and uses global information to evaluate the long-term value of the joint policy. This mechanism effectively mitigates the non-stationarity inherent in multi-agent environments. Because each agent’s policy updates alter the state distributions of others, global information is essential to accurately model this interdependence. However, during execution, agents rely solely on local observations and pre-trained decentralized actor networks to make independent decisions. This eliminates the need for real-time communication, thereby ensuring both privacy protection and communication efficiency in practical deployment [30].

4.2.1. MASAC Algorithm Principles [31,32]

The MASAC algorithm extends SAC to multi-agent environments, where each agent maintains an independent decentralized actor network and all agents share a centralized critic network. This centralized critic takes the global state and the joint actions of all agents as inputs to estimate the long-term cumulative reward.

(1) Maximum Entropy Reinforcement Learning Objective:

$$J(\pi )={\displaystyle \sum _{t=1}^T{\mathbb{E}}_{(s_t,a_t)~{\rho }_{\pi }}}\left[r(s_t,a_t)+\alpha \mathcal{H}(\pi (\cdot |s_t))\right]$$

(67)

where $J(\pi )$ is the single-agent maximum entropy objective function, ${\rho }_{\pi }$ is the joint state-action distribution induced by the policy $\pi$, $\mathcal{H}(\pi (\cdot |s_t))=-{\mathbb{E}}_{a_t~\pi }\log \pi (a_t|s_t)$ is the policy entropy, and $\alpha$ is the temperature parameter.

For multi-agent systems, the objective function of MASAC is extended to:

$$J({\pi }_1,\dots ,{\pi }_N)={\displaystyle \sum _{t=0}^{\infty }{\mathbb{E}}_{(s_t,a_t)~{\rho }_{\pi }}}\left[r(s_t,a_t)+\alpha {\displaystyle \sum _{i=1}^N\mathcal{H}}({\pi }_i(\cdot |s_t^i))\right]$$

(68)

(2) Definition of the Soft Value Function

The soft state value function represents the expected cumulative soft reward obtained at state $s_t$ when following policy $\pi$:

$$V(s_t)={\mathbb{E}}_{a_t~\pi }\left[Q(s_t,a_t)-\alpha \log \pi (a_t|s_t)\right]$$

(69)

The soft action value function satisfies the soft Bellman equation:

$$Q(s_t,a_t)=r(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}~p}\left[V(s_{t+1})\right]$$

(70)

where $\alpha \log \pi (a_t|s_t)$ is the entropy regularization term, $\gamma$ is the discount factor, and $p$ is the environmental state transition probability distribution.

(3) Critic Network Update:

MASAC employs two centralized Critic networks, $Q_{{\theta }_1}(s,a_1,\dots ,a_N)$ and $Q_{{\theta }_2}(s,a_1,\dots ,a_N)$, along with their corresponding target networks, $Q_{{\overline{\theta }}_1}$ and $Q_{{\overline{\theta }}_2}$. The double-Critic design estimates the target value by selecting the minimum of the two Q-values, thereby mitigating overestimation.

The loss function for the critic networks is defined as the soft Bellman residual:

$$L_Q({\theta }_j)={\mathbb{E}}_{(s,a,r,s_{t+1})~\mathcal{D}}\left[{\left(Q_{{\theta }_j}(s_t,a_t)-\overline{y}\right)}^2\right],j=1,2$$

(71)

The target value $\overline{y}$ is formulated as:

$$\overline{y}=r+\gamma {\mathbb{E}}_{a_{t+!}~\pi }\left[{\min }_{j=1,2}{Q}_{{\overline{\theta }}_j}(s_{t+1},a_{t+1})-\alpha {\displaystyle \sum _{i=1}^N\log }{\pi }_i(a_{t+1}^i|s_{t+1}^i)\right]$$

(72)

where $L_Q({\theta }_j)$ is the loss function of the $j$-th critic network, $(s,a,r,s_{t+1})$ is a transition tuple sampled from historical interactions, $\overline{y}$ is the critic target value, ${\min }_{j=1,2}{Q}_{{\overline{\theta }}_j}(s_{t+1},a_{t+1})$ is the minimum value between the two target critic networks, and ${\overline{\theta }}_j$ are the parameters of the target critic networks.

(4) Actor Network Update:

The Actor network ${\pi }_{{\varphi }_i}(a_t^i|s_t^i)$ of each agent is updated by minimizing the KL divergence, with the objective of aligning the policy distribution with the exponentially transformed soft Q-value distribution:

$$J_{\pi }({\varphi }_i)={\mathbb{E}}_{s~\mathcal{D},a_i~{\pi }_{{\varphi }_i}}\left[\alpha \log {\pi }_{{\varphi }_i}(a_t^i|s_t^i)-{\min }_{j=1,2}{Q}_{{\theta }_j}(s,a_1,\dots ,a_N)\right]$$

(73)

In gradient computation, the stochastic policy is sampled using the reparameterization trick:

$$a_t^i=f_{{\varphi }_i}({ϵ}_t^i;s_t^i)$$

(74)

Reparameterization enables gradients to be back-propagated directly through the sampling process:

$${\widehat{\nabla }}_{{\varphi }_i}{J}_{\pi }({\varphi }_i)={\mathbb{E}}_{ϵ~\mathcal{N}}\left[\alpha {\nabla }_{{\varphi }_i}\log {\pi }_{{\varphi }_i}(a_t^i|s_t^i)+\left(\begin{array}{l}{\nabla }_{a_i}\alpha \log {\pi }_{{\varphi }_i}(a_t^i|s_t^i)\\ -{\nabla }_{a_i}{\min }_j{Q}_{{\theta }_j}(s_t,a_t)\end{array}\right){\nabla }_{{\varphi }_i}{f}_{{\varphi }_i}({ϵ}_t^i;s_t^i)\right]$$

(75)

where $J_{\pi }({\varphi }_i)$ is the objective function of the $i$-th agent’s actor network, $f_{{\varphi }_i}$ is the reparameterized policy function, ${ϵ}_t^i$ is a noise vector sampled from a fixed distribution, and ${\nabla }_{{\varphi }_i}{J}_{\pi }({\varphi }_i)$ is the gradient of the Actor objective function with respect to the parameters ${\varphi }_i$.

(5) Adaptive Temperature Coefficient:

The temperature parameter $\alpha$ controls the intensity of entropy regularization. An excessively large value promotes over-exploration, whereas an excessively small value leads to insufficient exploration. To tackle this, SAC incorporates an adaptive adjustment mechanism that dynamically optimizes $\alpha$ by minimizing the following objective function:

$$L(\alpha )={\mathbb{E}}_{a_i~{\pi }_i}\left[-\alpha \log {\pi }_i(a_i|s_i)-\alpha \overline{\mathcal{H}}\right]$$

(76)

where $L(\alpha )$ is the adaptive loss function for the temperature parameter $\alpha$, and $\overline{\mathcal{H}}$ is the target entropy.

(6) Soft Update of the Target Network:

To stabilize training, the parameters of the target networks are updated via a soft-update mechanism, smoothly tracking the current network parameters:

$${\overline{\theta }}_j\leftarrow \tau {\theta }_j+(1-\tau ){\overline{\theta }}_j,\hspace{1em}j=1,2$$

(77)

where $\tau$ is the soft update coefficient.

4.2.2. Extraction of Marginal Contributions and Solving for P2P Transaction Payment Schemes

Once the MASAC algorithm converges, each DIEM’s actor network obtains an optimal cooperative operating policy. However, resolving the cooperative game further requires determining internal settlement prices to ensure a fair distribution of benefits. In economics, marginal contribution refers to the impact on total system revenue from adding a participant or a unit of a resource. The centralized critic network $Q\left(s,a;\theta \right)$ approximates the long-term cumulative value of the weighted Nash product. According to the envelope theorem, at the optimal strategy, the partial derivative of the value function with respect to a parameter equals the partial derivative of the Lagrangian function with respect to its corresponding constraint. Therefore, the partial derivatives of the critic network carry the economic significance of shadow prices. By treating the critic network as a close approximation of the true value function, the nodal marginal values can be extracted by calculating the partial derivatives:

$${\lambda }_t^{i,e}={\left.{\displaystyle \frac{\partial Q(s,a)}{\partial P_t^{\mathrm{P}2\mathrm{P},i,\mathrm{e}}}}\right|}_{(s,a^{*})}$$

(78)

According to the Nash bargaining solution, the transaction price must reflect the marginal valuations of both parties. The average marginal value of the two trading entities is adopted as the P2P settlement price:

$${\lambda }_t^{\mathrm{P}2\mathrm{P},ij,\mathrm{e}}={\displaystyle \frac{{\lambda }_t^{i,e}+{\lambda }_t^{j,e}}{2}}$$

(79)

Although the critic network is a neural approximation and its gradients may contain slight numerical inaccuracies, averaging the marginal valuations (as formulated in the settlement price equations) effectively mitigates unilateral approximation errors and balances the negotiation. The empirical stability of these derived prices is further examined in the subsequent case studies.

4.2.3. A Deep Reinforcement Learning Framework for Multi-Microgrids Based on CTDE-MASAC

Based on the above principles, the deep reinforcement learning framework for multi-microgrids using CTDE-MASAC is constructed as shown in Figure 4. This framework utilizes the environment module as the data hub to integrate all operational information from the multi-microgrid system and output local observations $s_t^i$ for each DIEM. Based on these local observations, the actor network of each DIEM generates continuous action strategies $a_t^i$. These actions encompass P2P transaction volumes, microgrid power outputs, and energy storage charging/discharging states. Upon receiving the joint actions of all DIEMs, the environment module updates the system state, computes the global reward $r(s_t,a_t)$, and stores the transition tuple $(s,a,r,s_{t+1})$ into the experience replay buffer. During the training phase, the centralized critic network has access to the states and actions of all DIEMs. It evaluates the long-term value of the joint policy by minimizing the soft Bellman residual. Concurrently, each actor network computes its policy gradients and updates its parameters using the critic’s value estimates and an entropy regularization term. Data decoupling through the experience replay buffer and the clipped double-critic mechanism effectively mitigates Q-value overestimation. Meanwhile, target networks smoothly track updates through a soft-update mechanism. Upon completing the training, the framework transitions to the decentralized execution phase. Each DIEM relies exclusively on its local observations and trained actor network to generate real-time dispatch commands independently, eliminating the need for online inter-agent communication. Through structured training iterations, this paradigm combines global guidance during centralized training with autonomous decision-making during decentralized execution, achieving adaptive convergence toward the Nash bargaining equilibrium. To further clarify the implementation process of the proposed CTDE-MASAC framework, the training and decentralized execution procedure is summarized in Algorithm 1.

Algorithm 1. CTDE-MASAC training and decentralized execution procedure

Input: Multi-microgrid environment; agent set $\mathcal{N}$; replay buffer $\mathcal{D}$; batch size $M$; discount factor $\gamma$; soft-update coefficient $\tau$; temperature coefficient $\alpha$.

Output: Trained actor policies ${\pi }_{{\varphi }_i}$ for all DIEM agents.

1.  Initialize actor networks ${\pi }_{{\varphi }_i}$ for all DIEM agents.

2.  Initialize centralized critic networks $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$.

3.  Initialize target critic networks $Q_{{\overline{\theta }}_1}$ and $Q_{{\overline{\theta }}_2},$ and initialize replay buffer $\mathcal{D}$.

4.  for each training episode do

5.   Reset the environment and obtain the initial joint state $s_t$.

6.   for each time step $t$ do

7.    Each DIEM observes the local state $s_t^i$ and generates an action $a_t^i$ using ${\pi }_{{\varphi }_i}$.

8.    Construct the joint action $a_t=[a_t^1,a_t^2,\dots ,a_t^N]$ and execute $a_t$.

9.    Calculate $R_t^i,$ and $r_t$ according to Equations (64)–(66).

10.     Store transition $(s_t,a_t,r_t,s_{t+1})$ in $\mathcal{D}$.

11.     Sample a mini-batch of $M$ transitions from $\mathcal{D}$.

12.     Update $Q_{{\theta }_1}$ and $Q_{{\theta }_2}$ by minimizing the critic loss in Equation (71).

13.     Update each actor policy ${\pi }_{{\varphi }_i}$ according to Equation (73).

14.     Update the temperature coefficient $\alpha$ according to Equation (76).

15.     Softly update $Q_{{\overline{\theta }}_1}$ and $Q_{{\overline{\theta }}_2}$ according to Equation (77).

16.    end for

17. end for

18. During decentralized execution, each DIEM uses $s_t^i$ and ${\pi }_{{\varphi }_i}$ to generate $a_t^i$.

5. Case Study

5.1. Basic Data

A typical industrial park microgrid cluster in the Yangtze River Delta region is selected for the case study. This cluster comprises three distributed, integrated energy microgrids, denoted DIEM 1, DIEM 2, and DIEM 3, each connected to the 10 kV distribution network via a 10 kV/0.4 kV transformer. The energy load and renewable power generation forecast data for each microgrid are based on statistical analyses of actual operational data from the park over the past three years, as shown in Figure 5. The basic market, device, and trading parameters are listed in

Table 2. Basic simulation parameters.

Parameters	Value	Parameters	Value	Parameters	Value
${\lambda }_0^{\mathrm{GCT}}$/(CNY/kWh)	0.15	$\rho$	0.4	$\lambda$	0.2
$k_1$	0.15	$\xi$	0.3	$\delta$	0.25
$k_2$	0.25	$\kappa$	1.05	$k_{\mathrm{chp}}$	0.035 CNY/kWh
$\theta$/(kg·kWh−1)	0.96	$\alpha$	3 × 10−4	$k_{\mathrm{gb}}$	0.02 CNY/kWh
${\omega }_1$	0.2	$\zeta$	0.6	$k_{\mathrm{p}2\mathrm{g}}$	0.015 CNY/kWh
${\omega }_2$	0.25	${\eta }_{\mathrm{ccus}}$	0.9	$a,b,c$	(Coal) 0.8, 0.05, 0.4; (Gas) 0.1, 0.002, 0.03
${\omega }_3$	0.3	$l$	2000 kg

, including carbon trading parameters, green certificate trading parameters, and device operation coefficients.

The software platform uses Windows 11 and Python 3.10. Deep neural networks are built with PyTorch 2.0, and parallel computing acceleration is enabled using an NVIDIA RTX 4060 GPU and CUDA 12.7.

To ensure the reproducibility of the proposed deep reinforcement learning method, the detailed neural network architectures and key training hyperparameters of the CTDE-MASAC algorithm are explicitly summarized in

Table 3. Neural network architecture and hyperparameters of the CTDE-MASAC algorithm.

Parameters	Value	Parameters	Value
Actor Network Architecture	MLP: [State_dim, 256, 256, Action_dim]	Batch Size	4096
Critic Network Architecture	MLP: [State_dim + Action_dim, 256, 256, 1]	Replay Buffer Capacity	100,000
Activation Function	ReLU (Hidden layers), Tanh (actor output)	Target Entropy	$-\dim \left(\mathrm{A}\right)$
Maximum Training Episodes	2000	Discount Factor $\gamma$	0.995
Actor Learning Rate (${\alpha }_{\pi }$)	1 × 10−4	Soft-update coefficient $\tau$	0.005
Critic Learning Rate (${\alpha }_Q$)	1 × 10−3

. Both the actor and critic networks utilize Multi-Layer Perceptron (MLP) architectures with two hidden layers.

Justification of modeling assumptions and key parameters: The parameters utilized in this study are carefully selected to reflect realistic engineering scenarios. The baseline carbon trading price and benchmark green certificate price are set according to the parameter values in Table 2, based on recent historical average clearing prices in the Yangtze River Delta regional pilot carbon market and the national green electricity market in China. Additionally, a fundamental modeling assumption in the cooperative game is the equal market status of the participants. Consequently, the Nash bargaining weights are set symmetrically ($w_1=w_2=w_3=1/3$), assuming that all three industrial park microgrids possess equal bargaining power in the decentralized P2P market without any monopolistic dominance. The capacities of the CHP, P2G, and ESS units are scaled proportionally to the actual peak load demands of typical medium-sized industrial parks to ensure the feasibility of the energy-carbon balancing mechanism.

5.2. Analysis of Algorithm Performance

To verify the adaptability of the proposed centralized training and decentralized execution model for multi-microgrid cooperative games in continuous action spaces, comparative experiments are conducted with three mainstream deep reinforcement learning algorithms: MADDPG, MASAC, and Multi-Agent Proximal Policy Optimization (MAPPO). Figure 6 shows the training reward curves of one representative random seed and is used to illustrate the convergence behavior of different algorithms in the continuous action space. The statistical results over five independent random seeds are reported in

Table 4. Comparison of operational cost, optimality gap, and computation time.

Algorithm	Average Daily Operational Cost (CNY)	Optimality Gap (%)	Computation Time (s)
Centralized MILP	66,837.02	0.00%	124.50
Proposed MASAC	67,673.24 ± 142.50	1.25%	0.12
MADDPG	69,521.15 ± 412.30	4.02%	0.15
MAPPO	68,914.80 ± 385.10	3.11%	0.11

. The curves indicate that the MASAC algorithm demonstrates a clear performance advantage. Compared with the deterministic policy used in MADDPG, MASAC’s stochastic policy provides stronger exploration. Compared to MAPPO’s online policy, the off-policy nature of MASAC enables full reuse of historical experiences from the replay buffer, thereby significantly improving data utilization efficiency. Therefore, MASAC better adapts to the interaction characteristics and cooperative game requirements of multi-microgrids, providing reliable methodological support for coordinated system optimization.

To rigorously evaluate the optimality and statistical robustness of the proposed MASAC framework, a centralized Mixed-Integer Linear Programming (MILP) model with perfect foresight is introduced as the benchmark for the global theoretical optimum. To avoid drawing conclusions from a single training run, all DRL algorithms (MASAC, MADDPG, and MAPPO) are trained and tested across five independent random seeds. Table 4 presents the mean and standard deviation of the daily total operating cost.

To quantitatively highlight the advantages of the proposed approach, the results in Table 4 demonstrate that MASAC significantly outperforms existing DRL methods. Specifically, MASAC reduces the optimality gap to a mere 1.25% relative to the centralized MILP baseline, whereas MADDPG and MAPPO exhibit larger gaps of 4.02% and 3.11%, respectively. Furthermore, the standard deviation of MASAC (±142.50 CNY) is roughly 65% lower than that of MADDPG, proving superior algorithmic stability. The primary advantage of the proposed CTDE-MASAC approach lies in its ability to achieve near-optimal economic dispatch with millisecond-level execution time (0.12 s) in a fully decentralized manner. This completely avoids the heavy communication overhead and strict data privacy issues inherent in centralized optimization methods like MILP, making it highly practical for real-world multi-agent microgrid operations. It should be noted, however, that the centralized MILP serves as a perfect-foresight, in-sample benchmark. Thus, the 1.25% optimality gap demonstrates the algorithm’s convergence capacity under known deterministic conditions, rather than a guaranteed bound for out-of-sample performance.

Regarding agent configuration, the multi-microgrid system comprises $N=3$ agents that are structurally homogeneous in terms of neural network topology and action dimensions, which simplifies real-world decentralized implementation. However, they are highly heterogeneous regarding their physical parameters, including distinct renewable energy penetration rates, industrial/commercial load profiles, and carbon emission boundaries. In terms of scalability, since the trained policies are executed locally by each agent, the online computational complexity scales linearly as $\mathcal{O}(N)$. Although this theoretical $\mathcal{O}(N)$ scalability highlights the architectural advantage of decentralized execution, rigorous empirical testing on large-scale multi-microgrid clusters is reserved for future research.

To rigorously demonstrate the robustness of the proposed CTDE-MASAC framework against source-load uncertainties, an out-of-sample evaluation is conducted. Using Monte Carlo simulations, 10 unseen testing scenarios are generated by injecting random Gaussian noise within a ±15% fluctuation range into the baseline WT, PV, and electrical/thermal/hydrogen load profiles shown in Figure 5. The pre-trained MASAC policies are then executed directly on these 10 unseen scenarios in a decentralized manner without retraining.

The execution results show that the trained agents can dynamically adjust energy storage outputs and P2P trading volumes to absorb stochastic fluctuations. Throughout all out-of-sample testing scenarios, the energy supply–demand balance constraints are satisfied without load shedding. As shown in

Table 5. Out-of-sample testing results across 10 stochastic scenarios.

Scenario	Source-Load Fluctuation	Total Operational Cost (CNY)	Fluctuation vs. Baseline (%)
Baseline	Deterministic (0%)	67,673.24
Scenario 1	Random noise (±15%)	67,125.40	−0.81%
Scenario 2	Random noise (±15%)	68,532.18	+1.27%
Scenario 3	Random noise (±15%)	66,210.55	−2.16%
Scenario 4	Random noise (±15%)	69,815.30	+3.17%
Scenario 5	Random noise (±15%)	67,890.12	+0.32%
Scenario 6	Random noise (±15%)	65,545.80	−3.14%
Scenario 7	Random noise (±15%)	68,102.75	+0.63%
Scenario 8	Random noise (±15%)	67,012.30	−0.98%
Scenario 9	Random noise (±15%)	69,105.45	+2.12%
Scenario 10	Random noise (±15%)	66,515.20	−1.71%

, compared with the baseline deterministic cost of 67,673.24 CNY, the out-of-sample costs range from 65,545.80 CNY to 69,815.30 CNY, with fluctuations between −3.14% and +3.17%. These results indicate that the proposed strategy maintains good generalization capability and robustness under source-load uncertainty.

5.3. Analysis of Optimized Operating Strategies

Figure 7 illustrates the electricity, heat, and hydrogen power supply–demand balance mechanisms for DIEM 1, DIEM 2, and DIEM 3 from a microgrid-level scheduling perspective. At the electric power level, each microgrid matches the electric load through the collaborative use of distributed energy resources, energy storage systems, and P2P transactions. During peak-load periods, local power sources and storage are prioritized for discharge. During off-peak periods, storage charging and power purchases are used for replenishment, thereby achieving a dynamic intra-day power balance. At the thermal energy level, microgrids rely primarily on CHP and GB as heat sources, combining thermal energy storage with P2P heat trading to form a coordinated heating system. In this system, DIEM 2 serves as the primary provider of a stable heat supply, while DIEM 1 and DIEM 3 meet their own heat demand through heat procurement. Overall, thermal output remains stable and closely aligns with the heat demand curve. At the hydrogen energy level, each microgrid constructs a supply–demand system based on P2G, hydrogen storage discharge, and P2P hydrogen trading. DIEM 2 serves as the primary hydrogen seller, while DIEM 1 acts as the primary buyer. DIEM 3 remains approximately balanced after a small initial hydrogen export. The hydrogen power output generally aligns with the hydrogen load curve. Through multi-energy complementarity and P2P transactions, microgrids effectively ensure stable operation and supply–demand balance across electricity, heat, and hydrogen systems.

Figure 8 illustrates the dynamic SOC variations in electrical energy storage in the multi-microgrid system. Between 1:00–10:00 and 15:00–20:00, the SOC ranges from 20% to 90%. Through frequent charging and discharging, the storage system responds rapidly to system power fluctuations and supports intra-day peak shaving and real-time energy balancing.

The interaction characteristics of energy trading power and prices among multi-microgrids are shown in Figure 9. From the perspective of electrical energy trading, the power exhibits high-frequency fluctuations between −100 kW and 100 kW, and the transaction price fluctuates synchronously with time-of-use (TOU) electricity prices. During the peak-load periods of 10:00–15:00 and 18:00–22:00, the P2P electricity trading prices rise toward the time-of-use purchase-price bound while remaining above the feed-in tariff, reflecting the price-guiding role of external electricity tariffs in P2P energy transactions. In terms of thermal energy trading, DIEM 2 consistently exports heat with negative power, whereas DIEM 3 imports heat with positive power. The transaction price fluctuates slightly within the upper and lower heat price limits, indicating a stable supply–demand matching relationship. In terms of hydrogen energy trading, DIEM 2 exports hydrogen with a constant negative power, while DIEM 1 purchases hydrogen with a constant positive power, and the transaction prices adjust smoothly within the upper and lower bounds of the hydrogen price. Overall, electricity, heat, and hydrogen exhibit different interaction characteristics in P2P trading. Electricity trading is most strongly affected by price signals, whereas heat and hydrogen trading show more stable supply–demand structures and pricing mechanisms.

To address potential concerns regarding the marginal values extracted from the neural approximation of the centralized critic network, the P2P settlement prices in Figure 9 are examined empirically. The derived trading prices for electricity, heat, and hydrogen do not exhibit divergent spikes and remain within their corresponding market bounds. These observations indicate that the gradients extracted from the converged centralized critic can provide stable approximate marginal-value signals for P2P settlement pricing.

Furthermore, a quantitative market-clearing check is conducted for decentralized P2P transactions. Based on the bilateral matching rule in Section 4.1.2, the net P2P trading volume among the three DIEMs is approximately zero at each time slot for electricity, heat, and hydrogen. The maximum absolute mismatch is below 10⁻⁶ kW, indicating that the total buying volume equals the total selling volume within numerical tolerance at the settled prices. In addition, the P2P carbon quota and green certificate trading volumes under Scenario 3 sum to zero across the three DIEMs. These results further confirm that the derived settlement prices support stable market clearing.

5.4. Analysis of Economic and Environmental Benefits

To further analyze the effects of the green certificate-tiered carbon trading mechanism on the economic and environmental performance of the multi-microgrid system, three comparative scenarios are established, as shown in

Table 6. Scenario settings.

Scenario	Carbon Trading Market Considered	Green Certificate Trading Market Considered
Scenario 1	No	No
Scenario 2	Yes	No
Scenario 3	Yes	Yes

.

Scenario 1: Neither the carbon trading mechanism nor the green certificate trading mechanism is considered, and only the total operating cost is minimized.

Scenario 2: The tiered carbon trading mechanism is considered, and the carbon trading cost is incorporated into the optimization model.

Scenario 3: The green certificate-tiered carbon trading mechanism is considered, with economic operating costs, carbon trading costs, and green certificate trading costs included simultaneously.

Table 7. Economic and environmental performance under different trading scenarios.

Scenario	DIEM	Carbon Emissions (kg)	P2P Carbon Quota Trading Volume (kg)	P2P Green Certificate Trading Volume (Certificates)	Carbon Trading Cost (CNY)	Green Certificate Trading Cost (CNY)	Total Operating Cost (CNY)
Scenario 1	DIEM 1	8535.47	/	/	/	/	23,599.18
	DIEM 2	7062.54	/	/	/	/	21,850.36
	DIEM 3	5290.89	/	/	/	/	20,120.57
Scenario 2	DIEM 1	6153.64	324.54	/	1030.79	/	24,632.99
	DIEM 2	5505.78	−221.51	/	930.48	/	22,775.33
	DIEM 3	3124.68	−103.03	/	526.58	/	20,645.52
Scenario 3	DIEM 1	5224.16	210.59	846.32	646.57	256.69	24,502.44
	DIEM 2	4463.99	−134.36	−351.29	557.07	212.08	22,619.51
	DIEM 3	2830.50	−76.23	−495.03	348.51	82.21	20,551.29

presents the economic and environmental benefit results of the DIEMs under the three scenarios. Following the introduction of the tiered carbon trading mechanism in Scenario 2, carbon trading costs are higher than in Scenario 1. However, while carbon emissions are reduced by 27.91%, 22.04%, and 40.94%, respectively, the total operating costs increase by only 4.38%, 4.23%, and 2.61%, respectively, reflecting the constraining and guiding role of carbon cost internalization in shaping the system’s low-carbon operation strategies. Scenario 3 introduces a green certificate trading mechanism on top of tiered carbon trading, further reducing each DIEM’s carbon emissions. Although green certificate trading costs are introduced, Scenario 3 further reduces the carbon trading costs of the three DIEMs by 37.27%, 40.13%, and 33.82%, respectively, compared with Scenario 2, and reduces their total operating costs by 0.53%, 0.68%, and 0.46%, respectively. This demonstrates that the green certificate-tiered carbon trading mechanism can constrain carbon emissions through carbon trading and incentivize the use of clean energy through green certificate trading. While achieving deep carbon emission reductions, it effectively mitigates the impact of carbon costs on system economic efficiency, providing robust support for multi-microgrids to realize the synergistic optimization of economic and environmental benefits under low-carbon constraints.

Table 8. Non-cooperative operating costs and Nash bargaining outcomes under Scenario 3.

Metric/Participant	DIEM 1	DIEM 2	DIEM 3	Total Coalition
Non-cooperative Operating Cost (CNY)	26,368.07	24,736.92	22,603.23	73,708.22
Final Cooperative Operating Cost (CNY)	24,502.44	22,619.51	20,551.29	67,673.24
Cooperative Cost Saving (CNY)	1865.63	2117.41	2051.94	6034.98
Final Cost Reduction Rate (%)	7.08%	8.56%	9.08%	8.19%

reports the non-cooperative operating costs and Nash bargaining outcomes under Scenario 3. To obtain the disagreement points, each DIEM is optimized independently under the same load profiles, renewable energy forecasts, device parameters, and external carbon and green certificate market settings as those adopted in Scenario 3. At the same time, all inter-DIEM P2P trading variables are fixed to zero. The resulting non-cooperative operating costs for DIEM 1, DIEM 2, and DIEM 3 are 26,368.07 CNY, 24,736.92 CNY, and 22,603.23 CNY, respectively, giving a total non-cooperative operating cost of 73,708.22 CNY. Under cooperative operation based on the Nash bargaining mechanism, the operating costs of DIEM 1, DIEM 2, and DIEM 3 decrease to 24,502.44 CNY, 22,619.51 CNY, and 20,551.29 CNY, respectively, and the total coalition operating cost decreases to 67,673.24 CNY.

The corresponding cooperative cost savings of DIEM 1, DIEM 2, and DIEM 3 are 1865.63 CNY, 2117.41 CNY, and 2051.94 CNY, representing cost reduction rates of 7.08%, 8.56%, and 9.08%, respectively. At the coalition level, cooperative operation yields a total cost saving of 6034.98 CNY and an overall cost reduction rate of 8.19%. In the proposed implementation, the Nash bargaining allocation is realized through the negotiated multi-commodity P2P settlement prices. The associated bilateral settlement payments are included in the comprehensive P2P trading cost $C_{\mathrm{total},i}^{\mathrm{P}2\mathrm{P}}$ and are therefore already reflected in the final cooperative operating cost of each DIEM. No additional ex-post lump-sum side-payment variable is introduced; therefore, separate side-payment values are not reported.

According to the payoff definitions $u_i=-C_i^{\mathrm{DIEM}}$ and $d_i=-C_i^{\mathrm{DIEM},\mathrm{non}}$, the individual rationality condition $u_i\ge d_i$ is equivalent to $C_i^{\mathrm{DIEM}}\le C_i^{\mathrm{DIEM},\mathrm{non}}$. As shown in Table 8, the final cooperative operating cost of each DIEM is lower than its corresponding independently optimized non-cooperative operating cost, and $u_i-d_i=C_i^{\mathrm{DIEM},\mathrm{non}}-C_i^{\mathrm{DIEM}}>0$ holds for all three DIEMs. Therefore, the numerical results directly verify the individual rationality condition of the Nash bargaining framework. Although equal bargaining weights are adopted, the realized cost savings differ because the three DIEMs have different operating characteristics and feasible P2P trading strategies.

Beyond the verified cooperative cost savings and individual rationality results, the proposed framework also has practical implications for industrial microgrid clusters. First, the green-carbon offsetting mechanism provides an additional value-conversion pathway between renewable energy consumption and carbon compliance, thereby reducing the economic pressure associated with carbon trading. Second, P2P cooperative trading enables local surplus–deficit complementarity among DIEMs, thereby reducing dependence on the external grid during peak periods. Third, the tiered carbon trading mechanism provides a flexible market signal to guide low-carbon operations. These findings indicate that the proposed framework can support coordinated economic and low-carbon operation in regional multi-microgrid systems.

In the sensitivity analysis, the baseline operating point is set according to the parameters in Table 2. The carbon quota coefficient and green certificate quota coefficient are varied around their baseline values, while the remaining parameters are kept unchanged. The resulting carbon price and green certificate price are expressed in CNY/kg and CNY/kWh, respectively. To investigate the effects of carbon quota constraint intensity and renewable energy consumption obligations on the system’s environmental and economic performance, a sensitivity analysis is conducted on the carbon quota coefficient and the green certificate quota coefficient, as shown in Figure 10. As shown in Figure 10a, the carbon price increases as the carbon quota coefficient decreases, and decreases as the green certificate quota coefficient increases. This is because when carbon quotas are tightened, high-emission DIEMs must purchase more carbon quotas, whereas the total volume of sellable carbon quotas from low-emission DIEMs decreases accordingly; this creates a market pattern of increased demand and shrinking supply for carbon emission rights, thereby driving up the carbon price. When the carbon quota coefficient remains constant, an increase in the green certificate quota coefficient raises the required green certificate purchase volume for the coal-fired generators within each DIEM, elevating generation costs and reducing their power outputs; this leads to a decline in the overall demand for carbon emission rights, ultimately causing the carbon market price to drop. As shown in Figure 10b, the green certificate price rises as the green certificate quota coefficient increases, and falls as the carbon quota coefficient decreases. This is because an increase in the green certificate quota coefficient significantly boosts the green certificate demand from coal-fired generators in each DIEM, while the total supply of green certificates available from renewable energy generators decreases; the tightening supply–demand balance directly pushes up the market price. Conversely, tightening the carbon quota coefficient suppresses the equilibrium output of coal-fired units in each DIEM while increasing renewable energy output. This increases the supply of green certificates and reduces demand, ultimately leading to a decline in the green certificate market price.

6. Conclusions

This paper proposes a CTDE-MASAC-based cooperative-game low-carbon optimal operation method for multi-microgrids. The proposed method integrates energy-carbon-green certificate P2P coordinated trading, green-carbon offsetting-based dual incentives, and Nash bargaining-based cooperative surplus allocation. A centralized training and decentralized execution framework is further developed to address cooperative games in continuous action spaces. The main conclusions are as follows:

(1)

The proposed CTDE-MASAC framework achieves near-optimal and stable decision-making performance in multi-microgrid cooperative operation. Compared with the centralized MILP benchmark, the proposed MASAC method achieves an average daily operating cost of 67,673.24 CNY with an optimality gap of only 1.25%, while reducing the computation time to 0.12 s. Compared with MADDPG and MAPPO, MASAC achieves lower operating costs and a smaller standard deviation across independent random seeds, indicating better training stability. In addition, the out-of-sample Monte Carlo tests show that the trained policies maintain supply–demand balance without load shedding, and the daily operating cost fluctuates within ±3.2%. These results verify the adaptability and robustness of the proposed method under source-load uncertainty.

(2)

The green certificate-carbon trading dual-incentive mechanism effectively strengthens the coupling between renewable energy consumption and carbon emission reduction. Compared with the scenario that considers only tiered carbon trading, the proposed green-carbon offsetting mechanism further reduces the carbon trading costs for DIEM 1, DIEM 2, and DIEM 3 by 37.27%, 40.13%, and 33.82%, respectively. Meanwhile, the total operating costs decreased by 0.53%, 0.68%, and 0.46%, respectively. These results indicate that the proposed mechanism can convert the value of green certificates into carbon-offset benefits, reduce the economic pressure associated with carbon compliance, and promote the coordinated optimization of economic and environmental benefits.

(3)

The Nash bargaining-based P2P cooperative trading model improves coalition efficiency and individual rationality. Under the energy-carbon-green certificate coordinated trading mechanism, electricity, heat, hydrogen, carbon quotas, and green certificates are jointly traded among DIEMs, enabling complementarity between surpluses and deficits across multiple energy carriers and market resources. Under Nash bargaining-based cooperative operation, the operating cost of each DIEM is lower than its corresponding non-cooperative operating cost, with reductions of 7.08%, 8.56%, and 9.08%, respectively. The total coalition operating cost decreases from 73,708.22 CNY to 67,673.24 CNY, yielding a cooperative cost saving of 6034.98 CNY and an overall reduction of 8.19%. These results numerically verify that all three DIEMs satisfy the individual rationality condition and demonstrate the economic effectiveness of the proposed cooperative game mechanism.

Although the proposed framework demonstrates good economic, low-carbon, and algorithmic performance, this study is still limited by the three-DIEM case setting, homogeneous-agent structure, and equal bargaining weights. Future work will extend the framework to larger-scale heterogeneous multi-microgrid systems and further consider communication delays, privacy protection, and real-time market clearing in practical applications.