A Bidding Strategy for Power Suppliers Based on Multi-Agent Reinforcement Learning in Carbon–Electricity–Coal Coupling Market

Abstract

The deepening operation of the carbon emission trading market has reshaped the cost–benefit structure of the power generation side. In the process of participating in the market quotation, power suppliers not only need to calculate the conventional power generation cost but also need to coordinate the superimposed impact of carbon quota accounting on operating income, which causes the power suppliers a multi-time-scale decision-making collaborative optimization problem under the interaction of the carbon market, power market, and coal market. This paper focuses on the multi-market-coupling decision optimization problem of thermal power suppliers. It proposes a collaborative bidding decision framework based on a multi-agent deep deterministic policy gradient (MADDPG). Firstly, aiming at the time-scale difference of multi-sided market decision making, a decision-making cycle coordination scheme for the carbon–electricity–coal coupling market is proposed. Secondly, upper and lower optimization models for the bidding decision making of power suppliers are constructed. Then, based on the MADDPG algorithm, the multi-generator bidding scenario is simulated to solve the optimal multi-generator bidding strategy in the carbon–electricity–coal coupling market. Finally, the multi-scenario simulation based on the IEEE-5 node system shows that the model can effectively analyze the differential influence of a multi-market structure on the bidding strategy of power suppliers, verifying the superiority of the algorithm in convergence speed and revenue optimization.

1. Introduction

Under the background of the establishment of the dual-carbon strategic goal, the construction process of the carbon trading market has been fully accelerated. At the same time, under the background of a high proportion of renewable energy grid connections, the correlation of price fluctuation between different markets has been enhanced, and the cross-influence of policy constraints has intensified. This presents a market pattern of multi-market coordinated operation, such as the electricity spot market, carbon trading market, and upstream coal market [1]. Power suppliers are faced with the problem of collaborative optimization of decision making in the multi-side market trading environment. The traditional single-market bidding strategy has been difficult to adapt to the bidding decision-making needs of power suppliers under the new power market system [2]. It is urgent to analyze the impact of multi-dimensional market structure on bidding clearing results and explore the coordination mechanism of the bidding decision making of power suppliers under the carbon–electricity–coal coupling environment so as to optimize the allocation of resources in the power market and improve the operating income of power suppliers [3].

The complexity of the market mechanism promotes the evolution of a bidding strategy from single-market equilibrium analysis to multi-market-coupling modeling. Early research on bidding strategies has mainly focused on the electricity spot market. Based on the short-term bidding game model, reference [4] verified the effectiveness of the marginal cost pricing mechanism of thermal power plants and clarified the bidding optimization path under the equilibrium of supply and demand in the spot market. With the promotion of the carbon trading mechanism, some scholars have studied the bidding strategy of electricity–carbon dual-market linkage. Reference [5] constructed a carbon–electricity collaborative demand-response model and revealed the marginal adjustment effect of carbon price fluctuation on generator output decision. By introducing a carbon tax constraint into the day-ahead market, the literature [6] realized the revenue co-optimization of thermal power unit energy and reserve service. In reference [7], the fuel procurement market was included in the analysis framework, and the risk hedging mechanism under the synergistic effect of fuel cost and electricity price fluctuation was clarified. However, the above research was limited to a single market or a specific dual-coupling market, and it is difficult to reflect the dynamic transmission effect of multi-market constraints on the bidding strategy of power suppliers, which may lead to the strategic loss risk of power suppliers in the real multi-market-coupling environment.

In order to solve the bidding strategy of power suppliers, agent-based algorithms have become a powerful tool for research in this field because of their ability to model and simulate the dynamic evolution of the market and the bidding behavior of power suppliers [8]. From the perspective of technological evolution, reinforcement learning based on the value function method has experienced an iterative leap from basic algorithms to deep architecture. In the early research, Q-learning was used to construct a bidding game model of discrete state-action space [9,10], but it was difficult to realize the representation of a high-dimensional market state. The deep Q-network (DQN) algorithm solves the problem of strategy generalization of Q-learning in high-dimensional state space by introducing a deep neural network and experience playback mechanism [11,12]. The double deep Q-network (DDQN) further decouples action selection and value evaluation, which effectively alleviates the overestimation bias in the solution of multi-generator bidding equilibrium [13,14]. The policy function law opens up a new path through continuous-action space modeling: deep deterministic policy gradient (DDPG) and its improved algorithm map the bidding strategy to the probability distribution space, which solves the discrete decision constraints of the traditional value function method [15,16,17]. However, the existing algorithms face the challenges of difficult cross-market dynamic interaction capture and high complexity of multi-agent game equilibrium solution in the multi-market-coupling environment, which leads to limited strategy generalization ability and decision accuracy, and further research on strategy stability and convergence efficiency in continuous-action space is still needed.

In order to solve the existing problems, this paper proposes a bidding decision optimization model for power suppliers based on a multi-agent deep deterministic strategy gradient algorithm in a carbon–electricity–coal multi-market-coupling environment. Firstly, the time scale difference of carbon–electricity–coal market decision making is analyzed, and the coordination mechanism of the carbon–electricity–coal coupling market decision-making cycle is established. Secondly, the upper-level day-ahead electricity spot market bidding model is established with the goal of maximizing the profit of power suppliers, and the lower-level day-ahead electricity spot market clearing model is constructed. Then, based on the MADDPG algorithm, the competition scenario of multi-generator entities in the power market is simulated. Finally, in the simulation environment of the IEEE-5 node system [18], the thermal power producer is taken as the research object, and several examples are set up to reveal the “low-carbon priority” characteristics of power system scheduling in the carbon trading market environment and the bidding process considering the role of the fuel market in improving the market competitiveness of power producers and also to verify that the decision optimization model proposed in this paper can bring higher benefits to power producers.

2. Carbon–Electricity–Coal Coupling Market Decision-Making Cycle Coordination

2.1. Analysis of Decision-Making Cycle of Power Suppliers Participating in Multi-Sided Market

Under the background of multi-policy coordination and control, such as power market reform and carbon trading, the operation of power suppliers faces the multi-objective coordination challenge of coordinating fuel cost control, power revenue optimization, and carbon quota balance: (1) Based on the power demand forecast, the monthly coal procurement plan of the upstream coal market is formulated, and its procurement cost directly affects the bidding strategy of the power market through the cost transmission mechanism. (2) By participating in the day-ahead bidding of the downstream power market to obtain the time-sharing power generation plan, the output adjustment will change the actual carbon emission intensity. (3) In the horizontal carbon trading market, power suppliers obtain the pre-allocation carbon quota at the beginning of the period, and at the end of the period, it is adjusted retroactively according to the actual power generation [5]. Through the cross-cycle policy design, the carbon market and power production form a feedback loop. The operation mode, decision-making cycle, management system, and policy direction of the coal market, the power market, and the carbon trading market are not the same, but carbon emissions is a transmission medium throughout the whole chain of fuel procurement, power production, and quota clearing, resulting in a coupling relationship between the upstream and downstream markets and the carbon trading market. The specific coupling relationship is shown in Figure 1.

The coupling relationship between the power market, the coal market, and the carbon trading market causes the power suppliers the problem of multi-time-scale coordinated bidding decision making in the multi-sided market operation. On the one hand, to avoid the influence of uncertain factors such as transportation delay and fuel price fluctuation on the cost of electricity production, the power suppliers need to predict the trend of the coal price and sign the monthly fuel procurement agreement before the monthly cycle, while the main power of the electricity is determined by the delivery of the day-ahead market on the day before the day of power operation, forming the time mismatch between fuel procurement and power decision making. On the other hand, the generation plan formed by day-ahead bidding needs to be implemented immediately, but the final accounting of carbon quotas needs to be delayed until the end of the monthly cycle, resulting in a vertical separation of the timing of market clearing and policy compensation. This multi-stage time-scale difference across month-ahead, day-ahead, and month-after leads to the need for power suppliers to coordinate fuel costs, power benefits, and policy risks under asymmetric timing constraints, which greatly increases the complexity of operation coordination. To this end, this paper proposes a carbon–electricity–coal coupling market decision-making cycle coordination mechanism to coordinate the operation decisions of power suppliers with multi-stage time scales in a complex market environment.

2.2. Carbon–Electricity–Coal Coupling Market Decision-Making Cycle Coordination Model

2.2.1. Coordination of Monthly Coal Market Cost Decision Making

At present, the electricity spot market is a game market with incomplete information. It is difficult for generators to obtain the unit quotation information of other generators, and it is difficult to obtain the maximum profit according to marginal cost quotation. Therefore, power suppliers need to reduce the upstream power generation cost as much as possible and take appropriate pricing strategies to obtain higher profits. This paper takes the example of coal-fired power plants to illustrate, and most of its day-ahead electricity spot market decisions use quadratic curve-fitting unit power generation cost as a decision reference [7]:

$$Cos{t}_{g,t}^G=(a_{\lambda }{P}_{g,t}^2+b_{\lambda }{P}_{g,t}+c_{\lambda })$$

(1)

In the formula, the winning power quantity of the generator in the period is P_g,t; a_λ, b_λ, and c_λ are the secondary term, primary term, and constant term coefficient of the generation cost, respectively.

However, the above traditional power generation cost accounting method based on the historical calculation data to fit the quadratic curve not only has a fixed error, but it has difficulty in reflecting the price changes [19] in the upstream coal market in real time. In order to fully tap the cost advantage of generators and transform it into competitiveness in downstream power market bidding, it is proposed to estimate the power generation cost by using the standard coal consumption per generation unit and the daily updated average unit price of standard coal in the furnace and dynamically update the power generation cost based on the average unit price of standard coal in the furnace. The specific method is as follows:

$$Cos{t}_{g,t}^G=k_T{p}_d{\psi }_t{P}_{g,t}$$

(2)

where k_T represents the cost conversion coefficient; p_d is the average unit price of standard coal in the furnace; p_dψ_t is the unit fuel cost of power generation; ψ_t represents the unit standard coal consumption of power generation in the t period of the unit, and the unit is g/kWh. The real-time coal consumption of power production equipment is difficult to directly measure, so the unit of the coal consumption of power generation is usually fitted by experimental method with the help of a quadratic function. The formula is as follows:

$${\psi }_t=a_T{P}_{\mathrm{g},t}^2+b_T{P}_{\mathrm{g},t}+c_T$$

(3)

where a_T, b_T, and c_T are the quadratic term, primary term, and constant term coefficients of the standard coal consumption curve of power generation unit, respectively.

2.2.2. Coordination of Monthly Carbon Trading Market Decision Cycle

The approved cost of the carbon emission trading market of the power suppliers is affected by the monthly electricity consumption and the original carbon quota at the beginning of the contract performance period, and the trading cycle belongs to the monthly settlement mechanism, which is contrary to the day-before auction clearance decision-making cycle. This paper considers the synergistic factors [20] of the decision cycle of the carbon trading market and the power market and determines the connection method of carbon trading policy simulation and the power market.

The carbon trading center calculates the initial carbon quota and pre-issues the carbon quota. The regional power dispatching center calculates the initial free carbon quota of the generators based on the baseline method and realizes the low-carbon clearing of the electric–carbon coupling market by combining the carbon emission intensity of the generators of each power supplier with the daily quotation information of the generators. The initial carbon emission quota, carbon emission, and carbon quota trading volume of generator g are calculated as shown in Equation (4):

$$\left\{\begin{array}{l}{M}_{g,t}^T=P_{g,t}{\delta }_T\\ M_{g,t}^G=P_{g,t}{\delta }_G\\ M_{g,t}=P_{g,t}({\delta }_G-{\delta }_T)\end{array}\right.$$

(4)

where $M_{g,t}^T$ is the inherent carbon quota of generator g, and δ_T is the inherent carbon emission quota per unit of power generation. $M_{g,t}^G$ is the carbon emission of generator g during period t, and δ_G is the carbon emission coefficient per unit power generation, that is, carbon emission intensity.

Carbon trading costs are related to carbon emissions. In view of the lack of a differentiated regulation mechanism for emission reduction incentives and violation punishment in traditional carbon emission methods, which is likely to lead to the lack of incentive for power suppliers to reduce emissions and the failure of market regulation, this paper proposes a stepped interval improvement based on carbon trading cost to achieve accurate guidance for power suppliers’ emission reduction behavior and efficient allocation of resources. The improved stepped carbon trading model is shown in Equation (5).

$${Cost}_{g,t}^{\mathrm{CO}2}=\left\{\begin{array}{l}{p}_{\mathrm{CO}2}(1+3\upsilon )(M_{g,t}+3l),-4l^{-}<M_{g,t}<-3l^{-}\\ p_{\mathrm{CO}2}(1+2\upsilon )(M_{g,t}+2l),-3l^{-}<M_{g,t}<-2l^{-}\\ p_{\mathrm{CO}2}{M}_{g,t}\theta ,-l<M_{g,t}<l^{+}\\ p_{\mathrm{CO}2}\left[k+(k-1)\theta \right]l^{+}+p_{\mathrm{CO}2}\left[1+k\theta \right](M_{g,t}-k{l}^{+}),k{l}^{+}<M_{g,t}\le (k+1)l^{+},k=1,2,3...\end{array}\right.$$

(5)

In the formula, ${Cost}_{g,t}^{CO2}$ is the carbon trading cost of generator g; p_CO2 is the carbon price; υ and θ are the reward coefficient and penalty coefficient of carbon trading, respectively; and l⁺ and l^- are the positive and negative interval carbon trading lengths, respectively.

2.2.3. Day-Ahead Decision-Making Process of Carbon–Electricity–Coal Coupling Market

Based on the coal market cost guidance and carbon trading volume calculation model, the decision-making cycle of the carbon–electricity–coal coupling market is herein coordinated, and the multi-sided market decision making is coordinated and reflected in the decision making of the electricity spot day-ahead market, which is helpful for power suppliers to optimize the bidding decision making of the power market according to the generation cost and carbon transaction cost so as to achieve the maximum profit. The market organization process based on carbon–electricity–coal multi-sided market coupling constructed in this paper is shown in Figure 2.

3. Bidding Decision Optimization Model of Power Suppliers in Carbon–Electricity–Coal Coupling Market

3.1. Bidding Decision Optimization Model of Power Suppliers

In order to focus on the optimization of quotation strategy of power suppliers in the day-ahead power market, this paper temporarily does not consider the impact of market transaction process on the error of user-side load declaration and instead only considers the day-ahead electricity energy market in the power spot market.

In this paper, it is assumed that the generators owned by the power suppliers are all thermal generators, and the purpose of maximizing the total profit of power generation is to purchase coal in advance of each month, modify the maximum unit output within the month, calculate the carbon trading cost in real time, and update the bidding decision-making scheme dynamically. The power generation income of the power suppliers in the multi-market-coupling environment can be expressed as follows:

$$\underset{{\phi }_{g,d},[a_{g,t},P_{g,t}^o]}{\max }{F}_{g,d}={\displaystyle \sum _{t=1}^T{\rho }_{g,t}{P}_{g,t}-Cos{t}_{g,t}^{\mathrm{CO}2}(P_{g,t})}-Cos{t}_{g,t}^G(P_{g,t})$$

(6)

Among them, φ_g,d is the maximum output of the D-day reporting unit; a_g,t and $P_{g,t}^o$ are the bidding curve parameters of the unit; F_g,d is total generation profit on D-day; T is the bidding cycle of D-day, and t is the bidding period; ρ_g,t and P_g,t are the clearing price and clearing power of marginal node in time period t, respectively; ${Cost}_{g,t}^{CO2}$ is carbon transaction cost; ${Cost}_{g,t}^G$ is the cost of power generation.

Generators generally use the “stepped multi-stage quotation” mode to design the quotation strategy. In this paper, the a_g,t times coefficient of the ladder quotation is used for quotation:

$$\left\{\begin{array}{l}{P}_{\min }\le P_{g,t}^b\le P_{\max },b\in \mathrm{B}=\{1,2,3\}\\ P_{g,t}={\displaystyle \sum _{b=1}^B{P}_{g,t}^b}\\ p_{g,t}^1=p_{g,t}\\ p_{g,t}^b=(1+a_{g,t})p_{g,t},b\in \mathrm{B}=\{2,3\}\end{array}\right.$$

(7)

where $P_{g,t}^b$ is the bid-winning electricity quantity of the generator g in the b capacity segment during the period t. The callable capacity of the generator is divided into three sections. In order to ensure the successful bidding of the unit, the first quotation coefficient is set to 1, and the second and third sections of each capacity [$P_{g,t}^{b-1}$, $P_{g,t}^b$] corresponding to the quote are $p_{g,t}^b$, which is composed of the unit power generation cost p_g,t of the generator and the quotation coefficient $a_{g,t}$.

3.2. Day-Ahead Electricity Spot Market Clearing Model

The dispatching agency usually constructs the power market clearing model based on the declared data of the generation side and maximizes social welfare and determines the generator start–stop plan together with the security constrained unit commitment (SCUT) model. Using the security constrained economic dispatch (SCED) to allocate the clearing electricity and finally combining the power transfer factor and Lagrange multiplier constrained by the power flow to determine the marginal node price, the specific clearing model is as follows [20]:

The SCUC model determines the generator start–stop combination:

$$\min {\displaystyle \sum _{g=1}^N{\displaystyle \sum _{t=1}^T\left\{\left.{\displaystyle \sum _{b=1}^B{p}_{g,t}^b{P}_{g,t}^b}+C_{g,t}^U+C_{g,t}^K\right\}\right.}}$$

(8)

In the formula, $P_{g,t}^b$ is the bid power of the generator g in the period t, $p_{g,t}^b$ is the quotation of the corresponding power segment, $C_{g,t}^U$ is the cost of generator start-up, and $C_{g,t}^K$ is the cost when the unit load power is 0.

The power balance and start–stop constraint are as follows:

$${\displaystyle \sum _{g=1}^N{P}_{g,t}{u}_{g,t}}=P_{D,t}$$

(9)

where u_g,t represents the operating state of generator g during the time period t; P_D,t is the system power load demand in time period t.

The unit power upper and lower limit constraints are as follows:

$$P_{g,\min }\le P_{g,t}\le P_{g,\max }$$

(10)

where P_g,max is the maximum output of the generator g, and P_g,min is the minimum output of the generator g.

The unit climbing constraints are as follows:

$$\left\{\begin{array}{l}{P}_{g,t}-P_{g,t-1}\le \Delta P_g^U{u}_{g,t-1}+P_{g,\min }(u_{g,t}-u_{g,t-1})+P_{g,\max }(1-u_{g,t})\\ P_{g,t-1}-P_{g,t}\le \Delta P_g^D{u}_{g,t-1}-P_{g,\min }(u_{g,t}-u_{g,t-1})+P_{g,\max }(1-u_{g,t-1})\end{array}\right.$$

(11)

where Δ$P_g^U$ is the maximum climbing rate of generator g, and Δ$P_g^D$ is the maximum downhill rate.

Consider branch power flow constraints:

$$\left\{\begin{array}{l}{F}_l-F_l^M\le 0\\ -F_l-F_l^M\le 0\end{array}\right.$$

(12)

where F_l is the power flow of the branch number l, and $F_l^M$ is the limit value of the power flow of the branch number l.

The SCED model determines the generator start–stop combination to calculate the clearing power of each generator, and the objective function is the maximum economic benefit of ISO, which is as follows:

$$\min {\displaystyle \sum _{g=1}^N{\displaystyle \sum _{t=1}^T{\displaystyle \sum _{b=1}^B{p}_{g,t}^b{P}_{g,t}^b}}}$$

(13)

The constraints of the economic dispatching model are power constraints, slope constraints, and line flow constraints.

$$\left\{\begin{array}{l}{\displaystyle \sum _{g=1}^N{P}_{g,t}}=P_{D,t}\\ P_{g,\min }\le P_{g,t}\le P_{g,\max }\\ P_{g,t}-P_{g,t-1}\le \Delta P_g^U\\ P_{g,t-1}-P_{g,t}\le \Delta P_g^D\\ F_l-F_l^M\le 0\\ -F_l-F_l^M\le 0\end{array}\right.$$

(14)

After the SCUC and SCED market clearing, each power generator will obtain the clearing price and the clearing electricity. The clearing price is calculated by the marginal node price in this paper:

$${\rho }_{k,t}={\lambda }_t-{\displaystyle \sum _{l=1}^L({\tau }_{l,t}^{\max }-{\tau }_{l,t}^{\min })G_{l-k}}$$

(15)

where λ and τ are Lagrangian multipliers of the power balance constraint and power flow constraint, respectively, and G represents the power transfer distribution factor of the generator.

4. Multi-Generator Bidding Strategy Based on MADDPG Algorithm

4.1. Bidding Strategy Design of Multi-Generator Market

In order to simulate the bidding decision-making process of multi-generators participating in the power market, the generator is modeled as an agent, and the bidding behavior of the generator is described as a Markov game (MG) process. The specific definition is as follows:

Environment: This paper constructs the trading environment of the electricity spot market according to the implementation rules of spot electricity energy trading in the Guangdong power market (2022). The power suppliers formulate the bidding strategy through three-stage bidding, and the ISO clears the clearing price and clearing electricity of each power supplier through the market clearing model and feeds back the private information to the corresponding power supplier.

State: The state space of the agent is set to the load information published by the ISO in the current period and the private information of the previous period, that is, the clearing electricity quantity and the clearing electricity price. Among them, the load of each period is directly published by the ISO, and the private information is calculated and fed back to the agent by the ISO through the clearing program.

Action: The action domain of the multi-agent is set to the quotation coefficient a_g,t of the second and third sections of the quotation curve.

Reward: The reward of the agent is defined by Equation (6); that is, the agent calculates the reward with the goal of maximizing the profit of the power supplier and updates the network parameters.

4.2. Multi-Agent Deep Deterministic Gradient Strategy Algorithm

The multi-agent deep reinforcement learning (MADRL) algorithm was gradually used to solve the bidding problem of multi-generator market with complex market rules and random market participants strategies due to its good simulation ability of multi-market players. In this paper, combined with MADRL and DDPG algorithms, a MADDPG algorithm based on the “centralized training, decentralized execution” framework is proposed. The deep neural network is used to realize the strategy learning of the deterministic strategy gradient and solve the problem of the multi-generator participating in market bidding strategy.

The networks of policy function μ (s|θ^μ) and Q-value function Q (s,a|θ^Q) are fitted based on different deep neural networks. The parameter of policy function μ (s|θ^μ) is θ^μ, and the parameter of value network function Q (s,a|θ^Q) is θ^Q [21].

$$Q^{\mu }(s_t,a_t)=E[r_t+\gamma Q^{\mu }(s_{t+1},a_{t+1})]$$

(16)

$${\nabla }_{{\theta }^{\mu }}J({\mu }_g)=E_{s,a}[{\nabla }_{{\theta }_g}{\mu }_g(a_g\left|s_g\right.){\nabla }_{a_g}{Q}_g^{\mu }(s,a)\left|a_g=\mu (s_g)\right.]$$

(17)

where μ (s_g) represents the current policy function, and s and a represent the joint state and joint action vector of multiple agents, respectively; that is, s = (s, s₂,…, s_N), a = (a₁, a₂,…, a_N).

The increase in the number of agents in the environment will exacerbate the difficulty of policy convergence. In this paper, the stability of the algorithm is improved by using the master–target network architecture: the main network generates actions and evaluates returns in real time, while the target network’s updates are delayed based on the main network parameters, and it gradually approximates the optimal value through linear interpolation, where the loss function of the value function main network can be defined as follows:

$$\left\{\begin{array}{l}L({\theta }^Q)=E_{s,a,r,s\prime }[{(Q(s,a\left|{\theta }^Q\right.)-y)}^2]\\ y=r_t+\gamma Q\prime (s\prime ,a\prime \left|{\theta }^{Q\prime }\right.)\left|{\mathit{a}}_g^{\prime }=\mu \prime (s_g^{\prime })\right.\end{array}\right.$$

(18)

where the main network parameter of the value function is θ^Q, r_t is the reward value of the environment feedback to the agent, γ is the discount coefficient, the target network parameter of the value function is θ^Q′, and the target network output value of the value function is expressed as Q′ (s′, a′, and θ^Q′). The target network of the policy function is μ′ ($s_g^{\prime }$), s_g′ represents the state vector at the next moment, and a_g′ represents the action vector at the next moment.

We can update the parameters of the policy function master network as well as the value function master network by Equation (19).

$$\left\{\begin{array}{l}{\theta }^{\mu }={\theta }^{\mu }+l{r}_a\nabla J\\ {\theta }^Q={\theta }^Q-l{r}_q\nabla L({\theta }^Q)\end{array}\right.$$

(19)

where l_ra and l_rq represent the learning rate of the policy network and the value network, respectively. The target network parameters of the policy function and the target value network parameters of the value function are soft-updated by Equation (20), and τ is the update rate.

$$\left\{\begin{array}{l}{\theta }^{\mu \prime }={\theta }^{\mu }\tau +(1-\tau ){\theta }^{\mu \prime }\\ {\theta }^{Q\prime }={\theta }^Q\tau +(1-\tau ){\theta }^{Q\prime }\end{array}\right.$$

(20)

4.3. Multi-Generator Bidding Strategy Based on MADDPG

The bidding strategy solving framework based on MADDPG is shown in Figure 3. Each generator as an independent agent first observes the current state s_t to formulate a strategy π = (a_t,|s_t). ISO collects the bidding information reported by each agent for market clearing and feeds back the clearing results to each agent. The MADDPG-based generator bidding strategy optimization algorithm is as follows:

(1)

The parameters of the main value function network, the target value function network, the main policy function network, and the target strategy function network are initialized as θ^Q, θ^Q′, θ^μ, and θ^μ′, respectively. The number of iterations is set to M, and the time step is set to T;

(2)

Initialize the number of training iterations k = 0, and set the experience pool overflow flag;

(3)

The initial state of the random agent and the time interval of the initial iteration rounds is t = 0;

(4)

According to the current state s_t, the action of each agent is output through the main network of the policy function a_t= (a_1,t, a_2,t,…,a_N,t). ISO calculates the clearing price and clearing power of each agent based on the market clearing program. Each agent calculates its own instant reward value according to Equation (6). Then, update the status value of the agent in the next period s_t+1 = (s_1,t+1, s_2,t+1,…, s_N,t+1);

(5)

Store experiences (s_t, a_t, r_t, s_t+1) in the experience pool and number them.

(6)

Check whether the current experience number is greater than the experience pool overflow flag. If the experience pool overflow flag is greater than the experience pool overflow flag, select Batch size samples at random to execute the training function. Otherwise, directly perform step 8;

(7)

Update the main network parameters of the policy function and the main network parameters of the value function according to Equation (19). After the update, the target network parameters of the policy function and the target network parameters of the value function are soft-updated according to Equation (20);

(8)

Let t = t + 1, and determine whether the number of iterations exceeds step T; if so, go to the next step; otherwise, go to step 4;

(9)

Let k = k + 1, and judge whether the number of iterations exceeds the total number of iterations. If so, end the iteration process and output the final clearing price and clearing power of each agent and save the parameters of the policy function model; otherwise, turn to step 3.

5. Results and Discussion

Through typical experimental cases, this paper studies the influence of multi-market structure on the bidding strategy of power suppliers and verifies the effectiveness and superiority of MADDPG algorithm in solving the bidding decision optimization model of multi-power suppliers in carbon–electricity–coal coupling environment. Based on the IEEE-5 node system, the bidding game behavior of multi-power suppliers is simulated. The system topology is shown in Figure 4a. The 24 h load curve and the parameters of the generator set are from the PJM website [22]. The 24 h load curve is shown in Figure 4b. Each node is configured with a power supplier, and each power supplier has a thermal generator. The upper limit of the bidding coefficient is set to 0.5, and the lower limit is set to −0.2. The learning rate of the critic network in the MADDPG algorithm is set to be higher than that of the actor network, and the values are taken as 1 × 10⁻³ and 1 × 10⁻⁴, respectively. The purpose is to provide accurate Q-value estimation through fast critic update and to ensure the stability of the strategy through conservative actor update. Assuming that the system is not blocked, the technical parameters of the generator are shown in

Table A1. Technical coefficient of generator.

Unit Number	a (CNY/MWh2)	b (CNY/MWh)	${\mathit{P}}_{\mathrm{g}}^{\min }$ (MW)	${\mathit{P}}_{\mathrm{g}}^{\max }$ (MW)	Ramp Rate (MW/min)	kT
G1	0.36	360	10	120	50	-
G2	0.42	360	10	130	50	-
G3	0.33	345	10	150	50	-
G4	0.39	360	10	110	50	1.5
G5	0.48	330	10	150	50	-

of Appendix A, the carbon emission information of the generator is shown in

Table A2. Generator unit carbon emission parameters.

Unit Number	pCO2 (CNY/t)	δG (t/MWh)	δT (t/MWh)
G1	120	0.975	0.575
G2	120	0.6	0.575
G3	120	0.975	0.575
G4	120	0.6	0.575
G5	120	0.975	0.575

, and the network structure and hyperparameters of the MADDPG algorithm are shown in

Table A3. Parameter setting of MADDPG algorithm.

Parameters of MADDPG	Critic Network	Actor Network
Number of neural network layers	3	3
Batch size	50	50
Hidden layer activation function	Relu	Relu
Output layer activation function	Relu	Sigmoid
Buffer size	50,000	50,000
Discount factor	0.95	0.95
Learning rate	1 × 10−3	1 × 10−4
Number of rounds	2000	2000
Soft update rate	0.01	0.01
Number of agents	5	5

[23,24].

5.1. Feasibility and Effectiveness Analysis of MADDPG-Based Generator Equilibrium Game

In this section, example 1 is set to verify the effectiveness of the proposed MADDPG algorithm for solving the multi-generator bidding decision optimization model. The example setting does not consider the factors of carbon trading market and coal market. Each generator formulates the bidding strategy based on the supply cost curve, and the independent system operator (ISO) performs the market clearing calculation.

Figure 5a shows the convergence characteristics of the cumulative reward of each agent in the multi-generator equilibrium game process based on MADDPG. Figure 5b,c show the game process of the bidding coefficient of the generator and the winning electricity at the time of t = 5 h with lower load. Figure 5d shows the iteration process of the system marginal price at the selected time point. As shown in Figure 5a, the initial training stage shows a significant return shock, which is due to the fact that agents accumulate market awareness by exploring new bidding strategies. With the continuous precipitation of strategic experience, the return value of learning shows an increasing trend and eventually tends to be stable. As the bidding coefficient of each generator gradually converges, the output of the generators and the market clearing price gradually stabilize.

In order to evaluate the superiority of the proposed MADDPG algorithm, this study compares it with multi-agent double deep Q-network (MADDQN) algorithm, proximal policy optimization (PPO) algorithm, and independent Q-learning (IQL) algorithm. The change curve of the total reward of the generator’s moving average is shown in Figure 6.

From the observation of convergence characteristics, the MADDPG algorithm and MADDQN algorithm converge at about 600 iterations, while the PPO and IQL algorithms converge after 300 iterations. The convergence speed of MADDPG and MADDQN is lower than that of PPO and IQL algorithm. The MADDPG algorithm and MADDQN show a more robust progressive learning curve, which verifies the dynamic balance characteristics of the multi-agent interactive environment in the power market. Although the PPO and IQL algorithms show faster initial convergence speed, their revenue growth models have obvious fluctuations, reflecting the lack of sensitivity of traditional reinforcement learning methods to market microstructure when dealing with continuous-action space.

In the stability dimension, thanks to the collaborative optimization of the actor–critic dual-network architecture, the MADDPG and PPO algorithms maintain the stability of the revenue curve in the long-term game, and the fluctuation of the revenue is maintained within 2000 yuan. The MADDQN algorithm has a discontinuous-action space, resulting in a maximum fluctuation of the total revenue of 5000 yuan after the model is stable, confirming the importance of continuous-action modeling to the real-time bidding scenario of the power market. At the level of optimization results, the optimal total revenue obtained by the MADDPG algorithm is 369,975.39 yuan. Compared with the IQL algorithm, PPO algorithm, and MADDQN algorithm, the revenue increases by 9.15%, 10.07%, and 17.49%, respectively, showing a significant total revenue advantage.

It can be seen that the MADDPG algorithm proposed in this paper shows high superiority in the calculation efficiency of game equilibrium and the accuracy of strategy convergence, which verifies its theoretical value as the optimal solution algorithm for the bidding decision model of power suppliers.

5.2. Bidding Decision of Power Suppliers Considering Carbon–Electricity–Coal Coupling Market

In this section, examples 2 and 3 are set to study the bidding decision-making behavior of multi-generators in the market-coupling scenario. Example 2 assumes that all generators participate in the power market and the carbon trading market, and the influence of the carbon trading market on bidding strategy of multi-generators is simulated and analyzed.

Figure 7a shows the equilibrium results of the generator output and the clearing price in the electricity–carbon coupling market. Figure 7b is a comparison of the total carbon emissions of generators in 24 periods.

Table 1. Comparison of bidding clearing results in electricity–carbon coupling market.

	Generation Revenue	Carbon Emissions	Carbon Trading Cost	Power-Production Cost	Gross Income
Power market	2,379,427.51	4431.80	0	2,009,452.54	369,975.63
Electricity–carbon market	2,500,657.82	4334.51	162,147.91	2,013,965.78	324,544.24

is the comparison between case 2 and case 1 in terms of power generation revenue, carbon emissions, carbon transaction costs, power generation costs, and total revenue.

From Figure 7a, it can be seen that the carbon trading mechanism significantly changes the bidding strategy of generators through the cost internalization effect: low-emission generators (such as generator 4) form a cost advantage by virtue of carbon emission quota savings. Under the condition of similar marginal power generation costs, the bid-winning electricity increases from 859.51 MWh to 1073.66 MWh, an increase of 24.91%; high-emission units (e.g., generator 1) are subject to rigid constraints on carbon emission costs, resulting in weakened price competitiveness and reduced output share.

Figure 7b shows that after considering the carbon trading market, the overall trend of total carbon emissions decreases significantly. Table 1 shows that carbon trading restructures the operating cost structure of market players—the purchase cost of carbon emission rights becomes an independent cost item, which reduces the total revenue of generators to a certain extent and promotes cost allocation to low-carbon attributes. This cost reconstruction is directly reflected in the bidding strategy: low-carbon generators hedge operating costs through carbon asset returns and enhance market competitiveness, while high-emission generators are forced to accept lower output levels due to cost rigidity. The system scheduling shows the characteristics of “low-carbon priority”. The carbon emission intensity is negatively correlated with the output probability, indicating that the carbon price signal has been deeply integrated into the market clearing mechanism. The essence of this structural adjustment is the reshaping of the marginal cost function of generators by carbon trading price signals, which promotes the formation of an endogenous linkage mechanism between market clearing prices and carbon emission costs.

In example 3, the bidding decision of generator 4 is set to consider the coal market additionally. Compared with generator 4 without considering the coal market in example 2, the moderating effect of coal market cost transmission on the bidding strategy of generator 4 is analyzed. The generator output, power generation income, and their comparison of the generator 4 in each period are shown in Figure 8a,b, which presents the comparison of the total carbon emissions in the whole period under the coupling of each market.

Table 2. Comparison of income results in carbon–electricity–coal coupling market.

	Generation Revenue	Carbon Emissions	Carbon Trading Cost	Power-Production Cost	Gross Income
Power market	2,379,427.51	4431.80	0	2,009,452.54	369,975.63
Electricity–carbon market	2,500,657.82	4334.51	162,147.91	2,013,965.78	324,544.24
Carbon–electricity–coal coupling market	2,503,293.88	4263.37	153,611.72	2,018,970.71	330,711.45

shows the income results of generators participating in the carbon–electricity–coal coupling market.

When generator 4 introduces the dynamic cost accounting mechanism of the coal market into the power market quotation model, the bidding power is significantly improved from 1073.66 MWh to 1362.83 MWh. This strategy transmits the unit power generation fuel cost to the bidding function by tracking the fuel price fluctuation in real time and directly reduces the marginal cost of electricity, thus forming a dynamic cost advantage with market competitiveness.

From the comparison of the income results in the coupling of the carbon–electricity–coal market in Table 2 with other market conditions, it can be seen that the increase in the output of generator 4 as a low-emission generator produces double benefits: on the one hand, the increase in the power generation of low-emission generators directly reduces the demand for an alternative output of high-emission generators, the overall carbon emission intensity of the power system decreases, and the total carbon emission decreases from 4334.51 t to 4263.37 t. The overall carbon emission level changes are shown in Figure 8b. On the other hand, the reduction in total carbon emissions has a chain reaction through the carbon quota allocation mechanism—the improvement of the marginal emission intensity of the system not only improves the efficiency of the use of carbon quotas but also reduces the total control cost of the carbon emission trading market. The cost of carbon trading decreases from 162,147.91 to 153,611.72 yuan. The synergistic realization of environmental and economic benefits is due to the endogenous effect of the market mechanism. The share of power generation obtained by generator 4 through cost advantage expands, which objectively promotes the transformation of the power system to low-carbon while improving individual operating income, forming a positive cycle of “cost optimization driving market selection–market structure change guiding emission reduction–emission reduction effect feeding back market efficiency”.

6. Conclusions

The results of this paper are applicable to the transitional electricity market with thermal power as the main body. In this paper, by constructing the decision-making framework for power suppliers to participate in the bidding of carbon–electricity–coal coupling market, the decision-making time-scale differences of the coal market, day-ahead power market, and carbon trading market were analyzed, and the decision-making cycle coordination mechanism of the carbon–electricity–coal market was established to realize the coordinated decision making of the multi-sided coupling market. The upper-level bidding model was established to maximize the revenue of power suppliers. The lower-level clearing model was established based on security-constrained unit commitment and security-constrained economic dispatch. Based on the MADDPG algorithm, the competition scenario of multi-power suppliers in the coupling environment of the carbon–electricity–coal market was simulated, and the influence of a multi-market structure on the bidding strategy of power suppliers was analyzed. The following conclusions were obtained through simulation analysis:

(1)

The results show that the proposed MADDPG algorithm-based multi-generator bidding strategy can effectively simulate the dynamic process of the bidding decision making of multi-generators. The performance of the MADDPG algorithm was compared with MADDQN, PPO, and IQL algorithms. The results show that the convergence performance of the MADDPG algorithm is better, and the optimized decision benefit is higher than that of other algorithms;

(2)

The coupling environment of the carbon–electricity–coal market has a significant impact on the bidding strategy of power suppliers. The carbon trading market reshapes the cost structure of power generation through cost internalization, which reduces the total carbon emissions while reducing the overall profit margin of the power generation side. Low-emission units rely on carbon quota income to form a competitive advantage, and high-emission units are subject to cost constraints. To reduce the share of output, the bidding process considers the coal market to update the power generation cost in real time, which can enhance the market competitiveness of power suppliers, enable power suppliers to obtain additional market advantages, and improve the clear power and profit income. The bidding decision optimization model of power suppliers in the carbon–electricity–coal coupling market as proposed in this paper will encourage power suppliers to upgrade and promote the energy conservation and emission reduction of generating units.