Multi-Agent Reinforcement Learning for Sustainable Integration of Heterogeneous Resources in a Double-Sided Auction Market with Power Balance Incentive Mechanism

Abstract

Traditional electricity market bidding typically focuses on unilateral structures, where independent energy storage units and flexible loads act merely as price takers. This reduces bidding motivation and weakens the balancing capability of regional power systems, thereby limiting the large-scale utilization of renewable energy. To address these challenges and support sustainable power system operation, this paper proposes a double-sided auction market strategy for heterogeneous multi-resource (HMR) participation based on multi-agent reinforcement learning (MARL). The framework explicitly considers the heterogeneous bidding and quantity reporting behaviors of renewable generation, flexible demand, and energy storage. An improved incentive mechanism is introduced to enhance real-time system power balance, thereby enabling higher renewable energy integration and reducing curtailment. To efficiently solve the market-clearing problem, an improved Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm is employed, along with a temporal-difference (TD) error-based prioritized experience replay mechanism to strengthen exploration. Case studies validate the effectiveness of the proposed approach in guiding heterogeneous resources toward cooperative bidding behaviors, improving market efficiency, and reinforcing the sustainable and resilient operation of future power systems.

1. Introduction

As the global energy transition accelerates, traditional power systems are gradually evolving toward market-oriented, flexible, and low-carbon directions [1,2]. The introduction of electricity markets has shifted electricity production, transmission, and consumption from a planned economy model to a more flexible competitive market model, which not only improves the efficiency of electricity resource allocation but also enhances the operational stability of power systems [3,4]. Early research primarily used supply–demand curves and classical equilibrium models to describe price formation, which, though simple and effective, did not fully consider the market’s dynamics and complexity, particularly the response of demand and the impact of other resources. Therefore, some studies add principal component analysis and other methods to the supply and demand curves to improve the reliability of the analysis [5,6]. With increasing competition in electricity markets, the unilateral bidding mechanism has revealed many shortcomings, especially in addressing the volatility of renewable energy and emerging resources such as energy storage systems, as traditional models struggle to handle the complex interactions between resources [7].

To address these shortcomings, researchers began to introduce game theory models to analyze the competitive strategies between power generation companies and extended these models that include demand response and energy storage behaviors [8,9]. Ref. [10] establishes a stochastic scheduling model to realize the synergetic optimization of day-ahead (DA) market bids. Ref. [11] presents both optimal bidding strategies and operating schedules of the integrated energy system with the consideration of the CO2 emission limits. The above research is mainly based on static game models. Dynamic game models are more suitable for describing long-term decisions and price fluctuations in electricity markets, taking into account feedback effects between participants. Hence, Ref. [12] presents a stochastic dynamic bi-level model for managing market participants. Ref. [13] proposes a bi-level model for non-cooperative game interactions among power generation companies. Bilateral bidding allows both electricity buyers and sellers to trade based on market conditions and their own needs, which better coordinates resource behaviors and optimizes market resource allocation. Hence, Ref. [14] considers bilateral contracting in electricity markets with demand response. It investigates how curtailment and shifting affect the energy quantity and energy cost of consumers that adopt a time-of-use tariff involving three block periods. Virtual computing needs can also become the subject of bilateral transactions. Ref. [15] uses the combination of barter and evolutionary game theory to model the exchange of instances among cloud service providers through an auction market. Peer-to-peer (P2P) energy markets are the other double-sided auction market. Ref. [16] proposes a bi-level optimization algorithm for trading quantity and surplus maximization in P2P energy trading, incorporating a double-sided carbon taxation scheme. The sheer number of participants in peer-to-peer transactions presents computational challenges. Consequently, game theory methods based on evolutionary algorithms have been extensively studied [17,18].

Nevertheless, game theory assumes that market participants behave rationally and have complete physical models, which limits its application in practice. In reality, there is often information asymmetry and decision-making bias. The operation of market participants themselves is difficult to model, and sometimes the reference for quotations can only be evaluated based on historical data [19,20]. In response to these complex market environments, reinforcement learning (RL), an intelligent algorithm capable of self-learning, has gradually become an important tool in electricity market research. Multi-agent reinforcement learning (MARL), an extension of RL, provides a new solution for electricity markets by simulating the interactions and competition between multiple agents in complex environments [21]. In the MARL framework, each participant continually adjusts its strategy to maximize its profits, thereby optimizing the entire electricity market [22,23]. In order to overcome challenges such as credit assignment and coordination, Ref. [24] proposes a parallelizable deep MT-MARL framework by incorporating multi-task learning (MTL) with MARL. Ref. [25] demonstrates that MARL can find superior bidding strategies for the market participants compared to traditional methods. Based on this, Ref. [26] presents the complete DA market using Multi-Agent Deep Deterministic Policy Gradient (MADDPG). Considering the shortcomings of MADDPG in terms of convergence reliability, Ref. [27] presents Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), which is a state-of-the-art method.

However, current multi-agent reinforcement learning (MARL) approaches for bidding decisions suffer from two critical limitations. First, they often homogenize diverse controllable resources—such as cold, heat, and electricity—failing to leverage their distinct operational characteristics. Second, and more significantly, in bilateral markets with high penetration of renewable energy, existing reward functions are typically myopic and self-interested, focusing solely on individual agent profits. They largely neglect system-level objectives, such as incentivizing the balanced consumption of renewable generation against load demand. While some approaches attempt to address this by applying simplistic, fixed penalty terms for grid imbalances [28] or by relying on a centralized critic to indirectly guide agents [29], these methods often fail to create a strong and direct coupling between an agent’s profit-seeking behavior and the collective system goal. The incentive for an individual agent to deviate for selfish gain often outweighs the weak, externally imposed penalty. This fundamental misalignment between individual agent incentives and collective system goals leads to significant power imbalances in the final market-clearing outcomes, severely limiting the practical application of MARL in modern power markets.

To fill this gap in existing research, this paper proposes an improved incentive MARL method for double-sided market using an improved MATD3. The novelty of our approach compared to closely related MARL-based market strategies is explicitly detailed in our contributions and summarized in Table 1. The main contributions of this paper are as follows:

(1)

In direct contrast to prevailing models that use separate, often static, penalties [28], this paper designs a novel market-clearing incentive mechanism to resolve the misalignment between individual profits and system-wide stability. This mechanism is genuinely new in that it dynamically integrates the system-level power balance residual directly into the profit calculation of each agent’s reward function. This creates an intrinsic, market-driven incentive where agents learn that maximizing their financial return is directly contingent on contributing to grid stability and renewable energy consumption. Hence, the profit-seeking behavior of individual agents will cooperate with the collective goal of market equilibrium and sustainability.

(2)

Conventional methods [26,27] rely on uniform experience replay, which treats all market outcomes equally and thus learns inefficiently from rare but critical events (e.g., moments of extreme power imbalance or highly profitable trades). This paper introduces a TD-error-based priority experience weighted replay mechanism. While prioritized experience replay is an established technique in single-agent RL [29], its adaptation and application to stabilize training and accelerate convergence in the volatile, high-dimensional, and sparse-reward context of multi-agent bilateral electricity markets represents a novel contribution. Our specific design weights experiences based not only on error but also on their relevance to system-level objectives, enables agents to focus on high-impact, information-rich experiences, thereby significantly enhancing training efficiency and accelerating convergence to robust bidding strategies in a dynamic, high-dimensional market landscape.

As detailed in Table 1, our work distinguishes itself from seminal approaches like MADDPG and MATD3 not by replacing the core algorithm, but by fundamentally redesigning the incentive structure and learning process around it. While they provide the architectural foundation, our novel incentive mechanism and prioritized replay scheme address the critical, unresolved issues of goal misalignment and learning inefficiency that are prevalent in their standard implementations for power market applications. This article is organized as follows. Section 2 presents the mathematical modeling of individual DERs and bilateral market clearing. Section 3 details the algorithm process of MATD3. Section 4 validates the proposed method through different cases in comparison to existing methods. Finally, Section 5 concludes the work and outlines future research directions.

Table 1. Comparison between existing models and methods.

	a	b	c	d	e	f	g
Refs. [13,30]	✓	✕	✕	✕	✕	✕	✕
Refs. [31,32]	✓	✓	✓	✕	✕	✕	✕
Refs. [21,26]	✓	✕	✓	✕	✓	✕	✕
Refs. [33,34]	✓	✕	✕	✓	✕	✕	✕
Ref. [25]	✓	✓	✓	✕	✓	✕	✕
This paper	✓	✓	✓	✓	✓	✓	✓

a: electricity market; b: heterogeneous resource; c: double-sided market; d: bidding price and quantity; e: RL decision framework; f: priority weighted experience replay mechanism; g: incentive mechanism.

Table 1. Comparison between existing models and methods.

	a	b	c	d	e	f	g
Refs. [13,30]	✓	✕	✕	✕	✕	✕	✕
Refs. [31,32]	✓	✓	✓	✕	✕	✕	✕
Refs. [21,26]	✓	✕	✓	✕	✓	✕	✕
Refs. [33,34]	✓	✕	✕	✓	✕	✕	✕
Ref. [25]	✓	✓	✓	✕	✓	✕	✕
This paper	✓	✓	✓	✓	✓	✓	✓

a: electricity market; b: heterogeneous resource; c: double-sided market; d: bidding price and quantity; e: RL decision framework; f: priority weighted experience replay mechanism; g: incentive mechanism.

2. Mathematical Modeling of Heterogeneous Multi-Resource and Bilateral Market Clearing

2.1. Mathematical Modeling of Heterogeneous Multi-Resource

In a bilateral bidding market, HMR can participate, which can be classified into generation, storage, and load sides. The generation side is divided into distributed photovoltaic (PV), wind power, and small hydropower, based on the type of operation. The load side is categorized into electrical load, cooling load, and heating load, depending on the type of demand. The storage side is represented by independent energy storage systems with charge and discharge capabilities, which can decide whether to charge or discharge based on current market conditions.

• The generation sides

For the generation side, the generation capacity of PV systems is strongly correlated with current environmental parameters, including temperature, irradiance, and wind speed. Since PV power plants are connected to the grid through power electronic devices, the power output can change rapidly between adjacent time steps. Therefore, this paper does not consider the impact of the ramp rate of PV power plants on their participation in bidding. Similar to PV systems, wind power generation is strongly correlated with environmental parameters. Unlike traditional pumped storage hydropower, run-of-river small hydropower (SH) plants generate power based on seasonal rainfall and the cascade relationship, and do not possess energy storage characteristics. Therefore, the bidding quantity of generation sides can be expressed as the following formula at time slot $t$:

$$0\le P_t^G\le P_t^{G,\mathit{max}}$$

(1)

$$C_{\mathit{min}}^{\mathit{market}}\le C_t^G\le C_{\mathit{max}}^{\mathit{market}}$$

(2)

where $P_t^G$ is the bid quantity of the generation sides, $P_t^{G,\mathit{max}}$ is the maximum power of the generation sides, $C_t^G$ is the bid price of the generation sides, $C_{\mathit{max}}^{\mathit{market}}$ is the maximum market price limitation, and $C_{\mathit{min}}^{\mathit{market}}$ is minimum market price limitation.

2.

The load sides

On the load side, electrical load, cooling load, and heating load all have adjustable capabilities. However, since load-side resources do not solely depend on environmental parameters, they are also influenced by the current demand on the load side. At the same time, conventional heating and cooling loads, such as large air conditioners and boilers, do not have fast response capabilities. Therefore, when participating in bilateral market bidding, the load side must also consider its ramp-up rate. Based on the above, the bidding on the load side can be expressed by the following formula:

$$P_t^{L,\mathit{max}}\le P_t^L\le P_t^{L,\mathit{max}}$$

(3)

$$P_t^L-P_{t-1}^L\le {Pr}_t^{L,up}\cdot \mathrm{\Delta }t$$

(4)

$$P_{t-1}^L-P_t^L\le {Pr}_t^{L,\mathit{down}}\cdot \mathrm{\Delta }t$$

(5)

$$C_{\mathit{min}}^{\mathit{market}}\le C_t^L\le C_{\mathit{max}}^{\mathit{market}}$$

(6)

where $P_t^L$ is the bid quantity of the load sides, $P_t^{L,\mathit{max}}$ is the maximum power of the load sides, $C_t^L$ is the bid price of the generation sides, ${Pr}_t^{L,up}$ is the ramp-up rate during the rising process, and ${Pr}_t^{L,\mathit{down}}$ is the ramp-up rate during the down process.

3.

The storage sides

For independent energy storage systems, they can either provide electricity as a generation side resource or absorb electricity as a load side resource. At the same time, due to the energy storage characteristics of storage plants, their participation in the electricity market must consider the energy consumption of the system during the power generation process. Therefore, the modeling of the storage side is as follows:

$$-P_t^{ch,\mathit{max}}\le P_t^{ch}\le 0$$

(7)

$$0\le P_t^{\mathit{dis}}\le P_t^{\mathit{dis},\mathit{max}}$$

(8)

$$SO{C}_t=SO{C}_{t-1}+\left(-P_t^{ch}\cdot {\eta }^{ch}-{\displaystyle \frac{P_t^{\mathit{dis}}}{{\eta }^{\mathit{dis}}}}\right)\cdot \mathrm{\Delta }t$$

(9)

$$SO{C}^{\mathit{min}}\le SO{C}_t\le SO{C}^{\mathit{max}}$$

(10)

$$P_t^{ch}\cdot P_t^{\mathit{dis}}=0$$

(11)

$$P_t^S=P_t^{ch}+P_t^{\mathit{dis}}$$

(12)

where $P_t^S$ is the bid quantity of the storage sides, $P_t^{\mathit{ch},\mathit{max}}$ and $P_t^{\mathit{dis},\mathit{max}}$ represent the maximum charge and discharge power, ${\eta }^{ch}$ and ${\eta }^{\mathit{dis}}$ represent the operation efficiency, $SO{C}_t$ is the state-of-charge for the storage sides, and $SO{C}^{\mathit{max}}$ and $SO{C}^{\mathit{min}}$ represent the maximum and minimum capacity.

2.2. The Double-Sided Auction Power Market Clearing Mechanism

In the bilateral bidding market, spot trading is the main trading method in the electricity market. To achieve effective clearing for all participants, this paper is set against the backdrop of a centralized clearing electricity spot market with the following assumptions:

(1)

Let there be N power generators, M load consumers, and Z energy storage systems participating in the market. Each market participant will submit fair bids and quantities based on publicly available market information.

(2)

Each power generator is assumed to be fully rational and risk-neutral, meaning they always strive to maximize their own profit.

(3)

The generation side, storage side, and load side engage in free bidding, meaning that for each trading period, each participant is required to submit a capacity–price pair.

Based on the above assumptions, the dispatching agency collects and calculates the capacity and corresponding bids. The clearing process is then performed according to the principle of “load with higher bids cleared first and generation with lower bids cleared first.” Compared to unilateral bidding markets, bilateral bidding markets not only need to ensure that each participant’s profit is maximized when participating in the electricity market but also require the balance of electricity after the clearing process. Therefore, this paper proposes an incentive method for each participant in the bilateral bidding clearing process in

Table 2. The incentive mechanism in the double-sided auction market clearing process.

1	initial the generation clearing quantity is $P_t^{G,c}$, load clearing quantity is $P_t^{L,c}$ and storage clearing quantity is $P_t^{S,c}$, market clearing price is $C_t^M$.
2	Calculate the normal unbalance power as $P_t^n={\sum }_{i=1}^N{P}_t^{G,n}-{\sum }_{i=1}^M{P}_t^{L,n}$
3	For the generation sides
4	If $P_t^n>0$
5	If $P_t^{G,c}<P_t^{G,n}$: $R_t^G=P_t^{G,c}\cdot C_t^M$ Else $P_t^{G,c}\cdot C_t^M\cdot k^G$
6	If $P_t^n\le 0$
7	If $P_t^{G,c}<P_t^{G,n}$: $R_t^G=P_t^{G,c}\cdot C_t^M$ Else $R_t^G=P_t^{G,c}\cdot C_t^M\cdot k^G$
8	For the load sides
9	If $P_t^n>0$ $R_t^L={\max (0,(P}_t^{L,c}-P_t^{L,n})\cdot C_t^M\cdot k^L)$
10	If $P_t^n<0$ $R_t^L={\max (0,(P}_t^{L,n}-P_t^{L,c})\cdot C_t^M\cdot k^L)$
11	If $P_t^n=0$
12	If $P_t^{L,c}!=P_t^{L,n}$ $R_t^L=-100$ Else $R_t^L=0$
13	For the storage sides
14	If $P_t^n>0$
15	If $P_t^s>0$ $R_t^s=0$ Else $R_t^s=P_t^{s,c}\cdot C_t^M\cdot k^G$
16	If $P_t^n<0$
17	If $P_t^s>0$ $R_t^s=P_t^{s,c}\cdot C_t^M\cdot k^G$ Else $R_t^s=0$
18	If $P_t^n=0$
19	If $P_t^s>0$ or $P_t^s<0$ $R_t^s=-P_t^{s,c}\cdot C_t^M\cdot k^G$ Else $R_t^s=0$

. The core of the incentive mechanism is the secondary adjustment of the profits of each market participant. Participants who contribute to the energy balance by compensating for the shortfall in the planned curve will receive additional rewards. Conversely, participants who are aware of the planned curve shortfall but fail to make targeted adjustments to their output will be penalized. These incentives will serve as the reward values for the reinforcement learning strategy updates of the corresponding participants. In the next chapter, this paper will focus on how to efficiently update the reinforcement learning agent strategies based on the aforementioned reward values.

3. Multi-Agent RL Method

In the previous section, the mathematical modeling of HMRs and the bilateral market clearing mechanism are presented. Traditional multi-agent bidding and pricing models in bilateral markets are typically based on game theory to establish a multi-agent bilateral bidding market clearing model for solution. However, these models overlook the dynamic characteristics of participants during actual operations. In reality, when the electricity market operates, participants will extensively learn from the behaviors of other participants and adjust their strategies accordingly. Traditional convex optimization-based modeling and solution methods are not well-suited to dynamically changing environments. To address this issue, this paper proposes a multi-agent reinforcement learning approach. In the following sections, we will introduce the action–state design and strategy parameter updates. The whole method construction and training process in actual operation is shown in Figure 1.

3.1. Action and State Space

The construction of the RL agent is specifically intended to determine the optimal bidding price and quantity for each entity in the market, which are evidently treated as the actions within the RL framework. The primary objective of bilateral power markets is to meet the power balance. Hence, the actions should align with the current power gap in the power system. Furthermore, states provide information for an agent to make decisions and evaluate the long-term benefits. Therefore, the state space should contain the data that are valuable for deciding on the next action. Specifically, for multi-agent reinforcement learning, each agent needs to learn the decisions of other agents. Therefore, the actions from the previous period and the market clearing results must be used as inputs for strategy learning.

The traditional multi-agent RL method employs a centralized training and decentralized execution approach, where during training, all agents can observe the actions and states of other agents, while during execution, each agent relies solely on its own information. However, in the context of a bilateral auction market, each participating agent can only access a subset of publicly available market information and cannot observe the states and actions of all other agents. Therefore, the proposed method in this paper introduces a synchronized training and decentralized execution mode based on partial market environment information. This modification allows each agent to observe publicly available market information during both training and execution. The consistency in state input for both training and execution helps accelerate convergence and enhances the stability of multi-agent training. Based on the analysis, we design the state space for each agent to include the previous period’s power balance gap, bid prices, bid quantities, awarded quantities, and clearing prices, as well as the power balance gap demand for the current period. Specifically, for the load side and generation side, the awarded quantities of each participant in the previous stage will be published as public market information, serving as a reference for other participants. The agent actions and states in time slot $t$ are as follows:

$$A_t^G=\left[P_t^G,C_t^G\right],G=PV,WT,\mathit{and} SH$$

(13)

$$A_t^L=\left[P_t^L,C_t^L\right],L=EL,HL,\mathit{and} CL$$

(14)

$$A_t^s=\left[P_t^s,C_t^s\right],S=ES$$

(15)

$$P_{t-1}^{G,c}=[P_{t-1}^{PV,c},P_{t-1}^{WT,c},P_{t-1}^{SH,c}]$$

(16)

$$P_{t-1}^{L,c}=[P_{t-1}^{EL,c},P_{t-1}^{HL,c},P_{t-1}^{CL,c}]$$

(17)

$$P_{t-1}^{s,c}=[P_{t-1}^{ES,c},P_{t-1}^{ES,c},P_{t-1}^{ES,c}]$$

(18)

$$S_t^G=\left[P_{t-1}^n,P_{t-1}^{G,c},C_{t-1}^G,,C_{t-1}^M,P_t^n\right],G=PV,WT,\mathit{and} SH$$

(19)

$$S_t^L=\left[P_{t-1}^n,P_{t-1}^L,C_{t-1}^L,P_{t-1}^{L,c},C_{t-1}^M,P_t^n\right],L=EL,HL,\mathit{and} CL$$

(20)

$$S_t^s=[P_{t-1}^n,P_{t-1}^s,C_{t-1}^s,P_{t-1}^{s,c},C_{t-1}^M,P_t^n],S=ES$$

(21)

where $A_t^G$ and $S_t^G$ are actions and states of the generation side, $A_t^L$ and $S_t^L$ are actions and states of the load side, and $A_t^s$ and $S_t^s$ are actions and states of the storage side.

3.2. RL Agent Optimization Algorithm

Based on the state space, an RL agent will decide the actions, and the environment will give back the final reward under the actions. Among the existing methods, the MADDPG algorithm is one of the most advanced algorithms [35,36]. However, MADDPG employs a centralized training and decentralized execution approach, which neglects the heterogeneity of individual agents and results in poor training stability in non-static and highly competitive multi-agent environments. To address these issues, this paper introduces an improved MATD3 algorithm. The core improvements of the proposed algorithm compared to MADDPG are as follows:

(1)

Double Q-networks

In MADDPG, the Q-value for a given state–action pair is updated based on the estimated value from the target Q-network. In the proposed algorithm, the double Q-learning is constructed to reduce overestimation bias by maintaining two separate Q-networks for each agent. The Q-value update rule in MATD3 is given by

$$y_i=R_t^G+\gamma \underset{A_t^G}{\min }>\{Q_1(S_{t+1}^G,A_{t+1}^G),Q_2(S_{t+1}^G,A_{t+1}^G)\}$$

(22)

where $y_i$ represents the target value for the Q-update, the expected future return of the agent based on its current state–action pair, $\gamma$ is the discount factor for balancing immediate rewards with future rewards, and $Q_1$ and $Q_2$ are the double Q-networks.

(2)

Delayed Target Networks

The proposed algorithm improves the stability of the learning process by introducing delayed target networks, a technique borrowed from TD3. The target networks for both the critic and actor are updated less frequently than the main networks, preventing rapid changes in the value estimates. The target network update rule for the critic is

$${\theta }_{\mathit{critic}}^{’}=\tau {\theta }_{\mathit{critic}}+\left(1-\tau \right){\theta }_{\mathit{critic}}^{’}$$

(23)

where ${\theta }_{\mathit{critic}}^{’}$ is the parameter of critic target network, $\tau$ is the update rate for the target network (a small value like 0.005), and ${\theta }_{\mathit{critic}}$ is the current critic network parameter.

(3)

Target policy construction

Like the design for the target network, the proposed method employs the use of target policies for both the critic and actor networks to prevent instability during training. These target policies are updated periodically, ensuring that the learning process remains smooth. The target policy update for the actor is

$${\mu }^{’}\left(S_t^G\right)=\tau \cdot \mu \left(S_t^G\right)+\left(1-\tau \right)\cdot {\mu }^{’}\left(S_t^G\right)$$

(24)

where ${\mu }^{’}\left(S_t^G\right)$ is the target policy for the agent of the generation side and $\mu \left(S_t^G\right)$ is the current policy, which is updated based on the gradient of the critic target network.

3.3. Priority Experience Weighted Replay Mechanism

The RL algorithm requires many samples to learn effective policies, which can increase training time and computational resource demands. In order to improve the exploring efficiency of the agents, this paper establishes a priority weighted experience replay (PWER) mechanism for the above optimization algorithm. This mechanism consists of two critical components. Firstly, a higher advantage function value indicates a greater potential for exploration associated with the corresponding exploring action. Hence, the samples generated are assigned a selection probability according to the above value. The larger the probability, the easier it is to be selected as an update sample; the smaller the probability, the less likely it is to be selected. Secondly, the trained samples are assigned the lower weight to reduce the effect on the training results, thereby further improving the efficiency of screening environmental samples.

It is important to note that the framework proposed in this paper is based on multi-agent reinforcement learning. Therefore, each agent maintains its own experience buffer, and the updates to its policy are solely based on the samples selected from its own experience pool. The complete pseudo code of the priority replay experience mechanism taking the generation agent as an example is shown in

Table 3. Priority replay experience mechanism for each agent.

1	initial the buffer capacity $L^{max}$, batch size $B$, priority radio $\alpha =0.4$, weight radio $\gamma =0.4$;
2
3	If buffer not full
4	Push new experience $L_e^G=(s_t^G,A_t^G,R_t^G,s_{t+1}^G)$ at position $po=t \% L^{\mathrm{m}\mathrm{a}\mathrm{x}}$
5	else
6	Replace the old experience with low weights
7	Push new experience $L_e^G=(s_t^G,A_t^G,R_t^G,s_{t+1}^G)$
8	Calculate the priority of $L_e^t$ with ${\delta }_t^G=y^G-Q^G(s_t^G,A_t^G;{\theta }_{critic})$
9	Calculate the probability for $L_e^t$ with $L_e^{t,p}={\displaystyle \frac{{\delta }_t^G}{{\sum }_{i=1}^{L^{max}}{\delta }_i^G}}$
10	randomly Sample the batch ${\mathit{L}}_{\mathit{e}}=\{L_e^1,L_e^2,\cdots ,L_e^B\}$ under probabilities $L_e^{t,p}$
11	Calculate the weights of sampled experience with $L_e^{w,t}={\displaystyle \frac{{(L^{max}\cdot L_e^{t,p})}^{-\gamma }}{\max \left(L_e^w\right)}}$ (For loss update, $c_1=c_1 \ast L_e^{w,t}$)
12	Calculate the priority of sampled experience with ${\delta }_t^G={{\delta }_t^G}^{-\alpha }$

.

4. The Simulation Cases

To further demonstrate the advantages of the proposed method, this paper presents corresponding case studies for comparison. The bilateral bidding market considered in this study is the day-ahead market, where the participants are consistent with those shown in Figure 2. The generation side includes PV, wind, and SH, while the consumption side consists of electricity load, cooling load, and heating load. Energy storage is represented by an independent lithium-ion battery storage station. In the multi-agent framework, the PV is agent 1, the wind is agent 2, the SH is agent 3, the electrical load is agent 4, the cooling load is agent 5, the heating load is agent 6, and the energy storage is agent 7. Each type of resource participates in the market with price and energy volume point per 15 min under the proposed mechanism, totaling 96 points. The planned generation and consumption curves for agent 1 to agent 6 are shown in Figure 3. The operating parameters for the power market, the load, and the energy storage station are listed in

Table 4. Key parameters in bilateral power market.

Key Parameters		Unit	Value
Market clearing	Maximum market price	$	1.4
	Minimum market price	$	0
	Time interval	min	15
Load	Power rate of electrical load	kW/min	10
	Power rate of heat load	kW/min	10
	Power rate of cooling load	kW/min	10
Storage	Storage capacity	kWh	300
	Storage maximum changing power	kW	30
	Storage maximum discharging power	kW	30
	Storage charging efficiency	/	0.95
	Storage discharging efficiency	/	0.95

. The neural network architectures and hyper-parameters for the MATD3 algorithm are presented in

Table 5. The neural network architectures and hyper-parameters.

Key Parameters		Value
Actor	No. of Hidden layers	2
	No. of Neurons	128
	Activation Function	ReLU
	Learning rate	0.001
	Optimizer	Adam
	Delay update	0.005
Critic_$Q_1$	No. of Hidden layers	2
	No. of Neurons	128
	Activation Function	ReLU
	Learning rate	0.001
	Optimizer	Adam
	Delay update	0.005
Critic_$Q_2$	No. of Hidden layers	2
	No. of Neurons	128
	Activation Function	ReLU
	Learning rate	0.001
	Optimizer	Adam
	Delay update	0.005
Training process	Replay buffer size	1 × 1010
	Batch size	64
	Discount factor	0.99
	Weights factor	0.4
	Sampling factor	0.4
	Clip-epsilon	0.2

. The algorithm benchmarks used in this paper are MADDPG and the original MATD3, and the parameters of their action and evaluation networks are the same as those in Table 5. All experiments are run on an industrial-grade server equipped with Intel(R) Xeon(R) Gold 6330 CPU @ 2.00 GHz, 4*Nvidia GeForce RTX 4090 GPU and 256 GB RAM.

4.1. The Convergence Performance

RL-based decision-making requires an analysis of convergence performance. Therefore, this study evaluates the convergence performance of multiple algorithms: (1) MADDPG, (2) original MATD3, and (3) the proposed: MATD3 with PWER. In Figure 4, there is a significant gap in the convergence performance of different algorithms. The algorithm proposed in this paper demonstrates superior performance in both convergence speed and average reward value. The superior convergence performance stems from two key algorithmic enhancements. First, the incorporation of PWER significantly improves sample efficiency by focusing learning on transitions with high TD error. This is particularly beneficial in the sparse-reward environment of electricity markets, where critical states—such as those involving volatile renewable generation or strategic bidding—are rare but highly informative. As a result, the proposed method achieves faster initial convergence, as evidenced by the steeper ascent of the learning curve in early training stages. Secondly, the inherent stability of the MATD3 framework, reinforced by the twin-critic design, reduces value overestimation bias and leads to smoother policy updates. This is reflected in the lower reward variance of the proposed method after convergence, compared to the oscillatory behavior of MADDPG. Such stability not only aids training but also supports better generalization during testing, as the learned policy is less sensitive to noisy value estimates. In contrast, while MADDPG attains high rewards quickly, its pronounced fluctuations indicate sensitivity to overestimation bias. The baseline MATD3, lacking prioritized replay, converges more slowly due to less efficient use of past experiences. These observations, supported by ablation studies, confirm that both the PWER component and the underlying MATD3 structure contribute critically to the overall performance improvement.

4.2. The Market Clearing Conditions

Figure 5 illustrates the behaviors of the generation side, load side, and energy storage side when participating in the bilateral bidding process over a 24-h period. The left axis shows the bid price, while the right axis shows the energy quantity. The agent submits its bid price (orange line) and bid quantity (blue line) for each hour. The black line represents the actual cleared quantity of energy that was accepted by the market operator. The figure illustrates a diverse ecosystem of bidding strategies within a simulated electricity market, with each agent embodying a distinct strategic archetype. The generation side agents align its bid quantity with a predictable diurnal generation profile and employs a slightly nuanced pricing strategy, raising its bid price from zero during anticipated high-demand periods to capture better market prices. The load agents are price-inelastic, often bidding at the market price cap to guarantee their demand is met, prioritizing reliability above cost. The storage exemplifies a sophisticated energy arbitrageur and price-maker. It pursues a high-risk, high-reward strategy by bidding extremely high prices just below the market cap. This tactic is designed to capture maximal profit margins by becoming the marginal dispatch unit only during periods of extreme scarcity and peak pricing. Its sporadic cleared quantity is not a sign of failure but the intended outcome of a classic peak-shaving strategy. Collectively, these behaviors showcase a complex interplay between volume-driven renewables, profit-driven dispatchable units, reliability-focused loads, and arbitrage-seeking storage assets.

Figure 6 shows the profits obtained by each agent in the bilateral bidding market. From the perspective of the equilibrium in cooperative games, the profits of the agents are relatively balanced, and no market monopoly has occurred. On the one hand, this indicates that the strategies of the agents are independent of each other, and none of the agents have adopted the strategies of their competitors. On the other hand, this demonstrates the success of the incentive mechanism in the bilateral bidding market, effectively achieving the goal of fostering both cooperation and competitive equilibrium among the agents. The reward curves for both the generation and load sides reveal that, because the generation level does not correlate with load changes, in actual bilateral transactions, there are instances where the generation side continues to supply electricity despite knowing there will be losses. This is because the average marginal cost of small hydropower and wind power generation is close to zero. If generation is stopped and restarted, the restart cost would be extremely high. Therefore, to avoid frequent start-ups and shutdowns, the generation side will try to maintain a stable generation capacity. Load-side users, on the other hand, are price-sensitive and are not easily subject to losses. Therefore, it can be seen that the reward for the load side in the bilateral trading market is essentially non-zero.

4.3. The Power Balance Conditions

Figure 7 illustrates the clearing price curve throughout the bilateral bidding process and the energy balance before and after the bidding. It can be observed that, under the constraints of the market clearing price, the participants in the entire bilateral bidding process have formed a consensus to set their bids slightly below the clearing price upper limit. Although the load side would typically respond to market price changes, such responses are based on the subjective intentions of each participant. Therefore, this study does not consider the price sensitivity of the load side’s energy usage. The values slightly below the clearing price upper limit align with the profit-maximizing behavior of the agents during the market bidding process.

Furthermore, comparing the market clearing price and the energy balance before the bidding, it is evident that, when the market has excess energy, it becomes a buyer’s market. In this scenario, generation-side participants lower their energy price bids to maximize their chances of being awarded power, leading to a decrease in the market clearing price. Conversely, when there is an energy shortage in the market, it becomes a seller’s market. In this case, generation-side participants raise their bids to achieve higher revenues, resulting in an increase in the market clearing price. This negative feedback effect helps prevent abnormal bid prices from market participants, and the incompleteness of public information further stabilizes the market price, avoiding large fluctuations. The proposed method in this study alleviates the power imbalance in the system. The figure also compares the power imbalance under different algorithms. With the method proposed in this paper, the power imbalance decreased by 92.8%. The current mainstream method, MADDPG, reduces the power imbalance by 54%, while the MATD3 method, which does not introduce an incentive mechanism, reduces the imbalance by 74.2%. From the data analysis, it is evident that the incentive mechanism proposed in this paper can effectively guide the bidding behavior of various market participants, showing great application potential.

5. Conclusions

This paper proposes a multi-agent reinforcement learning-based bidding decision method for bilateral markets. First, the common market participants in a bilateral market, including PV, wind turbines, hydropower, electrical load, cooling load, and heating load, are analyzed and modeled based on their specific physical processes. Second, an incentive mechanism aimed at renewable energy absorption and power balance is established and integrated with the market clearing process, thereby guiding and incentivizing each participant in the bilateral market. Next, a multi-agent reinforcement learning algorithm is introduced. This algorithm constructs a dual-critic network to enhance the robustness of state estimation in reinforcement learning. Additionally, a TD-error-based prioritized experience replay mechanism is developed to ensure efficient exploration of reinforcement learning agents within the environment. Finally, through the construction of typical operating scenarios, the superiority of the proposed method is validated. Compared to traditional methods, the proposed approach reduces the system’s power balance deviation by 92.8% and accelerates training time by 36%, demonstrating its significant application value.

Despite the encouraging results, this study has certain limitations that point to valuable directions for future research:

(1)

A primary constraint lies in the simplified market composition adopted for this initial investigation. To fully validate the scalability and generalizability of the proposed framework, it is imperative to test its performance within a more extensive and heterogeneous market environment.

(2)

The current market model does not incorporate complex physical network constraints, such as transmission line limits and nodal pricing. Future work will, therefore, focus on enhancing the model’s fidelity by integrating these critical elements of power system operation, thereby assessing the method’s robustness under more realistic market conditions.

(3)

The inherent non-stationarity of multi-agent learning environments remains a challenge. Advancing the algorithm to improve its convergence stability and sample efficiency in large-scale applications represents a crucial avenue for further algorithmic development.