Virtual Power Plant Optimization Process Under the Electricity–Carbon–Certificate Multi-Market: A Case Study in Southern China

Abstract

Over the past decade, China has vigorously supported the development of renewable energy and has initially established the electricity–carbon–certificate multi-market. As a typical market-oriented demand-side management model, studying the optimization process and cases of virtual power plants (VPPs) under the multi-market has significant importance for enhancing the operation level of VPPs, as well as promoting corresponding experiences. Based on the mechanisms and impacts of the electricity–carbon–certificate multi-market, this manuscript takes a VPP project in southern China as a case, constructs a sequential decision-making optimization model for the VPP under a diversified market, and solves it using reinforcement learning and Markov decision theory. The case analysis shows that, compared to energy supply income, although the proportion of income from certificate trading and carbon trading in the multi-market is relatively limited, participating in the electricity–carbon–certificate multi-market can significantly enhance VPPs’ willingness to accommodate the uncertainties of renewable energy and can significantly improve the economic and environmental performances of VPPs, which is of great significance for improving the energy structure and accelerating the process of low-carbon energy transformation.

1. Introduction

China’s power industry accounts for nearly 40% of the country’s total CO₂ emissions [1,2]. To address the challenges posed by climate change, China has embarked on a series of explorations aimed at transforming its energy supply structure. However, relying solely on the large-scale construction of centralized renewable energy bases on the supply side is insufficient to fully meet the requirements for carbon neutrality. It is also necessary to fully tap into the potential of demand-side resources [3].

As a demand-side management model capable of transcending geographical boundaries to widely aggregate distributed devices and multi-energy loads, the virtual power plant (VPP) promotes the safe, large-scale, and efficient utilization of renewable energy by effectively coordinating the aggregated resources. This increases the share of renewable energy in the power consumption structure. Conducting research on the optimization methods and processes of VPPs under the electricity–carbon–certificate multi-market is of great significance for enhancing the comprehensive benefits of VPPs and supporting the green and low-carbon transformation of energy sector [4,5].

From the perspective of research scenarios, current scholarly research on VPPs mainly falls into two categories: the coordinated scheduling of VPP internal resources and strategy optimization for VPPs’ participation in the external market [6,7]. Ding et al. [8], Liao et al. [9], and Elgamal et al. [10] have studied the optimization methods for internal resource scheduling within the VPP, aiming to promote the absorption level of renewable energy and enhance economic benefits. Based on the forecasting of renewable power output, they considered the technical characteristics of combined heat and power equipment, carbon capture equipment, power-to-gas (P2G) facilities, energy storage, and other typical equipment operated by the VPP. Pandzic et al. [11] and Zhou et al. [12] further incorporated demand response load resources into the optimization scope, and the results indicated that demand response can effectively improve the load curve, but its contribution to enhancing environmental benefits is relatively limited. Ju et al. [13] and Ge et al. [14] further incorporated typical distribution network-side constraints such as renewable power uncertainty, peer-to-peer energy trading, and carbon emission characteristics into the VPP internal resource scheduling problems. Wu et al. [15], Song et al. [16], and Aldegheishem et al. [17] conducted research on optimizing market bidding strategies and the output of distributed energy resources (DERs), energy storage, and electric vehicles by robust optimization and two-stage optimization, providing corresponding references for VPPs to enhance their economic performance when participating in the electricity market. Furthermore, in light of the construction and reform demands for multi-markets in China, as exemplified by the electricity market and carbon market, scholars have conducted a series of in-depth studies on multi-market mechanisms, including coupling mechanisms [18,19], price transmission [20], and joint clearing methods [21,22]. Correspondingly, Sun et al. [23] and Wu et al. [24] conducted research on the scenario where VPPs participate in both the electricity market and carbon market, and based on designing carbon pricing mechanisms and analyzing the impact of carbon trading, they also proposed corresponding bidding strategies and scheduling strategies.

From the perspective of research methods, considering the accumulated research, the modeling and solving of VPP optimization problems can be categorized into two approaches: holistic modeling and solving and hierarchical modeling and solving. Under the holistic modeling and solving approach, scholars typically adopt game theory and multi-stage coordinated optimization to construct nonlinear mixed-integer VPP optimization models. They further employ linearization techniques based on McCormick envelopes and intelligent algorithms to solve these problems [25,26,27,28]. Considering the difficulty and limitations of solving the holistic modeling problem, Liu et al. [29], Li et al. [30], and Mao et al. [31] decomposed the optimization problem into upper-level problems and lower-level subproblems based on theoretical methods such as Lagrangian dual relaxation, column-and-constraint generation (C&CG), and multi-agent architecture. By solving these problems hierarchically, they derived optimization strategies for the VPP, effectively reducing the difficulty of solution and improving the computational efficiency. Specifically, Xie et al. [32], Aghdam et al. [33], Shen et al. [34], and Liu et al. [35] incorporated uncertainty factors that may be involved in the VPP optimization process into their research scope by utilizing methods such as Information Gap Decision Theory (IGDT), stochastic optimization, and chance constraints, thereby further enhancing the robustness of the solution results.

Based on the above research, the contributions and original work of this paper are primarily manifested in two aspects.

From the perspective of research scenarios, this paper analyzes the implementation and interaction mechanisms of China’s diverse “carbon neutrality”-themed policies, including renewable energy development and carbon quota systems. It investigates how these mechanisms drive and guide the multi-market. Based on these mechanisms, an optimized VPP model is constructed within the electricity–carbon–certificate multi-market. This approach addresses the limitations of the existing literature in research scenario construction, which often deviates significantly from real-world energy policy mechanisms, neglects the guiding role of policy mechanisms on the market, and overlooks the influence of RPS and CA mechanisms on VPPs’ willingness to absorb renewable energy and participate in the multi-market transactions.

From the perspective of research methodology, this paper constructs a Markov Decision Process (MDP) model for VPP optimization process under the multi-market, utilizing the principles of reinforcement learning. The model incorporates factors such as the uncertainty of renewable energy and flexible loads. By sequentially optimizing VPP decisions throughout the dispatch period in chronological order, it simulates the interaction between VPP decision-making behaviors and the decision-making environment, as well as the impact of prior decisions on subsequent problem-solving environments and decision-making processes. This study overcomes the limitations of existing research approaches, which typically employ either an “integrated modeling and solving” or a “hierarchical modeling and solving” strategy to derive VPP strategies for each time instant through a single one-time solution. These conventional approaches often overlook the influence of prior VPP strategies, RPS compliance progress, and other factors on the problem-solving process in subsequent time periods. In contrast, our research offers a valuable reference for VPP strategy formulation, behavior simulation, and policy effectiveness evaluation.

The rest of this paper is organized as follows: Section 2 analyzes the electricity–carbon–certificate multi-market mechanism and its related impacts and proposes an optimization framework for VPPs under the multi-market. Section 3 constructs the VPP optimization model and proposes a solution method based on the concept of reinforcement learning. In Section 4, a case study and discussion are carried out on a VPP located in southern China. Finally, Section 5 draws the relevant conclusions.

2. VPP Optimization Framework Under the Multi-Market

2.1. Electricity–Carbon–Certificate Multi-Market

Over the past decade, guided by initiatives such as the Clean Development Mechanism (CDM) and energy security strategies, the Chinese government has made a series of efforts centered around the construction of the electricity–carbon–certificate multi-market. Initially, two primary mechanisms have been formed: the “CA + carbon trading” mechanism, which is overseen by the Ministry of Ecology and Environment (MEE), and the “RPS + green electricity + green electricity certificates (GECs)” mechanism, which is managed by the National Development and Reform Commission (NDRC). Below are the typical mechanisms and their impacts within this multi-market framework.

Firstly, renewable energy electricity (green electricity) trading can be conducted under two basic models: “certificate–electricity integration” and “certificate–electricity separation.” Moreover, GECs can serve as one of the important means for entities to meet their RPS targets. These mechanisms clarify the relationships among GREs, green electricity, and green environmental value, thereby addressing the potential risk of double-counting green environmental value.

Secondly, the electricity generated by distributed renewable energy units will also be eligible for the GEC issuance. This provides an important policy basis for tapping into the potential of distributed renewable energy to participate in GEC trading and meet RPS targets.

Thirdly, at this stage, carbon allowances in the carbon trading market will be distributed free of charge, and the consumption of renewable energy will not be covered under carbon allowance management. Carbon trading has emerged as a crucial approach for correcting the externalities associated with the production and operation activities of relevant entities.

Fourthly, the integration of the electricity and carbon market has emerged as an important development trend, and the multi-market mechanism and policies have explicitly stated that the consumption of renewable energy is not included in the scope of carbon emission control. This means that distributed renewable energy production, direct consumption of renewable energy, and GEC trading will directly or indirectly influence the settlement process of RPS and carbon allowances, thereby strengthening the interconnections and mutual influences among the electricity, GECs, and carbon markets.

Lastly, but most importantly, for VPPs’ optimization process, under constraints of the multi-market and relevant compliance policies, VPPs demonstrate significant dual attributes of both suppliers and demander. That is, while operating various types of generation equipment, VPPs also aggregate demand-side loads represented by electricity and heat. Therefore, according to relevant policy requirements, VPPs are one of the entities subject to the constraints of RPS and CA. This means that while VPPs need to control their carbon emissions, they also need to absorb a certain proportion of renewable energy when providing energy to demand-side loads. In this scenario, VPPs will act as market participants and become one of the hubs connecting the multi-market, which includes electricity, carbon, and certificates. This implies that under the electricity–carbon–certificate multi-market, VPPs can maximize their overall benefits by flexibly adopting diversified strategies such as “certificate–electricity integration” trading, “certificate–electricity separation” trading, renewable energy unit dispatch, etc. This also provides an important basis and scenario for this research.

2.2. Optimization Assumptions and Framework

Based on the analysis of the multi-market mechanisms in Section 2.1, the VPP is required not only to meet the energy consumption demands but also to comply with the obligations for RPS and CA as verified by relevant authorities. Taking a VPP project situated in southern China as a case study, we establish the research assumptions and the framework for the VPP optimization process, as illustrated in Figure 1.

(a) The VPP aggregates several types of resources, including photovoltaic units (PVs), gas combined heat and power units (CHP), and electric and heat loads. The electric load resources consist of rigid loads and elastic loads. Among these, the elastic loads are predominantly temperature-controlled loads represented by air conditioning, and the VPP can acquire a certain range of load regulation capabilities by adjusting the retail electricity price.

(b) In this case, the electricity consumption demand is met by PVs and CHP. In situations where there is a deficiency or surplus in electricity supply, the VPP can apply to purchase or sell electricity from/to the grid through the balancing market. However, the power exchange between the VPP and the grid is constrained by interconnection bus capacity. The thermal energy consumption demand is supplied by the CHP units, and the natural gas required for the CHP is purchased in the upstream wholesale market. In addition, since CHP is the primary equipment for thermal energy production, to ensure that heat load energy consumption needs are met, it is further stipulated that the VPP will operate the CHP in a “power determined by heat” mode.

(c) Both the VPP and the aggregated elastic loads are boundedly rational entities. Among these, the VPP pursues the maximization of overall benefits within the dispatch day, whereas the elastic loads pursue the optimization of the comprehensive benefits in terms of economy and comfort.

(d) China’s RPS, CA, carbon trading, and other related mechanisms are still in their initial stages. Therefore, based on international experience and existing research, the following assumptions are proposed accordingly.

• Regarding the RPS, firstly, as the responsible entity, the VPP shall be subject to certain economic penalties if it fails to meet the RPS target, with the unit penalty equivalent to one GEC. Secondly, the annual RPS target can be disaggregated into daily target based on the RPS weights approved by the government and the daily load, and assessments can be carried out accordingly.
• In terms of CA, although the primary emission sources during VPP operation are CHP and electricity purchased from the grid, the root cause of VPP’s carbon emissions, when viewed from the essence of economic activity, lies in its responsibility to supply electricity and heat to the aggregated loads it serves. Drawing on Section 2.1, carbon trading is recognized as a significant means to address the externalities associated with the activities of entities. In this paper, it is stipulated that the CA for VPPs is calculated based on their load levels. Additionally, given the differences in settlement cycles across the multi–market, carbon trading settlement and assessment are conducted on a daily basis, with the carbon market adopting a stepped pricing mechanism. The difference between actual emissions and carbon allowances can be sold or purchased in carbon market. Finally, although some scholars have conducted relevant research on the mutual recognition, quota offsetting, or deduction in transactions involving products such as GECs, CA, and green electricity [36,37], under China’s current policy mechanisms, the trading of green electricity and GECs are managed separately by the NDRC, while CA and carbon trading are managed by the MEE. The outcomes of GEC trading do not impact the issuance, trading, or compliance of carbon quotas. Therefore, this paper assumes that CA will not be influenced by transactions involving GECs.
• Concerning GECs, according to the multi-market mechanism, GECs can be traded through two modes: “certificate–electricity integration” and “certificate–electricity separation.” This implies that when the consumption of renewable energy by the VPP exceeds the RPS target, it can adopt the “certificate–electricity separation” mode to separate GECs from the renewable energy and generate profits by selling them; conversely, if the consumption falls short of RPS target, GECs need to be purchased to meet the target; otherwise, it will face penalties.

3. Modeling and Solution

3.1. VPP Modeling

3.1.1. Objective Function

Based on the assumptions outlined in Section 2, the VPP can pursue the maximization of comprehensive benefits over the scheduling period by optimizing the actions at each time step, including dispatching generation units (PV or CHP), regulating elastic loads, and engaging in transactions involving carbon, certificates, and electricity. Indeed, it is evident that optimizing the VPP’s strategies within a single scheduling day under the multi-market presents a typical sequential decision-making optimization problem. Based on this observation, the following model is constructed:

$$\max R=I^{\mathrm{sale}}-C^{\mathrm{CHP}}-C^{\mathrm{ele}}-C^{\mathrm{PV}}+C^{\mathrm{GEC}}+C^{\mathrm{C}\mathrm{O}2}$$

(1)

Here, $I^{\mathrm{sale}}$ represents the energy supply income. $C^{\mathrm{CHP}}$ represents the CHP cost. $C^{\mathrm{ele}}$ represents the cost/income of purchasing/selling electricity from/to the balancing market. $C^{\mathrm{PV}}$ represents the PV LCOE. $C^{\mathrm{GEC}}$ and $C^{\mathrm{C}\mathrm{O}2}$ represent the GEC trading item and carbon trading item, respectively, with positive values indicating income and negative values indicating cost.

The incomes and costs in (1) can be further derived as follows:

• Energy supply income

$$I^{\mathrm{sale}}={\displaystyle \sum _{t=1}^T(p_t^{\mathrm{sale},\mathrm{e}}\cdot L_t^{\mathrm{e}}+p_t^{\mathrm{sale},\mathrm{h}}\cdot L_t^{\mathrm{h}})}$$

(2)

Here, $p_t^{\mathrm{sale},\mathrm{e}}$ and $p_t^{\mathrm{sale},\mathrm{h}}$ represent the retail electricity price and retail heat price of the VPP at time t. $L_t^{\mathrm{e}}$ and $L_t^{\mathrm{h}}$ represent the electric load and heat load aggregated by the VPP.

Considering that the VPP can guide elastic loads to alter their electricity consumption characteristics by adjusting retail electricity price, this process can be depicted as follows:

$$\left\{\begin{array}{l}{L}_t^{\mathrm{e}}={\displaystyle \sum _{j=1}^J(P_{t,j}^{\mathrm{base}}+P_{t,j}^{\mathrm{DR}})}\\ p_t^{\mathrm{sale},\mathrm{e}}={\theta }_t^{\mathrm{DR}}\cdot p_t^{\mathrm{e}}\end{array}\right.$$

(3)

where $P_{t,j}^{\mathrm{base}}$ and $P_{t,j}^{\mathrm{DR}}$ represent the power of load j before participating in the VPP regulation/demand response (DR) and the responsive power. For rigid loads, $P_{t,j}^{\mathrm{DR}}=0$. ${\theta }_t^{\mathrm{DR}}$ represents the discount coefficient for the retail electricity price of the VPP price-based DR. When ${\theta }_t^{\mathrm{DR}}$ is greater than 0 and less than 1, it indicates that the retail electricity price is reduced to encourage an increase in load demand, at which point $P_{t,j}^{\mathrm{DR}}\ge 0$. Conversely, when ${\theta }_t^{\mathrm{DR}}$ is greater than 1, it signifies that the retail electricity price is increased to prompt a decrease in load demand, and $P_{t,j}^{\mathrm{DR}}\le 0$. $p_t^{\mathrm{e}}$ represents the retail electricity price before the VPP DR.

• CHP operational cost

$$\left\{\begin{array}{l}{C}^{\mathrm{CHP}}={\displaystyle \sum _{t=1}^T{p}_t^{\mathrm{gas}}\cdot V_t^{\mathrm{gas}}}\\ V_t^{\mathrm{gas}}={\displaystyle \frac{P_t^{\mathrm{CHP}}}{{\eta }^e\cdot q}}\end{array}\right.$$

(4)

Here, $p_t^{\mathrm{gas}}$ represents the natural gas price in wholesale market. $V_t^{\mathrm{gas}}$ represents the natural gas input volume of the CHP. $P_t^{\mathrm{CHP}}$ and $H_t^{\mathrm{CHP}}$ represent the electric output and heat output of the CHP at time t, respectively. ${\eta }^e$ represents the efficiency coefficients. $q$ represents the lower heating value (LHV) of natural gas.

• Balancing market cost

$$C^{\mathrm{ele}}={\displaystyle \sum _{t=1}^T{p}_t^{\mathrm{grid}}\cdot P_t^{\mathrm{grid}}}$$

(5)

Here, $p_t^{\mathrm{grid}}$ represents the real-time price in the balancing market. $P_t^{\mathrm{grid}}$ represents the electricity that the VPP declares to purchase/sell in the balancing market at time t.

• PV cost

$$C^{\mathrm{PV}}={\displaystyle \sum _{t=1}^T{c}^{\mathrm{PV}}\cdot P_t^{\mathrm{PV}}}$$

(6)

Here, $c^{\mathrm{PV}}$ represents the PV LCOE cost coefficient. $P_t^{\mathrm{PV}}$ represents the PV output at time t.

• GEC trading cost

$$C^{\mathrm{GEC}}=\left\{\begin{array}{l}{Q}^{\mathrm{RPS}}\cdot p^{\mathrm{GEC}}, Q^{\mathrm{RPS}}\ge 0\\ Q^{\mathrm{RPS}}\cdot p^{\mathrm{penalty}}, Q^{\mathrm{RPS}}<0\end{array}\right.$$

(7)

$$Q^{\mathrm{RPS}}=\omega \cdot {\displaystyle \sum _{t=1}^T{L}_t^{\mathrm{e}}}-{\displaystyle \sum _{t=1}^T{P}_t^{\mathrm{PV}}}$$

(8)

Here, $Q^{\mathrm{RPS}}$ represents the excess amount of RPS. When $Q^{\mathrm{RPS}}$ is greater than 0, it indicates that the corresponding RPS target has been achieved, and under the “certificate–electricity separation”, GECs from the renewable energy can be stripped and sold to obtain income. When $Q^{\mathrm{RPS}}$ is less than 0, it means that the VPP has failed to meet the RPS target and will face corresponding penalties. $p^{\mathrm{GEC}}$ represents the GEC price in the certificate market. $p^{\mathrm{penalty}}$ represents the penalty for failing to meet the RPS target. $\omega$ represents the RPS weighting.

• Carbon trading cost

The excess carbon emissions are calculated as the difference between the actual carbon emissions and the government-approved carbon allowances. As shown in Figure 2, under the stepped carbon trading mechanism, when excess carbon emission is positive, the VPP is required to purchase carbon allowances from the carbon market. Conversely, when it is negative, the VPP can sell its surplus carbon allowances in the carbon market to generate income [38].

$$C^{\mathrm{C}\mathrm{O}2}=\left\{\begin{array}{l}{p}^{\mathrm{C}\mathrm{O}2}\cdot d\cdot (2+\alpha )+p^{\mathrm{C}\mathrm{O}2}(1+2\alpha )\cdot \left|Q^{\mathrm{C}\mathrm{O}2}+2d\right| -2d<Q^{\mathrm{C}\mathrm{O}2}\le -d\\ p^{\mathrm{C}\mathrm{O}2}\cdot d+p^{\mathrm{C}\mathrm{O}2}(1+\alpha )\cdot \left|Q^{\mathrm{C}\mathrm{O}2}+d\right| -2d<Q^{\mathrm{C}\mathrm{O}2}\le -d\\ p^{\mathrm{C}\mathrm{O}2}\cdot \left|Q^{\mathrm{C}\mathrm{O}2}\right| -d<Q^{\mathrm{C}\mathrm{O}2}\le 0\\ -p^{\mathrm{C}\mathrm{O}2}\cdot Q^{\mathrm{C}\mathrm{O}2} 0<Q^{\mathrm{C}\mathrm{O}2}\le d\\ -p^{\mathrm{C}\mathrm{O}2}\cdot d-p^{\mathrm{C}\mathrm{O}2}(1+\beta )\cdot (Q^{\mathrm{C}\mathrm{O}2}-d)d<Q^{\mathrm{C}\mathrm{O}2}\le 2d\\ -p^{\mathrm{C}\mathrm{O}2}\cdot d\cdot (2+\beta )-p^{\mathrm{C}\mathrm{O}2}(1+2\beta )\cdot (Q^{\mathrm{C}\mathrm{O}2}-2d)2d<Q^{\mathrm{C}\mathrm{O}2}\le 3d\end{array}\right.$$

(9)

Here, $Q^{\mathrm{C}\mathrm{O}2}$ represents the excess carbon emissions of the VPP. $p^{\mathrm{C}\mathrm{O}2}$ represents the benchmark price for carbon trading. $d$ represents the step size of carbon emission intervals in the stepped carbon trading mechanism, indicating the proportion of carbon emissions exceeded or reduced relative to the government-approved carbon allowances. $\alpha$ and $\beta$ represents the coefficient for the increase/decrease of carbon price.

Furthermore, according to the current multi-market mechanism and assumptions, the carbon emissions of the VPP primarily stem from meeting the energy consumption demands of the aggregated loads. Consequently, the excess carbon emissions can be expressed as follows:

$$\left\{\begin{array}{l}{Q}^{\mathrm{C}\mathrm{O}2}=Q^{\mathrm{C}\mathrm{O}2,\mathrm{VPP}}-Q^{\mathrm{C}\mathrm{O}2,\mathrm{CA}}\\ Q^{\mathrm{C}\mathrm{O}2,\mathrm{VPP}}={\displaystyle \sum _{t=1}^T(Q_t^{\mathrm{C}\mathrm{O}2,\mathrm{CHP}}+{\mu }^{\mathrm{gird}}\cdot P_t^{\mathrm{grid}})}\\ Q_t^{\mathrm{C}\mathrm{O}2,\mathrm{CHP}}={\mu }^{\mathrm{CHP},\mathrm{a}}+{\mu }^{\mathrm{CHP},\mathrm{b}}\cdot P_t^{\mathrm{CHP}}+{\mu }^{\mathrm{CHP},\mathrm{c}}\cdot {(P_t^{\mathrm{CHP}})}^2\\ Q^{\mathrm{C}\mathrm{O}2,\mathrm{CA}}={\displaystyle \sum _{t=1}^T{L}_t^e}\cdot {\omega }^{\mathrm{C}\mathrm{O}2}\cdot {\omega }^{free}\end{array}\right.$$

(10)

where $Q^{\mathrm{C}\mathrm{O}2,\mathrm{VPP}}$ and $Q^{\mathrm{C}\mathrm{O}2,\mathrm{CA}}$ represent the actual carbon emissions and the government-approved carbon allowances of the VPP. ${\mu }^{\mathrm{gird}}$ represents the carbon emission factor of grid electricity issued by the government. ${\mu }^{\mathrm{CHP},\mathrm{a}}$, ${\mu }^{\mathrm{CHP},\mathrm{b}}$, and ${\mu }^{\mathrm{CHP},\mathrm{c}}$ are the carbon emission coefficients of the CHP. ${\omega }^{\mathrm{C}\mathrm{O}2}$ is the CA weight for the VPP. ${\omega }^{free}$ represents the free portion of the government-approved carbon allowances for the VPP.

3.1.2. Constrains

• Energy supply and demand balance

Based on the assumptions in Section 2, the VPP is required to meet the electricity and heat energy consumption demands of the aggregated loads, so the VPP must adhere to the energy balance constraints as follows:

$$L_t^{\mathrm{e}}=P_t^{\mathrm{PV}}+P_t^{\mathrm{CHP}}+P_t^{\mathrm{grid}}$$

(11)

$$L_t^{\mathrm{h}}=H_t^{\mathrm{CHP}}$$

(12)

• Interconnection bus/pipeline capacity

The flow of electricity/natural gas transmitted between the VPP and the grid/natural gas network must be less than the maximum bus/pipeline capacity, with the constraint expressed as follows:

$$\left\{\begin{array}{l}0\le P_t^{\mathrm{grid}}\le P_{\max }^{\mathrm{grid}}\\ 0\le V_t^{\mathrm{gas}}\le V_{\max }^{\mathrm{gas}}\end{array}\right.$$

(13)

where $P_{\max }^{\mathrm{grid}}$ and $V_{\max }^{\mathrm{gas}}$ represent the maximum transmission capacity of the interconnection bus and pipeline.

• PV output

$$\left\{\begin{array}{l}0\le P_t^{\mathrm{PV}}\le P_{t,\max }^{\mathrm{PV}}\\ P_{t,\max }^{\mathrm{PV}}={\tilde{P}}_t^{\mathrm{PV}}+\Delta P_t^{\mathrm{PV}}\end{array}\right.$$

(14)

$$\left\{\begin{array}{l}{f}_{\mathrm{Guassian}}\left(\Delta P_t^{\mathrm{PV}}\right)={\displaystyle \sum _k^K{\omega }_k\cdot f_{\mathrm{Guassian}}^k\left(\left.\Delta P_t^{\mathrm{PV}}\right|{\mu }_k,{\sigma }_k^2\right)}\\ 0<{\omega }_k<1\\ {\displaystyle \sum _k^K{\omega }_k=1}\end{array}\right.$$

(15)

Here, ${\tilde{P}}_t^{\mathrm{PV}}$ represents the predicted PV output at time t. $\Delta P_t^{\mathrm{PV}}$ represents the prediction error of PV output, and the error follows a mixed Gaussian distribution $f_{\mathrm{Guassian}}\left(·\right)$. ${\omega }_k$ is the weight of the k-th Gaussian model. ${\mu }_k$ and ${\sigma }_k^2$ represent the mean and variance of the k-th Gaussian model.

• CHP output

According to the assumptions, the CHP operates in the “power determined by heat” mode, where although there is no strict one-to-one coupling relationship between the electrical and heat power outputs of the CHP, a constraint relationship exists within a certain range of flexibility [39].

$$\left\{\begin{array}{l}{P}_t^{\mathrm{CHP}}\ge \min \left\{{\lambda }_1^{\mathrm{CHP}}\cdot H_t^{\mathrm{CHP}}, {\lambda }_2^{\mathrm{CHP}}\cdot P_{\min }^{\mathrm{CHP}}-{\lambda }_3^{\mathrm{CHP}}\cdot H_t^{\mathrm{CHP}}\right\}\\ P_t^{\mathrm{CHP}}\le P_{\max }^{\mathrm{CHP}}-{\lambda }_4^{\mathrm{CHP}}\cdot H_t^{\mathrm{CHP}}\end{array}\right.$$

(16)

$$H_{\min }^{\mathrm{CHP}}\le H_t^{\mathrm{CHP}}\le H_{\max }^{\mathrm{CHP}}$$

(17)

$$Ram{p}_{\mathrm{CHP}}^{\mathrm{down}}\le P_t^{\mathrm{CHP}}-P_{t-1}^{\mathrm{CHP}}\le Ram{p}_{\mathrm{CHP}}^{\mathrm{up}}$$

(18)

Here, ${\lambda }_1^{\mathrm{CHP}}$, ${\lambda }_2^{\mathrm{CHP}}$, ${\lambda }_3^{\mathrm{CHP}}$, and ${\lambda }_4^{\mathrm{CHP}}$ are the thermoelectric coupling coefficients, which represent the technical characteristics of the CHP. $P_{\min }^{\mathrm{CHP}}$, $P_{\max }^{\mathrm{CHP}}$, $H_{\min }^{\mathrm{CHP}}$, and $H_{\max }^{\mathrm{CHP}}$ represent the minimum and maximum electrical and heat outputs of the CHP, respectively. $Ram{p}_{\mathrm{CHP}}^{\mathrm{up}}$ and $Ram{p}_{\mathrm{CHP}}^{\mathrm{down}}$ are the upward and downward power ramp rate limits for the CHP.

• Typical elastic load

Based on the assumptions, the temperature-controlled loads aggregated by the VPP are typical elastic loads, which could adjust their electricity consumption characteristics within a certain range in response to the retail price and temperature.

The regulation capability of temperature-controlled loads is shown below:

$$\left\{\begin{array}{l}{P}_{t,j}^{\mathrm{down},\theta }\le P_{t,j}^{\mathrm{DR}}\le 0 1<{\theta }_t^{\mathrm{DR}}\\ 0\le P_{t,j}^{\mathrm{DR}}<P_{t,j}^{\mathrm{up},\theta } 0<{\theta }_t^{\mathrm{DR}}<1\end{array}\right.$$

(19)

where $P_{t,j}^{\mathrm{up},\theta }$ and $P_{t,j}^{\mathrm{down},\theta }$ represent the maximum upward and downward adjustable powers of elastic load j when the retail electricity price discount factor is ${\theta }_t^{\mathrm{DR}}$.

According to (19), it is evident that the ability of the VPP to utilize temperature-controlled loads for operational optimization primarily hinges on the regulation potential of these loads under varying price discount factors. As boundedly rational entities, the regulation potential of temperature-controlled loads is often influenced by expectations regarding energy economics and comfort. According to the reference point dependence principle in decision-making optimization problems, temperature-controlled loads can be characterized as follows [40]:

$$\min f_j=c_j^{\mathrm{actual}}+c_j^{\mathrm{comfort}}+{\mu }^{\mathrm{cost}}(c_j^{\mathrm{actual}}-c_j^{\mathrm{base}})$$

(20)

$$\left\{\begin{array}{l}{c}_{t,j}^{\mathrm{actual}}={\theta }_t^{\mathrm{DR}}\cdot p_t^{\mathrm{e}}\cdot (P_{t,j}^{\mathrm{base}}+P_{t,j}^{\mathrm{DR}})\\ c_k^{\mathrm{comfort}}={\mu }^{\mathrm{comfort}}\cdot \left|T_{t,j}-T_{set,j}\right|\\ c_k^{\mathrm{base}}=p_t^{\mathrm{e}}\cdot P_{j,t}^{\mathrm{base}}\end{array}\right.$$

(21)

where $\min f_j$ is the objective function of temperature-controlled load j. $c_j^{\mathrm{actual}}$ represents the actual energy consumption cost. $c_j^{\mathrm{com}}$ represents the comfort deviation cost. ${\mu }^{\mathrm{comfort}}$ and ${\mu }^{\mathrm{cost}}$ are the deviation coefficients for the energy consumption comfort and economy. ${\mu }^{\mathrm{cost}}(c_j^{\mathrm{actual}}-c_j^{\mathrm{base}})$ represents the deviation cost of energy consumption economy. $T_{t,j}$ and $T_{set,j}$ represent the real-time temperature and the set baseline temperature of the load.

Furthermore, regarding the mechanism of temperature control characteristics for elastic loads, using engineering thermodynamics, the temperature deviation of a single temperature-controlled load after participating in VPP regulation/DR can be described as follows [41]:

$$\left\{\begin{array}{l}{T}_{t,j}=T_{t-1,j}\cdot \gamma +(1-\gamma )\cdot (T_{t,j}^{\mathrm{env}}-{\omega }_t\cdot \eta \cdot P_{t,j}^{\mathrm{elastic}}\cdot R)\\ \gamma =e^{-{\displaystyle \frac{\Delta t}{Rc}}}\\ {\omega }_t\cdot P_{\min }^{\mathrm{elastic}}\le P_{t,j}^{\mathrm{elastic}}\le {\omega }_t\cdot P_{\max }^{\mathrm{elastic}}\end{array}\right.$$

(22)

where $\gamma$ is the heat dissipation function. $T_{t,j}^{\mathrm{env}}$ represents the ambient temperature. ${\omega }_t$ represents the operating status of the elastic load, which is either 0 or 1. $\eta$ represents the energy efficiency ratio of the elastic load. $P_{t,j}^{\mathrm{elastic}}$ represents the power of the elastic load j at time t. $R$ is the thermal resistance. $\Delta t$ is the time step for VPP regulation. $c$ represents the specific heat capacity of air. $P_{\max }^{\mathrm{elastic}}$ and $P_{\min }^{\mathrm{elastic}}$ represent the upper and lower power limits.

It is important to note that for a cluster of temperature-controlled loads aggregated by the VPP, their regulation potential can be characterized as follows:

$$\min F={\displaystyle \sum _{j\in J}{f}_j}={\displaystyle \sum _{j\in J}(c_j^{\mathrm{actual}}+{\mu }^{\mathrm{comfort}}\cdot \left|T_{t,j}-T_{set,j}\right|}+{\mu }^{\mathrm{cost}}(c_j^{\mathrm{actual}}-c_j^{\mathrm{base}}))$$

(23)

where J represents the cluster of temperature-controlled loads aggregated by the VPP.

3.2. Solution

Based on the assumptions and optimization framework outlined in Section 2, under the multi-market, the VPP strategy optimization process essentially constitutes a mixed logical–continuous decision optimization problem across a time series. At any given time t within the scheduling period, the VPP formulates units, output plans, and load regulation plans based on factors such as the PV output, energy price levels, load levels, and RPS target progress. The implementation of these plans subsequently alters the load characteristics, RPS target achievement, and other factors in the subsequent time step, which, in turn, influences the formulation of future units, output plans, and load regulation plans.

Based on the preceding analysis, the core idea of conducting strategy optimization for a VPP is consistent with the approach in reinforcement learning that optimizes the mapping relationship between “behavior policy” and “reward” through “planning–exploration–improvement.” Therefore, reinforcement learning concept can be utilized to solve the VPP strategy optimization problem. Therefore, the VPP strategy optimization problem is typically formulated as an MDP, denoted as $M=\left(I,\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\lambda \right)$ [42,43].

• I represents the agent in the MDP, which is the VPP.
• $\mathcal{S}$ represents the state set, which reflects the decision-making environment faced by the VPP at each time step. $\mathcal{S}=\{S_1,S_2,\dots ,S_T\}$, where $S_t=(s_t^1,s_t^2,\dots ,s_t^m)$, and m describes the characteristic dimensions of the environment in which the VPP operates, namely the constraints mentioned in Section 3, such as the PV output, balancing market prices, the interconnection bus/pipeline capacity, etc.
• $\mathcal{A}$ is the action set, representing the possible decisions that the VPP can take at each time step. $\mathcal{A}=\{a_1,a_2,\dots ,a_T\}$, where $a_t=(a_t^1,a_t^2,\dots ,a_t^n)$, and n represents the dimensions of the VPP decision variables. Based on the assumptions, the decision variables include the CHP output $P_t^{\mathrm{CHP}}$, PV output $P_t^{\mathrm{PV}}$, electricity that the VPP declares to purchase/sell in the balancing market $P_t^{\mathrm{grid}}$, and elastic load scheduling $P_{t,j}^{\mathrm{DR}}$. In the MDP, the decision variables mentioned above can be denoted as $a_t^{\mathrm{CHP}}$, $a_t^{\mathrm{PV}}$, $a_t^{\mathrm{grid}}$, and $a_t^{\mathrm{DR}}$. Each decision $a_t^n$ must satisfy the constraints outlined in Section 3.
• $\mathcal{P}$ represents the transition process to the next decision-making environment state after the VPP takes certain decisions (actions).
• $\mathcal{R}$ is the immediate reward function, representing the reward that the VPP can obtain after taking certain decisions/actions. According to Section 3, the reward that the VPP can receive in the environmental state at time t is as follows:

  $$R_{S_t}^{a_t}=\left\{\begin{array}{l}{I}_t^{\mathrm{sale}}-C_t^{\mathrm{CHP}}-C_t^{\mathrm{ele}}-C_t^{\mathrm{PV}}t\ne T\\ I_t^{\mathrm{sale}}-C_t^{\mathrm{CHP}}-C_t^{\mathrm{ele}}-C_t^{\mathrm{PV}}+C_t^{\mathrm{GEC}}+C_t^{\mathrm{C}\mathrm{O}2} t=T\end{array}\right.$$

  (24)

It should be noted that the uncertainty of PVs has a direct impact on the VPP operation. Especially when the actual PV output is less than the forecasted output, the VPP needs to purchase electricity from the grid to ensure the energy balance. At this point, the VPP’s renewable energy consumption, carbon emissions, and RPS compliance progress will all change, thereby affecting related revenue and cost items.

In light of this, considering the uncertainty of PVs, from the perspective of MDP, the constraints in the model regarding PV costs need to be revised as follows:

$$C_t^{\mathrm{PV}}=\left\{\begin{array}{l}{s}^{\mathrm{PV},\mathrm{c}}\cdot a_t^{\mathrm{PV}}{a}_t^{\mathrm{PV}}\le s_t^{\mathrm{PV}}\\ s^{\mathrm{PV},\mathrm{c}}\cdot s_t^{\mathrm{PV}}+(a_t^{\mathrm{PV}}-s_t^{\mathrm{PV}})\cdot s_t^{\mathrm{grid},\mathrm{p}}{a}_t^{\mathrm{PV}}>s_t^{\mathrm{PV}}\end{array}\right.$$

(25)

where $s^{\mathrm{PV},\mathrm{c}}$ represents the LCOE cost state coefficient for PVs. $a_t^{\mathrm{PV}}$ represents the PV output scheduled by the VPP. $s_t^{\mathrm{PV}}$ represents the maximum technical output state of PVs. $s_t^{\mathrm{grid},\mathrm{p}}$ represents the electricity price in the balancing market.

• $\lambda$ is the discount factor, representing the VPP’s preference for immediate rewards versus long-term rewards. For $\lambda \in (0,1]$, the closer $\lambda$ is to 0, the more the VPP values immediate rewards. Conversely, the more it values long-term rewards.

The key to solving the MDP lies in following Bellman’s optimality principle to find the sequence of actions (strategy $\pi$) that maximizes the expected cumulative rewards over the scheduling period T. By referring to the relevant literature on the derivation process of reinforcement learning, it can be stated that, according to Bellman’s optimality principle, the value function can represent the mapping relationship between the current state, actions, and the cumulative rewards under strategy $\pi$, as shown in (26) [44,45].

$$V_t^{\pi }\left(S\right)=R_{S_t}^{a_t}+\gamma {\displaystyle \sum _{S_t\in S}{T}_{S_t\to S_{t+1}}^{a_t}}{V}_{t+1}^{\pi }\left(S_{t+1}\right)$$

(26)

Here, $V_t^{\pi }\left(S\right)$ is the state value function, representing the immediate rewards and accumulation of expected future rewards after taking action $a_t$ at time t. $T_{S_t\to S_{t+1}}^{a_t}$ represents the process of transitioning from state $S_t$ to state $S_{t+1}$ after taking action $a_t$.

Furthermore, the cumulative rewards of the VPP under the optimal strategy ${\pi }^{\ast }$ over the entire scheduling period can be characterized as follows:

$$F^{{\pi }^{\ast }}=V_t^{{\pi }^{\ast }}\left(S\right)=F_t\left(S_t,{\pi }^{\ast }\left(S_t\right)\right)+\gamma \cdot \mathbb{E}\left\{\left.V_{t+1}^{{\pi }^{\ast }}\left(S_{t+1}\right)\right|S_t\right\}$$

(27)

The optimal sequence of actions ${\pi }^{\ast }$ can be extracted from the optimal value function based on the greedy algorithm, as shown below.

$${\pi }^{\ast }=\underset{a_t\in A,S_t\in S}{\arg \min }{V}_t\left(S_t,a_t\right)$$

(28)

Combining the idea of Bellman’s optimality, the VPP strategy optimization problem can be summarized as a backward iterative process, and evolutionary algorithms can be used to extract the optimal actions for each scheduling period to achieve the solution. The solution process is shown in Figure 3. Specifically, the evolutionary algorithm adopted is the genetic algorithm (GA). The mathematical essence of this algorithm involves treating each individual within the population as a feasible solution (feasible decision in the MDP) within the solution space (state constraints in the MDP). By simulating the biological evolution process, it conducts genetic operations such as inheritance, mutation, crossover, and replication to search for the optimal solution (optimal actions) within the solution space, ultimately forming the optimal sequence of actions ${\pi }^{\ast }$ over time [46,47].

4. Results and Discussions

4.1. Basic Data

The case study selected in this manuscript is a park-level VPP project located in southern China. The architecture and operational model of the VPP are described in Section 2, and the relevant equipment parameters are specified in

Table 1. Parameter information.

Items	Value	Units
CHP capacity	10	MW
CHP ramp rate $Ram{p}_{\mathrm{CHP}}^{\mathrm{up}}$ , $Ram{p}_{\mathrm{CHP}}^{\mathrm{down}}$	±2	MW/h
CHP carbon emission coefficients	${\mu }^{\mathrm{CHP},\mathrm{a}}$ = 0.01 ${\mu }^{\mathrm{CHP},\mathrm{b} }$ = 0.2 ${\mu }^{\mathrm{CHP},\mathrm{c}}$ = 0.012	/
CHP thermoelectric coupling coefficients	${\lambda }_1^{\mathrm{CHP}}$ = 0.44 ${\lambda }_2^{\mathrm{CHP}}$ = 45.4 ${\lambda }_3^{\mathrm{CHP}}$ = 0.23 ${\lambda }_4^{\mathrm{CHP}}$ = 0.5	/
CHP efficiency coefficients	0.9	/
PV capacity	10	MW
PV LCOE cost $c^{\mathrm{PV}}$	250	Yuan/MWh
Bus capacity $P_{\max }^{\mathrm{grid}}$	5	MW
Pipeline capacity $V_{\max }^{\mathrm{gas}}$	1500	Nm3
Grid carbon emission factor ${\mu }^{\mathrm{gird}}$	0.85	tCO2e/MWh

. Notably, the stepped carbon trading mechanism implemented has a maximum floating tier of ±3 levels. For quantities that exceed or fall short of the carbon quota by more than 30%, the trading will still be carried out at the price corresponding to the ±3rd tier.

The loads aggregated by the VPP are shown in Figure 4; the retail electricity and heat prices, as well as the transaction prices in electricity balancing market and natural gas market, are presented in Figure 5; and the forecasted PV output is illustrated in Figure 6. Additionally, for elastic load control, the retail electricity price discount coefficients are set at ±5%, ±10%, and ±15%, making a total of six levels. The regulation range of the load at each level is derived from the accumulated data in the VPP dispatching system.

4.2. Scenarios and Results

To analyze the VPP strategy optimization problem under the electricity–carbon–certificate multi-market, four scenarios are established, as outlined in

Table 2. Scenario settings.

Scenarios	GEC Market (RPS)	Carbon Market (CA)
S1	Unimplemented	Unimplemented
S2	RPS weighting $\omega$ = 15%, $p^{\mathrm{GEC}}$ = $p^{\mathrm{penalty}}$ = 50 CNY/MWh.	Unimplemented
S3	Unimplemented	100% free CA; benchmark price $p^{\mathrm{C}\mathrm{O}2}$ = 100 CNY/tCO2e; carbon emission intervals in the stepped carbon trading $d$ = 10%; pricing adjustment factor $\alpha$ = $\beta$ = 10%; CA weight ${\omega }^{\mathrm{C}\mathrm{O}2}$ = 0.7.
S4	RPS weighting $\omega$ = 15%, $p^{\mathrm{GEC}}$ = $p^{\mathrm{penalty}}$ = 50 CNY/MWh.	100% free CA; benchmark price $p^{\mathrm{C}\mathrm{O}2}$ = 100 CNY/tCO2e; carbon emission intervals in the stepped carbon trading $d$ = 10%; pricing adjustment factor $\alpha$ = $\beta$ = 10%; CA weight ${\omega }^{\mathrm{C}\mathrm{O}2}$ = 0.7.

. The parameter settings for RPS weighting, GEC price, and carbon trading price are based on data from China’s official institutions, including NDRC (https://www.ndrc.gov.cn/), China’s Green Certificate Trading Platform (https://www.greenenergy.org.cn/), and Shanghai Environment and Energy Exchange (the designated organization for carbon trading management in China by MEE) (https://www.cneeex.com/).

In addition, for each scenario in the solution, the discount factor $\lambda$ is set to 0.7, with a maximum iteration count of 150. For the evolutionary algorithm, a genetic algorithm is employed, with a population size of 80, a maximum number of generations set to 150, a mutation probability of 0.01, and a crossover probability of 0.7. Based on the objective function outlined in Section 3, the results for each scenario are presented in

Table 3. Optimization results of objective function items (Units: CNY).

Items	S1	S2	S3	S4
Energy supply income $I^{\mathrm{sale}}$	156,633.56	157,974.54	158,005.38	157,905.99
CHP cost $C^{\mathrm{CHP}}$	44,285.50	52,524.36	58,362.51	63,983.05
Balancing market cost $C^{\mathrm{ele}}$	20,669.13	10,708.5	2305.43	−4,325.41
PV cost $C^{\mathrm{PV}}$	14,259.25	15,739.25	17,581.75	18,696.00
GEC item $C^{\mathrm{GEC}}$	/	1449.74	/	2044.58
Carbon trading item $C^{\mathrm{C}\mathrm{O}2}$	/	/	28,657.37	30,553.59
VPP revenue	77,419.68	80,452.19	108,413.06	112,150.51

.

To analyze the convergence characteristics of the model and solving method, we employed the “holistic modeling and solving” approach outlined in Section 1 and selected the GA for comparative analysis. The convergence values (measured by VPP revenue) and convergence times (computational duration) of both methods are presented in

Table 4. Comparison of convergence characteristics.

Items	Convergence Characteristics	MDP	GA
S1	Value (CNY)	77,419.68	75,901.65
	Time (s)	21.64	15.52
S2	Value (CNY)	80,452.19	76,621.13
	Time (s)	26.92	16.12
S3	Value (CNY)	108,413.06	96,797.37
	Time (s)	27.51	15.91
S4	Value (CNY)	112,150.51	97,522.18
	Time (s)	29.19	16.22

.

It can be observed that, due to differences in iteration patterns and problem complexity during the solving process, the proposed method requires a longer convergence time than GA, indicating its higher demand for and reliance on computational resources to some extent. However, the proposed method exhibits superior performance in terms of convergence values, particularly as the variety of market types increases. In such cases, its achieved convergence values significantly outperform those obtained by GA.

4.3. Discussions

To analyze the specific impacts of the multi-market on VPPs, the following discussion will focus on three aspects: VPP strategy, environmental performance, and economic performance.

(a)

VPP Strategy

The VPP’s strategies in the four scenarios are shown in Figure 7.

In Scenario S1, firstly, from 1:00 to 10:00 and from 21:00 to 24:00, due to the fact that the electricity price in the balancing market is lower than or close to the CHP cost, the VPP tends to minimize the CHP output and prioritizes purchasing electricity from the grid to meet the load demand. However, influenced by the rapid price increase in the balancing market at 11:00 and the CHP ramp rate constraints, at 10:00, even though the balancing market price is significantly lower than the CHP cost, the VPP notably increases the CHP output at 10:00 to ensure the electrical output capability at 11:00. This may even require selling excess generated electricity to the balancing market at a price below cost.

Secondly, from 11:00 to 20:00, the VPP increases the output of both CHP and PV because the balancing market prices during this period are mostly significantly higher than the generation costs of CHP and PV. Specifically, from 11:00 to 15:00, the VPP even sells electricity to the grid through the balancing market to generate revenue. It is worth noting that from 13:00 to 14:00, even though the VPP is selling electricity to the balancing market and the CHP still has ample spare generation capacity, the VPP still adopts a strategy of reducing elastic load power through DR. The reason for this is that during this period, the balancing market price is significantly higher than the VPP’s retail electricity price, generation costs, and DR cost. Compared to supplying electricity to the load, selling as much electricity as possible in balancing market brings more revenue to the VPP.

In S2, from 1:00 to 10:00 and from 16:00 to 24:00, although the VPP is constrained by the RPS, its strategies are relatively similar to that in S1, as these periods do not coincide with the peak output of PV. From 11:00 to 15:00, which is the peak period for PV output, under the RPS, influenced by the potential economic penalties and the potential income from GEC transactions, the VPP increases the PV output level, aiming to reduce the expected penalties and maximize the expected GEC transaction income at the end of the dispatch period.

In S3, from 1:00 to 10:00 and from 21:00 to 24:00, compared to S1 and S2, the VPP significantly reduces the amount of electricity purchased from the balancing market. This is because, under CA, both the operation of CHP and the purchase of electricity from the balancing market incur corresponding carbon emission costs. During these two periods, the average carbon emission factor of CHP is lower than that of grid electricity, making the total cost of supplying electricity through CHP (including fuel cost and carbon emission cost) lower than that of purchasing electricity from balancing market (including transaction cost and carbon emission cost). Therefore, within the constraints of ramp rates, power balance, and other factors, the VPP tends to maximize CHP output to reduce comprehensive costs.

During the peak PV output period (11:00 to 16:00), compared to S2, the VPP further increases PV output and the amount of electricity sold in the balancing market. This is because the current unit carbon trading price in China is significantly higher than the unit GEC price, leading to a higher expected carbon trading income under CA compared to the expected GEC trading income under RPS.

In S4, the strategies of the VPP from 1:00 to 8:00 and from 20:00 to 24:00 is relatively similar to those in S3, both tending to maximize the CHP output to reduce the expected carbon emission costs at the end of the dispatch period (and increase the expected income from carbon trading). Different from S1, S2, and S3, during 9:00 to 18:00 when PVs have generating capacity, the VPP significantly increases PV output. This is due to the combined influence of RPS and CA, through which the VPP can obtain potential multiple benefits by consuming PV power. From 11:00 to 12:00 and from 15:00 to 16:00, the VPP further enhances its ability to absorb PV power by increasing the power of elastic loads. Correspondingly, from 11:00 to 14:00, the VPP also increases its electricity sales through the balancing market. In addition to being influenced by the “RPS + CA”, these strategies are also affected by the difference between the VPP’s retail electricity price, the balancing market price, and the DR cost.

(b)

Environmental performance

Figure 8 presents the change rates of environmental indicators in S2, S3, and S4 compared to S1. Compared to S1, in terms of renewable energy consumption, S2 shows an increase of 10.38%, S3 an increase of 24.31%, and S4 an increase of 30.21%. As for carbon emissions, S2 dropped to approximately 76,300 metric tons, a decrease of 17.16%; S3 fell to around 54,900 metric tons, a 33.37% reduction; and S4 decreased to roughly 46,900 metric tons, a 43.08% decline.

In terms of carbon emissions, S2 decreased by 17.16%, S3 by 33.37%, and S4 by 43.08%.

It is evident that participating solely in the GEC market or carbon trading market (that is, implementing either RPS or CA alone), or in combination, significantly enhances renewable energy consumption and reduces carbon emissions. Notably, compared to participating solely in the GEC market, participating solely in the carbon trading market can achieve more significant environmental benefits. One of the key reasons for this is that the price of GECs is significantly lower than that of carbon trading in China.

Additionally, compared to participating solely in GEC market or the carbon trading market, participating in the electricity–carbon–certificate multi-market can further enhance environmental performance, but the marginal increase is diminishing. Apart from market price factors, the limitations in total installed PV capacity, as well as the scale and regulation capabilities of elastic loads, are important reasons for this phenomenon.

As illustrated in Figure 9, the description of PV output across the four scenarios can be summarized as follows:

As RPS, CA, and the combination of “RPS + CA” are implemented (that is, the multi-market), the VPP enhances the PV output level, particularly evident during the period from 11:00 to 15:00. The reason for this is that the enforcement of RPS and CA imposes mandatory constraints on the VPP to absorb renewable energy and reduce carbon emissions. Simultaneously, the resulting GEC trading and carbon trading broaden the VPP’s income channels, enhancing its willingness and ability to bear the risks associated with PV uncertainty. Especially during the PV output peak period from 11:00 to 15:00, when the VPP’s retail electricity price is significantly higher than the balancing market price and the PV cost, the VPP adopts a more aggressive PV output strategy.

However, it is important to note that in S4, at 13:00, the PV output does not show a significant increase compared to S3. This is because the VPP’s retail electricity price during this “mid-peak period” is lower than the balancing market price, limiting the VPP’s willingness to tolerate the PV uncertainties. To some extent, this demonstrates that relying solely on strong policy compliance constraints is not sufficient to continuously enhance the VPP’s capacity to absorb renewable energy. Other factors, such as market trading prices, also need to be taken into account.

(c)

Economic performance

From an overall perspective, the growth rates of revenue among the four scenarios are illustrated in Figure 10. Compared to S1, the total revenues of the VPP in S2, S3, and S4 have increased by 3.91%, 34.75%, and 44.86%, respectively. This indicates that, whether it is participating in the carbon market/GEC market alone or participating in the multi-market (that is, implementing RPS and CA either separately or in combination), this can boost the total revenue of the VPP. Although this is closely related to the capacity and structural ratio of the PVs, CHP, and aggregated load operated by the VPP in the case, when combined with the discussions of environmental performance, it suggests, to a certain extent, that with reasonable planning and allocation of unit capacities and loads, participating in the multi-market can promote both economic and environmental benefits.

From a structural perspective, the cost and income structures for the four scenarios are as depicted in Figure 11.

In terms of cost structure, the primary cost for the VPP in all four scenarios is the CHP cost, accounting for more than 50% in each case. The proportion of CHP cost gradually increases from S1 to S4. In contrast, the proportions of PV cost and DR cost do not exhibit significant changes. This is attributed to the fact that the carbon emission factor of CHP is lower than that of electricity from the grid, as well as the CA weight. As CA constraints tighten, the VPP tends to replace electricity purchased from the grid through the balancing market with CHP, which offers more comprehensive advantages.

In terms of income structure, under the multi-market, energy supply income remains the main source of income for the VPP in all four scenarios, accounting for more than 80% in each case. Notably, the income from GEC trading is significantly lower than that from carbon trading, and in S4, it is even less than the income from the balancing market. To some extent, this reflects the impact of the current multi-market mechanism and policy environment, where there exists a disparity and imbalance in prices between GECs and carbon trading.

5. Conclusions

This manuscript analyzes the potential impacts of the electricity–carbon–certificate multi-market and related mechanisms on the operation of VPPs. It proposes an optimization framework and model for VPPs under the multi-market. Furthermore, based on the concept of reinforcement learning, it characterizes the VPP optimization process as an MDP for solution. Finally, a case study is conducted on a VPP located in southern China, and the results are as follows:

• Whether participating solely in the certificate market, the carbon trading market, or engaging in the electricity–carbon–certificate multi-market, VPPs can significantly enhance their environmental performance. This includes improving their renewable energy consumption capacity and reducing total carbon emissions. Additionally, participating in the multi-market can broaden VPPs’ income streams and notably increase their total revenues. However, compared to energy supply income, the proportion of income derived from carbon trading, electricity sales in the balancing market, and certificate trading within the total income remains relatively limited. In the four scenarios analyzed in the case study, energy supply revenue accounted for over 80% of the total income in each case. Notably, the income from certificate trading accounts for an extremely minor portion of VPPs’ total income. Specifically, in the case analysis, green certificate trading income accounted for less than 1%, significantly lower than both carbon trading income and electricity sales income in the balancing market. This is attributed to factors such as the currently low price of GECs in China.
• Participating in the electricity–carbon–certificate multi-market can enhance VPPs’ willingness and ability to undertake the uncertainty risks associated with renewable energy, engage in DR programs, and participate in electricity sales in balancing market. Specifically, in the case study, the VPP under the multi-market adopted more aggressive dispatch strategies during peak PV output periods, resulting in a 31.11% increase in renewable energy consumption compared to the scenario without multi-market participation. This is of great importance for tapping into demand-side resources, facilitating the source–grid–load interaction, enhancing the level of renewable energy integration, and improving the overall energy structure.
• From a dispatch strategy perspective, under the multi-market, VPP entities can moderately increase the output plans for CHP and renewable energy generators within a certain range. This is because the potential revenues from certificates trading, carbon trading, and reduced electricity procurement costs can effectively offset—or even surpass—the risks and costs associated with renewable energy uncertainty. However, the timing and extent of output plan adjustments must be analyzed based on the differential margins among retail electricity price, balancing market price, and generation costs.
• Compared to participating solely in the certificate market, engaging in the carbon trading market or the multi-market yields more significant improvements in both the economic and environmental performances of VPPs. Taking the case study as an example, when the VPP participated in the carbon trading market, the profits increased by 33.65%, renewable energy consumption rose by 11.71%, and carbon emissions decreased by 27.97%. When participating in the multi-market, profits increased by 38.49%, renewable energy consumption rose by 18.79%, and carbon emissions decreased by 38.47%. On one hand, this can be attributed to the relatively high prices in carbon trading market and the stepped pricing mechanism. On the other hand, it is linked to the multiple benefits derived from consuming renewable energy electricity under the carbon market or the multi-market. These benefits include the increase in expected income from GECs and carbon trading at the end of the dispatch period, as well as a reduction in the expected penalties for non-compliance with RPS.