Power Shift: Decarbonization and the New Dynamics of Energy Markets

Abstract

This paper examines the transformative effects of decarbonization on electricity market design, emphasizing the challenges and opportunities posed by the rapid integration of renewable energy sources such as wind and solar. It analyzes the evolution of key wholesale market segments—including day-ahead, real-time, capacity, long-term purchase agreements, ancillary services, and transmission markets—highlighting their critical roles in managing the variability of renewable energy generation through efficient price signals and resource coordination. Variable renewable energy integration introduces significant operational challenges, including overgeneration risks, ramping capacity demands, forecast inaccuracies, and transmission constraints. Addressing these issues requires enhanced market flexibility, dynamic pricing mechanisms, and advanced real-time balancing strategies. This paper assesses these challenges, offering strategies to align generation with demand and optimize market outcomes. As electricity systems evolve, legacy market structures must adapt to incorporate carbon-free resources while maintaining grid reliability and economic sustainability. By exploring case studies such as Chile and California, this paper demonstrates the importance of targeted innovations in market design, regulatory frameworks, and operational technologies. It advocates for a holistic approach to ensure a reliable, affordable, and equitable transition to a decarbonized energy future.

1. Introduction

Decarbonization aims to reduce CO₂ equivalent emissions (CO₂e), the principal driver of climate change, significantly. This involves replacing conventional CO₂-emitting fossil fuel (FF) technologies with renewable energy (RE) technologies like solar, wind, and nuclear, and FF technologies equipped with carbon capture and storage (CCS). It also demands energy efficiency and reduced CO₂ emissions related to land use, land-use change, and forestry (LULUCF), and other human activities [1].

Decarbonization strategies shape energy markets in impactful ways. Civil society and regulatory frameworks play crucial roles by leveraging public pressure and implementing mandatory regulations, such as renewable energy targets (RETs), renewable portfolio standards (RPSs), or clean electricity Standards (CESs). These policies create markets for renewable energy certificates (RECs) or clean electricity certificates (CECs) and push for the phased retirement of fossil fuel plants. CO₂ taxes make FF-based energy more expensive, incentivizing investments in cleaner energy technologies and energy efficiency. Cap-and-trade systems create a market for carbon allowances, allowing emissions reductions where they are most economically viable. This top-down approach guides the transition to sustainable energy. In parallel, efforts to improve the efficiency of existing FF power plants through CCS technologies [2,3,4,5], combined heat and power (CHP) systems, and retrofitting for lower emissions aim to mitigate environmental impacts while maintaining energy security during the transition to net zero emissions.

Although complex to administrate, CO₂ quotas or licenses individually make consumers and businesses accountable for their carbon emissions, fostering energy-saving behaviors and cleaner energy choices. Mandated energy efficiency improvements offer another path toward reduced emissions across various sectors. While costly, subsidies for renewable energy, such as feed-in tariffs or premiums, lower the financial barriers to clean energy adoption. Research and development (R&D) investments are essential to spur innovation in energy generation, energy storage solutions (ESSs), systems integration and operation, and efficiency, unlocking long-term solutions and accelerating the transition to a decarbonized future.

Each instrument’s impact varies, affecting market dynamics, investment decisions, and the pace of transition towards a decarbonized energy future. The expected evolution of the power sector should be characterized by the gradual substitution of FF generation, those lacking CCS, with renewable sources like solar photovoltaic (PV), wind, nuclear, and other low CO₂e emission technologies.

Historically, the power system was designed for baseload plants to meet demand on continuous high-output operations. Thermal power stations, including coal, geothermal, and nuclear power, typically exhibit availability factors [6] ranging between 70% and 90% or higher. In contrast, renewables such as solar and wind offer variable output contingent upon weather, daylight, and wind conditions, resulting in minute, hourly, daily, seasonal, and yearly variations in energy supply. The increasing variability and uncertainty in power generation—both from utility-scale sources and a growing share of distributed energy resources (DERs)—across different geographic regions and temporal scales necessitates a more adaptive and responsive power system infrastructure.

This transition also occurs with a growing electrification of end-use sectors, such as transportation, heating, and industry. Electrification is a key strategy to decarbonize these sectors, as it allows for the direct use of renewable electricity and the integration of energy storage solutions (ESSs). The shift towards electric vehicles (EVs), electric heating systems, and electrified industrial processes increases electricity demand. While EVs offer cleaner operations by reducing greenhouse gas (GHG) emissions compared to internal combustion engine vehicles (ICEVs), life cycle assessments [7] indicate increased environmental and human toxicity impacts during their manufacturing phase, primarily due to the production of batteries and powertrain components.

On the other end of RE, geothermal energy [8] is a promising renewable energy source that provides consistent baseload power with minimal emissions without the problems of variable renewable energy (VRE) like wind and solar PV. Enhanced geothermal systems (EGSs) and binary cycle technologies are expanding the potential of geothermal energy beyond traditional high-heat areas. Despite exploration risk, high initial costs, and location-specific limitations, it offers significant potential for decarbonizing the energy market, especially when integrated with district heating systems or hybridized with other renewables. Countries like Iceland and New Zealand [9,10] demonstrate its viability, and ongoing developments could make it a more accessible and transformative solution globally. Similarly, nuclear power generation can provide consistent baseload power with low CO2 emissions. Advanced nuclear technologies, such as small modular reactors (SMRs), offer enhanced safety features, reduced costs, and greater flexibility in deployment. These innovations could be crucial in transitioning to a low-carbon energy system, complementing other renewable energy sources and ensuring a stable and reliable power supply.

Thus, the primary challenges for power system development and operation involve incorporating, integrating, and managing increasing shares of VRE and distributed generation (DG) sources. While traditional market structures are centralized, decentralization via DERs like rooftop solar photovoltaics, batteries, and microgrids disrupts this model. This shift introduces a more sizable number of generation units, several with small plate capacity and variable output profiles, geographically dispersed and out of the system operator’s (SO) reach. At the same time, the SO (independent systems operators—ISOs; regional transmission organizations—RTOs) [11,12,13,14] must ensure system reliability. To address these challenges, expanding transmission infrastructure, ESSs, accommodating VRE, and implementing resource-enhancing technologies, such as advanced communication and operational and control technologies that facilitate the coordinated dispatch of all resources to match supply and demand in real-time, are crucial. The baseload paradigm relying on large, centralized plants must evolve into a more flexible system capable of adapting to demand and renewable generation variability.

In the U.S., Independent System Operators (ISOs) emerged in response to Federal Energy Regulatory Commission (FERC) Orders 888 [12] and 889 [13] (1996) to ensure non-discriminatory access to transmission within integrated power pools, restructuring vertically integrated utilities. Later, Order 2000 [14] (1999) encouraged the voluntary formation of Regional Transmission Organizations (RTOs) to oversee transmission at a regional level, including Canada, outlining specific criteria for RTO qualification. Functionally similar to ISOs, RTOs, and ISOs play a crucial role in ensuring reliable system coordination, efficient market operations, transparent planning, and impartial grid access, mitigating conflicts of interest. Notable ISOs/RTOs in North America include Alberta Electric System Operator (AESO); Pennsylvania-New Jersey-Maryland (PJM); Electric Reliability Council of Texas (ERCOT); ISO New England (ISO-NE); New York Independent System Operator (NYISO); Midcontinent Independent System Operator (MISO); California Independent System Operator (CAISO); Southwest Power Pool (SPP).

Electricity markets are pivotal in facilitating trade and resource allocation by providing a structured framework for valuing and pricing tradable items. They can develop organically or designed to enable the transfer of ownership rights. In the electric power industry, various electricity market structures exist. Some rely on competitive bidding to determine market-based prices, while others depend on administratively defined prices.

Physical and financial markets exist in electricity and other markets. Physical electricity markets encompass the infrastructure, institutions, and participants in, for example, trading energy and related services like capacity and ancillary services. They are distinguished and identified by location-based physical attributes and system conditions. Locational Marginal Prices (LMPs) represent energy prices at specific grid nodes and times (spatial distribution of electricity prices). Trade can occur in real-time spot markets to balance instantaneous supply and demand of energy and forward markets for future delivery; depending on market constructs, timeframes can range from 5 min to multiple years. The market design aims to efficiently manage grid constraints and electrical losses while matching generation with load across the transmission system. Both spot and forward physical markets are essential for reliable grid management, planning, and investors’ decisions.

In contrast, financial electricity markets, also known as futures and derivatives, center around trading financial instruments whose values derive from underlying physical electricity prices. Financial instruments or derivatives in the electric industry are contracts that derive their value from an underlying financial asset, commodity, index, market metric, or other measurable event like a weather or climate event. In the electricity sector, derivatives include forwards, futures, swaps, options, securities, and insurance based on electricity prices. A derivatives transaction does not typically involve the transfer of any physical product—just money. Electricity futures and similar derivatives enable market participants to mitigate or hedge price risks, like LMPs, within a competitive electricity market. These futures contracts, being legally binding and negotiable, stipulate the future delivery of a commodity. In many instances, physical delivery does not occur, and the closure of the futures contract is executed through buying or selling on or around the delivery date. Financial instruments include derivatives [15,16], securities, options, swaps, forward and futures, and insurance products linked to energy assets. These markets have distinct structures, institutions, participants, and products influenced by financial supply and demand drivers. Participants may use financial markets for risk management, speculation, or portfolio optimization without owning physical assets. These markets provide flexibility and liquidity.

Thus, the key distinction between physical and financial electricity markets is in the nature of the traded products and participants’ motivations. Recognizing this difference is vital for the electricity market’s design and analysis. EIA data show that 2021 US electric industry sales to ultimate customers reached USD 422 billion, while for the Americas, the energy derivative market size, mainly the US, by notional value, registered by the World Federation of Exchanges for CME Group [17] was USD 33 trillion, which highlights the difference in market size between the physical market and the derivatives market, with the last being about 80 times bigger.

Flexibility has become essential in decarbonized markets, as the fluctuating output from VRE sources requires energy storage (ES), demand response (DR), and flexible generation to maintain grid stability. Accommodating the growth of VRE and distributed energy resources (DERs), significant investments in grid infrastructure, including transmission upgrades, ES, more granular pricing, flexible markets, and advanced smart metering and management technologies, are crucial. Decarbonization also calls for comprehensive policy and regulatory reforms, such as RPSs, carbon pricing, and streamlined permitting, to facilitate an efficient integration of renewable energy. These regulatory shifts have and continue to influence market dynamics and investment decisions, steering capital away from FF infrastructure towards RE projects and technologies.

This paper analyzes trends on the impact of decarbonization on wholesale electricity markets, on markets where supply and demand dynamics, bilateral agreements, and administrative price-setting models determine prices. However, the decarbonization of the energy matrix is not the only driver of electricity markets [18]. As economies worldwide transition toward low-carbon energy sources, electricity markets must adapt to accommodate these new realities (e.g., rising VRE integration, price volatility, and challenges for baseload power plants). While this analysis primarily emphasizes the effects of decarbonization on competitive physical wholesale electricity markets, it is essential to briefly mention the interconnected nature of financial markets, given their significant role in hedging risk and supporting investment in the evolving energy landscape.

To effectively navigate the complexities of decarbonizing electricity markets, this paper adopts a conceptual framework that integrates key components driving market evolution and operational adaptation. The framework centers on three interconnected pillars: the integration of VRE, the impact on markets and structures, and the role of policy and technological innovation. These pillars interact dynamically, shaping how markets address intermittency, grid flexibility, and financial sustainability while pursuing decarbonization goals. By connecting operational challenges with economic and environmental objectives, this framework provides a cohesive lens to analyze the evolving dynamics of electricity markets, emphasizing the importance of holistic strategies to ensure reliability, affordability, and sustainability in the energy transition. This approach also highlights feedback loops where policy and technological advances influence market outcomes, fostering resilience in an increasingly decentralized and renewable-powered energy landscape.

The results emphasize that existing market designs, incentives and pricing mechanisms, and grid infrastructure must adapt to integrate VRE while ensuring reliability and affordability efficiently. DERs are disrupting market structure, and price volatility is expected to increase, as VRE can cause significant price drops during periods of overgeneration and low demand, as well as sharp spikes during shortages. This volatility impacts power plant revenues, with FF generators seeing reduced operating hours and declining revenues, while VRE projects, supported by long-term contracts and evolving market incentives, become more economically viable. This paper also underscores the role of policy and regulatory frameworks, such as CO₂ taxes, RETs, RPSs, RECs, and CESs, in shaping market dynamics and investment patterns. Additionally, the transition towards a low-carbon energy system is introducing new market players and creating opportunities for innovation. Still, it also poses economic challenges, including the risk of stranded assets in the FF sector.

This paper advances the understanding of renewable energy integration and electricity market design by building on and extending prior research. Previous studies, such as [19], underscore the importance of flexibility for managing the variability of VRE. Similarly, ref. [20] highlights the critical role of market design elements, including dynamic pricing, ancillary services, and cross-border trading. This paper builds on these frameworks by analyzing specific market segments—day-ahead, real-time, capacity, long-term purchase agreements, centralized dispatch systems, transmission markets, and ancillary markets—and their effectiveness in managing variability through pricing signals and coordination mechanisms. This paper also addresses operational challenges, such as overgeneration risks, ramping demands, and forecast inaccuracies, identified by [21]. It proposes dynamic pricing and real-time balancing strategies, leveraging advancements in machine learning and grid automation, as discussed by [22]. Real-world validation is provided through case studies: Chile’s market integration, as analyzed by [23], demonstrates how infrastructure investments and regulatory frameworks drive renewable deployment and stabilize prices, while California’s flexible resource adequacy policies and advanced forecasting methods, highlighted by [24], serve as a model for integrating high shares of renewables. Finally, this paper builds on the works of [25,26] by advocating for research and development investments in hybrid energy storage systems and green hydrogen technologies. These innovations address critical intermittency and grid flexibility challenges, providing a forward-looking approach to the decarbonization of electricity markets.

This paper is organized into distinct sections to address the transformative effects of decarbonization on electricity markets systematically. It begins with an Introduction, which discusses the shift toward renewable energy, the challenges in market designs, and the importance of policy and technological advancements. The Materials and Methods Section outlines a mixed-methods approach that integrates qualitative and quantitative analyses, focusing on three core pillars: the integration of variable renewable energy (VRE), market structure transformation, and the role of policy and innovation, with case studies from regions like Chile, the US, and the EU. The Wholesale Electricity Markets Section examines how key market segments adapt to decarbonization, including energy, capacity, ancillary services, and transmission. The Decarbonization in Electricity Markets and Trends Section explores the implications of increasing VRE integration, such as more significant capacity requirements and operational reserves, with detailed case studies from Chile and California. Finally, the Discussion and Implications Section emphasizes the need for policy innovation and market resilience. At the same time, the Conclusion synthesizes the findings, offering strategies to align renewable integration with goals of efficiency, reliability, and sustainability. This comprehensive structure ensures a cohesive analysis of decarbonization’s challenges and solutions.

2. Materials and Methods

This study adopts a mixed-methods approach to examine the transformative impact of decarbonization on electricity market design, integrating qualitative and quantitative analyses within a cohesive conceptual framework. By synthesizing insights from the peer-reviewed literature, policy reports, and case studies, this research identifies emerging trends, evaluates proposed solutions, and provides an evidence-based analysis of electricity market dynamics. Key data sources include reports from government agencies and leading organizations such as the International Energy Agency (IEA), RTOs/ISOs, and open-access repositories. Case studies from regions undergoing significant energy transitions, including the European Union, the United States, and Chile, highlight regional challenges and opportunities in integrating high shares of variable renewable energy (VRE).

The analysis focuses on three interconnected pillars: (1) the integration of VRE, (2) the transformation of market structures, and (3) the role of policy and technological innovation. These pillars are examined to address critical challenges such as intermittency, grid flexibility, and financial sustainability while advancing decarbonization goals. The conceptual framework synthesizes trends in market dynamics, operational challenges, and policy adaptations. It illustrates how market design, renewable integration, and system flexibility interact to shape the future of decarbonized energy systems. This comprehensive approach emphasizes the importance of holistic strategies to ensure reliability, affordability, and sustainability while fostering resilience in increasingly decentralized and renewable-powered energy landscapes.

3. Wholesale Electricity Markets

The design of electricity markets varies across countries and regions, yet a common overarching framework prevails. This framework typically encompasses energy, capacity, ancillary services, transportation, storage, long-term purchase agreements (LTPAs) or power purchase agreements (PPAs), and distribution markets, some of which operate at wholesale and retail levels. Retail markets pertain to selling electricity directly to consumers in distribution markets. In contrast, wholesale markets predominantly revolve around electricity transactions among generation and transmission sectors, electric utilities, and electricity traders before the final sale to consumers or large consumers, typically at high voltages. In contrast, retail electricity markets center around the sale of electricity within an electricity distribution grid, often at lower voltages, targeting end consumers, including households, commercial establishments, and smaller industrial consumers. The local distribution company and third-party aggregators or suppliers utilizing the distribution grid commonly manage this sale.

Historically, real-time operations in the energy sector were defined by the constant variation of demand, where capable generation units adapted their output in a process known as load following to accommodate these ongoing demand shifts. ISO/RTOs utilize a centralized economic dispatch based on generators/demand bids, or power plants’ operational costs merit order, to optimize the use of these generators and minimize real-time operational costs.

Key wholesale market segments include bilateral long-term contracting, day-ahead scheduling, real-time dispatch, capacity auctions, ancillary services procurement [27], and transmission markets. When the conditions enable competitive markets, each segment establishes prices through competitive mechanisms based on supply and demand fundamentals. Well-designed and complete markets provide incentives and price signals to attract investment in a flexible, low-carbon resource mix aligned with policy goals. Wholesale electricity markets are organized into several segments where prices are determined:

• Electricity energy markets—Long-term power purchase agreements, day-ahead markets, and real-time/balancing markets facilitate the buying and selling of electricity for final consumption.
• Reliability/Capacity markets—Capacity obligations are purchased through centralized auctions and bilateral contracts to ensure adequate generation and demand-side resources are available to meet system reliability requirements.
• Ancillary services markets—Grid resources bid or agree on bilateral contracts to provide essential reliability services like frequency regulation, operating reserves, voltage support, and black start.
• Transmission markets—Prices for transmission networks, like access rights, are determined to manage grid congestion and losses efficiently.

3.1. Day-Ahead and Real-Time/Balancing Markets

Day-ahead and real-time wholesale electricity markets facilitate reliable grid management and transparent pricing. In the day-ahead market, generators offer their availability, and buyers bid on their demands for the next day, resulting in financially binding schedules and prices. Then, in real-time, actual generator dispatch and pricing are determined every 5–15 min based on market design and precise system conditions. The day-ahead projections provide price certainty for participants to plan, while the real-time market ensures supply and demand are balanced. These organized markets dynamically coordinate competitive resources on the complex challenges of electric grid operations.

Some electricity systems that utilize day-ahead and real-time market structures are PJM, which covers parts of the Mid-Atlantic and Midwest US states [28,29]; ISO-NE, responsible for the six New England states [30]; CAISO, which manages most of California [31]; and NYISO, covering New York state [32]. ERCOT oversees most of Texas [33]. Power Exchanges organize day-ahead and intraday electricity markets in the European Union [34], where the day-ahead market uses a daily blind auction with participants submit locational orders to buy or sell electricity for each hour of the next day, matching supply and demand to determine locational market clearing prices. The intraday market enables continuous trading, allowing adjustments closer to delivery time and enhancing flexibility. Both markets are integrated across European borders, with clearing houses ensuring transaction fulfillment and secure operations.

Other regions include Europe, with countries like Germany, the UK, and the Nordics participating in these systems [35,36,37]. In Australia, the Australian Energy Market Commission (AEMC) manages electricity markets [38,39,40], where generators submit locational offers over 5-min intervals to the Australian Energy Market Operator (AEMO), which uses a security-constrained economic dispatch model every 5 min to determine generation merit order. In 2021, AEMC transitioned the electricity spot price settlement interval from 30 min to 5 min to provide a more precise signal for fast-response technologies, such as batteries and gas peaking plants. Norway also follows a similar market structure, with generators submitting bids based on their short-run marginal costs into the Nord Pool day-ahead and intraday markets. Nord Pool calculates hourly market clearing prices based on supply and demand fundamentals [41].

Day-ahead and real-time markets launched in Singapore in 2003, with wholesale electricity prices changing every half-hour [42]. Canada features two distinct systems: Alberta, where generation is competitive while transmission and distribution remain regulated, and Ontario, which employs a hybrid model under the management of the Independent Electricity System Operator (IESO) [43,44,45]. Finally, New Zealand has adopted similar market structures [46,47].

In the US [48], restructured competitive wholesale markets have dominated in the Northeast, Mid-Atlantic, Texas, and parts of the Midwest. Seven RTOs operate wholesale electricity markets in the United States: ISO-NE, NYISO, PJM, MISO, CAISO, SPP, and ERCOT. Of these, only ERCOT remains under state jurisdiction, as its operations are contained entirely within Texas. The other six RTOs are subject to the oversight of FERC.

Meanwhile, traditional cost-based utility dispatch, explained below, continues to be used across much of the Southeast, Southwest, and Northwest, as well as some isolated systems where vertically integrated utilities have persisted after deregulation. Thus, most utilities and retail choices have generally embraced day-ahead and real-time market structures in large market-oriented economies with larger interconnected systems and diverse resources.

While ISO/RTO runs day-ahead (or other time intervals) and in real-time/balancing electricity markets, some general features or main architectural characteristics, which vary according to market design rules, are as follows:

• Day-Ahead Markets:

• Operate the day before electricity delivery (hence “day-ahead”).
• Participants submit bids to buy or offers to sell electricity for each hour of the next day.
• The system operator matches supply and demand for each hour to determine the market clearing price at the minimum cost, safeguarding system reliability and electricity transmission constraints.
• Provides price signals for generators and load-serving entities to plan their operations and bidding for the next day.
• Results in financial commitments to buy or sell electricity at the day-ahead price.

• Real-Time/Spot Markets:

• Operate in real-time as electricity is delivered, with intervals that can be 5 min.
• The ISO/RTO uses it to balance differences between day-ahead schedules and real-time demand/generation at the minimum cost, safeguarding system reliability and electricity transmission constraints.
• Prices fluctuate minute-to-minute based on actual system conditions.
• Generators are paid the real-time price for any extra electricity produced.
• Load-serving entities pay the real-time price for any extra electricity consumed.
• Provides price signals to incentivize generators to follow dispatch instructions.

• Key Differences:

• Day-ahead prices are based on expected conditions and bids, whereas real-time prices reflect actual conditions.
• Day-ahead markets provide financial certainty, while real-time markets balance physical differences.
• Day-ahead markets have higher trade volumes, while real-time markets ensure reliability.
• Day-ahead prices are generally less volatile than real-time prices.

The actual physical operation of balancing electricity supply and demand often deviates from the projections made in the day-ahead electricity market. These disparities can arise due to unforeseen shifts in supply or demand conditions that differ from the anticipated scenarios envisioned during the day-ahead electricity market forecasting and closure.

In summary, the day-ahead market facilitates forward price discovery and financial commitments, while the real-time market ensures the physical balance of supply and demand. Together, these markets contribute to an efficient and reliable wholesale electricity market.

3.2. Capacity Market

A capacity market is a pivotal mechanism ensuring an ample supply of future generation, encompassing potential demand-side capacity aligned with projected needs. Its core mission is to stimulate investments in new capacity and secure commitments through auctions, often scheduled three or more years ahead. Participants receive capacity payments for pledging their capacity, irrespective of actual energy dispatch.

In the United States, some ISOs/RTOs feature capacity market mechanisms. However, it is crucial to note that not all ISOs/RTOs in the US [49] adopt capacity markets; some rely solely on energy markets. Four RTOs—MISO, PJM, NYISO, and ISO-NE—currently administer capacity markets through their footprint. These markets secure future resource adequacy commitments to meet peak demand forecasts. The other three RTOs (CAISO, SPP, ERCOT) fulfill capacity needs through alternative mechanisms and do not operate formal capacity market constructs.

PJM’s capacity market, known as the reliability pricing model (RPM) [50], stands out prominently. Operated by PJM Interconnection, the largest wholesale electricity market operator in the US, RPM plays a critical role in ensuring a reliable and secure electricity supply. RPM operates through periodic auctions designed to procure commitments from generators, load-serving entities, large consumers, and other resources to provide a specific amount of generation capacity in the future, typically three years ahead. Participants in RPM auctions commit to delivering capacity during specified future periods, assuring the grid operator of sufficient resources, especially during peak periods. The RPM, as a capacity market, serves to provide investment signals, with the auction process intending to send price signals encouraging new investments where and when needed for system reliability. Generators that clear the auction and commit to providing capacity receive compensation, regardless of whether they produce electricity during the contracted period. In PJM, the RPM may also allow demand-side resources, like demand response programs, to participate, offering a comprehensive approach to meeting capacity needs.

Some ISO/RTOs, such as PJM [51], incorporate a penalty structure enforcing capacity commitments. Resources committed to supply capacity face financial penalties if unavailable during the delivery year. This structure ensures resources meet their reliability obligations, with potential penalties serving as a crucial incentive for standing behind auction commitments.

The capacity mechanisms in the EU [52] have evolved as tools to ensure adequate electricity generation capacity is available to meet demand, especially during peak times. They have been a subject of regulatory development since the early 2000s, with key moments in 2003 and 2009 and a significant overhaul in the 2019 electricity market reform. Initially introduced to address the “missing money” problem—where energy-only markets did not provide sufficient incentives for investment in new capacity—the EU has progressively aimed to limit their use on a perceived risk of potential market distortions. In response to the energy crisis following Russia’s invasion of Ukraine, the EU implemented several emergency measures, and the European Commission proposed comprehensive electricity market reforms in March 2023, which were agreed upon by EU member states in December 2023, where among the key elements is the support for long-term capacity mechanisms aligned with decarbonization goals.

Chile, which pioneered the liberalization of electricity markets in the 1980s [53], established a payment mechanism for capacity, currently a power sufficiency mechanism, to ensure sufficient generation capacity is available to satisfy the maximum demand for the electrical system. This capacity payment is determined based on each plant’s contribution to system reliability, availability during peak demand periods, and the overall needs of the electrical system to maintain an adequate reserve margin. However, the capacity payment is not made irrespective of whether a generator unit is called in the day-ahead operations planned dispatch. Instead, the system coordinator calculates the capacity payment annually to adjust for changes in the electricity market, demand forecasts, and the entry or exit of generation units [54]. Additionally, Law Decree No. 70 (Decreto Supremo No. 70 of 2023-DS 70-) from the Chilean Ministry of Energy [54] introduced key updates to recognize better and compensate energy storage systems and hybrid plants with storage capacity. These changes include a methodology to evaluate and recognize the power capacity of stand-alone energy storage systems, rules for determining capacity payments for renewable plants with storage, and improved availability of data and studies to identify peak hours for calculating and paying for capacity more accurately.

The debate over electricity market design—specifically between energy-only models with scarcity pricing and energy-plus-capacity markets—remains complex and contentious, with several critical points of dispute:

• Market Efficiency: Arguments favoring capacity markets emphasize their ability to ensure sufficient generation capacity to meet peak demand, thereby maintaining grid reliability [55,56,57,58]. Energy-only markets depend on scarcity pricing during high-demand periods to provide the “missing money” needed to support capacity investment. Without effective scarcity pricing, the market fails to incentivize adequate capacity, creating a risk of supply shortages. Conversely, arguments against capacity markets highlight their potential to distort price signals, which can lead to costly overinvestment in unnecessary capacity and contribute to a suppressed price problem, where energy prices are artificially lowered due to capacity payments [57,59,60,61,62].
• Cost to Consumers: Capacity markets are argued to promote long-term price stability and secure supply, benefiting consumers in the long run [55,56,63]. Conversely, some arguments contend that these markets increase consumer costs by compensating generators for potential capacity rather than actual production, which may not always be economically justified [57,59,61,62,64].
• Impact on Renewable Energy: Arguments in favor suggest that capacity markets can facilitate renewable integration by ensuring a reliable backup capacity, supporting the transition to cleaner energy systems [57,58,62,63]. However, opposing arguments warn that capacity markets often favor traditional fossil fuel generators, potentially hindering the transition to renewable energy sources [57,59,61,62,65].
• Complexity of Market Design: Capacity markets are argued to be adaptable, allowing for tailoring to meet specific regional needs and policy goals while stabilizing prices that might otherwise become volatile under scarcity pricing [56,57,66,67]. In contrast, arguments against capacity markets suggest they add unnecessary complexity to electricity markets, creating opportunities for manipulation by market participants [57,62,64,68].
• Impact on Innovation: Some arguments suggest that capacity markets provide revenue certainty, which can encourage investment in new technologies and support innovation [55,59,60]. However, counterarguments claim that capacity markets may stifle innovation by favoring established technologies and existing market structures over more disruptive advancements [57,61,62,68].

While capacity markets may offer stability and reliability, they also bring concerns about costs, complexities, and potential drawbacks for market power, innovation, and renewable energy integration. This controversy reflects broader debates about how best to ensure electricity reliability, affordability, and sustainability in evolving power markets. The appropriateness of capacity markets often depends on specific regional circumstances and policy priorities, where we should acknowledge that both energy-only and energy-plus-capacity markets are subject to market power and imperfections, which must be addressed in any market design. Notwithstanding, dynamic modeling that integrates energy and capacity pricing can offer significant benefits, capturing the effects of long-term investment decisions and technological advancements. While simpler market structures have the advantage of lower transaction costs, they often impose constraints, such as setting upfront capacity prices to zero, which can reduce optimization variables and welfare compared to more flexible systems that allow for dynamic energy and capacity pricing.

Innovation incentives and risk allocation are vital considerations in any market design, whether it follows an energy-only or energy-plus-capacity framework. These factors should be explicitly accounted for, as they influence market adaptability and encourage new technologies.

Setting prices optimally in complex markets is challenging, yet arbitrarily constraining prices—such as eliminating capacity pricing—can lead to even more significant losses in welfare. Thus, with a broader range of pricing options, the theoretical advantages of energy-plus-capacity markets may yield tangible benefits if these market designs can navigate real-world challenges effectively. In this way, markets with greater flexibility and nuanced pricing mechanisms can better support long-term goals of stability, innovation, and welfare. While both market designs face challenges, a well-designed market might aim to maximize social welfare, particularly when considering long-term, dynamic effects. However, the effectiveness of either approach ultimately depends on the specific implementation and how it addresses real-world market complexities.

3.3. Long-Term Purchase Agreements

In general, long-term purchase agreements (LTPAs) are bilateral contracts between buyers and sellers, facilitating the purchase of electricity over extended periods, typically one year or longer. These agreements are voluntary and privately negotiated, existing externally in ISO/RTO markets where the ISO/RTO is not directly involved.

Nevertheless, the ISO/RTO considers the impact of the LTPA on scheduling, resource dispatch, and market prices. Load-serving entities and large consumers utilize these agreements to secure generating capacity and energy, aligning with their anticipated demand projections.

In PJM, generators engaged in the ISO/RTO capacity market may offer surplus capacity not committed in the LTPA to ISO/RTO reliability pricing model (RPM) auctions.

During day-ahead market scheduling, resources bound by LTPA submit contracted MW amounts on their offers/bids, which ISO/RTO utilizes for dispatch purposes. In real-time operations, resources under long-term contracts can be self-scheduled or economically dispatched. LMPs remain unaffected by the nature of bilateral transactions under long-term agreements.

Thus, while the practice is that the ISO/RTO acknowledges the existence of an LTPA, they refrain from direct negotiation or regulation of these external transactions. However, the ISO/RTO incorporates the quantities from these agreements in its scheduling and dispatch processes to uphold system reliability.

3.4. Ancillary Services Market

Ancillary services (ASs) help manage the various technical challenges that arise in generating, transmitting, and distributing electricity; the ancillary services market sometimes called the reliability market, is a crucial component of electricity markets, providing essential support services to ensure the power system’s reliability, stability, and flexibility. Reliability Markets focus on securing essential grid reliability services, such as frequency regulation, operating reserves, and voltage support, to manage system variability and contingencies. These markets typically value resources that enhance grid stability, with participants committing to providing reliable services and receiving payments based on real-time availability and performance.

The provision of AS is critical to real-time grid operations. Operators continuously monitor the grid and, based on system conditions, call upon resources that provide ASs to address specific needs. These services specifically target frequency control, voltage support, and grid stability issues, operating within short timeframes, even microseconds, extending beyond electricity generation and delivery. By swiftly addressing sudden shifts in demand, like the loss of sizeable industrial consumption, unforeseen outages, or disruptions, AS plays a pivotal role in averting voltage instability, frequency deviations, and other potential challenges that might escalate to a system-wide blackout, thereby enhancing grid reliability.

ASs are typically procured through auctions or competitive bidding processes facilitated by the ISO/RTO. The system operator conducts competitive auctions to secure ASs that align with operational requirements. Participants, encompassing generators, demand response providers, and resources such as energy storage facilities capable of instantly or rapidly delivering energy, submit economic offers detailing the quantity and price of their services. The operator compiles aggregate supply curves from these offers and identifies clearing quantities and prices at the intersections with demand requirements. Subsequently, service providers receive dispatch instructions, prompting them to stand by or activate services based on real-time system conditions.

Regulation resources respond to automated signals to stabilize fluctuations, ensuring a continuous balance in the system. In parallel, reserve resources stand ready for contingencies. Regulation markets undergo frequent clearings with pricing dynamically adjusted in real-time, whereas reserve markets generally feature longer clearing intervals with fixed pricing. Payments are determined by both market clearing prices and the energy supplied upon activation. Aligning AS commitments with energy market dispatch enables the system operator to uphold reliability efficiently and cost-effectively.

The specific payment structure is contingent on the type of AS and the governing market rules. In some markets, participants offering AS receive compensation for their contributions, possibly in capacity payments, energy payments, or a combination.

ASs are intricately linked with energy markets. Resources such as flexible generators, energy storage facilities, or demand response units can provide energy and AS. This integration fosters a more streamlined and coordinated operation of the entire power system.

3.5. Cost Base Centralized Systems

In theory, a system with a centralized dispatch based on reported marginal costs holds the potential to replace conventional day-ahead and real-time electricity markets.

In this model, generators would submit their short-run marginal costs directly to the system operator instead of participating in market auctions. These declared costs reflect the expense of producing each facility’s next megawatt (MW) of power. An auditing process for submitted generator marginal costs would be essential to ensure transparency and prevent manipulation. Incentives and penalties based on accuracy would guarantee the integrity of cost declarations.

The overarching goal of the ISO would be to minimize total system production costs. This entails dispatching the lowest-cost resources to meet demand requirements and ensure reliability. The system operator would employ security-constrained unit commitment and economic dispatch algorithms to schedule resources, aligning with forecasted load across various time horizons and resource availability, from years and months to day-ahead and real-time.

While generators undergo central dispatch in merit order, prioritized by their declared marginal costs, ranging from the lowest to the highest, market clearing prices would be removed, and generators would receive compensation based on a price defined by the reported marginal costs of the most expensive dispatched unit for the energy exchanged during the actual dispatch. Depending on the design of government rules, load-serving entities, and aggregators would then remit average rates to final consumers, encompassing total production costs. Alternatively, prices for large energy consumers or load-serving entities might be predetermined through bilateral contracts with power generators or aggregators. In such cases, the real-time system marginal cost, determined by declared costs, becomes the basis for the price on power exchanges between producers facing deficits in their long-term agreements and supply commitments with significant energy consumers or load-serving entities.

As crucial for grid stability, it could also be procured based on systems costs, long-term auctions, central dispatch, and energy. Competing investment in new resources could be achieved through long-term planning processes, capacity payments, private investment initiatives, auctions, and procurement mechanisms overseen by the SO or a regulatory body.

Some countries and regions that use or have used electricity dispatch systems based on declared marginal costs rather than market pricing include France [69,70,71]; vertically and formerly vertically integrated utilities in the United States, such as those in the Southeastern US [48], served by companies like Southern Company [72], Duke Energy [73], and Tennessee Valley Authority [74]; the Western US, including Colorado, Wyoming, and Montana [48]; Hawaii, where the State of Hawaii Public Utilities Commission oversees four electric utility firms generating, acquiring, transmitting, distributing, and selling electric energy. Hawaii’s six primary islands operate independent electrical grids without interconnection, with the HECO Companies (HECO, MECO, and HELCO) serving approximately 95% of the state’s population. At the same time, KIUC caters to the remaining 5% on Kauai. Notably, the islands of Niihau and Kahoolawe lack electric utility services [75,76]; Alaska [77,78]; parts of the Midwest, including Missouri and Oklahoma [48]; Chile, which relies on a capacity payment system and a marginal cost-based dispatch. Where on December 2023, the national electric system (Sistema Eléctrico Nacional, SEN) has an installed capacity of 33,831.6 MW, with 62.7% corresponding to renewable sources (21.9% hydraulic, 26.4% solar, 13.7% wind, 2.2% biomass, and 0.3% geothermal) and 35.1% to thermal sources (11.0% coal, 15.7% natural gas, and 8.4% oil). [79]; Argentina [80]; Peru [81], where the Committee for Economic Operation of the System (Comité de Operación Económica del Sistema Interconectado Nacional, COES SINAC) acts as the ISO. The country relies on marginal cost-based economic dispatch, where generators submit availability, technical parameters, and short-run marginal costs for unit commitment and dispatch processes from the day-ahead to real-time. Additional revenues come from spot energy, capacity, and ancillary services markets based on marginal costs and auction clearing price; and China [82,83], which is the transition from a planned dispatch approach to a more market-oriented system but still heavily relies on a regulated cost-based dispatch model in many regions.

In the past, some hydro-heavy systems have effectively used centralized optimization relying on generator cost submissions, as was the case in Norway, which later transitioned to a wholesale market system. However, in the last three decades, a centrally optimized security-constrained economic dispatch, relying on audited marginal cost declarations, has led in many places to the new day-ahead and real-time market price signals. Still, smaller systems use effective centralized dispatch based on marginal costs. Nonetheless, large and diverse markets generally rely on competitive day-ahead and real-time pricing, like the US, market-based pricing, and Norway, which also depends on competitive pricing via Nord Pool’s Day-ahead/Intraday markets.

3.6. Transmission Markets

The transmission market plays a critical role in ensuring the efficient delivery of electricity, especially as renewable energy sources become more integrated into the grid. Transmission facilitates the flow of electricity from remote generation sites to consumption centers, maintaining system reliability and market efficiency. By connecting renewable-rich regions with consumption hubs, the transmission market enables the full utilization of distant generation sources, such as hydro, wind, and solar. It reduces regional price disparities by alleviating congestion and allowing for electricity to flow to areas of higher demand.

The classification and regulation of transmission lines vary across jurisdictions and are influenced by several key factors. Voltage levels and their impact on the system play a critical role, with higher-voltage lines typically subject to stricter open-access regulations due to their significant contributions to grid reliability and regional connectivity. Investment structures also shape regulatory approaches, such as the funding source—whether public or private—and the mechanisms for cost recovery and risk allocation influence market dynamics and policy decisions. Additionally, market design elements such as nodal versus zonal pricing, congestion management strategies, and system operation requirements are pivotal in determining how transmission networks are organized and managed, reflecting each jurisdiction’s unique needs and priorities.

Also, transmission lines are typically characterized by their capacity, exclusivity, or shared usage and their role in linking various regions or connecting major consumption hubs with significant production centers.

(a)

Open-access interconnection lines are essential for interregional power exchanges and market integration. They provide non-discriminatory access to all qualified market participants, usually under regulated tariffs. Examples include the following:

• US interstate transmission lines under FERC Order 888 [84].
• European cross-border interconnectors are regulated by EU Regulation 714/2009 [85].
• Chile’s north-south interconnection system, Sistema Eléctrico Nacional (SEN) [86].
• Australian interstate connectors in the National Electricity Market (NEM) [87].

(b)

Dedicated transmission lines (generator lead lines) connect specific generation facilities to the main grid or load centers. Usually funded and owned by the generator, these lines often have limited third-party access. Examples include lines that connect remote renewable projects, large thermal power plants, and mining operations.

(c)

Shared access (limited participation) lines serve multiple users under restricted access schemes. Costs for these lines are typically shared among predefined users, and private agreements with possible regulatory oversight often govern them. Examples include shared infrastructure for mining companies or industrial parks.

(d)

Hybrid access lines blend dedicated use with open access requirements, where the primary capacity is reserved for an anchor user, but excess capacity is available for others through open access provisions. This often requires regulatory approval and may include compensation mechanisms for the original investor.

• Transmission Market Structures, Financial Instruments vs. Centralized Expansion:

Energy markets utilizing transmission rights and financial instruments, such as those in the US (PJM, ERCOT, CAISO, MISO), use financial products like congestion revenue rights (CRRs) or financial transmission rights (FTRs) to allow for market participants to hedge against congestion costs. The Nord Pool region in Europe uses similar mechanisms to hedge against transmission congestion. This decentralized approach incentivizes efficient grid use and encourages investment in reducing congestion. Australia’s NEM operates with LMPs and is moving toward adopting financial transmission rights.

In contrast, markets with centralized transmission expansion (postage stamp system), such as Chile, Brazil, and Mexico, manage transmission centrally. A “postage stamp” system is used, where all users share costs equally regardless of location or contribution to congestion. In these systems, a central authority plans and oversees transmission expansion, ensuring grid reliability and capacity for renewable integration.

• Examples:

• Chile: The Coordinador Eléctrico Nacional (CEN) [88] centrally plans transmission expansions, applying uniform tariffs across all users.
• Brazil: Transmission is centrally regulated by Agência Nacional de Energia Elétrica (ANEEL) [89], with expansion procured through public auctions.
• Mexico: Centro Nacional de Control de Energía (CENACE) [90] manages grid operations and expansion, using a postage stamp system to allocate transmission costs.

• Key Differences:

• Decentralized Markets: The US and parts of Europe use financial instruments (FTRs/CRRs) to manage congestion, offering flexibility for market participants to hedge against transmission risks.
• Centralized Markets: Chile, Brazil, and Mexico rely on centralized planning with uniform cost distribution and without financial tools for hedging congestion risks.

• System Reliability and Congestion Management:

Tools such as congestion charges and LMPs manage grid congestion, ensuring efficient use of transmission capacity. Financial instruments like FTRs allow market participants to hedge against congestion risks. Interregional coordination is critical in planning investments supporting electricity flows across broader regions, facilitating RE integration, and maintaining grid stability.

• Transmission Market Evolution:

Technological advancements in transmission and other technologies, such as DERs and ES, reshape the transmission market. Future developments must incorporate smart grid technologies, adapt to the changing energy mix, and improve planning for uncertainty and risk management. Market operators are also considering cross-border integration to optimize resources across wider areas.

Market structures vary globally, from decentralized financial instruments like FTRs in the US to centralized planning and cost-sharing models like Chile and Brazil.

4. Decarbonization in Electricity Market and Trends

The energy transition (ET) is producing comparable trends across energy markets, which include increased plate capacity and operational reserve requirements, greater demand for flexibility and ramping capacity, changes in system demand, overgeneration curtailment risks, with instances of zero or negative prices, forecast-increasing challenges, and risks of increased price volatility. These challenges are becoming more prominent as the integration of VRE accelerates.

4.1. Larger Capacity and Operating Reserves per GWh

As VRE penetration increases, substantial capacity and supporting infrastructure—and consequently significant investment—are required to meet demand, underscoring the economic and infrastructural challenges of integrating high levels of VRE. This situation necessitates innovative policy design to avoid inefficiencies, excessive redundancies, and unwarranted system costs.

The OECD and NEA [91] demonstrate that increasing shares of VRE, such as solar and wind, require a substantially higher capacity to produce the same electricity as conventional thermal power plants due to lower load factors and capacity credits. As VRE penetration increases to 10%, 30%, 50%, and 75%, the required installed capacity and associated investments grow substantially. Total system costs, encompassing grid infrastructure, balancing, and connection expenses, also escalate with higher VRE shares, ranging from less than USD 10 per MWh at 10% VRE to over USD 50 per MWh at 75% VRE. These costs can be significantly influenced by the availability of system flexibility, such as interconnections and flexible hydropower resources, which help mitigate the challenges associated with VRE integration.

(a)

Chile’s Case Study

Chile has made significant strides in expanding its VRE generation capacity over the past two decades. Installed plate capacity has grown from around 10 GW in 2000 to 33 GW in 2023, a threefold increase reflecting the country’s efforts to meet rising electricity demand and enhance energy security while addressing climate change objectives with increasing shares of VRE.

As of the first quarter of 2024, 66.1% of Chile’s electricity comes from renewable sources, with renewable energy, different from large hydropower plants, contributing 40.7%, up from 2% in 2010. In 2023, wind and solar alone accounted for 40% of installed capacity and 28% of total electricity generation; see Figure 1 and Figure 2. This substantial share highlights the effectiveness of policies and market dynamics in promoting VRE, helping reduce greenhouse gas emissions, and leveraging Chile’s abundant natural resources.

However, integrating these VRE sources into the grid comes with challenges, particularly in ensuring the reliability and stability of the power supply. Chile’s power system has experienced a significant increase in spare capacity (Figure 3), measured as the ratio of peak generation to installed capacity. This ratio has climbed from approximately 40% in the early 2000s to 65% in 2023. Thus, while peak generation has increased by 103% and annual generation by 115%, installed capacity has grown by 254% since 2000 (or 36%, 43%, and 116%, respectively, since 2010), indicating a growing buffer between plate capacity and peak demand, as also had happened within PJM [92].

This capacity increase is primarily attributable to the intermittent nature of wind and solar power, which is unavailable 24/7 and must be complemented with other generation sources, storage, backup, or reserve capacity to maintain grid stability. However, the requirement for this complementary capacity comes at a cost. While the naked average investment cost per MW of installed capacity has decreased by 28% since 2000, when calculated at new replacement value (NRV), the average investment cost to be amortized per MWh of generation has increased by 26% over the same period. Interestingly, despite this increase in investment cost per MWh, the weighted average of the levelized cost of electricity (LCOE) has decreased by 14% since 2000. The EIA [93] has noticed that the LCOE is a limited measure because it only reflects the cost to build and operate a plant, not the value of the plant to the grid.

This scenario presents a growing challenge in many markets. As the share of VRE increases, with its characteristic zero marginal cost, it becomes increasingly difficult to secure a stable stream of revenues to finance investment costs, and merchant power plants must increasingly shift to revenues that need to be secured by AS markets, capacity markets, and LTPAs. This situation underscores the need for innovative market designs and policy frameworks to ensure the long-term financial viability of the power system while continuing to integrate higher proportions of VRE.

Chile’s increasingly diverse electricity generation mix, less reliant on imported FF, enhances its energy security from FF geopolitical risks, like foreign FF market price manipulation, but requires sophisticated grid management and operational strategies.

(b)

CAISO Case Study

Over the past two decades, California has significantly expanded its VRE capacity, reflecting global trends in sustainable ET. Figure 4 shows that CAISO installed capacity has grown from 55.5 GW in 2001 to nearly 88 GW in 2023, a notable increase driven by the state’s ambition to meet electricity demand while addressing climate objectives. The increasing share of solar and wind explains this growth, which accounted for over 30% of installed capacity by 2023, compared to just 3.5% in 2001. Also, Figure 4 further illustrates the gradual reduction in the reliance on nuclear and FF, with renewables, particularly solar PV, playing an increasing role in CAISO’s energy mix.

As of 2023, wind and solar contributed 26% of total electricity generation, marking a substantial shift from traditional FF sources. Figure 5 further illustrates the gradual reduction in the reliance on FF and nuclear, with renewables, particularly solar PV, playing an increasing role in CAISO’s energy mix.

Despite the rise in VRE capacity, integrating these resources into the grid remains challenging. CAISO’s power system requires substantial backup and reserve capacity to compensate for the intermittent nature of VRE, as illustrated by the increased spare capacity over time. Peak generation as a percentage of installed capacity has decreased, signaling growing buffer plate capacity—essential to ensure grid stability. In 2001, the ratio of peak consumption to installed capacity was 75%, whereas by 2023, it had dropped to 51%, reflecting significant capacity increases to accommodate VRE intermittency.

As in Chile, California’s experience also provides key insights for other regions seeking to diversify their energy mix and reduce reliance on imported FF. However, SEN-Chile and CAISO, with different market constructs, face similar challenges related to grid management, storage, and flexible generation investment, as well as developing innovative market designs to support continued growth in renewable energy.

4.2. Ramping Capacity Needs and Operating Reserves

The increasing integration of VRE sources and DERs introduces significant variability in energy production, demanding a more flexible power system and better forecasting. While market designs are evolving to accommodate VRE, new plants often benefit from existing system capabilities without fully accounting for the additional costs they impose, such as expanded operating reserves, transmission upgrades, and enhanced operational capabilities to manage higher VRE penetration.

For example, some of the system costs associated with VRE integration include the following:

• Balancing costs: The expenses for ancillary services needed to stabilize the grid include load-following and managing unforeseen fluctuations.
• Capacity costs: Backup generation and storage are required during low VRE output periods to ensure reliability.
• Transmission and distribution costs: Upgrading the grid infrastructure to accommodate the increasing number of small power plants, DG systems, and behind-the-meter VRE sources.
• Operational costs: Coordinating a diverse and larger array of power generators, energy storage, prosumers, and demand management systems.
• Curtailment costs: Managing surplus generation when VRE output exceeds demand, leading to curtailment or the need for investment in energy storage.

Addressing these challenges requires thoughtful planning and policy design to optimize renewable energy integration while maintaining system reliability and controlling costs.

As a result, while increasing shares of VRE are aimed at contributing to reducing greenhouse emissions, the stability and security of energy supply, with characteristics of a public good, along with the expansion of transmission lines and other services needed to operate an electrical system, with an increasing share of VRE, are not always fully compensated by VRE investors. These renewable sources, such as DERs, are typically more dispersed and smaller than traditional energy sources, further complicating system management. For example, as of May 2024, the Chilean electricity system SEN had 1008 power generation units (in 2010, the power system had about 150 units). Of these, 740 were defined as small distributed generation facilities (with a capacity of up to 9 MW), with an average installed capacity of 3.2 MW, not counting behind-the-meter (BTM) installations, such as solar PV rooftop, which in the case of Chile are still in their infancy.

This new landscape necessitates crucial flexibility and pricing schemes that assist with quickly ramping output up or down to accommodate fluctuations in supply and demand, anticipated or not. Traditional baseload power plants, designed to provide constant output, often struggle with this requirement for rapid ramping. Consequently, there is a growing need for flexible generation resources, ESSs, DR programs, and adaptive grid infrastructure.

Ref. [94] provides a comprehensive analysis of thermal unit statistics in the United States, examining the historical operating costs of hundreds of conventional electrical generation units and identifying key factors influencing costs associated with providing flexibility services. These factors allow for estimating indirect ignition costs, load following costs, and increased unit unavailability resulting from more frequent cycling, focusing on coal and gas plants (combined cycle, simple cycle, and aero-derivative). Ref. [95] also reviews hydropower plants’ start and stop costs by highlighting the increased operational stresses and costs from frequent start and stop cycles due to intermittent renewable energy sources. It categorizes cost estimation approaches into economic and technical–economic types, emphasizing the need for comprehensive, auditable models that integrate material and non-material costs. Future research should aim to develop representative models for various components and aggregate these to estimate system-level costs. Ref. [96] quantifies the costs incurred by conventional power plants in Germany due to increased VRE penetration, finding notable increases in direct and indirect ignition costs with higher VRE penetration. Additionally, Ref. [97] examines the impact of greater VRE penetration on the startup frequency of conventional thermal units, concluding that total operating hours of thermal units decrease in scenarios with higher renewable penetration, primarily due to changes in the generation portfolio, such as the phase-out of nuclear plants. These studies collectively provide valuable insights into the complex relationship between increasing VRE penetration and conventional power plants’ operational patterns and costs, highlighting the need for a nuanced understanding of power system flexibility in ET.

Table 1. Typical generator ramp rates, start-up and shut-down Costs, and wear costs.

Fuel Type	Ramp Rate (% of Rated Capacity Per Min) *	Start-Up Costs (USD/MW/Start) *	Shut-Down Costs (USD/MW/Shut-Down) *	Start-Up Time (Hours) *	Shut-Down TIME (Hours) *	Minimum On Time (Hours) *	Minimum Off Time (Hours) *	Cost of Wear and Tear (USD/MWh) For Primary and Secondary Frequency Control **
Coal	0.6–8	100–250	10–25	4–9	2–9	0–12	0–12	1.087–1.760
Natural gas	0.8–30	20–150	2–15	2–4	1–4	0–2	0–1	1.243–1.824
Nuclear	0–5	1000	1000	24	24	Days	Days	0.029
Hydro	15–25	0–5	0–0.5	0–1	0–1	0–1	0–1	2.722–4.714 Dum; 0.029 Run of River
Wind	Non-dispatchable	-	-	-	-	-	-	0.006–0.070
Solar	Non-dispatchable	-	-	-	-	-	-	0.006–0.235 PV 0.017 CSP
Battery storage	0–100	Varies	Varies	<0.25	<0.25	<1	<1	0.042 (PV + BEES)
Diesel	2–5	5–20	1–5	0.25–0.5	0.17–0.33	1–2	1–2	0.043

Source: * [98]; ** [99,100,101].

highlights the operational characteristics of various power generation technologies, showing significant differences in flexibility, start-up and shut-down costs, and wear and tear costs. Start-up Costs refer to the expenses incurred to bring a power plant from a non-operational state to an operational one; Shut-down Costs refer to the costs associated with ceasing the operation of a power plant and transitioning it to a non-operational state; and wear and tear costs refer to the costs related to the wear and tear of equipment during operation, often expressed as a cost per megawatt-hour of electricity generated. Due to frequent cycling, wear and tear costs are considered physical degradation and operational expenses such as fuel consumption and maintenance. The differences in operational characteristics of various power generation technologies, illustrated by Table 1, influence the technologies’ suitability for different roles within the power grid, from steady baseload generation to dynamic peaking and backup power.

As Table 1 illustrates, the flexibility and response time of the different technologies and start-up and shut-down costs vary significantly across technologies, with nuclear and coal having the highest due to their complexity and safety requirements. Natural gas, hydro, and diesel are lower, reflecting simpler and quicker processes. And, on wear and tear costs, nuclear, wind, and solar have the lowest wear costs due to fewer mechanical stresses and steady operations. Hydropower has higher wear costs due to mechanical wear from water flow, while coal and natural gas have moderate wear costs due to thermal and mechanical stresses. This underscores the importance of considering system costs and operational flexibility when planning for a future of decarbonized energy.

Historically, baseload generation has relied on stable, continuous-output technologies like nuclear and coal, which efficiently meet the grid’s constant demand but lack flexibility. In contrast, peaking power has been best served by flexible and fast-ramping technologies such as natural gas, hydro, and, recently, battery storage, which can quickly respond to demand spikes and shut down during low-demand periods. Diesel generators excel with their rapid start-up capabilities for backup and emergency power, providing immediate support during outages or unexpected demand surges. Each technology plays a distinct role in ensuring a balanced and reliable electricity supply.

(a)

Chile’s Case Study

For Chile, Figure 6, Figure 7, and Figure 8, on a 24-h window timeframe for 1 January 2000, 2010, and 2023, depict the hourly electricity generation by technology. In 2000, the electricity generation mix was dominated by coal, hydro (dam and run-of-river), and natural gas, with relatively small contributions from other sources such as diesel and biomass. In 2010, the mix saw a significant increase in natural gas and a diversified contribution from coal, hydro, and a tiny but growing share of renewables like wind and biomass—when the transition towards increasing participation of VREs began. In 2023, the generation mix showed a substantial increase in solar and wind energy contributions alongside a significant presence of flexible generation and incipient BESSs. Figure 6, Figure 7 and Figure 8 illustrate that the demand for power sources that can quickly accommodate solar and wind generation variability in 2023 is evident.

From the significant penetration of solar and wind energy, which creates pronounced fluctuations in the residual demand faced by natural gas, coal, and large hydropower plants, Figure 6, Figure 7 and Figure 8 highlight the challenges faced in 2023 by the Chilean electric system. The system had to manage a morning ramp of 3000 MW within 3 h and an evening ramp of 2000 MW as solar output decreased, partially offset by wind generation. On average, solar and wind contributed 52% of the electricity generation at 10:00 AM and 7:00 PM. Interestingly, while the system’s peak daily demand between 8:00 PM and 10:00 PM necessitated an additional generation increase of 736 MW, the VRE ramping capacity needs exceeded the system ramping capacity needed to satisfy the system’s daily peak load.

Figure 9 and Figure 10 illustrate the daily variations in power generation by different technologies, reflecting the critical role of flexibility in maintaining grid stability amidst increasing renewable energy integration. To meet demand, peak load, VRE following, and ramping requirements, the system primarily relies on conventional FF technologies, including LNG power plants and, to a lesser extent, coal power plants and some hydropower plants. These conventional technologies are essential for providing flexibility to accommodate the variability in renewable energy production. The use of fast-response gas turbines, although effective, incurs high marginal costs, emphasizing the need for more sustainable flexibility solutions such as competitive BESSs and DR mechanisms.

As net demand in Figure 9 illustrates, ramping capacity use has increased significantly in the Chilean electric system due to solar and wind power integration, following VRE supply fluctuations exceeding the resources needed to follow the daily system load fluctuations. VRE has created notable volatility in the net or residual demand curve faced by natural gas, coal, and hydropower plants, often called the “duck curve”. The cost implications of this ramping are important.

(b)

CAISO Case Study

For CAISO, Figure 11 and Figure 12, for 1 January 2023, depict hourly electricity demand, forecasts, net demand, and flexibility providers by technology.

CAISO demand has been steadily declining due to the increase in BTM PV production, as shown in Figure 11 and Figure 12. In the morning, the electric system faced a ramp-down of 8000 MW in less than three hours, followed by a ramp-up of 11,000 MW in the evening as PV output decreased. Between 10:00 a.m. and 7:00 p.m., solar and wind supplied an average of 52% of the electricity. These ramping capacity needs are exacerbated further by BTM PV production, which reduces demand during daylight hours, intensifying the need for ramping both in the morning and evening. At peak production, renewable sources like wind and solar can supply more than 65% of market demand.

In CAISO, flexibility for ramping up and down is primarily provided by a combination of natural gas power plants, large hydropower, energy imports, and battery storage, as illustrated in Figure 11 and Figure 12. These resources play a critical role in meeting fluctuating demand, particularly during rapid changes in renewable generation.

Fascinatingly, on 8 April 2024, a solar eclipse highlighted contrasting approaches to managing solar generation drops between California and Texas. In CAISO, battery storage systems played the primary role, switching from charging (2.6 GW) to discharging (2.7 GW) to compensate for the roughly 27% drop in solar output (from 14.5 GW to 9.1 GW), while curtailment of excess solar temporarily disappeared. In contrast, ERCOT, which experienced a more severe 94% solar reduction (13.8 GW to 0.8 GW), relied mainly on natural gas generation, which ramped up from 19 GW to 27+ GW. This difference exemplifies broader grid transition strategies: California leverages energy storage charged by excess renewables, while Texas and other regions depend primarily on natural gas for grid balancing.

As Chile, CAISO, and ERCOT cases illustrate, the dependency on conventional FF for flexibility further complicates the transition to a fully RE system [102,103], highlighting the importance of developing regulatory frameworks to manage new technologies, electronics, and storage costs efficiently, where ramping requirements with fast-response solutions can have high-levelized costs, impacting the overall economics of the power system.

Further, the transition from an electromechanical system to an electromagnetic one, especially in a system dominated by inverter-based resources (IBRs), indeed requires significant shifts across multiple levels of grid planning, operation, and regulatory frameworks. In electromechanical systems dominated by synchronous machines, conventional FF, or hydropower plants, the inherent physical characteristics of these machines, such as inertia, provide natural stability to the power grid. However, as the energy system integrates more power electronics and digital control through IBRs like solar PV, wind, and battery storage, the traditional methods for ensuring grid stability are no longer sufficient.

The transition towards a more renewable-based energy system highlights the need for a flexible and adaptive power grid. As VRE becomes more prevalent, the role of flexible generation resources, energy storage, demand response, and advanced grid infrastructure becomes increasingly important. These elements are crucial for ensuring grid reliability and resilience in the face of the inherent variability of renewable energy sources. Chile’s 2000, 2010, and 2023 figures (Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10) and CAISO (Figure 11 and Figure 12) vividly depict the significant shifts in the energy mix, emphasizing the growing need for technologies to support and stabilize VRE integration.

4.3. Ramping Capability and Risk of Market Power

The energy transition, characterized by increasing shares of VRE sources, leads to greater generation volatility—both anticipated and unanticipated—creating a critical need for technologies with ramping capabilities, much more significant than in systems primarily characterized by FFs, large hydro, and nuclear baseload generation. Units with ramping capabilities, such as natural gas turbines, hydroelectric plants, BESSs, diesel generators, and, to a lower extent, coal plants, can adjust output in response to changes in residual demand through scheduled operations, on-the-spot adjustments or as ancillary services (e.g., a faster response such as frequency regulation and spinning reserves), where demand response and, to some extent, VRE can also provide some of these adjustments. The residual or net demand for ramping and ancillary services that flexible power units and BESSs face is the net demand, the market demand minus the supply provided by VRE, and other inflexible baseload generation or committed supply on the day ahead.

In this context, flexible units with ramping capabilities can engage in a market where those who are unresponsive to price fluctuations or committed in advance to fixed outputs cannot participate. Market demand reflects the relationship between price and total quantity consumed, influenced by factors such as consumer behavior and system needs. Residual demand, faced by flexible power units and BESS, represents the remaining demand after accounting for supply from VRE and other inflexible baseload generation. While market demand elasticity measures the sensitivity of demand to price changes, residual demand elasticity captures the responsiveness of the remaining demand handled by flexible generators, BESS, and demand response.

When residual demand becomes less elastic (less responsive to price changes), flexible units gain more market power, as they can adjust prices without significantly reducing the quantity demanded. This situation typically arises when few competitors or alternatives are available to ISOs/RTOs or the grid. The elasticity of residual demand for these flexible power units is crucial, where the more significant the need for and the fewer the units capable of providing ramping—especially considering the inflexibility of VRE and other baseload generation—the more inelastic the residual demand becomes, enhancing the market power of flexible power units and BESSs [104,105].

A more significant and inelastic residual demand allows these units to influence market prices more effectively, as the grid has fewer alternatives when adjusting their output. Market power can be enhanced when demand and net demand are highly variable, and a VRE supply is hard to predict, where the influence of flexible power units and BESSs can be more impactful on prices. Transmission constraints and conditions affecting demand and power equipment performance can further enhance local market power. Here, regulation is fundamental in enabling as many technologies and agents to participate in ramp-following and ancillary services as possible [106,107].

4.4. Overgeneration and Curtailment Risk

The increasing share of VRE, supported by economic incentives and limited transmission capacity and ES, introduces the risk of overgeneration and curtailment during high renewable output and low demand. Such conditions have resulted in zero or negative locational marginal prices (LMPs) where excess energy cannot always be immediately utilized. While this market response reflects the underlying incentives schemes and supply-demand dynamics, effective management strategies—including ES, curtailment mechanisms and rules, improved grid infrastructure, and well-designed VRE incentives—are essential to mitigate wasted energy and ensure grid reliability.

In the United States, overgeneration risk is observed, but with regional variations due to the diverse structure of US electricity markets. Some US markets [108], particularly those with high wind penetration, like the Midwest, have experienced negative pricing during periods of high renewable generation and low demand. The frequency and impact of overgeneration vary significantly across different ISOs and RTOs [109]. Where VRE subsidies, state-level RPSs, and federal tax incentives have all played a role in encouraging RE growth and contributing to these market dynamics for utility-scale and BTM solar PV and other RE deployment. According to [110], almost half of VRE generation capacity growth since 2000 is associated with state RPS requirements in the US, and the combined demand for clean electricity from RPS and clean energy standards (CES) policies will grow from roughly 500 TWh in 2024 to 1700 TWh by 2050. Additionally, some markets have implemented or are reforming their AS and capacity markets [111] to ensure sufficient flexible resources are available to manage variability.

For example, CAISO [112] has faced significant challenges with the “duck curve” phenomenon, leading to periods of potential overgeneration from solar during midday and steep ramp-up requirements in the evening. In 2023, CAISO experienced a reduction in energy demand attributed to milder summer conditions compared to previous years and an increase in behind-the-meter (BTM) and PV generation. This trend has continued since 2020, reflecting how decentralized PV and changing weather patterns influence overall energy consumption across the state. ERCOT [113] has also experienced similar challenges with high wind penetration.

In 2023, CAISO faced increased price volatility, marked by a 73% rise in negative prices, Figure 13, due to oversupply from growing solar and wind generation, while high price spikes diminished amid milder weather and fewer extreme demand events. Further, in the CAISO and WEIM areas, total downward dispatch in 2023 increased by 9.5 percent and 18.2 percent, respectively, relative to 2022. In both areas, most of the downward dispatch was economic. Economic downward dispatch accounted for about 2688 GWh (95.5 percent) of curtailment during the year, while self-scheduled curtailment accounted for about 53 GWh (2 percent). Exceptional dispatch curtailments for self-scheduled and economic bid resources remained low at about 2.4 GWh (less than 1 percent). The roughly 70 GWh (2.5 percent) of remaining curtailment came from “other” economic and self-scheduled curtailment.

PJM Interconnection often commits more generation capacity than is ultimately needed [114,115], partly due to the minimum offer price rule (MOPR) and its interaction with the variable resource requirement (VRR) curve in the base residual auction (BRA). The MOPR, designed to mitigate buyer-side market power, establishes a price floor for specific resources, often clearing additional capacity beyond the VRR curve’s target. When the MOPR sets the clearing price, PJM procures all resources subject to the MOPR at that price, even if their combined capacity exceeds system reliability needs. To respect lower-priced offers from non-MOPR resources, PJM commits additional capacity, leading to over-procurement. While this ensures reliability and peak demand readiness, it can result in overgeneration during off-peak periods. During off-peak periods, the excess capacity procured in the capacity market might remain online due to inflexible generation commitments or economic pressures to operate. This oversupply can force grid operators to curtail renewable resources (e.g., wind or solar) due to their lack of dispatchability, as thermal generators may have minimum operating thresholds that require them to remain online. Generators may bid at negative prices to avoid costly shutdown and restart cycles of their plants, effectively paying consumers to take excess electricity off the grid. This practice is particularly relevant for nuclear and coal-fired plants with high operational transition costs, as running at a temporary loss during low-demand periods is more economical than cycling the plant. Additionally, VRE generators may be incentivized to continue producing during negative pricing periods when production tax credits and renewable certificates can offset these losses.

Thus, overgeneration pressures market prices downward and may force the curtailment of VRE due to their non-dispatchable nature, lack of ES, and transmission constraints [114,116], causing economic inefficiencies, such as higher consumer costs and underutilization of committed resources. This dynamic [66,115,117,118,119,120] highlights the trade-off between ensuring reliability and avoiding the unintended economic consequences of excess capacity procurement.

On 1 December 2023, the U.S. Court of Appeals for the Third Circuit [115] upheld PJM Interconnection’s Focused Minimum Offer Price Rule (MOPR), rejecting the rule’s and FERC’s approval challenges. The Focused MOPR, which mitigates buyer-side market power in cases of market manipulation or state subsidies conflicting with the Federal Power Act (FPA), replaced the Expanded MOPR (2019) and took effect by law on 29 September 2021, after FERC failed to act within the statutory deadline. The court ruled that FERC’s acceptance was neither arbitrary nor capricious, as it was supported by substantial evidence, including arguments from FERC Commissioners. It also affirmed that judicial review of FERC’s explicit or implicit actions follows the same deferential standards under the FPA and Administrative Procedure Act (APA). The ruling acknowledged the Focused MOPR’s balanced approach in addressing market concerns while accommodating state policies.

While facing overgeneration risk challenges, US markets [111] have implemented various measures to address these issues, including improved forecasting, more flexible ramping products, and market rule changes to accommodate VRE. Areas with transmission limitations [111,121] exacerbate local overgeneration problems, particularly in regions with rapid RE growth, such as West Texas and the Southwest Power Pool (SPP) region. Such transmission constraints also affected the implementation and refinement of Ramp Products.

In the European Union [122,123,124,125,126,127,128,129,130], substantial subsidies and support schemes for renewable energy have accelerated the deployment of technologies like wind and solar, increasing the risk of overgeneration. In 2023, negative hours reached a record 6870 in European markets, compared to just 569 in 2022—a staggering increase of 1107% [131]. This overgeneration has led to negative electricity prices or LMPs, particularly when supply from these variable renewable sources exceeds demand. For instance, in Q3 2023 alone, there were 3064 h of negative wholesale prices, and in October, a record of 1685 h with negative prices was observed. While these policies have significantly expanded renewable capacity, they underscore the need for enhanced grid flexibility, improved transmission infrastructure, and energy storage solutions to manage excess supply during high renewable output and lower demand.

In Chile, zero marginal location costs have become increasingly common. While these were once rare occurrences, typically triggered by extreme events like heavy rains leading to the release of water from hydroelectric reservoirs, they have become a regular and daily phenomenon in recent years, particularly in nodes with large installed solar capacity facing transmission constraints. Small energy projects (≤9 MW of capacity) benefit from a stabilized price (PE) that often exceeds the locational marginal cost, and they also benefit from having precedence in system dispatch over utility-scale power plants. Also, their revenues are not negatively impacted by excess supply. This situation further incentivizes oversupply and curtailment, dropping local marginal prices to zero, as the Cardones node exemplifies in Figure 14.

Figure 14 illustrates the locational marginal costs for the Chilean SEN electric system on 26 May 2024 at four locations, showing that the Cardones node reaches zero marginal cost during peak solar PV generation. This reflects overgeneration and transmission constraints that necessitate the curtailment of solar energy. According to [132], these losses amounted to 9.72% of the output from VRE plants throughout 2023, nearly a tenth of their total generation. In the first quarter of 2024, wasted energy surged to more than 18% of the country’s wind and solar production. These data underscore the need for storage solutions and grid upgrades to manage surplus power, prevent economic losses, and reassess incentives for future small power plants (≤9 MW). However, the timeline for building new high-voltage transmission lines lags the construction pace of renewable generation projects—a critical issue, especially as small power plants face an incentive to flood the market given the PE, which expires in 2034.

These situations in the US, EU, and Chile highlight the global nature of the challenges associated with the incentive schemes and market design when integrating high levels of VRE into electricity grids and markets. They also underscore the need for continued innovation in market design, incentive schemes, grid operations, and energy storage technologies to manage increasing levels of VRE [107].

4.5. Discrepancies Within Day-Ahead and Real-Time Markets

The actual physical operation of balancing electricity supply and demand often deviates from projections made in the day-ahead electricity market. These discrepancies arise from unforeseen shifts in supply or demand conditions that differ from the scenarios anticipated during day-ahead forecasting and market closure. Several factors contribute to these unexpected changes, including the inherent variability of renewable energy resources, the behavior of market participants, the design and mechanisms of the market, force majeure events, and the regulatory framework in place. The market impact of these deviations is influenced by the drivers of supply and demand and their respective elasticities [133], which shape the system’s ability to adapt to such fluctuations. Thus, the impact of unexpected discrepancies on the real-time/balancing market depends on the flexibility of demand and supply, the presence of responsive demand-side management programs, the variability of renewable resources, and the effectiveness of market design and regulatory frameworks.

This underscores that forecast precision is vital in enhancing system and cost efficiency. Moreover, the same degree of forecasting error, whether overestimating or underestimating, can lead to unevenly higher operational costs [134] when addressing real-time energy demand.

In the case of PJM [135], which runs a day-ahead and a real-time energy market, with a total installed capacity of 196,380.2 MW as of 31 December 2023, the discrepancies between day-ahead and real-time demand averaged 21,589 MWh in 2023 and 14,818 MWh in 2022, with a median of 22,606 MWh and 15,527 MWh, and a standard deviation of 3787 MWh and 2346 MWh, respectively. Thus, on average, day-ahead scheduling was planned for 19% and 14% more generation than what was required in real time in 2023 and 2022, respectively, which can be influenced by the grid and operational constraints but also the conservative approach of the RPM to prioritize the security of supply.

In the case of Chilean SEN, the ISO [136] centrally plans the dispatch of power plant units based on informed demand commitments and audited marginal costs, with a total installed capacity of 33,831.6 MW as of 31 December 2023; the discrepancies between day-ahead and real-time demand averaged −68.4 MWh in 2023 (against an average planned demand of 9346.03 MWh) and −103.6 MWh in 2022 (against an average planned demand of 9448.1 MWh), with a median of −58.3 MWh and −93.1 MWh, and a standard deviation of 267.6 MWh and 268.3 MWh, respectively. Thus, on average, the Chilean ISO, contrary to PJM, day-ahead scheduling underestimated the real-time demand by 0.73% in 2023 and 1.11% in 2022, indicating a slight but consistent underestimation of demand in their planning process.

• Chile’s Case Study

Figure 15 for 2023 depicts the disparities between scheduled and actual generation for Chile’s SEN system as a percentage deviation. The deviation between planned and actual generation ranges from a minimum of −15% to a maximum of 13%, with a standard deviation of 2.83, an average of −0.73, a median of −0.62, and a mean absolute percentage error (MAPE) of 2.23%. Scheduled generation exceeded actual generation by 2797 h, while it fell short by 4294 h. The bias where scheduled generation fell short of actual generation triggers the operation of fast-response units, which in the SEN are typically gas turbines and diesel engines. This, in turn, triggers high locational marginal costs, influencing the payments for energy transfers between power generators within the ISO.

For the Chilean SEN system, on 26 May 2024, specifically for the Cardones node, Figure 16 illustrates the deviation between the scheduled locational marginal cost (USD/MWh) and the actual locational marginal cost (USD/MWh), with discrepancies reaching up to 100%.

In the case of ERCOT [137], the increasing penetration of wind and solar generation has led to larger swings in real-time electricity prices. As the percentage of wind and solar generation in the electricity mix has increased, price volatility, measured at 15-min intervals for price changes exceeding +/− USD 25/MWh, has increased, too. These fluctuations can be further exacerbated by forecast inaccuracies and biases in system operational practices, which contribute to inefficient resource allocation and elevated operational costs.

When ISOs underestimate generation needs, as seen in the Chilean SEN system, fast-response units like gas turbines or diesel engines must be called on demand to maintain grid balance, leading to higher locational marginal prices and increased operational costs. As the share of VRE grows, forecast errors can further heighten the need for flexibility, making it essential for grids to integrate responsive and flexible resources to manage price volatility effectively.

Precise next-day energy demand and supply forecasts ensure that day-ahead scheduling matches real needs, reducing operational costs. However, inaccuracies in forecasts or the ISO’s operational practices, either conservative or because of overestimation, underestimating power needs, or force majeure situations, increase operational expenses by causing less or more than the necessary unit commitments, as is in the case of PJM, which conservatively tends to overcommit, or in Chile, where ISO tends to under-commit, requiring costly quick-response units.

In systems with increasing shares of VRE, the transition to renewable energy emphasizes the critical need for advanced forecasting and grid flexibility. ES, DR, and improved market mechanisms with accurate price signals will help ensure stability as renewable adoption continues to rise. Despite improvements in forecasting techniques, uncertainty remains, requiring a flexible and responsive resource mix to maintain system balance. Evolving electricity markets to provide the right incentives and granularity will support a resilient grid with diverse energy sources and services.

5. Implications of Decarbonization for Electricity Market Design

Decarbonizing the electric system involves phasing out CO₂-emitting technologies in favor of zero or low-emission technologies aligned with the NetZero goals by 2050. This necessitates the widespread adoption of renewables such as solar and wind, storage technologies, carbon capture and storage (CCS), nuclear, and other low CO₂-emitting technologies in the power sector. Simultaneously, the electrification of transportation and production processes, shifting away from internal combustion engines and other CO₂-emitting technologies, will further increase electricity consumption.

This section underscores the multifaceted implications of decarbonization on electricity market design, focusing on how transitioning to a low-carbon power system reshapes market structures, operations, and financial dynamics. Key areas such as day-ahead and real-time markets, capacity and ancillary services, transmission, centralized dispatch, and financial instruments are examined, highlighting the transformative shifts required to support a clean energy future.

(1)

Day-Ahead and Real-Time/Balancing Markets: Decarbonization significantly impacts day-ahead and real-time/balancing electricity markets, requiring enhanced flexibility, forecasting, and market design. The variability and uncertainty of increased VRE requires flexible generation, ramping capabilities, and operating reserves, supported by improved weather and climate forecasting for day-ahead planning. However, real-time markets must address persistent uncertainties and risks of overgeneration, where oversupply may lead to zero or negative LMPs. Strengthened transmission networks and innovative technologies like dynamic line ratings can alleviate constraints and optimize renewable integration. The evolving resource mix, replacing baseload FF plants with renewables, DER, storage, hydrogen, and flexible nuclear, necessitates upgraded scheduling and operational, as well as dispatch approaches. Granular pricing intervals, advanced ICT, smart metering, and control technologies enable markets to better align with system dynamics, balancing flexibility and revenue adequacy. These reforms, coupled with mechanisms to mitigate oversupply and incentivize ancillary services, are essential for integrating high shares of renewables across day-ahead and real-time markets.

(2)

Capacity Market: Decarbonization fundamentally reshapes capacity markets, necessitating adaptations to ensure grid reliability and resource adequacy amid increased reliance on VRE and DERs. The low marginal costs and weather-dependent nature of VRE stress reserve margins threaten the financial sustainability of conventional power plants, with reduced hours of operation and, as such, leading to higher average energy costs, which require support that can come from capacity market revenues. These markets must integrate emerging resources like ES, DR, and DER while valuing their contributions to reliability. Shorter development timelines for renewables and batteries offer flexibility in capacity procurement but require alignment with the slower expansion of transmission infrastructure. As renewable sources with lower capacity factors replace baseload and FF plants, capacity markets must procure more nameplate capacity and embrace multi-year commitments with updated performance requirements to incentivize flexible and reliable resources. Enhanced forecasting and interregional transmission coordination will be essential for addressing grid bottlenecks and optimizing capacity procurement. Decarbonization’s dual impact—driving up capacity prices due to reduced reserve margins and downward energy price pressures from low-cost renewables—requires holistic market redesigns to ensure revenue adequacy for conventional and emerging technologies, securing grid stability in a low-carbon future.

(3)

Long-term Purchase Agreements: Decarbonization significantly reshapes LTPAs, making them a part of the equation to secure financing for capital-intensive renewable projects while introducing new complexities. As VRE with zero marginal costs increases, an LTPA provides revenue certainty but faces challenges like shorter contract terms due to rapid technological advancements and the risk of stranded assets. Hybrid structures combining RE with ES, hydrogen, or DERs might become more common to enhance flexibility and capacity commitments. Corporate renewable PPA may perhaps increase, expanding access to LTPAs. However, evolving risks such as counterparty challenges from utilities with high-carbon assets and pricing pressures from declining clean technology costs necessitate more dynamic agreements, often with renegotiation clauses or price adjustments. Despite these shifts, the LTPA remains critical for securing investment in a renewable-focused energy landscape.

(4)

Ancillary Services Market: Decarbonization fundamentally transforms ancillary services markets, increasing demand for services like operating reserves, frequency regulation, ramping, and reactive power to manage the variability and dispersion of RE. Faster response times and higher accuracy become critical, necessitating new ancillary services such as capacity synchronization, stability support, and congestion relief. Emerging resources like battery storage, advanced inverters, intelligent appliances, and electric vehicles are poised to play a significant role, requiring smarter grids and operations. Co-optimization of ancillary services and energy dispatch can leverage shared flexibilities, but market designs must evolve to introduce new products and compensation mechanisms for advanced technologies. Prices in ancillary services markets are likely to experience greater volatility, with potential spikes due to flexibility scarcity and the need for faster, more dynamic responses. Significant upgrades in ancillary services markets are essential to ensure system stability and reliability in a high-renewables grid.

(5)

Centralized Dispatch System: Decarbonization introduces significant complexity to centralized dispatch systems based on declared marginal costs, as the variability of VRE and DG challenges traditional supply-demand balancing. The predominance of low or zero-marginal-cost resources, coupled with overgeneration risks during periods of high renewable output and low demand, necessitates enhanced curtailment, load flexibility, and revenue mechanisms to ensure financial sustainability. Long-term planning becomes critical for capital-intensive projects like nuclear, pumped hydro, and transmission while managing security constraints and diverse generation profiles, such as DERs requiring advanced algorithms. Central Dispatch Systems are not exempt from controversy regarding declared marginal costs, particularly concerning the complex interplay between generators’ fuel procurement strategies and their influence on declared marginal and system marginal costs [138]. For example, one critical question is whether the natural gas combined cycle (NGCC) natural gas “take or pay” contracts inherently establish a must-run generation condition with a fuel opportunity cost of zero. Integrating advanced technologies and additional ancillary services through dynamic auctions will be essential to maintaining system stability. Declining marginal costs and clearing prices in merit-order dispatch systems could undermine conventional generators’ revenues, raising the need for innovative capacity and ancillary service procurement strategies. Ultimately, a centralized dispatch system could coordinate an efficient zero-carbon grid, and success hinges on addressing flexibility needs, optimizing marginal costs, and leveraging competitive electricity generation to deliver transparent, dynamic price signals.

(6)

Transmission Markets: Decarbonization places unprecedented demands on transmission markets, highlighting their central role in integrating renewable energy and ensuring the energy transition’s success. Utility-scale renewables, often sited far from demand centers, require robust and flexible transmission systems to deliver energy efficiently, reduce congestion, and prevent renewable curtailment. However, the rapid deployment of renewable generation outpaces the slower transmission infrastructure development, creating significant bottlenecks and curtailing clean energy’s full potential. This disconnect undermines economic viability, exacerbates grid stability challenges, and necessitates sophisticated planning and investment. Modern grids must adapt to bidirectional power flows, integrate DER, and lose the natural inertia from traditional power plants, requiring advanced control systems and AS. Solutions such as grid-enhancing technologies, dynamic line ratings, strategically deployed energy storage, and innovative market mechanisms like financial transmission rights are helping optimize current infrastructure and incentivize investment. Addressing this challenge requires earlier and proactive transmission planning, updated regulatory frameworks for fair cost allocation, and enhanced coordination between transmission and distribution systems. Accelerating transmission development through technical, policy, and market reforms is essential to achieving a sustainable and decarbonized energy future.

(7)

Effects of Power Sector Decarbonization on Power Plant Revenues: Decarbonization fundamentally reshapes power plant revenues, creating challenges and opportunities depending on the resource type and market participation. VRE introduces increased market variability, with low or even negative prices during high output periods, reducing traditional FF plants’ revenues as their operating hours decline. Price volatility and regulatory costs, like carbon pricing, further pressure FF plants, which may need to shift from a baseload to a more flexible peaking role with less predictable but higher short-term earnings during low renewable output periods. Capacity markets and ancillary services offer new revenue streams for flexible resources like gas peakers, hydropower, and storage, rewarding their ability to stabilize the grid amidst renewable variability. Renewable plants benefit from LTPAs and priority dispatch, but overgeneration risks and curtailment can undermine their earnings without sufficient storage or demand-side flexibility. As storage and DR technologies become cost-effective, competition in capacity and ancillary markets intensifies. Adapting to these dynamics by embracing flexibility, advanced technologies, and new market roles will be essential for sustaining power plant profitability in a decarbonized energy system.

(8)

Implications of Decarbonization on financial or derivatives markets and prices: Decarbonization significantly impacts financial and derivatives markets by increasing the demand for risk management tools to address heightened variability and uncertainty in energy output and prices. Integrating VRE introduces sharper price fluctuations, reflected in energy derivatives such as futures and options, which traders and investors use to hedge against market volatility. Capacity derivatives may adjust to shifts in reserve margins and the growing influence of weather-dependent resources. At the same time, AS markets could drive new financial instruments like options or swaps to manage risks tied to grid stability. Transmission congestion, a frequent challenge in renewable-heavy systems, heightens the relevance of instruments like CRRs and FTRs to hedge against localized price spikes. LTPA and its associated financial products could also evolve and be influenced by shifting technology costs and market structures. Additionally, derivatives tied to centralized dispatch systems may reflect the uncertainties of marginal costs and renewable integration efficiency. Decarbonization prompts the development of sophisticated financial tools to manage risks, ensure market stability, and capitalize on emerging opportunities in a transitioning energy landscape.

(9)

Emerging Technologies: The transformation of electricity markets necessitates a comprehensive approach integrating cutting-edge technologies and innovative market designs. Artificial intelligence and machine learning [139] can significantly enhance market forecasting and operational optimization, particularly for managing VRE uncertainties and DERs. Digital twin technologies offer crucial virtual simulation capabilities for testing grid scenarios, VRE integration strategies, and market responses.

Blockchain, too, presents transformative potential for decentralizing energy trading and settlement [140], enabling peer-to-peer (P2P) energy trading and virtual power plants (VPPs) [141]. A VPP is an advanced network of DERs that extends beyond traditional solar and battery storage, incorporating grid-interactive appliances, smart buildings, EV charging infrastructure, and thermal energy storage. Using sophisticated control systems, aggregators, utilities, and grid operators remotely coordinate these assets under contractual frameworks to optimize clean energy generation, enhance grid reliability, and provide essential grid services. By integrating advanced software and hardware, VPPs transform decentralized energy assets into responsive, utility-scale resources, redefining conventional energy management. Thus, with widespread smart meter deployment, prosumers—individuals or businesses producing and consuming energy—can directly buy and sell electricity, circumventing traditional utility intermediaries.

Policy frameworks must actively facilitate the integration and aggregation of DERs by adopting adaptive market mechanisms that incentivize grid flexibility, enable local and peer-to-peer energy transactions, implement dynamic pricing models, and establish dedicated local flexibility markets for trading ramping capabilities and load-following resources. Hybrid market structures are essential to achieve this balance, combining centralized reliability with the innovation potential of decentralized systems. Achieving successful implementation requires holistic policy reforms, including comprehensive data-sharing protocols, robust cybersecurity measures to safeguard the system, and incentives to drive innovation and technological advancement.

Pioneering regions in Europe [142,143], Australia [144,145], Japan [146], and the US [147] are making progress in these approaches, highlighting the potential of digital and decentralized technologies to reshape electricity market operations. Policymakers and market regulators must proactively design adaptive frameworks that accommodate technological disruption while maintaining grid stability and advancing sustainable energy transitions.

Blockchain and decentralized energy platforms fundamentally reimagine electricity market infrastructure by shifting from centralized, top-down coordination to a more distributed, peer-to-peer transactional model. This technological innovation enables direct energy trading between producers and consumers, potentially disrupting traditional market operators and grid management systems. The core breakthrough lies in creating transparent, verifiable, and instantaneous transaction settlement mechanisms that allow for granular participation from small-scale energy producers like residential solar owners. Real-world examples, such as LO3 Energy’s Vermont Green program [148], demonstrate how these platforms create micro-markets that traditional grid systems cannot accommodate, enabling net-metered solar customers to sell renewable energy credits to local businesses. Major energy companies are taking notice, with Energy Web [149,150,151] boasting over 100 partnerships, including industry giants like Tepco, Siemens, Shell, and Exelon. Technologically, this is achieved through blockchain-based ledgers, decentralized operating systems, and innovative consensus mechanisms that ensure network integrity. These platforms address the decentralization of electricity networks, encompassing a wide range of distributed energy resources from smart appliances and climate control systems to residential solar + battery installations and electric vehicles. They envision a networked electricity ecosystem where grid operators, consumers, and diverse energy assets interact and collaborate. While significant challenges remain—including technological scalability, complex regulatory landscapes, and market integration—the growing momentum suggests this represents more than a speculative concept. Instead, it signals a potential fundamental transformation of energy infrastructure, shifting from a traditional top-down, unidirectional utility model to a bottom-up, democratic marketplace that empowers individual actors with greater economic agency, technological flexibility, and sustainability potential.

The European Commission placed the accent on energy security after Russia invaded Ukraine. In March 2023, it proposed significant reforms to the EU’s electricity market design to enhance consumer protection, boost renewable energy integration, and improve industrial competitiveness [152]. These reforms were adopted by the EU member states on 21 May 2024 and entered into force on 16 July 2024. The central focus of these reforms is ensuring affordability, reliability, and resilience in an energy system increasingly dominated by VRE. Key measures include enhanced support for contracts for difference (CfDs) to stabilize revenues for renewable energy producers and incentivize clean energy investments, alongside promoting PPA to allow for industrial and consumer actors to hedge against price volatility while directly supporting capacity expansion. Capacity mechanisms have been refined to align with decarbonization goals, incorporating new flexibility services to address grid stability issues. Cross-border market integration has also been expanded, facilitating the efficient use of renewable resources across the EU and reducing congestion and regional price disparities.

Additionally, the reforms prioritize demand-side flexibility through dynamic pricing and incentivized demand response programs, empowering consumers to participate actively in grid balancing. These measures are complemented by support for advanced grid technologies, such as smart grids and AI-driven forecasting tools, to optimize grid operations. Collectively, these reforms position the EU as a leader in designing electricity markets that balance reliability, affordability, and sustainability, offering a model for regions pursuing similar transitions.

• Economic Incentives:

The transition to decarbonization and increasing RE integration fundamentally reshapes electricity markets, catalyzing profound transformations in pricing, resource allocation, and market design. VRE introduces unprecedented price volatility, characterized by overgeneration (triggering low or negative prices) and subsequent shortages that cause dramatic price spikes. This volatility, coupled with DERs, progressively displaces traditional baseload power generation and elevates the strategic value of flexible resources like ES, DR mechanisms, and fast-ramping generation technologies.

Traditional revenue models face challenges as conventional power generation paradigms erode. Concurrently, RE sources benefit from RETs, RPSs, CO₂ taxes and incentive mechanisms, LTPAs, capacity markets, and innovative business models like VPPs. The decentralization of energy production and growing demand for ancillary services underscore the importance of grid modernization, granular pricing strategies, and market designs that leverage emerging technologies. Market structures are dynamically evolving to balance scarcity pricing—which incentivizes system flexibility—with capacity markets that provide generator revenue certainty. For instance, in the European Union’s electricity markets, introducing capacity mechanisms ensures system adequacy while enabling the integration of renewables. LTPAs offer financial stability but may limit real-time market responsiveness, creating a complex optimization challenge between long-term predictability and short-term adaptability. Grid operators face intricate challenges in integrating VRE, DERs, and flexibility requirements within centrally dispatched systems, balancing system stability and economic efficiency.

Economic models such as CO₂ pricing or taxes, cost-benefit analyses, and market adaptations like dynamic pricing are integral to designing electricity markets in a decarbonized future. Carbon pricing—implemented through instruments like the European Union Emissions Trading System (EU ETS)—directly incentivizes emission reductions by attaching a monetary cost to carbon, driving investment in low-carbon technologies. Dynamic pricing further supports the efficient integration of VRE by encouraging demand-side flexibility, enabling consumers to adapt their usage in response to real-time price signals.

Technological advancements, including energy storage systems, smart grids, and digital monitoring tools, are critical to modern electricity market design. For example, lithium-ion and flow batteries mitigate the intermittency of VRE, while smart grids equipped with advanced monitoring and control systems enhance system reliability and flexibility. In the United States, Tesla’s large-scale battery installations in California and South Australia demonstrate how energy storage can stabilize grids and reduce reliance on FF peakers. Additionally, CCS provides transitional pathways for FF-dependent markets, as seen in Norway’s Longship CCS project.

The successful design of electricity markets in a decarbonized economy relies on the interplay of policy, economic incentives, and technology. Regulatory frameworks, such as RPS in the US or feed-in tariffs in Germany, establish foundational market rules that promote low-carbon energy adoption. Economic tools like carbon pricing and market redesigns incentivize behavior changes, while technological advancements enable operational efficiency and reliability. These elements reshape electricity markets to align with decarbonization goals, searching for a sustainable, flexible, and economically efficient energy future.

6. Conclusions

This study explores the transformative impact of large-scale VRE integration on electricity markets, emphasizing the challenges and opportunities posed by decarbonization. The findings highlight the disruptive influence of VRE, such as wind and solar, on traditional market operations, necessitating substantial shifts across technical, economic, and policy domains. It comprehensively explores the transformative effects of decarbonization on electricity markets, addressing technical, regulatory, and operational challenges associated with integrating high shares of variable renewable energy (VRE). Foundational studies, such as [19], emphasize the importance of market flexibility and [20] discuss the critical role of market design elements like dynamic pricing, ancillary services, and cross-border trading. This paper advances their work by offering detailed strategies for specific market segments—day-ahead, real-time, and ancillary services markets—to manage variability through targeted pricing and coordination mechanisms. Operational challenges identified by [21], including overgeneration risks and ramping demands, are addressed through innovative solutions such as dynamic pricing and real-time balancing strategies. These strategies align with machine learning and automation advancements discussed by [22], providing practical methods for optimizing resource utilization while maintaining grid stability and cost efficiency. Case studies offer empirical validation for these contributions. Analyzed by [23], Chile’s market integration demonstrates how regulatory and infrastructure interventions can accelerate renewable deployment and stabilize prices. Similarly, California’s adoption of flexible resource adequacy policies and advanced forecasting methods, highlighted by [24], illustrates the potential of regulatory and technical solutions for achieving sustainability goals. Finally, this paper emphasizes the role of next-generation technologies, building on the works of [25,26]. It advocates increased investments in hybrid energy storage systems and green hydrogen technologies, which address intermittency and enhance grid flexibility, ensuring long-term sustainability. By synthesizing these elements, this paper provides a holistic roadmap for electricity market adaptation, marking a significant contribution to the discourse on renewable energy integration and decarbonization.

• Key Insights:

1

Market Dynamics and Flexibility: The variability and uncertainty introduced by VRE challenge traditional market structures, driving a shift toward markets that value flexibility. Resources capable of rapid response, such as energy storage and demand-side solutions, are becoming pivotal in maintaining system balance and reliability.

2

Forecasting and Real-Time Operations: Accurate forecasting and responsive real-time operations are essential to bridge the gaps between day-ahead schedules and actual conditions. Improved forecasting techniques are critical to reducing costly system imbalances caused by discrepancies in renewable output.

3

Generation Mix and Investment Patterns: A significant transition is evident, with traditional FF-based generation giving way to VRE, ES, and DERs. Investment patterns are shifting towards grid modernization and technologies that enable VRE integration.

4

Transmission and Overgeneration: Transmission constraints and overgeneration issues are highlighted as critical bottlenecks. Overgeneration leads to curtailment and wasted clean energy, undermining renewable project economics. This study stresses the importance of expanded transmission infrastructure, better integration of storage solutions, and refined renewables incentive schemes to address these challenges.

5

Capacity and Ancillary Services Markets: Decarbonization redefines capacity markets, which should prioritize flexible resources like storage and DR over a model designed to favor traditional baseload generators. Ancillary services markets are evolving to address the challenges posed by high VRE penetration, such as frequency regulation, ramping, and voltage support.

6

Economic Dynamics and Policy Focus: The transition to VRE has implications for investment costs, levelized cost of electricity (LCOE), and system costs. Policymakers must incentivize clean energy adoption, encourage energy efficiency, and support the adoption and research into emerging technologies while addressing the system’s costs, the public good nature of energy security, and system flexibility.

7

Financial Markets and Instruments: VRE variability underscores the importance of financial tools like LTPAs, carbon pricing, and derivatives for managing risks and stabilizing revenues. These instruments and the adoption of new technologies, like blockchain and smart contracts, will play a growing role in adapting to the evolving electricity market landscape.

• Concluding Implications:

This study emphasizes the critical need for updated market designs, regulatory reforms, and infrastructure investments to support a VRE-dominated system. Enhanced forecasting, greater system flexibility, and optimized transmission planning are essential to ensure reliability and economic efficiency. It highlights the evolving roles of capacity and ancillary services markets and financial instruments in addressing the challenges of integrating VRE while maintaining system stability.

Economically, decarbonization disrupts traditional market structures with increased price volatility, necessitating granular pricing mechanisms and incentives for flexibility. Financially, the shift directs investments toward clean technologies, energy storage, and grid modernization, with greater reliance on financial instruments like derivatives and LTPAs to mitigate risks and stabilize revenues. Legally, contract frameworks must adapt to address curtailment risks, enhance financial stability for energy producers, and include flexible pricing and renegotiation mechanisms to align with dynamic market conditions. Capacity market agreements and ancillary service contracts should clearly define flexibility and reliability obligations, especially for hybrid resources like storage-integrated renewables.

This study also calls for innovative legal frameworks to support emerging technologies, such as blockchain-based smart contracts for peer-to-peer energy trading, ensuring regulatory compliance and efficient dispute resolution. These economic, financial, and legal transformations underscore the need for cohesive strategies to facilitate a stable, sustainable, and equitable energy transition.