The Next-Generation Retail Electricity Market in the Context of Distributed Energy Resources: Vision and Integrating Framework ^†

Abstract

The increasing adoption of distributed energy resources (DERs) and smart grid technologies (SGTs) by end-user retail customers is changing significantly both technical and economic operations in the distribution grid. The next-generation retail electricity market will promote decentralization, efficiency, and competitiveness by accommodating existing and new agents through new business models and transactive approaches in an advanced metering infrastructure (AMI). However, these changes will bring several technical challenges to be addressed in transmission and distribution systems. Considerable activities have been carried out worldwide to study the impacts of integrating DERs into the grid and in the wholesale electricity market. However, the big vision and framework of the next-generation retail market in the context of DERs is still unclear. This paper aims to present a brief review of the present retail electricity market, some recent developments, and a comprehensive vision of the next-generation retail electricity market by describing its expected characteristics, challenges, needs, and future research topics to be addressed. A framework of integrating retail and wholesale electricity markets is also presented and discussed. The proposed vision and framework particularly highlight the necessity of new business models and regulatory initiatives to establish decentralized markets for DERs at the retail level as well as advances in technology and infrastructure necessary to allow the widespread use of DERs in active and effective ways.

1. Introduction

The traditional way of generating, transmitting, and distributing electricity has been changing significantly as generating units become more distributed, efficient, and closer to consumption centers. The use of distributed energy resources (DERs), such as distributed photovoltaic (PV), energy storage systems (ESSs), and demand response (DR), combined with innovative smart grid technologies (SGTs) has been growing every year. DERs are not the most cost-effective option from a power system perspective, but many investments have been made worldwide in developing more affordable, flexible, and efficient solutions that enable greater adoption of these technologies at customer level, thus improving power system efficiency, reliability, flexibility, and helping several countries reach decarbonization targets [1]. However, market efficiency is necessary to make the smart grid environment completely smart [2]. This includes business models designed to maximize the social welfare of all participants by monetizing energy products and services with the highest possible quality for the lowest possible price.

Many states and countries have partially liberalized and decentralized their retail electricity markets by adopting retail choice programs that enable retail customers to choose electricity suppliers, services, and tariff schemes according to their needs and preferences. Such a paradigm has encouraged innovation and flexibility in both pricing and services offered to retail customers. In addition, many liberalized and non-liberalized retail markets have implemented compensation programs for customer-side distributed generation (DG) and DR to encourage the adoption of behind-the-meter renewable DG and peak load reduction, respectively. In the United States (U.S.), over 40 states adopted compensation programs for customer-side DG, such as net metering [3], and the number of net-metered customers increased from fewer than 22,000 in 2005 to more than 1,500,000 in 2016 [4,5]. Utilities in the California Independent System Operator (CAISO) area reported 5.4 GW of net-metered distributed solar capacity as of December of 2016 [6]. Moreover, some independent system operators (ISOs) and regional transmission organizations (RTOs) in the U.S. have been adopting DR programs for end-user retail customers through direct load control [7].

Europe has also been experiencing significant growth in DERs in the last years. Residential PV installations are expected to reach 90 GW by 2021. Belgium currently has around 150 W of PV residential capacity per capita, the highest in Europe [8]. In Australia, small-scale PV systems produced 3.4% of the country’s total electricity in 2017. More than 20% of the Australian houses are currently provided with PV systems [9].

As advanced metering infrastructures (AMIs) become more affordable, accessible, and spread in the distribution grid, more entities will take advantage of DERs, transforming conventional loads into smart loads, and conventional buildings into smart buildings and microgrids. Such technologies are provided with effective power management and control systems, and communication schemes that promote energy efficiency and allow direct interaction with external agents and systems. Thus, conventional passive customers will become active agents in the retail electricity market, by providing DERs in efficient and effective ways. Such active agents are defined as DER agents and their main technical capabilities are described as follows [1]:

• DG: Small-scale production of electricity at distribution voltage levels. It can be based on variable renewable sources (e.g., PV systems), or systems that can be dispatched at any time (e.g., small gas turbine generators).
• Load management (LM): Capability to receive and quickly respond to external signals to manage real-time electricity consumption, perform load shifting and scheduling, etc., through smart metering systems (SMSs), intelligent energy management systems (IEMSs), smart loads, etc. [10,11,12].
• Energy storage systems (ESSs): Capability to import, store, and export electricity to the grid via battery energy storage systems (BESSs), plug-in electric vehicles (PEVs) operating in the vehicle-to-grid (V2G) mode, and other energy storage technologies.

The management of DERs is performed through IEMSs and SMSs, whereby DER agents can receive and send information, set preferences, and control energy production, consumption, and storage in real-time. Figure 1 illustrates the concept of DER agents in the smart grid environment.

The increasing adoption of DERs and SGTs in the distribution grid has led researchers, policy-makers, and other stakeholders to define the concept of transactive energy as a reliable, affordable, and sustainable system that promotes the dynamic balance of supply and demand and maximizes the social welfare of all its agents [13]. Under this environment, all decisions have economic and/or engineering “values,” such as profit or performance, as key operational parameters [14].

Power system flexibility from both supply and demand sides are becoming a necessity in several countries facing fuel price uncertainties, growth in variable renewable energy production, and higher load variability [15]. The management of DERs for power system flexibility is still very limited, but is expected to play a big role in the next years, opening opportunities for reforms in retail electricity markets worldwide.

Although the recent advances in DG, LM, and ESSs have been transforming the distribution grid in many ways, it is evident that the conventional electric system framework, which includes the physical infrastructure, system operations, and power market operations, does not fully take advantage of the technical and economic benefits that all DERs can provide, and is not ready to accommodate and manage the dynamic flow of power and information among the existing and the emergent electricity market agents.

In the literature, several studies have proposed new methods to integrate DERs in the wholesale market [16,17,18,19,20,21] and in the retail market [22,23,24,25,26,27]. Reference [28] provided a comprehensive bibliographical survey of recent works related to the next-generation U.S. retail electricity market with consumers and prosumers, which are end users with the ability to produce and consume electricity.

Most of those proposals, to be implemented successfully, involve important changes in the electric system, such as new market business models, new technologies, and advanced transmission and distribution infrastructures. Some of those changes are discussed in [29,30], with great emphasis on implications for electric utilities. With the increasing penetration and participation of DER agents in the distribution grid, what are the expected technical and economic characteristics, challenges, and needs of the next-generation retail electricity markets? How can DERs integrate retail and wholesale markets as well as transmission and distribution grids? To address these open questions in a comprehensive manner and provide researchers and engineers with an overview of the future characteristics, changes, challenges, and future research topics associated with increasing market integration of DERs, this paper expands the work in [1] by presenting an overview of the present retail electricity market, some recent developments on transactive energy, a comprehensive vision of the next-generation retail electricity market, a framework of integrating retail and wholesale markets, and the main roles and responsibilities of retail market agents under the proposed vision and framework. In addition, this paper contributes to the energy policy research area by providing examples of regulatory initiatives in different jurisdictions and discussing important technical and economic aspects necessary to develop new policies for decentralized markets with DERs. Although the U.S. has taken the lead in implementing new retail market business models with transactive energy mechanisms and most examples in this paper refer to the U.S., retail electricity market evolution with widespread DERs is a global trend. Thus, the proposed vision and framework can be applied to any retail electricity market without loss of generality.

The remainder of this paper is organized as follows. Section 2 briefly reviews the present retail electricity market, its recent developments, and some transactive energy demonstration projects. Section 3 presents a framework and the main characteristics of the next-generation retail electricity market in the context of DERs. Section 4 describes the challenges and needs that should be addressed as well as potential solutions in the proposed vision and framework. The main roles and responsibilities of retail market agents are described in Section 5. A framework of integrating wholesale and retail markets is presented in Section 6. Finally, relevant conclusions are presented in Section 7.

2. Present Retail Electricity Market and Recent Developments

The retail electricity market is the economic platform that facilitates the provision of electricity to retail customers. Historically, most retail customers worldwide were subjected to electricity rates defined by electric utilities and regulated by governmental entities. However, the liberalization of some retail electricity markets has promoted competition and diversification of services, enabling customers to choose from different services and suppliers. According to a study of the retail electricity in the U.S. [31], the effect of moving to a competitive retail electricity market was mixed across states, but led to lower electricity prices in states with high customer participation, and higher prices in states with little customer participation. In that study, customer participation was referred only to the ability to switch electricity suppliers. Liberalization made the retail electricity market more efficient by lowering the markup of retail prices over wholesale costs. In addition, the development of DER technologies enabled new types of retail tariffs and mechanisms for the integration of DR and DG into the distribution grid.

Current DR programs adopted worldwide can be classified into price-based DR, in which customers control their energy consumption in response to pricing arrangements, and incentive-based DR, in which retail customers directly control or let grid operators control part of their loads in exchange for financial incentives. The present price-based DR programs in most retail markets include time-of-use (TOU) pricing, real-time pricing (RTP), critical peak pricing (CPP), and increasing block price (IBC). The present incentive-based DR programs include direct load control (DLC), emergency demand response (EDR), and demand bidding/buyback (DB) [22].

Table 1. Main demand response (DR) programs in the present retail electricity market. TOU: time-of-use; RTP: real-time pricing; CPP: critical peak pricing; IBC: increasing block price (IBC); DLC: direct load control; EDR: emergency demand response; DB: demand bidding/buyback.

Price-Based DR		Incentive-Based DR
DR Program	Characteristics	DR Program	Characteristics
TOU	Retail rates depend on the time of the day (e.g., peak and off-peak), season (e.g., summer and winter), and day of a week (e.g., weekday, weekend, and holiday). Retail customers have full knowledge on TOU rates before adjusting their consumption pattern.	DLC	Grid operators remotely control customers’ loads during the periods of high electricity prices or when the grid reliability is threatened.
RTP	Retail rates reflect the real-time wholesale market price changes at every hour or half-hour. Some RTP programs simply pass along the hourly wholesale market prices with no markup whereas other programs define a retail rate based on the wholesale market price variations.	EDR	EDR programs usually offer rebates or other incentives to retail customers for load reduction during critical events. Retail rates are generally not changed.
CPB	Retail rates are significantly raised during times of critical events (e.g., periods of extremely high wholesale market prices and power system emergency conditions). CPP programs are usually limited to a fixed number of critical events during a specific time period.	DB	Large retail customers provide load curtailment at a price at which they are willing to be curtailed.
IBP	Retail rates are divided into several increasing tiers or blocks. Retail customers are charged based on the electricity consumption levels and the prices for the corresponded blocks.

describes the main characteristics of these DR programs.

DG is integrated into the present retail electricity market worldwide primarily through net metering and feed-in tariff mechanisms. Both mechanisms credit small-scale DG producers, such as residential and commercial facilities with solar PV systems, for the electricity they add to the grid to which they are connected. The term net metering refers to the value of the difference between the energy delivered and received at a retail customer, which is usually measured via a unidirectional meter that only measures electricity consumption. In feed-in tariff mechanisms, the rate utilities pay for the electricity exported to the grid might be different from the consumption rate. This requires a bidirectional meter that measures both electricity generation and consumption. In many states and countries, retail customers are paid the full retail rate for the energy exported to the grid. However, as the number of self-producers of energy increases every year, there has been increasing discussions among electric utilities, regulatory commissions, industry stakeholders, and retail customers concerning DG compensation tariffs and the real value of DG for both the retail markets and distribution grids. On the one hand, some utilities argue that self-producing retail customers are not paying for grid and consumer costs, thus shifting costs onto other retail customers [32]. On the other hand, a coalition of partners in the solar energy industry claims that net metering tariffs should not be underestimated; otherwise, it would discourage customers from generating their own clean electricity [33]. In 2014, Minnesota became the first state in the U.S. to adopt a “value of solar” policy, by defining solar energy tariffs based on all the costs avoided by utilities when using solar energy from retail customers [34]. Other states, such as Arizona, Maine, and Indiana are currently reviewing their regulation policies on net metering and defining new tariff schemes for DG.

In contrast to traditional market integration schemes for DERs, transactive energy approaches have been receiving increasing attention from both industry and academia in the last few years due to the need for greater integration of DERs into the electricity grids as well as new market mechanisms to facilitate such integration. It should be noted that in this paper “electricity grids” include the cyber-physical infrastructure of distribution and transmission grids.

Some demonstration projects in the U.S. analyzed some values of DERs in a transactive energy framework. Some of these projects include the Pacific Northwest Smart Grid Demonstration in the Olympic Peninsula and the GridSMART Demonstration Project in Ohio. The former evaluated technologies and proposed a regional transactive coordinating system to optimize the energy dispatch and the demand management in an environment replete with intermittent energy sources and SGTs [35]. The latter integrated several customer-oriented SGTs and studied the impacts of different DR pricing mechanisms to the grid reliability and to customers’ electricity bill [36]. Both projects involved several entities such as utilities, universities, and technology companies. They concluded that transactive mechanisms contributed to peak power demand and energy cost reductions under high customer engagement. Some other demonstration projects under development include the National Institute of Standards and Technology (NIST) Transactive Energy Challenge in the U.S. [37] and the Ecogrid 2.0 in Denmark [38]. Additionally, the Gridwise Architecture Council (GWAC) has been taking an important role in studying transactive energy projects. The GWAC is an independent organization supported by the U.S. Department of Energy that includes representatives from different sectors of the U.S. electric system [39].

Furthermore, some U.S. states have been adopting transactive energy approaches in their planning and vision for the future. For instance, the Reforming the Energy Vision (REV) strategy intends to change the way energy is produced, consumed, and transacted throughout the state of New York. This program intends to reduce 40% of total carbon emissions, reduce 23% of the energy consumption in buildings, and generate 50% of the total electricity from renewable energy sources by 2030 through various approaches, including transactive energy initiatives that allow interactions between consumers and self-producers of energy. The REV project intends to make market agents take full advantage of every type of energy resource [40]. Moreover, in California, environmental and energy policies combined with increasing integration of DERs are forcing substantial changes in the local electricity market [41]. CAISO is currently implementing a new market mechanism to integrate DERs into its wholesale market through new market agents, called distributed energy resource providers (DERPs). Such agents will be able to aggregate DERs at the retail level and provide energy and ancillary services in the wholesale market. This new mechanism will provide more trading options for retail market agents and enable more integration of DERs into the grid [42]. Other initiatives, such as the Distribution Resource Plan (DRP) and the Integrated Distribution Energy Resources (IDER) efforts will explore transactive energy mechanisms, among various alternatives, to support services provided by DERs and improve the reliability of the Californian distribution grids [43]. In the European Union (E.U.), the Renewable Energy Directive is currently establishing policies for the promotion and production of renewable energy. This directive intends to ensure that at least 27% of the electricity consumption in E.U. is supplied by renewable energy sources by setting specific targets for each country. E.U. countries, on the other hand, have been developing action plans that include investments and policies to facilitate market integration of DERs to meet those targets [44].

Finally, there is a growing interest in the global industry in exploring the utilization of bulk electricity market concepts to provide economic signals that can be used to compensate DER-driven transactions. For instance, the introduction of distribution locational marginal pricing (DLMP) [45,46,47,48,49] has received growing attention as an option to estimate the value of DER and evaluate the feasibility of non-wire alternatives (NWAs).

3. Key Characteristics of the Next-Generation Retail Electricity Market in the Context of DERs

The recent retail market developments described in the previous section as well as the trends and technical studies for the market integration of DERs mentioned in Section 1 indicate that the current retail electricity market is evolving from a centralized and passive environment to a decentralized and interactive platform, where market participants can interact with each other and provide technical services to the electricity grid. This entails not only in new business models, but also in innovative regulatory paradigms and infrastructure upgrades. Based on such studies, trends, and developments, an expanded vision of the next-generation retail electricity market in the context of DERs is illustrated in Figure 2. This vision is formed by key characteristics as well as challenges and needs that will require future research and development [1].

The key characteristics of the proposed vision are described as follows:

• Operation optimization: Market agents will be able to minimize costs, maximize revenues, and optimize the grid operation through timely, cost-effective, and safe transactive energy mechanisms that will promote the effective, reliable, and efficient integration of DERs for commercial and technical purposes. Furthermore, an efficient integration between retail and wholesale markets will allow DERs to be aggregated and effectively integrated into the wholesale market and the transmission system. The optimal management of DERs will facilitate the provision of affordable and fast-response ancillary services to distribution and transmission grids. The implementation of this paradigm would require high levels of real-time monitoring, protection, automation, and control to keep track and process potentially millions of transactions. These are the capabilities that are not currently available in distribution grids and would require large investments in enabling intelligent infrastructure (e.g., sensors, information systems, etc.). The approval and recovery of investments of this magnitude typically involves thorough regulatory scrutiny. Therefore, this vision will need to be accompanied by regulatory modernization and modern recovery mechanisms that should go beyond traditional approaches based on cost of service.
• Flexibility: A flexible retail electricity market will be ready to accommodate new technologies and market agents by enhancing and expanding its infrastructure. Market agents will cope with uncertainties, such as electricity demand, electricity prices, and renewable energy generation through effective forecasting techniques and comprehensive decision-making strategies. In addition, a flexible retail electricity market will utilize its local resources to manage uncertainties, variations, and unforeseen events over various time horizons. Here it is important to emphasize that flexibility must be accompanied by robustness to handle the uncertainties derived from market complexity, which may be difficult to capture even through advanced forecasting techniques.
• Customer integration: A fully integrated retail electricity market will be designed to maximize the social welfare of all market agents, accommodate a large number of diverse agents, and satisfy their needs and preferences in a competitive environment. Market platforms should be designed to allow the active participation and interaction of all agents and avoid market power and conflicts of interest by considering the strengths and limitations of each agent. An integrated retail electricity market will also promote social engagement to facilitate the participation of agents from different socio-economic strata. This is a critical task, given that the proliferation of DER and the introduction of related concepts such as microgrids, may prompt end-users to consider the possibility of defecting from the grid or to largely use it as a backup source [50]. Therefore, a customer-focused market that has the capability to provide customized solutions to end users may help retain end users.
• Sustainability: A sustainable retail electricity market will maximize the use of clean and renewable energy sources and promote energy efficiency initiatives through market mechanisms designed to reduce greenhouse gas emissions, mitigate load and network losses, and incentivize the use of environment-friendly technologies that will contribute to sustainability and energy efficiency.

4. Opportunities, Challenges, and Needs

In order to achieve the key characteristics described in Section 3, the present retail electricity market must be reformed not only to bring new opportunities for DERs integration but also to properly address several challenges and needs. Such aspects are described as follows [1]:

4.1. New Business Models

As the number of DER agents increase in the distribution grid, comprehensive business models designed to accommodate, integrate, and allow the dynamic interaction of all retail electricity market agents will be needed. Such business models include clear rules and plans describing the market participants, their roles and interactions, products and services, as well as strategies to generate revenue, minimize costs, and maximize profits. The main elements and aspects of the new business models are described as follows:

• Regulatory Modernization: The next-generation retail electricity market will depend massively on regulatory modernization in different jurisdictions. This includes market liberalization, decentralization as well as policies and incentives aimed to reduce the current dependency on centralized and pollutant power plants and incentivize the use and adoption of DERs. Today, in most countries, the concept of retail market liberalization and decentralization refers to the capacity of consumers to have freedom and independence to choose electricity suppliers and services according to their needs and preferences. Such a paradigm has promoted competition and cost reductions and increased innovation. However, in the context of DERs, liberalization and decentralization also include the provision of DG, LM, and ESSs for important energy services in a decentralized market framework. The increasing integration of DERs makes the retail market liberalization and decentralization even more challenging due to several technical and economic complexities, and uncertainties that are introduced. The extended process of retail electricity market liberalization and decentralization is not a simple event. It is a long-term process that, to be successful, requires coordinated planning among governmental entities and all other involved sectors and agents. Furthermore, regulatory changes along with plans for investments in infrastructure and technology necessary to ensure all the characteristics described in Section 3 should take into account all possible economic and social benefits. For instance, aspects such as innovation, expected societal benefits of renewable integration, resiliency improvement, and customer choice must need to be accounted for in project justification. In this regard, the RIIO model (Revenue = Incentives + Innovation + Outputs) implemented by Ofgem, the U.K. regulatory agency, represents an initial step in this direction [51].
• New Agents: The increasing adoption of SGTs and DERs has brought several DER agents in the distribution grid. Such agents include smart buildings, microgrids, and PEVs. However, their current participation in the retail electricity market is very limited since they are still under programs and tariffs imposed by electric utilities. Such programs and tariffs limit the active market participation and the provision of important energy services from DER agents, which can provide technical and economic benefits to the grid and to other market agents. The studies in [22,52,53,54] illustrated how load serving entities (LSEs), such as utilities and competitive retailers, can benefit from integrating DERs in their decision-making models and, thus, mitigate the impacts of wholesale market volatility on their procurement costs. However, the massive adoption of DERs will depend directly on governmental subsidies and tax breaks designed to reduce greenhouse gas emissions and control the electricity demand growth. Such aspects are still big challenges in countries with high dependency on centralized and nonrenewable generation and unclear or inexistent decarbonization targets. The increasing number of DER agents will give rise to new entities to operate the distribution grid and facilitate market mechanisms in the retail level. In most countries, the electricity market is categorized into wholesale and retail markets, and the power grid is divided into transmission and distribution grids with different agents performing technical and economic operations. In the U.S., for example, both wholesale market and transmission grid operations are performed by ISOs/RTOs. Electric utilities, on the other hand, coordinate both retail market and distribution grid operations. The concept of distribution system operator (DSO) has been extensively discussed in the academic, industry, and regulation sectors [55]. The DSO is envisioned as an entity that can operate the distribution grid, facilitate transactions involving DERs in a neutral retail market, and work cooperatively with existing transmission grid and wholesale market operators [56,57,58,59,60,61,62,63]. This concept is different from some present European utilities also called DSOs, which mainly operate the distribution grid and facilitate customer choice of load serving entities, with no complete interaction with transmission service providers and wholesale market operators [57]. Two types of DSOs have been extensively discussed in the literature. The utility DSOs (UDSOs) are referred to as evolved electric utilities, which will have expanded responsibilities and capabilities to provide new products and services to retail customers and provide new ways for DER owners to increase the return on their investments [64,65]. The independent DSOs (IDSOs), on the other hand, are referred to as independent and regulated entities that are not affiliated with any other electricity market participant [66], and are responsible for coordinating technical operations in the distribution system and facilitating open access retail markets for DERs, similar to how ISO/RTOs operate the transmission grids and coordinate the wholesale market in the U.S. In the IDSO concept, utilities continue to own and maintain the physical assets in the distribution grid [60]. A complete discussion of the roles and responsibilities of the market agents in the next-generation retail electricity market is provided in Section 5.
• Transactive energy mechanisms: To promote liberalization and decentralization, and accommodate the increasing number of DER agents in the retail electricity markets, effective transactive energy mechanisms should be developed to monetize the value of DERs. Examples of transactive energy mechanisms include interactive bidding and offering platforms, and market clearing models capable to incorporate uncertainties from DG production, electricity demands, and wholesale market prices over different time horizons [67]. To ensure the real-time balancing of supply and demand, minimize the impacts of abrupt changes and unforeseen events in the distribution grid, and ensure that DER agents can respond to such events promptly, the time frame of each transactive energy mechanisms should be determined properly. The profitability of flexible products and services via transactive energy mechanisms is a critical point that should be studied carefully before implementing new retail market business models. Moreover, DSOs should be able to review, approve, deny, and modify a large number of transactions by considering their impacts on the distribution grid [68]. This will be possible with high speed sensing and processing capabilities. All transactive energy mechanisms should promote the information exchange among agents, facilitate the receipts and payments for energy services in real-time, and ensure price transparency. Thus, the real-time cost of power and ancillary services can be accessible to all market agents.
• Competitiveness: The increase in DER use and adoption will stimulate competition in two ways. First, the incentives, subsidies, and other regulatory programs aimed to promote local energy efficiency, flexibility, and sustainability will increase the competitiveness of DERs over centralized and non-renewable generation units. Without such regulatory programs, system operators are prone to use more centralized generation units since the widely use of DERs increases power system complexity and requires substantial infrastructure upgrades. Second, the creation of new business models and transactive energy mechanisms will stimulate retail market competition and, thus, promote lower electricity prices and better services. However, to ensure the social welfare maximization, the market rules and mechanisms should be clear, transparent, and fair to all market agents.

4.2. Innovative DER Technologies

The massive adoption of DERs in the distribution grid will rely on innovative technologies aimed to increase efficiency, resilience, and flexibility to both technical and market operations in affordable ways. Existing technologies will be enhanced and expanded to provide additional functionalities. Some opportunities for enhancements and advances in DER technologies are described as follows:

• DG Systems: In the last few years, DG systems have become more efficient and affordable. PV systems are currently the most used DG systems worldwide. Grid-tied PV systems are directly connected to the grid at all times and do not use storage systems. On the other hand, hybrid PV systems can operate in grid-tied form, charge batteries during the periods of low electricity prices, discharge batteries during the periods of high electricity prices, and even operate in the off-grid mode. Figure 3 shows the residential and commercial PV costs evolution from 2010 to 2018 in the U.S. [69]. In 2016, the National Renewable Energy Laboratory (NREL) and the Swiss Center for Electronics and Microtechnology (CSEM) have jointly set a new world record for converting non-concentrated sunlight into electricity with 29.8% efficiency using a dual-junction III-V/Si solar cell [70]. However, the reduction in the size of solar modules and the adaptability and customization to different places and surfaces are examples of improvements required to satisfy customer needs and preferences. Solar shingles and tiles currently in development by Tesla [71] may be a potential solution for these issues if proven to be cost-effective and durable for different environments. Moreover, other types of existing clean DG systems such as wind and hydrogen fuel cell need improvements in flexibility and affordability. The development of innovative and clean DG system will promote a diversified portfolio of sustainable energy in the distribution grid. Furthermore, the power electronics of DG systems should be designed to provide high quality power supply, voltage regulation, and ancillary services in effective ways. Particularly, PV inverters should mitigate voltage rises, caused by reverse power flow from PV systems, through effective control strategies [72].
• ESSs: Lithium-ion batteries are a good example of electrical energy storage technologies that have been improved significantly in the last few years. However, not all technologies are widely available and economically feasible. Cost, specific energy, specific power, safety, performance, and life span of ESSs still need improvements and should be considered carefully for ESSs to become widely adopted in the distribution grid and, thus, increase the flexibility of variable DG. The next-generation retail electricity market will rely on efficient and affordable energy storage technologies to manage a large share of intermittent resources. Furthermore, other existing ESSs such as flywheels and supercapacitors may also be improved and adapted to DER agents in the distribution grid.
• PEVs: The recent advances in technologic, affordable, and customer-oriented PEV models boosted PEV sales in the last few years. The total U.S. PEV sales increased from fewer than 20,000 in 2009 to nearly 200,000 in 2017 [73]. PEVs can provide important DERs to the grid since they can act as loads and dispatchable ESSs. When not needed for transportation, PEVs can participate in DR programs and contribute to peak load reduction while being charged and can also operate in the vehicle-to-grid (V2G) mode, when charged, to support the electricity grid needs such as energy dispatch, voltage regulation, and reactive power support [74,75]. In addition, PEVs can also operate in the vehicle-to-building mode under a grid voltage [76], and exchange power with charging stations and other PEVs. Some key factors that will allow PEVs to become large-scale DER providers include evolution of ESSs, advanced information and communication technologies, regulatory initiatives, as well as public and private incentives and investments aimed to improve technical performance and grid interoperability with reduced costs [74,77]. Upgrades on functionalities and grid interconnection schemes of existing PEV charging stations represent an initial step towards this end.
• Microgrids: DER proliferation and growing requirements from end users regarding reliability, resiliency, power quality, and efficiency have prompted a growing interest in both utility and customer-owned microgrids as a means to take advantage of some of the key benefits of DERs. While DER technology, analysis, standards, and relevant technical aspects required for microgrid deployment are evolving fast, existing regulatory frameworks are not ready to address the challenges associated with microgrid implementation. Aspects such as DER compensation, performance guarantees, liabilities, and participation of third-party providers in community or utility microgrids are areas that are still either undefined or not clearly defined, depending on the jurisdiction. This can only be addressed with clear viability studies and pilot projects that require efforts and collaboration from different stakeholders. Figure 4 illustrates the potential complexity of this endeavor, where every DG is expected to be compensated according to the value and benefits it provides to the grid and transactions. Estimating these revenue streams requires the ability to potentially process massive amounts of data to ensure that individual profit maximization objectives do not trump the overall operational performance goals.
• IEMSs and SMSs: The coordination of energy production, consumption, storage, and the energy exchange with the grid is performed through IEMSs and SMSs, which allow DER agents to be active agents in the retail electricity market. Future IEMSs and SMSs will be designed to minimize procurement costs, maximize revenues and energy efficiency actions while maintaining acceptable level of customers’ comfort, and provide two-way flow of power and information that will allow DER agents to interact directly with other market agents. Decision-making algorithms will need to be designed and embedded in IEMSs and SMSs to satisfy customer preferences and provide services to the grid. Transactive control, also called market-based control, might be a potential solution to allow IEMSs and SMSs interact more actively on market operations. Transactive control is a distributed control approach that uses market mechanisms to engage generating units, storage systems, and flexible loads to provide energy services to the grid. In this approach, DER agents and DSOs would only exchange the prices and the quantities of electricity to be purchased/sold during a certain period, thus preserving the preferences and the privacy of DER agents [78]. Some relevant transactive control approaches for commercial buildings, residential buildings, and PEVs are presented in [78,79,80,81,82], respectively.
• Smart loads: Future low-voltage loads, such as electric vehicles, heating, ventilation, and air conditioning (HVAC) systems, lighting systems, and household appliances, should be designed to provide affordability, flexibility, energy efficiency, and easy communication with IEMSs, SMSs, and DSOs through adaptive and timely power control. Flexibility can be achieved through the provision of different levels of comfort and user preferences in interactive ways.

4.3. Grid Infrastructure and Operations

Presently, a large amount of DERs are behind-the-meter assets that are “invisible” to grid operators due to limited distribution system awareness. These DERs “mask” the load tracked by meters, thus making load forecasting and power system operations even more challenging [83]. The existing transmission and distribution grid infrastructures are not ready to accommodate the increasing number of DERs and allow their transition from passive customers to active retail market agents. Several advances in grid infrastructure and operations will be needed in the proposed vision of the next-generation retail electricity market to ensure the efficiency, reliability, and resiliency of distribution grids and support the needs of transmission grids through energy services coordinated by DSOs and transmission grid and wholesale market operators. Such advances are summarized into five aspects as follows:

• Control and optimization: The complexities associated with grid integration of DERs, transactive energy approaches, and the implementation of DSOs will challenge the existing control and optimization architectures in the electricity market. In particular, future DSOs will need to deal with a great number of DER agents and energy transactions in different buses of the distribution grid by producing adaptive control actions. Some of the technical aspects that should be considered in the decision-making strategies of DSOs include reactive supply and voltage control (Volt/Var control), self-healing actions, and many power flow constraints [84]. On the market operations, DSOs should operate pricing mechanisms that reflect the real cost of electricity in each distribution node, such as market clearing mechanisms with DLMP that reflect the real-time value of DERs in the distribution grid. In general, all retail market agents should be able to collect, store, and analyze large sets of data through advanced architectures for data acquisition, transmission, storage, and processing. Big data analytics will play an important role for both technical and market operations in the distribution grid [85,86]. An innovative supervisory control and data acquisition (SCADA) system based on the Internet of Things (IoT) may be a potential solution to enable the observability and controllability of multiple DERs in the distribution grid [64].
• Grid expansion planning: Transmission and distribution grid expansion planning analyzes the expansion or reinforcement needs of an existing infrastructure to adequately serve system loads with minimal cost and acceptable quality standards over a certain time horizon [87]. The massive and widespread penetration of DERs in the distribution grid will bring more uncertainties and challenges to the transmission and distribution grid expansion planning strategies. The proper integration of DERs along with the development of transactive energy mechanisms may help defer investments on costly distribution and transmission grid assets. The REV strategy is a practical example of grid expansion planning considering DERs. This strategy focuses on the integration of DERs and nontraditional approaches, also referred to as NWAs, into the planning of the distribution grid in the state of New York [88]. Moreover, new methodologies for increasing the hosting capacity (i.e., the maximum amount of DERs that can be accommodated without jeopardizing the system) will need to be developed. This can be addressed with technical studies and methodologies adaptable to different power system topologies and scenarios. Future grid expansion planning mechanisms should take into account different DER penetration scenarios and ensure the long-term reliability of transmission and distribution grids.
• Forecasting: The uncertainties associated with the variability of short-term renewable energy production, energy consumption, energy storage levels, and electricity prices will challenge technical and market strategies in the distribution grid, thus requiring the development of advanced and accurate forecasting techniques to improve the decision-making strategies of retail market agents and allow the management of risks caused by unforeseen events [89]. New computational intelligence approaches capable of considering a vast number of uncertainties may be a potential solution towards this end.
• Communications: Transactive energy mechanisms should be able to handle the vast amount of data coming from DER agents dispersed in the distribution grid and real-time information from wholesale markets and transmission grids. This will require advanced communication infrastructures as well as interconnectivity, interoperability, and scalability capabilities to allow real-time information exchange among retail market agents in different points of the distribution grid. Advanced wireless communication infrastructures with standard communication protocols will be essential to ensure information collection, dissemination, processing, and security [90,91]. Energy Internet (EI) communication schemes may be a potential solution to allow effective communications and transactive energy mechanisms among market agents. EI, which is considered the evolution of smart grid, is an integrated grid of DERs, real-time monitoring, information sharing, and market transactions that provides energy packing and routing functions, similar to the Internet [92,93,94].
• Cybersecurity: As the communication networks become more interconnected and interoperable, they become more vulnerable to deliberate attacks, such as those from disgruntled employees, espionage, and terrorists, as well as inadvertent compromises of the information systems due to natural disasters, equipment failures, and human errors. Such vulnerabilities might allow attackers to penetrate a network and destabilize the system in several ways [95]. In addition, in the event of an unauthorized disclosure and access to private and confidential data, market agents can orient their strategies to gain advantage illicitly and, thus, destabilize competitive energy trading mechanisms. Prevention should be the main goal, but preparation to respond and recover quickly should also be carefully planned. This will require advanced monitoring, sensing, and control mechanisms as well as standardized authentication and authorization strategies to ensure the system security and integrity [95]. Blockchain and smart contracts may be a potential solution to facilitate auditable multiparty transactions based on prespecified rules between market agents and, thus, increase the trustworthiness, integrity, and resilience of energy transactions [96]. Recently, the U.S. Department of Energy announced over $20 million awards to national laboratories and partners to conduct cybersecurity projects for the energy sector. Among such projects, the KISS (Keyless Infrastructure Security Solution) project, to be conducted by the Pacific Northwest National Laboratory, will develop blockchain cybersecurity technology for DERs at the grid’s edge [97].

4.4. Distributed Grid Services

The next-generation retail electricity market will not only open economic transactions involving DERs, but also maximize the benefits of DERs to transmission and distribution grids through energy services designed to enhance and support the grid reliability and resilience. The main challenges and needs associated with the implementation of such services are described as follows:

• Congestion and Losses Management: Presently, most electricity markets worldwide do not consider energy congestion and losses in distribution grids in the electricity price determination. However, the adoption of DLMP has been receiving increasing attention in the last few years [49,98,99,100] as a potential solution to price congestion and losses in the distribution grid. In transmission grids, transmission LMPs (TLMPs) consider three components namely system energy price, congestion cost, and marginal losses cost. The first component considers only the optimal energy dispatch and, thus, ignores congestion and losses. The second component represents the prices of congestion, which are calculated on each bus of the system. Load generally pays the congestion prices to generation. The third component represents the price of losses caused by power injection or withdrawal and it is also calculated on each bus. In the distribution grid, DLMPs can be decomposed into marginal costs for active power, reactive power, congestion, voltage support, and losses while considering the real value of DERs on each node of the distribution grid [49]. This will require powerful market clearing mechanisms capable of handling a large amount of decision variables while considering the imbalances and nonlinearities of the distribution grid.
• Markets for ancillary services: Ancillary services are essential energy services provided in addition to power generation to guarantee power system reliability, safety, and stability. In the present electricity market framework, such services are provided by large synchronous generators and coordinated by system operators in transmission grids via wholesale market operations. Figure 5 shows the six ancillary services defined by the U.S. Federal Energy Regulatory Commission (FERC) through its orders 888 and 2000, issued in 1996 and 1999, respectively. Several studies have shown potential ancillary services that can be provided by DERs through effective active and reactive power control of inverter-based DG, prompt load control, and proper management of ESSs, respectively [101,102,103,104,105,106,107]. However, the adoption of ancillary services provided by DERs should be accompanied by transactive energy mechanisms that facilitate the provision of those services locally to a distribution grid or aggregated to serve the needs of a transmission grid. Volt/VAR control is an example of ancillary service that may be provided to a local distribution grid by inverter-based DG. Volt/VAR control is currently performed by using capacitors in distribution lines and substations. Figure 6 illustrates a framework of integrating ancillary services provided by DERs in distribution and transmission grids. In this framework, the DSO coordinates ancillary services exclusively for the distribution grid, such as Volt/VAR control, and also aggregates potential ancillary services to be integrated in the transmission grid by the ISO/RTO, when prices are favorable. In addition, DERs can be aggregated by other market agents to provide ancillary services to the transmission grid through the wholesale market. However, the provision of ancillary services by DERs will depend on regulatory initiatives for DER integration and aggregation, new standards for DER interconnection and interoperability with the electricity grid, and a cooperative work between DSOs and transmission grid and wholesale market operators. The IEEE 1547 has been recently revised to allow some new grid support functionalities of DERs [108]. The next-generation retail electricity market should take advantage of the technical capabilities of DER agents and integrate effective transactive mechanisms for ancillary services. Thus, distribution and transmission grids may benefit from emerging DER agents to become more efficient and reliable.
• Load management: The existing price-based and incentive-based DR programs described in Section 2 play an important role in the operations of several power systems worldwide, especially during the periods of high wholesale market prices, peak load demands, or when system reliability is threatened [109]. However, such programs do not explore all functionalities and capabilities provided by existing and emerging SGTs to allow real-time load management as well as dynamic and effective interactions among DER agents and between DER agents and grid operators. Transactive energy mechanisms for load management will be extremely important in the next-generation retail electricity market as SGTs continue to evolve and DER agents have their capabilities expanded to become active market agents and directly participate in real-time load management programs in effective ways.

5. Retail Market Agents: Main Roles and Responsibilities

The roles of retail market agents are changing dramatically as SGTs evolve and DERs spread in the distribution grid [110]. The next-generation retail electricity market will accommodate new agents, which will play important roles in the effective operation of transactive energy mechanisms and the provision of services to the electricity grids. The main roles and responsibilities of active market agents in the next-generation retail electricity market are briefly illustrated in Figure 7 and described below. Hereinafter, without loss of generality, transmission grid and wholesale market operators are represented by a single entity, which is the case in the U.S., for illustrative purposes.

• DER agents: DER agents will play an important role in making the retail market more competitive, efficient, and sustainable. They will be able to interact and exchange energy with other market agents as well as provide DERs and important energy services to electricity grids through transactive energy mechanisms.
• Load serving entities (LSEs): LSEs consist of competitive retailers and utilities. The main difference between retailers and utilities is that utilities generally supply electricity to their customers and maintain power lines, transformers, substations, and other equipment in the distribution grid whereas retailers only resell electricity to their customers, having no ownership over any assets in the distribution grid. LSEs will continue to ensure the supply of energy at competitive prices and high-quality services to passive and active market agents, promote incentives for energy efficiency and DG, and bill retail customers for energy consumption and delivery charges.
• Aggregators: For the sake of generality, aggregators and LSEs are represented as separate entities in Figure 7 since DER aggregation can be of two types: LSE aggregation and third-party aggregation. LSE aggregation refers to the capacity of LSEs to aggregate local DERs in its territory to support the needs of a local distribution grid or a transmission grid in cooperation with ISOs/RTOs. On the other hand, third-party DER aggregation is performed by any entity other than LSEs and their individual customers [111,112], such as third-party investors. Third-party aggregation has been discussed extensively in the last few years as a way to make DERs competitive resources in the wholesale electricity market, but its implementation still faces regulatory barriers in several jurisdictions. The Regulatory Assistance Project (RAP) recently published a comprehensive report to the Public Service Commission of the state of Arkansas that shows the advances, challenges, and needs required for the proper implementation of third-party aggregators in that state [111]. In general, aggregators will actively participate in both retail and wholesale markets by aggregating DERs and trading those resources through effective and dynamic transactive energy mechanisms. Aggregators will work cooperatively with DER agents and ISOs/RTOs to ensure an effective integration of DERs in both the wholesale markets and transmission grids [112].
• Distribution system operators (DSOs): DSOs will be responsible for balancing supply and demand at the distribution level and close the gap between retail and wholesale markets [13]. They will provide new functionalities to coordinate the participation of DER agents in the electricity markets (as market operators) and optimize technical operations in distribution grids (as grid operators). For UDSOs, a firewall separation between the DSO block and the merchant block of UDSOs needs to be established to ensure a fair, nondiscriminatory, and competitive retail market and avoid market power and eventual conflicts of interest especially in the cases that UDSOs own or invest in DERs [64]. On the other hand, IDSOs should be nonprofit market facilitators with no ownership of the distribution system assets. In both DSO concepts, regulated utilities continue to own and maintain the distribution grid assets and LSEs continue to retain billing function to their customers [60]. The implementation of DSOs, however, is a gradual process of technical and regulatory modernization towards the integration of transactive energy mechanisms in the distribution grid. Figure 8 illustrates the roles and responsibilities of comprehensive DSOs in the distribution grid and retail market operations. Working in the distribution grid operations, DSOs will be responsible for maintaining the safety and reliability of the distribution system through efficient protection and control mechanisms, coordinating ancillary services provided by DER agents, planning the distribution system expansion, and islanding their territory partially or totally when black-start capacity from DERs is available in the event of a transmission system disturbance [58,113]. Working in the retail market operations, DSOs will balance supply and demand in the distribution grid, facilitate energy scheduling and settlements, coordinate the exchange of power with the wholesale market, and ensure the integrity and transparency of all transactions in the retail market [113]. The common responsibilities of DSOs in both grid and market operations include the forecasting, planning, and integration of DERs as well as the social welfare maximization of all agents in their territory.
• ISOs/RTOs: ISOs/RTOs are primarily responsible for the control and operation of transmission grids and wholesale markets, respectively. However, in the next-generation retail electricity market, such entities will work cooperatively with DSOs to ensure the transmission grid reliability by using DERs located in the distribution grid through transactive energy mechanisms. Note that in Figure 7, ISOs/RTOs are represented by the dotted box hierarchically above all other agents. A framework of integrating retail and wholesale markets that shows the interaction between ISOs/RTOs and retail market agents is described in Section 6.

6. A Framework of Integrating Retail and Wholesale Electricity Markets

The advances in technology, infrastructure, and the provision of energy services described in Section 4 will change not only the interactions among retail market agents, but also the way in which wholesale and retail markets interact. An information exchange framework for the wholesale and the next-generation retail electricity markets is illustrated in Figure 9a. The solid arrows indicate the flows of information between agents in the present electricity market framework, which remain unchanged. The hatched arrows indicate new two-way flows of information among existing and emerging market agents. In this framework, DER agents can interact with DSOs, LSEs, aggregators, and other DER agents. In Figure 9a, the DSO and the LSEs are separated for the sake of generality. However, in the UDSO concept, they become a single entity. Moreover, aggregators are included in both retail and wholesale markets as explained in Section 5. In such a framework, DSOs and ISOs/RTOs may interact with each order and work coordinately for the provision of energy services to ensure the efficiency, reliability, and resilience of the electricity grids. A coordinated work between DSOs and ISOs/RTOs is necessary to close the existing gaps between wholesale and retail markets as well as between transmission and distribution grids, respectively [13]. Unlike the present electricity market framework in which energy production and DR are mostly centralized in large producers and consumers, the next-generation retail electricity market will integrate active DER agents in a decentralized manner. Figure 9b illustrates the flows of energy among wholesale market agents and the next-generation retail electricity market agents in a decentralized framework. Wholesale market participants and LSEs will continue to generate bulk power to be provided to retail market agents. However, a decentralized retail market will allow the exchange of energy among DER agents and the efficient provision of DERs to the wholesale market.

7. Discussion and Conclusions

The retail electricity market worldwide is changing due to the increasing integration of DERs in the distribution grid, the development of innovative SGTs, and the necessity to ensure the reliability, resilience, and efficiency of transmission grids, and provide new services and functionalities to retail market agents. This paper aimed to fill an existing gap in the literature regarding the comprehensive vision of the next-generation retail electricity in the context of DERs and the integration of retail and wholesale electricity markets. In this paper, an overview of the present retail electricity market and some recent developments on transactive energy were discussed. Then, a comprehensive vision of the next-generation retail electricity market in the context of DERs has been presented. The proposed vision described the main technical and regulatory characteristics, challenges, opportunities, and needs that will arise as passive customers provided with DERs become active agents in the electricity market. In addition, a framework of integrating retail and wholesale markets and the main roles and responsibilities of retail market agents under the proposed vision and framework have also been presented and discussed. In particular, this work highlighted the necessity of new regulatory approaches to establish new business models with decentralized markets for DERs and the expected technology advances and infrastructure upgrades required to allow a reliable and efficient integration of DERs. These two points are the big challenges to reach the proposed vision and framework since they require concerted efforts and collaboration from different stakeholders. Government agencies, LSEs, energy policy makers, DER agents, and technology providers must agree on a common strategy to start the shift towards an efficient, flexible, sustainable, and decentralized retail electricity market. Through this way, DER agents will have active participation in the electricity market, and DERs will be used more efficiently in a competitive way. Therefore, more research, development, and practical initiatives are needed to make this vision and framework partially or fully implementable to ensure the social welfare of market agents as well as to maximize the technical benefits of DERs to both distribution and transmission grids.