Coalitional Demand Response Management in Community Energy Management Systems

Abstract

With the advent of the Distributed Energy Resources within smart grid systems, traditional demand response management (DRM) models need to be redesigned to capture prosumers’ energy consumption requests and dynamic behavior within the energy market. In this paper, a coalitional DRM model is introduced based on the principles of Game Theory and reinforcement learning to dynamically determine prosumers’ formation in local energy trading communities and their optimal energy consumption. A hedonic game-theoretic model is introduced to enable prosumers to autonomously and dynamically select an energy trading community considering the partially available information regarding prosumers’ energy generation and consumption characteristics and utility companies’ provided rewards per community. Then, a log-linear reinforcement learning model is proposed to enable each prosumer to distributedly determine their optimal energy consumption. A detailed evaluation of the proposed coalitional DRM model is performed based on real data collected from the southwest area of the USA.

1. Introduction

In recent years, growing technological advancements have provided a wide variety of appliances to consumers, improving their living standards. However, this ever-increasing number of appliances, being used in homes in consumers’ every day lives, has resulted in an unprecedented energy demand [1]. The traditional retail energy market struggles to support consumers’ energy demand in a real-time manner, and a large portion of total energy consumption stems from household appliances, e.g., 42% in the U.S. [2]. Towards addressing this supply–demand challenge, Distributed Energy Resources (DERs), such as electric vehicles (EVs), residential rooftop solar photovoltaic (PV) panels, and energy storage systems (ESSs), have transformed traditional consumers into prosumers who are capable of generating and consuming energy and exchanging/selling energy with other prosumers in a peer-to-peer manner or with the main grid [3]. The joint problem of satisfying the energy demand of residential users with a wide variety of energy consumption patterns in order to avoid high energy demand peaks is addressed by demand response management (DRM) models [4,5].

1.1. Background and Motivation

Several sophisticated DRM models have been introduced in the recent literature, such as time-of-use (TOU), price-based home management systems, critical peak pricing (CPP), and real-time pricing (RTP). These DRM models have been applied in energy markets consisting of both consumers and prosumers, considering their energy consumption and generation characteristics [6]. Focusing on efficiently exploiting prosumers’ renewable energy production and DERs flexibility, a new energy market paradigm has recently been introduced based on the principles of peer-to-peer (P2P) transactive energy trading. In this new energy market paradigm,  prosumers are organized in communities forming P2P energy trading systems which are coordinated following either system-centric or prosumer-centric approaches [7]. However, the existing prosumer community creation approaches mainly consider prosumers’ physical proximity to group them in communities, without exploiting their energy generation and consumption characteristics to reveal the full potential of the communities in terms of balancing customers’ energy needs in a smart grid system [8].

In this paper, towards addressing the above challenges, we introduce a novel coalitional demand response management model to reveal the full potential of prosumer communities in terms of addressing their energy demands. The proposed coalitional DRM model consists of two main components: (i) the hedonic game-theoretic communities formation framework and (ii) the reinforcement-learning-based DRM mechanism. Specifically,  prosumers’ energy generation and consumption characteristics are exploited along with the smart grid operator’s/utility company’s provided rewards and the information availability in the overall smart grid system in order to form communities among prosumers following the principles of hedonic games [9]. Then, a reinforcement-learning-based mechanism is designed to determine each residential prosumer’s optimal consumption in order to optimize its perceived satisfaction while accounting for its energy demand characteristics. A detailed evaluation of the proposed coalitional DRM model is performed based on real data analysis in the southwest area of the USA.

1.2. Related Work

Several artificial-intelligent-inspired DRM models have been introduced in the recent literature focusing on residential consumer use case scenarios [10,11]. A time-of-use DRM model is introduced in [12] enabled by a model-free deep reinforcement learning mechanism with dueling deep Q network structure to optimize the management of the interruptible load considering a variety of consumers energy consumption patterns. A similar reinforcement-learning-enabled time-of-use DRM model is proposed in [13] focusing on the utility companies’ perspective in terms of predicting the optimal energy prices and discounts offered to consumers. Focusing on the home energy management systems as part of the DRM models, a double deep-Q-learning mechanism is proposed in [14] to perform scheduling of home energy appliances considering consumers’ energy needs and utility companies’ announced prices. A similar mechanism is designed in [15] following a deep-Q-network approach and focusing on the time-shiftable appliances, such as electric vehicles, lighting systems, and air conditioners. A residential DRM model based on reinforcement learning and fuzzy reasoning is discussed in [16] considering consumers’ preferences in order to schedule the operation of smart home appliances, also as  a function of energy price. An hour-ahead DRM model is analyzed in [17] based on an artificial neural network approach to predict energy price and a multiagent reinforcement learning mechanism in order to perform scheduling of home appliances. A federated learning DRM model is introduced in [18] by aggregating local models from multiple utility companies to train a global smart grid model that performs energy price prediction.

Several recent approaches have also focused on privacy issues related to the design of DRM models targeted at consumers or prosumers [19]. A modified vector homomorphic encryption is analyzed in [20] in order to perform a secure load profiling of consumers based on encrypted meter data. Aiming at protecting consumers’ privacy, a DRM model is designed in [21] to learn an intelligent multi-microgrid system’s energy price response by implementing a deep neural network without direct access to consumers’ private energy consumption  information.

Focusing on the DRM models specifically designed to address residential prosumers energy needs, great attention has been given to the analysis of their energy generation and consumption characteristics [22]. A deep reinforcement learning model is designed in [23] to optimize the energy consumption of a household equipped with several DERs towards reducing prosumers’ energy cost while accounting for their comfort-level characteristics. The retail and wholesale energy markets are analyzed in [24] by introducing a reinforcement-learning-based, price-based DRM mechanism that enables the energy management system to determine its optimal retail market energy price and prosumers’ energy consumption to jointly maximize profit and prosumer utility. A reinforcement-learning-based DRM model is introduced in [25] to perform energy scheduling of smart homes’ energy storage systems in order to minimize energy cost given announced energy prices.

The P2P transactive energy trading concept has recently attracted the interest of the research community, enabling prosumers to be organized in groups and directly exchange energy among each other through the grid [26]. A P2P transaction model is designed in [27], including a participant model, an equipment model, and a price model. A community energy management system is introduced in [28] based on a multiagent reinforcement learning approach to handle uncertainty in renewable energy and minimize energy cost. A deep reinforcement learning approach is proposed in [29] to minimize energy costs experienced by prosumers in P2P energy trading considering the dynamic nature of their energy availability. The users’ preferences and their level of engagement in a P2P energy trading market is studied in [30] based on a reinforcement learning approach that optimizes the system’s performance in terms of matching the energy resources.

1.3. Contributions and Outline

Though a tremendous amount of research work has been performed in the fields of DRM and community energy management systems, the problem of forming prosumers’ communities based on their energy generation and consumption characteristics, as well as accounting for the smart grid operator’s provided rewards, remains highly unexplored. Even more complicated is the problem of optimally forming prosumers’ communities while jointly determining prosumers’ energy consumption accounting for their preferences and characteristics.

Towards addressing the above challenges, in this paper, a coalitional demand response management model is introduced to support the operation of community energy management systems and optimize prosumers’ payoff via optimally forming communities among each other and determining their optimal energy consumption. To the best of our knowledge, this is the first research work in the existing literature combining the theory of hedonic games with the principles of reinforcement learning in order to create prosumer coalitions and determine optimal energy consumption, respectively. Specifically, the main contributions of this manuscript are summarized as follows:

• A community energy management system is introduced consisting of prosumers who are autonomously organized in communities. The community formation process accounts for prosumers’ energy generation and consumption characteristics, the available information about the prosumers’ characteristics in the overall system, and the smart grid operator’s/utility company’s allocated reward to consumers for participating in the DRM process. The community formation process is formulated as a hedonic community formation game, and the existence of a Nash-stable and individual-stable partition is proven.
• A reinforcement-learning-based framework is introduced to determine prosumers’ energy consumption towards fulfilling their energy needs and maximizing their payoff from purchasing their remaining energy demand from the smart grid. Two different types of reinforcement learning mechanisms are adopted, i.e., log-linear reinforcement learning algorithms, in order to test their accuracy, performance, and complexity in terms of determining prosumers’ optimal energy consumption.
• A detailed evaluation of the proposed coalitional DRM model is performed based on real data collected from the southwest area of the USA. The performance evaluation demonstrates the operational characteristics of the proposed DRM model along with its superiority compared with the state of the art in terms of optimizing prosumer payoff.

The remainder of this paper is organized as follows. Section 2 and Section 3 present the prosumers’ and their communities’ models, respectively. Section 4 introduces the hedonic communities formation game-theoretic framework. Section 5 analyzes the reinforcement-learning-based DRM model to determine prosumers’ optimal energy consumption. The performance evaluation is demonstrated in Section 6, and Section 7 concludes this paper.

2. Prosumer Characteristics

In this section,  prosumers’ energy generation and consumption characteristics are analyzed. We consider a set of residential prosumers $\mathcal{N}=[1,\dots ,n,\dots ,N]$ residing in a geographical area and being able to exchange energy among each other through the infrastructure of the smart grid network. Each prosumer is equipped with several DERs that enable them to generate, e.g., residential rooftop solar photovoltaic panels, and store energy, e.g., electric vehicles and energy storage systems. Based on the available DERs, each prosumer can generate $g_n^t$ (KWh) energy per timeslot t, e.g., one hour. We study the overall system for a total set of timeslots $\mathcal{T}=[1,\dots ,t,\dots ,T]$, e.g., $T=24$ h one-day duration.

Each prosumer has a set of appliances ${\mathcal{A}}_n$ that consume energy $a_n^t$ (KWh) when they continuously operate. Some of the appliances can be characterized by shiftable operation, e.g., charging of electric vehicles and running dishwashers, and some of them by nonshiftable operation, e.g., refrigerators. Based on each prosumer’s shiftable and nonshiftable appliances’ energy demand characteristics, they can adapt their energy consumption following an intelligent DRM model. The total energy demand of a prosumer is determined by the energy consumption of their appliances, if the latter ones operate, as follows:

$$d_n^t=\sum _{\forall a_n\in {\mathcal{A}}_n}{\delta }_{a_n}^t·a_n^t$$

(1)

where $d_n^t$ (KWh) denotes the prosumer’s energy demand in timeslot t, $a_n$ denotes the prosumer’s appliance with $a_n\in {\mathcal{A}}_n$, and ${\delta }_{a_n}^t$ denotes an operator, where ${\delta }_{a_n}^t=1$ if the appliance $a_n$ operates during the timeslot t; otherwise, ${\delta }_{a_n}^t=0$. The prosumers’ energy demand vector in a timeslot t in the examined area is denoted as ${\mathit{D}}^t=[d_1^t,\dots ,d_n^t,\dots ,d_N^t]$ and their corresponding energy generation vector as ${\mathit{G}}^t=[g_1^t,\dots ,g_n^t,\dots ,g_N^t]$.

A prosumer is characterized as self-sufficient in timeslot t, if $g_n^t\ge d_n^t$. In this case, the prosumer can satisfy their own energy needs without purchasing energy from other prosumers or the utility company. The potential surplus of energy can be stored in the prosumer’s energy management system $b_n^{t+1}=b_n^t+(g_n^t-l_n^t)$, where $b_n^t$ (KWh) denotes the prosumer’s battery availability in timeslot t, and $l_n^t$ (KWh) captures the prosumer’s actual energy consumption in timeslot t, where in general $l_n^t\le d_n^t$ given appliances with shiftable operation. The prosumer can use the energy surplus to cover their energy needs in a future timeslot or sell it to other prosumers or to the utility company, while the latter point is out of the scope of this research work.

A prosumer is characterized as non-self-sufficient if $g_n^t+b_n^{t-1}<d_n^t$. In this case, the prosumer cannot cover their energy needs in timeslot t and buys energy from the utility company. The utility company’s price of the energy is controlled by state/country-level regulations [31], and is denoted as $c\left[\frac{\$}{\mathrm{KWh}}\right]$. The prosumer decides to buy an amount $l_n^t\le d_n^t-g_n^t-b_n^{t-1}$ from the utility company in timeslot t in order to cover their energy needs considering the energy price and flexibility with respect to the energy consumption stemming from their appliances with shiftable operation. In the rest of our analysis, we focus our study on the non-self-sufficient prosumers who need to join a community in order to cover their energy needs by purchasing energy.

3. Communities Model

The goal of the smart grid operator is to efficiently handle the prosumers’ energy demand in order not to experience a brownout or even a blackout, while at the same time maximizing its profit. A brownout occurs when the voltage supplied by a utility company drops below the normal level for a brief period. Brownouts are often caused by factors such as high demand for electricity or grid instability. Unlike a blackout, where power is completely lost, a brownout involves a partial reduction in voltage levels, which can impact the performance of electrical devices. Thus, if the prosumers’ energy demand from a utility company is not handled by intelligent demand response management mechanisms, the smart grid can experience a brownout or even a blackout. Similarly, the goal of the prosumers is to organize themselves in communities in order to experience the maximum possible benefit from the energy purchase process, e.g., discounts from the utility company. We consider that N prosumers can be organized into M communities, where the set of communities is denoted as $\mathcal{M}=[1,\dots ,m,\dots ,M]$. The prosumers are organized in communities by considering their energy generation and consumption characteristics, the rewards provided by the utility company in the form of energy price discounts, and the available information to make their decision, as analyzed below.

A community m is characterized by its value, which is defined as follows:

$$V\left({\mathcal{N}}_m^t\right)=\sum _{\forall n\in {\mathcal{N}}_m^t}{l}_n^t$$

(2)

where ${\mathcal{N}}_m^t=[1,\dots ,n,\dots ,N_m^t]$ is the set of prosumers that joined the community m in timeslot t. The physical meaning of the community’s value $V\left({\mathcal{N}}_m^t\right)$ captures the energy consumption volume that is requested by the prosumers of this community in timelsot t.

The utility company allocates a reward $r_m$ (USD) in the form of energy price discounts at each community m to incentivize the prosumers to purchase energy directly from the utility company. The utility company can sophisticatedly decide the allocation of the rewards $r_m$ per community based on multiple factors, such as energy demand in a geographical area, distribution cost of the energy in specific geographical areas, service priority of specific areas (e.g., hospitals, schools, and in general critical infrastructure), and others. The reward is allocated among the prosumers of the community following the principles of proportional fairness as follows:

$$\frac{l_n^t}{V\left({\mathcal{N}}_m^t\right)}·r_m^t=\frac{l_n^t}{{\displaystyle \sum _{\forall n\in {\mathcal{N}}_m^t}}{l}_n^t}·r_m^t$$

(3)

However, the prosumers that have selected a community m in timeslot t are not aware of the reward that other prosumers received by joining another community $m^{\prime }$ based on their energy consumption characteristics $l_{n^{\prime }}^t,\forall n^{\prime }\in {\mathcal{N}}_{m^{\prime }}^t$. Thus, we define the energy consumption uncertainty among the prosumers’ communities as $u_{m^{\prime },m}^t\in {\mathbb{R}}^{+}$. The lower the value of $u_{m^{\prime },m}^t\in {\mathbb{R}}^{+}$, the more noisy the information; thus, the prosumers have very vague information about the energy consumption characteristics and experienced rewards in other communities. Based on the consumption uncertainty at each community $m^{\prime }\in \mathcal{M}\setminus \left\{m\right\}$, the prosumers belonging to each community $m^{\prime }\in \mathcal{M}\setminus \left\{m\right\}$ are informed in a noisy manner about the energy consumption characteristics and experienced rewards in community m. Thus, we define the noisy energy consumption as follows:

$$y_{{\mathcal{M}}^{\prime }\to m}^t={\displaystyle \sum _{\forall m^{\prime }\in \mathcal{M}\setminus \left\{m\right\}}}{u}_{m^{\prime },m}^t\frac{{\displaystyle \sum _{\forall n\in {\mathcal{N}}_m^t}}{l}_n^t}{{\displaystyle \sum _{m=1}^M}{\displaystyle \sum _{\forall n\in {\mathcal{N}}_m^t}}{l}_n^t}$$

(4)

where ${\mathcal{M}}^{\prime }=\mathcal{M}\setminus \left\{m\right\}$. The noisy energy consumption parameter captures the level of information incompleteness experienced by the prosumers belonging to a community $m^{\prime }\in {\mathcal{M}}^{\prime }$ regarding the energy consumption characteristics and rewards of the prosumer belonging to community m.

Also, the most recent information regarding the energy consumption characteristics and rewards experienced by the prosumers in a community m values more in other prosumers’ decision to join this community. Thus, we define the freshness fading function to weigh the most recent information more as follows:

$${\theta }_{\tau }=z^{t-\tau }$$

(5)

where $z\in (0,1]$, t denotes all the timeslots until the current timeslot that the system had studied, and $\tau$ denotes the individual passed timeslot up to the current timeslot t.

Based on the freshness fading function, we define the fading-aware consumption uncertainty as follows:

$${\tilde{u}}_{m^{\prime },m}^t=\frac{{\displaystyle \sum _{\forall m^{\prime }\in {\mathcal{M}}^{\prime }}}{\displaystyle \sum _{i=1}^t}{\theta }_i{u}_{m,m^{\prime }}^i}{{\displaystyle \sum _{i=1}^t}{\theta }_i}$$

(6)

which captures the level of uncertainty regarding prosumers’ energy consumption and rewards in a community $m^{\prime }$ experienced by the prosumers belonging to community m.

Thus, we introduce the fading-aware noisy energy consumption as follows:

$${\tilde{y}}_{{\mathcal{M}}^{\prime }\to m}^t=\frac{{\displaystyle \sum _{i=1}^t}{\theta }_i{y}_{{\mathcal{M}}^{\prime }\to m}^i}{{\displaystyle \sum _{i=1}^t}{\theta }_i}$$

(7)

that captures the freshness of information within prosumers’ noisy energy consumption parameter.

By combining the fading-aware consumption uncertainty (Equation (6)) and the fading-aware noisy energy consumption (Equation (7)), we introduce the concept of community’s m captivation parameter as follows:

$$c_m^t=w_1{\tilde{y}}_{{\mathcal{M}}^{\prime }\to m}^t+w_2{\tilde{u}}_{m^{\prime },m}^t$$

(8)

where $w_1,w_2\in [0,1]$, with $w_1+w_2=1$. The captivation parameter captures the attractiveness of a community in terms of attracting prosumers to join it, given that they will experience high rewards and they have the potential of exchanging energy with other prosumers who have the potential of high energy surpluses. The captivation parameter (Equation (8)) depends on the fading-aware noisy energy consumption ${\tilde{y}}_{{\mathcal{M}}^{\prime }\to m}^t$ and the fading-aware consumption uncertainty ${\tilde{u}}_{m^{\prime },m}^t$, which, respectively, depend on the energy consumption uncertainty $u_{m^{\prime },m}^t\in {\mathbb{R}}^{+}$. Thus, its value is determined based on the prosumers’ energy consumption characteristics and the information availability within the examined smart grid system.

By combining prosumer’s experienced reward and captivation parameter from joining a community m, we can define their corresponding pure payoff by belonging to a community as follows.

$$P_n^{{\mathcal{N}}_m^t}=\frac{l_n^t}{{\displaystyle \sum _{\forall n\in {\mathcal{N}}_m^t}}{l}_n^t}·r_m^t+\alpha c_m^t$$

(9)

where $\alpha =1\left[USD\right]$ captures the equivalent revenue benefit of the prosumer by joining a community.

Also, the prosumer has a corresponding cost from purchasing energy $l_n^t$ regarding the community that it belongs to, and it is defined as follows:

$$c_n^{{\mathcal{N}}_m^t}=c·l_n^t$$

(10)

Thus, the prosumer’s payoff from joining a community m is derived as follows:

$${\mathcal{U}}_n^{{\mathcal{N}}_m^t}=P_n^{{\mathcal{N}}_m^t}-c_n^{{\mathcal{N}}_m^t}$$

(11)

4. Hedonic Communities Formation

In this section, the theory of hedonic games is exploited in order to autonomously organize prosumers into communities, thus forming a community energy management system. The prosumers’ energy generation and consumption characteristics, along with the utility company’s provided rewards and information availability in the overall smart grid system, are jointly considered in the communities formation process, which is autonomously performed by the prosumers.

Definition 1.

(Community): A community of prosumers is denoted as ${\mathcal{N}}_m^t\subseteq \mathcal{N}$, where m is the community’s index.

Definition 2.

(Prosumers’ Partition): Considering the total number of communities M, with ${\mathcal{N}}_m^t\cap {\mathcal{N}}_{m^{\prime }}^t=\varnothing ,\forall m\ne m^{\prime },m,m^{\prime }\in \mathcal{M}$, and ${\mathcal{U}}_{m=1}^M{\mathcal{N}}_m^t=\mathcal{N}$, the partition of $\mathcal{M}$ that spans all the prosumers in $\mathcal{N}$ is defined as $\Pi =\{{\mathcal{N}}_1^t,\dots ,{\mathcal{N}}_m^t,\dots ,{\mathcal{N}}_M^t\}$.

The available prosumers’ partitions can be categorized into the following special cases.

Definition 3.

(Grant Community): If all the prosumers are organized in only one community, then this community is called a grant community.

Definition 4.

(Singleton Community): If each prosumer creates its own community without any other prosumers belonging to it, then this community is called a singleton community.

Definition 5.

(Empty Community): If no prosumers belong to a community, then this community is called an empty community.

Each prosumer $n\in \mathcal{N}$ has its own preferences over all the communities that they can possibly join, as derived by their experienced payoff (Equation (11)). Thus, the prosumers compare the potential payoffs that they can enjoy by joining different communities and order the latter ones in terms of their preferences.

Definition 6.

(Preference Order): A preference relation ${\ge }_n$ is defined for each prosumer n, $\forall n\in \mathcal{N}$, as a reflexive, complete, and binary relation over all the potential communities that each prosumer can join as follows:

$${\mathcal{N}}_n^t\ge {\mathcal{N}}_{m^{\prime }}^t\iff {\mathcal{U}}_n^{{\mathcal{N}}_m^t}\ge {\mathcal{U}}_n^{{\mathcal{N}}_{m^{\prime }}^t}$$

(12)

where ${\mathcal{U}}_n^{{\mathcal{N}}_m^t}$ is given by Equation (11). If a prosumer strictly prefers to join community ${\mathcal{N}}_m^t$ over ${\mathcal{N}}_{m^{\prime }}^t$, then the following expression should hold true:

$${\mathcal{N}}_n^t>{\mathcal{N}}_{m^{\prime }}^t\iff {\mathcal{U}}_n^{{\mathcal{N}}_m^t}>{\mathcal{U}}_n^{{\mathcal{N}}_{m^{\prime }}^t}$$

(13)

The prosumers participate in a non-cooperative game in order to determine their optimal community choice, as quantified by the corresponding payoff that they experience, as captured by Equation (11).

Definition 7.

(Hedonic Game): A hedonic game is defined by the pair $(\mathcal{N},>)$, where $\mathcal{N}$ denotes the set of prosumers, and $>=[{>}_1,\dots ,{>}_n,\dots ,{>}_N]$ is the preference order vector of the prosumers. The prosumers’ payoff ${\mathcal{U}}_n^{{\mathcal{N}}_m^t}$ depends only on the prosumers of the community ${\mathcal{N}}_m^t$ that the prosumer n belongs to, and the communities partition $\Pi =\{{\mathcal{N}}_1^t,\dots ,{\mathcal{N}}_m^t,\dots ,{\mathcal{N}}_M^t\}$ results from the prosumers’ preferences over all the potential communities M.

During the execution of the hedonic game, the prosumers dynamically switch communities until they converge to a stable partition.

Definition 8.

(Switching Operation): Given a prosumers’ partition $\Pi =\{{\mathcal{N}}_1^t,\dots ,{\mathcal{N}}_m^t,\dots ,{\mathcal{N}}_M^t\}$, a prosumer $n\in {\mathcal{N}}_m^t$ switches communities from ${\mathcal{N}}_m^t$ to ${\mathcal{N}}_{m^{\prime }}^t$ if and only if: ${\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\}{>}_n{\mathcal{N}}_m^t$ thus, ${\mathcal{N}}_m^t\to {\mathcal{N}}_m^t\setminus \left\{n\right\}$ and ${\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\}$.

The goal of the prosumers is to converge to a Nash-stable and individual-stable partition so they cannot further improve their payoff given the communities’ choices that the rest of the prosumers have selected, thus achieving the maximum personal payoff by following their own community choice.

Definition 9.

(Nash-stable Partition): A partition $\Pi =\{{\mathcal{N}}_1^t,\dots ,{\mathcal{N}}_m^t,\dots ,{\mathcal{N}}_M^t\}$ is Nash-stable if ${\mathcal{N}}_m^t{>}_n{\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\},\forall m\ne m^{\prime },\forall n\in {\mathcal{N}}_m^t,\forall {\mathcal{N}}_m^t\in \Pi$.

The physical meaning of a Nash-stable partition is that no prosumer wants to change community, given the communities’ choices of the rest of the prosumers, as they will not receive a higher payoff (Equation (11)).

Definition 10.

(Individual-stable Partition): A partition $\Pi =\{{\mathcal{N}}_1^t,\dots ,{\mathcal{N}}_m^t,\dots ,{\mathcal{N}}_M^t\}$ is individual-stable if and only if the following conditions do not hold true:

(i) 

${\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\}{>}_n{\mathcal{N}}_m^t\forall m\ne m^{\prime }$, meaning there does not exist a prosumer n in community ${\mathcal{N}}_m^t$ who strictly prefers another community ${\mathcal{N}}_{m^{\prime }}^t$;

(ii) 

${\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\}{>}_{n^{\prime }}{\mathcal{N}}_{m^{\prime }}^t,\forall n^{\prime }\in {\mathcal{N}}_{m^{\prime }}^t$, meaning that the formation of a new community ${\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\}$ does not reduce the preference payoffs of the members of the new community ${\mathcal{N}}_{m^{\prime }}^t$.

Based on Definitions 9 and 10, it is easily observed that a Nash-stable partition is also an individual-stable partition, while a vice versa observation does not always hold true.

Theorem 1.

(Existence of a Nash-stable and Individual-stable Partition): Given a random initial partition $\Pi =\{{\mathcal{N}}_1^t,\dots ,{\mathcal{N}}_m^t,\dots ,{\mathcal{N}}_M^t\}$, the proposed hedonic game $(\mathcal{N},>)$ always converges to a Nash-stable and individual-stable partition ${\Pi }^{*}=\{{\mathcal{N}}_1^{t*},\dots ,{\mathcal{N}}_m^{t*},\dots ,{\mathcal{N}}_M^{t*}\}$.

Proof. 

The proof of Theorem 1 follows the reductio ad absurdum analysis. Suppose that the final partition ${\Pi }^{*}$ is not Nash-stable. Then, some of the prosumers have the incentive to switch communities in order to experience a higher payoff. Thus, some prosumers will follow the switching operation process (Definition 8) and join some other communities, where they have the potential of enjoying a higher payoff. Given that switching operations still go on, the partition ${\Pi }^{*}$ is not final, which contradicts our assumption that the final partition ${\Pi }^{*}$ is not Nash-stable. Therefore, the hedonic game always converges to a final partition ${\Pi }^{*}$, which is Nash-stable; thus, it is also individual-stable.    □

The hedonic game’s algorithm in order to determine the Nash-stable and individual-stable partition is presented in the next section, along with the reinforcement learning algorithm to determine prosumers’ energy consumption.

5. Reinforcement-Learning-Based Demand Response Management

In this section, a reinforcement-learning-based mechanism is introduced to enable prosumers to determine their optimal energy consumption $l_n^t$ to maximize their utility from purchasing and consuming energy, while also considering the energy tariff announced by the utility company.

The prosumer’s utility from purchasing and consuming energy is defined as follows:

$$U_n(l_n^t,{\mathbf{l}}_{-n}^t)=\beta f(l_n^t,{\mathbf{l}}_{-n}^t)g\left(l_n^t\right)-c\left(l_n^t\right)+\frac{l_n^{t-1}}{{\displaystyle \sum _{\forall n\in {\mathcal{N}}_m^t}}{l}_n^{t-1}}{r}_m$$

(14)

where ${\mathbf{l}}_{-n}^t=[l_1^t,\dots ,l_{n-1}^t,l_{n+1}^t,\dots ,l_N^t]$ denotes the energy consumption vector of all prosumers, excluding prosumer n, and $\beta =1\left[\frac{\$}{{\mathrm{kWh}}^{3/2}}\right]$ is used as a constant unit-mapping parameter. The function $f(l_n^t,{\mathbf{l}}_{-n}^t)$ is the prosumer’s pure utility from consuming the purchased energy, and it is a strictly increasing and concave function with respect to the prosumer’s energy consumption $l_n^t$, e.g., $f(l_n^t,{\mathbf{l}}_{-n}^t)=\frac{\sqrt{l_n^t}}{\sum _{\forall n\in \mathcal{N}{n}^{\prime }\ne n}{l}_{n^{\prime }}^t}$. The physical meaning of the prosumer’s pure utility function is that it increases as more energy is purchased by the prosumer to cover its nonshiftable energy demand, and the curve is also concave as the prosumer reaches its maximum energy demand $d_n^t-g_n^t-b_n^{t-1}$ at timeslot t. Also, the prosumer’s pure utility from consuming energy is decreasing with respect to the total energy consumption of the rest of the prosumers in the examined smart grid system. This formulation is reasonable given that if the overall energy consumption in the smart grid system increases, then the utility company will eventually increase the energy price, which will decrease the prosumer’s utility. The function $g\left(l_n^t\right)$ captures the discount dissatisfaction, and it is a strictly decreasing and concave function with respect to the prosumer’s energy consumption $l_n^t$, e.g., $g\left(l_n^t\right)=-{l_n^t}^2+a$, $a>0$. The physical meaning of the discount dissatisfaction function captures the phenomenon of having the communities reward $r_m$ shared in a larger portion among the prosumers, who will finally enjoy a smaller percentage of the allocated reward if the prosumers in the same community increase their energy consumption (Equation (3)). The function $c\left(l_n^t\right)$ captures prosumer’s cost to purchase energy at a price $c\left[\frac{\$}{KWh}\right]$, e.g., $c\left(l_n^t\right)=c·l_n^t$. The last term of Equation (14) quantifies the prosumer’s received reward from participating in community m, as determined in Section 4.

The goal of each prosumer is to maximize its experienced utility by purchasing an optimal amount of energy, while considering its energy demand characteristics. Thus, the corresponding optimization problem is formulated as follows:

$$\begin{array}{c}\hfill \underset{l_n^t}{max}{U}_n(l_n^t,{\mathbf{l}}_{-n}^t)\end{array}$$

(15a)

$$\begin{array}{c}\hfill s.t.0\le l_n^t\le d_n^t-g_n^t-b_n^{t-1}\end{array}$$

(15b)

Given the decentralized nature of the smart grid system and respecting the prosumers’ privacy concerns, there is no centralized entity in the system to address the optimization problems (15a) and (15b) and determine prosumers’ optimal energy consumption ${\mathbf{l}}^{t*}=[l_1^{t*},\dots ,l_n^{t*},\dots ,l_N^{t*}]$ at each timeslot t. Thus, a reinforcement-learning-based mechanism is a natural choice in order to determine in an autonomous and distributed manner prosumers’ optimal energy consumption ${\mathbf{l}}^{t*}$ at each timeslot t. The theory of log-linear reinforcement learning is adopted by exploiting the B-logit and the Max-logit algorithms. Both of them enable prosumers to select the optimal strategy $l_n^{t*},\forall n\in \mathcal{N}$ that maximizes their utility (Equation (14)) by autonomously performing the exploration and exploiting processes. The benefit of the Max-logit algorithm is that it can determine the Pareto-optimal solution ${\mathbf{l}}^{t*}$, if it exists. The probability update rules of selecting a strategy $l_n^t$ for the Max-logit and the B-logit algorithms are presented in Equations (16a), (16b), (17a) and (17b), respectively, while $\beta \in {\mathbb{R}}^{+}$ denotes the learning parameter, and i denotes the reinforcement learning algorithm’s iteration until it converges to ${\mathbf{l}}^{t*}$.

$$\begin{array}{c}\hfill P(l_n^t{|}_i=l_n^t{|}_{i-1})=\frac{e^{\beta U_n\left(l_n^t{|}_{i-1}\right)}}{max\{e^{\beta U_n\left(l_n^t{|}_{i-1}\right)},e^{\beta U_n\left(l_n^{t^{\prime }}{|}_i\right)}\}}\end{array}$$

(16a)

$$\begin{array}{c}\hfill P(l_n^t{|}_i=l_n^{t^{\prime }}{|}_i)=\frac{e^{\beta U_n\left(l_n^{t^{\prime }}{|}_i\right)}}{max\{e^{\beta U_n\left(l_n^t{|}_{i-1}\right)},e^{\beta U_n\left(l_n^{t^{\prime }}{|}_i\right)}\}}\end{array}$$

(16b)

$$\begin{array}{c}\hfill P(l_n^t{|}_i=l_n^t{|}_{i-1})=\frac{e^{\beta U_n\left(l_n^t{|}_{i-1}\right)}}{e^{\beta U_n\left(l_n^t{|}_{i-1}\right)}+e^{\beta U_n\left(l_n^{t^{\prime }}{|}_i\right)}}\end{array}$$

(17a)

$$\begin{array}{c}\hfill P(l_n^t{|}_i=l_n^{t^{\prime }}{|}_i)=\frac{e^{\beta U_n\left(l_n^{t^{\prime }}{|}_i\right)}}{e^{\beta U_n\left(l_n^t{|}_{i-1}\right)}+e^{\beta U_n\left(l_n^{t^{\prime }}{|}_i\right)}}\end{array}$$

(17b)

The physical meaning of the above probability update rules of selecting a strategy $l_n^t$ is that prosumers select with equal probability an alternative strategy $l_n^{t^{\prime }}{|}_i$ during the exploration phase and receive a corresponding payoff $U_n\left(l_n^{t^{\prime }}{|}_i\right)$. Then, during the exploitation phase, they learn through the probabilistic rules (16a) and (16b) for the Max-logit and (17a) and (17b) for the B-logit, which is the best strategy $l_n^t$ to probabilistically select in the next iteration of the reinforcement learning algorithm. The Max-logit and B-logit algorithms both converge to the optimal solution ${\mathbf{l}}^{t*}$. Detailed results are presented in Section 6.

The coalitional DRM algorithm that determines both the Nash-stable and individual-stable prosumers’ partition ${\Pi }^{*}$ to communities following the principles of hedonic games and the prosumers’ optimal energy consumption ${\mathbf{l}}^{t*}$ based on the proposed reinforcement learning mechanism is presented in Algorithm 1. An overview of the overall proposed model is presented in Figure 1.

Algorithm 1 Coalitional DRM Algorithm

1: Input: $\mathcal{N},\mathcal{M},{\mathbf{l}}^t,t$   2: Output: ${\Pi }^{*}=\{{\mathcal{N}}_1^{t*},\dots ,{\mathcal{N}}_m^{t*},\dots ,{\mathcal{N}}_M^{t*}\}$, ${\mathbf{l}}^{t*}$   3: Initialization: ${\Pi }^{*}=\varnothing$, create an initial partition $\Pi$ by randomly allocating prosumers N to communities M. ${\delta }_n=1$ if prosumer n switches communities, and ${\delta }_n=0$ if prosumer n does not switch communities. $i=0$, $Convergence=0$, ${\mathbf{l}}^t{|}_{i=0}$.   4: while  $\mathit{Convergence}==0$do   5:     while $\sum _{\forall n\in \mathcal{N}}{\delta }_n\ne 0$ do   6:        for $n=1$ to N do   7:           Determine ${\mathcal{U}}_n^{{\mathcal{N}}_m^t}$;   8:           for each community ${\mathcal{N}}_{m^{{}^{\prime }}}^t$$\in \Pi$, $m\ne m^{\prime }$ do   9:              Prosumer n joins ${\mathcal{N}}_{m^{\prime }}^t$, ${\mathcal{N}}_{m^{\prime }}^t$$\leftarrow {\mathcal{N}}_{m^{\prime }}^t\cup \left\{n\right\}$ 10:              Update $V\left({\mathcal{N}}_{m^{\prime }}^t\right)$ and determine ${\mathcal{U}}_n^{{\mathcal{N}}_{m^{\prime }}^t}$; 11:              if ${\mathcal{U}}_n^{{\mathcal{N}}_{m^{\prime }}^t}$$>{\mathcal{U}}_n^{{\mathcal{N}}_m^t}$ and ${\mathcal{N}}_{m^{\prime }}^t$$\notin h\left(n\right)$ then 12:                  Prosumer n switches from ${\mathcal{N}}_m^t,{\mathcal{N}}_m^t\leftarrow {\mathcal{N}}_m^t\setminus \left\{n\right\}$ and joins ${\mathcal{N}}_{m^{\prime }}^t$, ${\delta }_n=1$; 13:              else 14:                  Prosumer n does not switch to ${\mathcal{N}}_{m^{\prime }}^t$, ${\mathcal{N}}_{m^{\prime }}^t\leftarrow {\mathcal{N}}_{m^{\prime }}^t\setminus \left\{n\right\}$ and remains at ${\mathcal{N}}_m^t$, ${\delta }_n=0$; 15:              end if 16:            end for 17:        end for 18:        Update ${\Pi }^{*}=\Pi$, and determine ${\mathcal{U}}_n^{{\mathcal{N}}_m^t}$, $\forall n\in \mathcal{N}$; 19:     end while 20:     for $n=1$N do 21:         Prosumer n selects $l_n^{t^{\prime }}{|}_i$ with equal probability among all the possible energy consumption strategies and the rest of the prosumers keep their previous consumption, i.e., ${\mathbf{l}}_{-n}^t{{|}_i={\mathbf{l}}_{-n}^t|}_{i-1}$. 22:         Prosumer n receives a utility $U_n(l_n^{t^{\prime }}{|}_i,{\mathbf{l}}_{-n}^t{|}_i)$ and updates $l_n^t{|}_i$ based on Equations (16a) and (16b), (Equations (17a) and (17b)). 23:     end for 24:     if $|\frac{{\displaystyle \sum _{i=0}^T}{\displaystyle \sum _{\forall n\in \mathcal{N}}}{U}_n{|}_i}{T}-{\displaystyle \sum _{\forall n\in \mathcal{N}}}{U}_n{|}_i|\le ϵ$, $ϵ$ small positive number then 25:        $Convergence=1$ 26:     end if 27:     $i=i+1$ 28: end return 29: return ${\Pi }^{*}$, ${\mathbf{l}}^{t*}$

6. Numerical Evaluation

In this section, a detailed numerical evaluation of the proposed coalitional DRM model in community energy management systems in presented. Specifically, the pure operation and performance of the proposed model is presented in Section 6.1, while the impact of the utility company’s provided reward and the captivation parameter on the communities’ creation process is analyzed in Section 6.2. A scalability analysis is provided in Section 6.3 to demonstrate the efficiency and robustness of the proposed model, while a detailed comparative evaluation to the state of the art is performed in Section 6.4. The following simulation parameters have been adopted for the numerical evaluation following a real data analysis in the southwest area of the USA [32]: $M=5$, $N=20$, the non-self-sufficient prosumer’s energy demand vector, i.e., $d_n^t-g_n^t-b_n^{t-1}$ in an indicative timeslot t is $[1.00,1.03,1.05,1.08,1.11,1.13,1.16,1.18,1.21,1.24,1.26,1.29,1.32,1.34,1.37,1.39,1.42,1.45,1.47,1.5]$ kWh, $w_1=0.7,w_2=0.3,c=0.1495\frac{\$}{\mathrm{KWh}},{\mathbf{r}}^t=[1,2,3,4,5]·{10}^3$ [¢], and ${\mathbf{c}}^t=[3.5,4.0,4.5,5.0,5.5]·{10}^2$. In the rest of the numerical evaluation, prosumers’ payoff and utility is presented in [¢]. The proposed framework’s evaluation was conducted using a Dell XPS desktop with 11th Gen Intel core i9-11900 K 5.3 GHz processor, and 64 GB available RAM.

6.1. Pure Operation and Performance

In this section, the pure performance and operation of the proposed coalitional DRM model are presented. Figure 2a,b demonstrates prosumers’ initial and final payoff (Equation (11)) and utility (Equation (14)), respectively, based on the B-logit and the Max-logit DRM algorithms as a function of the prosumer’s ID. It is noted that a higher prosumer ID corresponds to a higher energy demand from the prosumer’s side. Figure 2c presents prosumers’ initial and final utility (Equation (14)) under the B-logit and Max-logit algorithms as a function of the prosumers’ optimal energy consumption.

The results reveal that prosumers intelligently select the communities that they belong to by following the principles of hedonic games (Section 4), and they ultimately achieve a higher payoff (Equation (11)) under both the B-logit and Max-logit algorithms (Figure 2a). Also, it is observed that the resulting prosumers’ payoff under the hedonic games is on average very similar, either following the B-logit or the Max-logit algorithms. Furthermore, prosumers with higher energy consumption achieve a higher payoff, as they better exploit the provided rewards by the utility company (Equation (3)). Focusing on the RL-based DRM model (Section 5) that determines prosumers’ optimal energy consumption, the results show that the Max-logit algorithm determines the Pareto-optimal solution better than the B-logit algorithm, thus resulting in a higher prosumer utility (Equation (14)). Also, both RL-based algorithms converge to an optimal solution, thus achieving a higher prosumer utility compared with the initial one (Figure 2b). Furthermore, the higher the prosumers’ energy demand (i.e., higher prosumer’s ID), the higher the corresponding achieved utility, as the prosumers cover a larger portion and absolute amount of their energy needs. By taking a closer look at the findings of Figure 2c, we observe that the Max-logit DRM algorithm achieves on average a higher utility compared with the B-logit algorithm, as it converges to a Pareto-optimal solution. Also, it is highlighted that higher prosumer utility is achieved by the Max-logit DRM algorithm with a corresponding lower energy consumption, thus supporting prosumers’ energy needs in a manner more beneficial to them (Figure 2c).

Figure 3a–c demonstrate the average prosumers’ utility (Equation (14)), their average energy consumption, and the execution time of the RL-based DRM model under the B-logit and Max-logit algorithms, respectively. The box plots capture the 0th percentile, i.e., the lowest data point, the 100th percentile, i.e., the highest data point, the 25th percentile, i.e., the lowest part of the box, the 75th percentile, i.e., the highest part of the box, the median value, i.e., the horizontal line inside the box, and the mean value, i.e., the circle point in the box. The results show that the Max-logit DRM algorithm results in higher achieved prosumer utility (Figure 3a) with a relatively lower average consumption for prosumers (Figure 3b) and corresponding shorter execution time (Figure 3c). Thus, the Max-logit DRM algorithm outperforms the B-logit algorithm both in terms of computational efficiency and achieved results for prosumers.

6.2. Impact of Reward and Captivation Parameter

In this section, an analysis of quantifying the impact of the utility company’s provided rewards and the captivation parameter on the communities formation process is performed. Figure 4 presents the average size of the five examined communities in terms of prosumers as a function of the utility company’s allocated rewards per community and the captivation parameter for the B-logit and Max-logit DRM algorithms. The results show that as the allocated rewards and the captivation parameter increase, more prosumers are attracted to join the community, as they have the potential to receive a higher reward (in the form of energy price discount) and have the potential to exchange energy with many more prosumers in the same community. Also, the results clearly show the prosumers’ sensitivity to the provided rewards and captivation parameter, as a small increase in the latter ones drives more prosumers to join the corresponding community. Finally, we observe that the choice of the RL-based DRM model that determines prosumers’ optimal energy consumption does not really affect prosumers’ formation process, given that their resulting energy consumption has small differences that cannot drive prosumers to join a different community.

6.3. Scalability Analysis

In this section, a scalability analysis is conducted to quantify the efficiency and robustness of the proposed coalitional DRM model. We consider a large smart grid system consisting of up to 100 prosumers who purchase energy from a utility company. All the prosumers are considered as non-self-sufficient, and they purchase energy, while their energy needs are equally distributed in the interval $l_n^t\in [1,1.5]$ KWh. Figure 5a illustrates the prosumers’ average switching operations, their average payoff, and the hedonic game’s execution time as the number of prosumers increases. Figure 5b presents the average prosumers’ utility and the Max-logit DRM algorithm’s execution time as a function of the number of prosumers. Also, Figure 5c shows the overall execution time of the proposed coalitional DRM model adopting the Max-logit or B-logit RL-based algorithms for an increasing number of prosumers, as well as the corresponding average values over the maximum number of prosumers (N = 100).

The results show that as the number of prosumers increases, the execution time of the hedonic game for the communities formation process (Figure 5a), the Max-logit DRM algorithm (Figure 5b), and the overall execution time of the proposed coalitional DRM model (Figure 5c) increase. However, it is highlighted that the execution time of the coalitional DRM model remains at low realistic values, i.e., a few seconds, even for a large population of prosumers, i.e., N = 100. Also, the results confirm that the Max-logit algorithm converges faster to an optimal solution than the B-logit algorithm (Figure 5c). Focusing on prosumers’ achieved payoff (Equation (11)) and utility (Equation (14)), the results show that both of them decrease as the number of prosumers increases. Specifically, the prosumers’ payoff decreasing trend stems from the utility company’s rewards sharing among a larger number of prosumers (Figure 5a). It is noted that the reward is considered constant per community in order to capture the impact of the increasing number of prosumers to the average number of switch operations, the average prosumers’ payoff and utility, and the execution time of the proposed coalitional demand response management model. The reward that is assigned to each community is ${\mathbf{r}}^t=[1,2,3,4,5]\times {10}^3$ [¢]. Also, the prosumers’ utility decreasing trend stems from the increase in the energy demand from the utility company due to the increasing number of prosumers, which results in sharing the utility company’s energy supply among a larger number of prosumers (Figure 5b).

6.4. Comparative Evaluation

In this section, a comparative evaluation to the state of the art is performed, both regarding the community formation process and the RL-based DRM framework. Figure 6 presents the prosumers’ achieved average payoff (Equation (11)) for four comparative scenarios: (i) proposed coalitional DRM model, (ii) random community formation process, and (iii)–(iv) reward and captivation parameter-based community selection, where the prosumers select the community with the highest reward and captivation parameter, respectively. It is highlighted that the captivation parameter and the allocated reward per community are the same per community under the reward-based and captivation-based community formation processes, respectively, to capture the impact of each individual parameter. The results reveal that the proposed coalitional DRM model achieves the highest average prosumers’ payoff compared with all the comparative scenarios for an increasing number of prosumers (Figure 6). In contrast, the reward-based community formation scenario results in the worst payoff for the prosumers, as all of them are trying to share the provided reward in one community. The captivation-based community formation process presents better prosumer payoff compared with the reward-based scenario, as the captivation parameter has the same impact to all prosumers and it is not shared among them as the utility company’s provided reward is. The random-based community formation scenario presents intermediate results between the best- and worse-case scenarios, as expected. Specifically, the proposed coalitional DRM model achieves a 7% increase in the average prosumers’ payoff compared with the random coalitional formation scenario for the considered Monte Carlo scalability analysis, which, however, results in substantial improved payoff in terms of profit, i.e., USD, for the prosumers.

Figure 7 demonstrates the prosumers’ energy consumption as a function of the prosumers’ ID for two comparative scenarios: (i) the proposed coalitional DRM model and (ii) the no-discount dissatisfaction model, where, in the latter one, the prosumers are not sensitive to sharing the communities’ rewards with other prosumers (as captured by the $g\left(l_n^t\right)$ function in Equation (14)). It is noted that in both comparative scenarios, the rewards are shared among the prosumers of the same community following the principles of proportional fairness, as presented in Equation (3). The results reveal that the proposed coalitional DRM model results in lower prosumers’ energy consumption compared with the no-discount dissatisfaction model, as it sophisticatedly captures the impact of the utility company’s allocated reward per community in the prosumers’ decision-making process.

7. Conclusions

In this paper, a novel coalitional demand response management model in community energy management systems is introduced based on the principles of game theory and reinforcement learning. Specifically, the principles of hedonic game and log-linear reinforcement learning are adopted to enable prosumers to get organized in communities and determine their optimal energy consumption, respectively. A community energy management system is introduced consisting of prosumers who intelligently select to participate in a community accounting for the partial available information of their own and other prosumers’ energy generation and consumption characteristics, as well as the rewards provided by the utility company to each community in terms of energy price discount. The prosumers’ community formation process is based on the theory of hedonic games, and the existence of a Nash-stable and individual-stable partition is proven. Then, two log-linear reinforcement learning algorithms, named B-logit and Max-logit, are introduced to enable each prosumer to determine their optimal energy consumption in a distributed and autonomous manner and compared among each other in terms of their accuracy and computational complexity. A detailed evaluation of the proposed coalitional DRM model is performed based on real data collected from the southwest area of the USA.

Part of our current and future work is the extension of this model considering prosumers’ risk-aware decision-making characteristics considering utility companies’ capacity to serve prosumers’ energy demands. Towards this direction, a multivariable prospect-theoretic model will be developed that quantifies prosumers’ risk-aware behavior in scenarios of limited energy supply that can drive the system to brownout or even to blackout scenarios [33]. Also, part of our current and future work is the extension of the proposed coalitional demand response management model towards capturing the energy exchange among prosumers who can buy and steal energy from their peers.