Multi-Agent Reinforcement Learning for Smart Community Energy Management

Abstract

This paper investigates a Local Strategy-Driven Multi-Agent Deep Deterministic Policy Gradient (LSD-MADDPG) method for demand-side energy management systems (EMS) in smart communities. LSD-MADDPG modifies the conventional MADDPG framework by limiting data sharing during centralized training to only discretized strategic information. During execution, it relies solely on local information, eliminating post-training data exchange. This approach addresses critical challenges commonly faced by EMS solutions serving dynamic, increasing-scale communities, such as communication delays, single-point failures, scalability, and nonstationary environments. By leveraging and sharing only strategic information among agents, LSD-MADDPG optimizes decision-making while enhancing training efficiency and safeguarding data privacy—a critical concern in the community EMS. The proposed LSD-MADDPG has proven to be capable of reducing energy costs and flattening the community demand curve by coordinating indoor temperature control and electric vehicle charging schedules across multiple buildings. Comparative case studies reveal that LSD-MADDPG excels in both cooperative and competitive settings by ensuring fair alignment between individual buildings’ energy management actions and community-wide goals, highlighting its potential for advancing future smart community energy management.

1. Introduction

The global push for adopting distributed and renewable energy resources alongside increased electrification and the gradual retirement of dispatchable fossil-fueled power plants is introducing new operational challenges to existing electric power grids [1]. As a result, there is a growing need for a new paradigm of energy management that shifts from supply-side to demand-side control [2]. Under this new paradigm, it becomes critical to comprehensively understand the evolving energy consumption patterns and controllability of diverse electric grid loads and their associated energy assets, particularly heating and cooling systems, which account for the largest portion of residential electricity consumption [3]. Failing to leverage energy consumption flexibility via strategically adapting consumer behaviors can significantly strain the power grid, potentially leading to unexpected generation capacity shortages during this energy transition. A notable example occurred in the summer of 2022, when a heatwave in California brought the state’s power grid to the brink of rolling blackouts, impacting millions of residents. Authorities responded by urging residents to refrain from charging their electric vehicles (EVs) to conserve energy, challenging the feasibility of California’s ambitious goal to eliminate all gas-powered vehicles by 2035 [4,5]. On the other hand, implementing effective demand-side energy management enabled by smart grid technologies can optimize energy consumption, assisting in a stable and reliable power supply while improving consumer economic and comfort outcomes [6,7].

Conventional mathematical methods have been thoroughly studied for effective demand-side energy management systems (EMSs), including rule-based methods often derived from expert knowledge of well-studied systems, which provide a static, definite, and easy-to-implement solution but lack the flexibility and adaptability required for more complex dynamic environments [8]. Optimization-based techniques, such as linear programming [9,10], mixed-integer linear programming [11,12,13,14,15,16], and quadratic programming [17,18,19], utilize mathematical formulations to model and manage objective systems over a fixed time horizon, but can be computationally intensive, hard to scale, and may struggle to adapt to changing conditions. Heuristic algorithms like Particle Swarm Optimization [20,21,22] and Genetic Algorithms [23,24,25] offer alternative approaches for non-linear optimization, but despite handling intricate systems, they often demand significant computational resources and are complex to implement and generalize. Model Predictive Control (MPC) complements these methods by incorporating system dynamics into the optimization process, predicting future states, and making real-time decisions over a moving horizon [26,27]. However, implementing MPC at the community level, which requires coordination among stakeholders, poses challenges due to the evolving conditions and the high costs of modeling large-scale energy assets [28]. Additionally, the performances of all the above-mentioned model-based methods heavily rely on accurate system modeling, meaning that model inaccuracies and constrained data sharing can lead to suboptimal decisions, thereby highlighting the limitations of these methods in demand-side EMSs that require coordination among numerous independent energy consumers.

Unlike model-based methods, Reinforcement Learning (RL) addresses issues including adaptation and generalization to dynamic environments and reduces reliance on precise system models, making it well-suited to explore its potential for demand-side EMS [29]. By leveraging machine learning (ML) capabilities, RL-algorithm-equipped agents (representing heterogenous energy consumers) incrementally learn and understand complex systems through interactions with the environment, which reflect the energy system’s dynamics and external conditions, allowing them to adapt their learned policies to evolving system conditions. RL agents can be implemented via either centralized methods, where a single entity coordinates the learning and decision-making for all agents, or decentralized methods, where multiple agents learn and make decisions independently.

Several centralized methods, including Q-learning [30,31,32], Deep Q-Network (DQN) [33,34], Double Deep Q-Network [35], Deep Deterministic Policy Gradient (DDPG) [36,37,38,39,40], and Proximal Policy Optimization [41], have been explored for both supply-side and demand-side EMSs. A centralized EMS provides a shared global perspective, enabling coordinated decision-making and comprehensive system understanding for efficient energy management. However, a centralized controller managing an entire community’s energy consumption has widely recognized drawbacks: (a) It imposes significant communication overhead due to large datasets shared among many energy consumers and the controller [42]. (b) It faces scalability issues due to limited processing power and increasing computational complexity as the system grows. (c) It represents a single point of failure, making the system vulnerable to outages and reliability issues. (d) Centralized data storage raises concerns about data privacy and security, making the system a target for cyber-attacks.

In contrast, decentralized methods rely primarily on local information for decision-making, with limited or no communication between multiple RL agents. This setting can improve problem-solving efficiency through parallel computations and enhance reliability by employing redundant agents [43,44]. Multi-agent reinforcement learning (MARL) approaches are typically categorized into three frameworks: Independent Learning, Centralized Critic, and Value Decomposition. Figure 1 illustrates the three categories and the information that is shared or exchanged between the Q-value function that evaluates actions and an actor–critic that involves: the actor (which makes decisions) and the critic (which evaluates them). Independent Learning in Figure 1a involves the Q-function and actor–critic to only have access to their individual observations, updating their policies without any data exchange, which simplifies the learning process and may improve energy management performance, but only if agents’ actions are non-conflicting [45]. However, this approach does not address two significant technical challenges in MARL: (i) interactions between agents give rise to a non-stationary environment, which inherently complicates learning convergence; and (ii) accurately attributing credit for an action to the responsible agent is difficult under the simultaneous actions of other agents [46]. As a result, it often struggles with achieving objectives that require coordination and typically underperforms compared to centralized strategies. Therefore, the main objective of many decentralized methods is to help each agent build a belief system that aligns its decision-making with its teammates and thus achieves a consensus on the overall system management goals [47].

Centralized Training with Decentralized Execution (CTDE) addresses these challenges by allowing agents to use shared information during training while relying solely on local observations during execution, thus mitigating issues of non-stationarity and credit assignment. Centralized Critic, a CTDE framework shown in Figure 1b, requires access to all agents’ observations and actions for optimization after selecting actions, so the critic can accurately assess the impact of each agent’s actions. The decentralized actors (policies) only access local observations. This approach is effective in both competitive and cooperative environments because it can model adversarial interactions where agents compete, as well as collaborative ones. Another CTDE, Value Decomposition (VD), as shown in Figure 1c, differs from Centralized Critic in that agents are required to share only their predicted Q-values or critic values. This framework decomposes the global value function into individual agent-specific value components, enhancing coordination and aligning individual agents’ objectives with overall system goals. This approach is particularly effective in cooperative multi-agent systems. All three approaches only rely on their actors’ learned policies and local observations to make decisions in the execution phase, while the critic and Q-function are removed, as depicted in Figure 1d.

Various MARL algorithms have demonstrated improvements in energy trading, microgrid operation, building energy management, and other applications. Notable algorithms such as MA-DQN [48], MADDPG [49,50], MAAC [51,52], MAA2C [53], MAPPO [54,55,56], MATD3 [57], and MA-Q-learning [45,58,59,60] have been implemented in MARL, enabling RL agents to independently model, monitor, and control their respective environments. Such approaches could naturally solve a complex system’s decentralized control problems, such as the community EMSs we would like to study.

This paper explores a new decentralized MARL-based smart community EMS solution, aiming to address key challenges related to data overhead during execution, single-point failure risks, a nonstationary environment, solution scalability, and the interplay between cooperation and competition among individual community energy consumers. Specifically, a Local Strategy-Driven MADDPG (LSD-MADDPG) is proposed to manage the energy consumption of individual smart buildings equipped with different energy assets in a community. The EMS objective is to flatten the smart community electricity demand curve in a coordinated fashion while reducing the energy costs of individual buildings, all within their respective levels of user satisfaction.

In the MADDPG framework, each RL agent is equipped with an actor responsible for executing actions and a critic that evaluates those actions to optimize the policy learning. When multiple decentralized RL agents are introduced for community energy management, individual agents’ access to the information of the entire community’s energy system could be limited due to physical and/or privacy constraints. Unlike conventional Centralized Critic methods, in the proposed LSD-MADDPG, the critic of an agent does not have access to other agents’ observations; instead, RL agents only share their “strategies”, which are specifically designed quantitative indicators representing each agent’s prioritized intent for energy consumption—i.e., the scheduling of HVAC usage and EV charging—based on the current state of its resources or demands. The critic of each agent leverages these shared “strategies” from others to improve its policy evaluation while ensuring data privacy, whereas its actor’s execution remains locally decentralized. For benchmarking purposes, the proposed demand-side community EMS, i.e., LSD-MADDPG, is tested and compared against the common practice of a naïve controller (NC), a centralized single-agent controller, a fully decentralized MADDPG controller with access to only local observations, and a conventional CTDE MADDPG with a centralized critic.

The major contributions of this work are summarized as follows:

1. A new LSD-MADDPG is proposed for a smart community EMS, which modifies the conventional MADDPG framework to only share “strategies”, instead of the global state for RL agent critic training, demonstrating enhanced control performance. Via an innovatively designed Markov game, LSD-MADDPG achieves superior solution scalability while offering more equitable reward distribution, emphasizing fairness in competition and coordination. Additionally, it maintains data privacy while achieving competitive results against conventional MADDPG in terms of reward, energy cost, EV charging, and building comfort satisfaction, as well as community peak demand reduction.

2. A simulation environment for RL implementations is rigorously modeled, incorporating dedicated HVAC systems and EV charging stations to facilitate the development and testing of advanced control strategies in different community EMS scenarios.

This paper is organized as follows: Section 2 presents a detailed modeling of the smart community EMS problem. Section 3 discusses the Markov game formulation, encompassing the state-action space and reward design for the smart community EMS. Section 4 introduces the proposed MARL framework. Section 5 presents the case studies on different EMS controllers, as well as a comparison of their training and evaluation results, highlighting the performance of the proposed LSD-MADDPG. Finally, the paper concludes with Section 6, summarizing the key findings and outlining future work.

2. Smart Community EMS Modeling

The objective of the smart community EMS is to minimize ${\mathrm{C}}_{total}$, including the community’s building energy costs, penalties for EV charging deviations from the target state of charge (SoC), and the community’s peak demand costs, thereby encouraging energy demand shifting. This optimization problem can be expressed as (1):

$$\mathrm{M}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e} {\mathrm{C}}_{total}={\sum }_{n=1}^N{\sum }_{t=1}^T{\lambda }_1(E_{n,t}^{ev,in}+E_{n,t}^{hc})\times P_t)+{\lambda }_2{\sum }_{n=1}^N{{(SoC}_n^{tgt}-{SoC}_{n,final})}^2+{\lambda }_3{P}_{pk}\underset{t\in \{1,\dots ,T\}}{\max }({\sum }_{n=1}^N\left(E_{n,t}^{ev,in}+E_{n,t}^{hc}\right))$$

(1)

where $E_{n,t}^{ev,in}$ and $E_{n,t}^{hc}$ denote the average energy consumption of EV charging and the HVAC system, respectively, for building n during time period t. $P_t$ represents the electricity price during time t and the peak demand price is $P_{pk}$. The term λ₁ is a weighting factor that emphasizes the importance of minimizing energy cost, while λ₂ is a weighting factor penalizing deviations from the EVs’ charging targets ${SoC}_n^{tgt}$. ${SoC}_{n,final}$ is defined as the final SoC of the EV in building n by the end of its charging period. Lastly, λ₃ is a weighting factor for reducing community daily peak demand charge.

2.1. Building Electric Vehicle Charging Modeling

The building EV charging model focuses on optimally scheduling energy usage to ensure EVs are adequately charged before departure. Charging occurs only during EV charging periods ${CP}_n$, defined as the time intervals when the EV is available for charging (2), and within its charging capacity limits (3). $E_{n,t}^{ev,limit}$ represents the constrained energy available for charging, while $E_n^{ev,min}$ and$E_n^{ev,max}$ are the minimum and maximum charging capacities of the EV charger.

$$E_{n,t}^{ev,limit}=0, if t\notin {CP}_n$$

(2)

$${E_n^{ev,min}\le E}_{n,t}^{ev,limit}\le E_n^{ev,max}$$

(3)

The average amount of energy consumed by the charging station is further constrained by the SoC of EVs plugged in, as modeled in (4). This limit,$E_{n,t}^{ev,in},$ is derived from the Tesla Model 3’s charging behavior, as it takes roughly 2 h to charge from 0% to 80% and an additional 2 h to charge from 80% to 100% using a 220V charger [61].

$$\left\{\begin{array}{c}{E}_{n,t}^{ev,in}=E_{n,t}^{ev,limit}\times 0.25, if {SoC}_{n,t}>80\%\\ E_{n,t}^{ev,in}=E_{n,t}^{ev,limit}\times 1.0, if {SoC}_{n,t}<80\%\end{array}\right.$$

(4)

The energy losses during EV charging, primarily due to the onboard inverter’s efficiency, and the active cooling of the EV battery when charging at higher amps, for a longer time, or in hot climates, are modeled to fit the charging data published by Tesla Motors Club Forum [61]. The total energy charged to the EV battery, accounting for such charging inefficiency, is calculated by (5), with SoC updated by (6) and constrained by (7), where $B_{max,n}$ is the EV battery capacity.

$$E_{n,t}^{ev}=-0.40478\times {\left({\displaystyle \frac{11\times E_{n,t}^{ev,in}}{E_n^{ev,max}}}\right)}^2+6.2306\times {\displaystyle \frac{11{\times E}_{n,t}^{ev,in}}{E_n^{ev,max}}}+66.8633$$

(5)

$${SoC}_{n,t}={SoC}_{n,t-1}+{\displaystyle \frac{E_{n,t}^{ev}}{B_{max,n}}}\times 100\%$$

(6)

$${0\le SoC}_{n,t}\le 100$$

(7)

2.2. Building HVAC System Modeling

The building HVAC system modeling ensures temperature comfort levels, providing heating and cooling to the building occupants, every minute m, but the temperature settings can only be adjusted every hour. This modeling approach maintains a consistent optimal hourly scheduling for both building EV charging and HVAC settings while accurately reflecting real-world thermodynamic conditions, where the HVAC system does not run continuously for an entire hour nor ramp up or down every second. Building resistance $R_n$ is used to quantify the building’s ability to resist the flow of heat between the inside and outside environments, considering windows, insulations, and building size [62]. A higher $R_n$ value indicates better insulation and lower heat transfer.

The building temperature T_n,m is updated via (8), accounting for thermal changes from the outdoor temperature $T_{o,t}$ and the heating gain H_g,n,m or cooling loss C_g,n,m from the building’s HVAC. In (8), ${\psi }_n$ is the mass of air and $c$ is the specific heat capacity of air. Building indoor temperature is strictly constrained by the occupants’ comfort levels (9).

$$T_{n,m+1} = T_{n,m} {\displaystyle \frac{1}{{\psi }_n\times c}}\times (H_{n,g,m} - C_{n,g,m} {\displaystyle \frac{T_{n,m}-T_{o,t}}{R_n}})$$

(8)

$${19.5 °\mathrm{C}<T}_{n,m}<23.9 °\mathrm{C}$$

(9)

The control logic for heating and cooling is enabled by setting the thermostat temperature $T_{F,n,t}$ between the minimum $\underset{\_}{T}$and maximum $\overline{T}$ thermostat setpoints (10)–(11). In this context, $b_{n,m}^c$ and $b_{n,m}^h$ are binary values indicating cooling and heating.

$${\underset{\_}{T}<T}_{F,n,t}<\overline{T}$$

(10)

$$\left\{\begin{array}{c}{b}_{n,m}^c=1 if T_{F,n,t} – T_{n,m}\ge 0.6 °\mathrm{C}\\ b_{n,m}^h=1 if T_{F,n,t} – T_{n,m}\le -0.6 °\mathrm{C}\\ b_{n,m}^c=0 and b_{n,m}^h=0 otherwise \end{array}\right.$$

(11)

The heating and cooling thermal energy inputs are calculated using (12) and (13), respectively. The HVAC’s heating output is a constant temperature T_h,n and its cooling output is T_c,n, both supplied at air flow rate M_n.

$$H_{g,n,m} = (M_n \times c \times (T_{h,n} - T_{n,m}) \times b_{n,m}^h)$$

(12)

$$C_{g,n,m} = (M_n \times c \times (T_{n,m} - T_{c,n}) \times b_{n,m}^c)$$

(13)

The efficiency of the HVAC system, i.e., the coefficient of performance (COP) $f_{hc,n,t}$, exhibits a reciprocal relationship with outdoor temperature, with 273.15 added to convert Celsius to Kelvin. The HVAC system’s COP and efficiency factor ${\eta }_n$ determine the amount of thermal energy that can be supplied from electrical energy (14).

$$f_{hc,n,t}=\left\{\begin{array}{c}{\eta }_n\times {\displaystyle \frac{T_{h,n}+273.15}{T_{h,n}-T_{o,t}}}, if b_{n,m}^h=1\\ {\eta }_n\times {\displaystyle \frac{T_{c,n}+273.15}{T_{o,t}-T_{c,n}}}, if b_{n,m}^c=1\end{array}\right.$$

(14)

The constraints on heating (15) and cooling (16) are further restricted by the maximum HVAC power consumption $E_n^{hc,max}$ and $f_{hc,n,t}$. The COP is constrained by the maximum COP, $f_{hc}^{max}$, as shown in (17).

$$0\le H_{g,n,m}\le E_n^{hc,max}\times f_{hc,n,t}$$

(15)

$$0\le C_{g,n,m}\le E_n^{hc,max}\times f_{hc,n,t}$$

(16)

$$1\le f_{hc,n,t}\le f_{hc}^{max}$$

(17)

The per minute (18) and per hour (19) electrical energy consumptions of a building HVAC system are then calculated as follows:

$$E_{n,m}^{hc}={\displaystyle \frac{H_{g,n,m}+C_{g,n,m}}{f_{hc,n,t}\times 3.6\times {10}^6}}$$

(18)

$$E_{n,t}^{hc}={\sum }_{m=1}^{60}{E}_{n,m}^{hc}$$

(19)

3. Markov Game Formulation of the Community EMS

A Markov Game could be designed to reformulate the community EMS problem as elaborated in Section 2 into a model-free RL control framework including N agents (i.e., EMS controllers), a joint action set {A_1:N}, a joint observation set {O_1:N} representing the local information that each agent can perceive from the global state S, and a joint reward set {R_1:N}, as well as a system global state transition function $\mathbb{P}$.

State Space Design: Each agent n receives a private observation $o_n^t$at time t, which is influenced by the current state of the system environment S → $o_n^t$, where state S is affected by the joint actions of all agents. The initial state S₀ is sampled from a probability distribution p(S) over the range [Smin, Smax]. The observation for agent n at t is defined as

$$o_n^t=[d_t, T_{o,t},P_t ,{T_{n,t},K}_t, E_{n,t}^{hc} , E_{n,t}^{ev},{cp}_{n,t}, {SoC}_{n,t}]$$

(20)

where $d_t$ represents the current date and time, including month, day, and hour. Since the charging period of an EV at the building charging station is assumed to be unknown to the agent, ${cp}_{n,t}$ is a binary value indicating the presence of an EV. $T_{n,t}$ is the average temperature of T_n,m within the hour. $K_t$ represents the total demand or “peak” for current time period t (hour in our study) of the community. It is not possible to predict the future precisely, so the daily peak cannot be optimized for a full 24-h period in advance. Instead, $K_t$ provides an hourly “peak” that can be reduced and shifted through learning from historical measurements from the community electricity distribution system.

Action Space Design: The action $a_n^t$ taken by each agent n is defined as an hourly decision on EV charging and HVAC setting at time t to influence the system environment, i.e., $a_n^t$ = [$a_{n,t}^{ev}$, $a_{n,t}^{hc}$]. Each agent n employs a deterministic policy π_n: $o_n^t$ → $a_n^t$, which maps its local observations $o_n^t$ to actions $a_n^t$. Actions are normalized to the interval [−1, 1] to ensure consistent scaling and facilitate the neural network learning process. The designed agent actions $a_{n,t}^{ev}$ and $a_{n,t}^{hc}$ are transformed into EMS control variables $E_{n,t}^{ev,in}$and $T_{F,n,t}$, as defined in (21)–(23), representing the energy supplied to EV charging and the HVAC thermostat setting, respectively.

$$E_{n,t}^{ev,a}={\displaystyle \frac{{(a}_{n,t}^{ev}+1)}{2}}\times E_n^{ev,max}$$

(21)

$$E_{n,t}^{ev,in}=\left\{\begin{array}{c} E_{n,t}^{ev,a} if E_{n,t}^{ev,a}\ge E_n^{ev,min}\\ 0 otherwise\end{array}\right.$$

(22)

$$T_{F,n,t}={\displaystyle \frac{a_{n,t}^{hc}\times (\overline{T}-\underset{\_}{T})}{2}}+{\displaystyle \frac{\overline{T}+\underset{\_}{T}}{2}}$$

(23)

The EV charging action is clipped and mapped to a value between the minimum and maximum charging capacities, naturally constraining power as per (3). Similarly, the HVAC action is mapped within the thermostat’s operational range, ensuring compliance with (10).

Reward Design: The reward function is used to guide an RL agent’s learning process by mapping the immediate reward for each action to the state-action-next state tuple (S, A, S’). In the community EMS, each agent’s objective is to find the optimal control policy (π) that maximizes its cumulative reward, i.e., contribution to the EMS objective (1). Thus, the agent-level reward, $r_n^t$, is designed to include the HVAC reward component $r_{n,t}^{hc}$, the EV charging reward component $r_{n,t}^{ev}$, the electricity cost reward component $r_{n,t}^p$, and peak demand cost reward component $r_{n,t}^k$, as shown in (30).

EV charging and SoC constraints (2) and (7) are soft-constrained, with penalties ${\varphi }_{n,t}^{ev}$ applied to the reward components when constraints are violated (24).

$${\varphi }_{n,t}^{ev}=\left\{\begin{array}{c}-1 if constraint \left(2\right) is violated\\ {\displaystyle \frac{{SoC}_{n,t}-95}{5}} if constraint \left(7\right) is violated\end{array}\right.$$

(24)

The reward for EV charging is based on the energy supplied $E_{n,t}^{ev,in}$to the vehicle, as shown in (25). If the EV is not present and the agent chooses not to charge, a small positive reward is given. This design provides feedback even during non-charging periods, reducing reward sparsity and ensuring appropriate rewards for inaction. Additionally, the agent is rewarded for charging during the designated charging period, independent of the time or EV SoC, encouraging completing charging within allowed time periods while penalizing charging outside, as described in (24). Decoupling the reward from specific time periods and SoC levels allows for flexible time-shifting, as each reward is independent of the charging status at any given time, with the electricity costs and peak demand cost being addressed in a later reward.

$$r_{n,t}^{ev}=\left\{\begin{array}{c} {\displaystyle \frac{E_{n,t}^{ev}}{E_n^{ev,max}}}-{\varphi }_{n,t}^{ev} if E_{n,t}^{ev,in}>0\\ {\displaystyle \frac{1-{cp}_{n,t}}{10}} if E_{n,t}^{ev,in}=0 \end{array}\right.$$

(25)

The HVAC reward component provides positive feedback when the temperature is maintained within the comfort range and uses a soft-constraint to penalize the agent for deviations from this range, encouraging efficient HVAC management.

$$r_{n,t}^{hc}=\left\{\begin{array}{c}-1 if constraint \left(11\right) is violated \\ 1 otherwise \end{array}\right.$$

(26)

The electricity cost reward (27) is designed to penalize the agent based on the cost of electric energy used for EV charging and HVAC operation, scaling them separately to prevent imbalances in different system capacities:

$$r_{n,t}^P=P_t\times ({\displaystyle \frac{E_{n,t}^{ev,in}}{E_n^{ev,max}}}+{\displaystyle \frac{E_{n,t}^{hc}}{E_n^{hc,max}}})$$

(27)

Peak demand calculated in (28) will be rewarded revolving around a predefined threshold D, which could be derived from historical data or based on distribution system operational experience. While D is set as a constant in our daily EMS model, it can be adjusted periodically to account for seasonal variations or system operation condition changes in peak demand. Each building agent is designed to be rewarded or penalized according to its contribution to reducing or increasing the peak demand, as expressed by (29). The reward design works by incentivizing agents to reduce peak demand by providing positive rewards when the total energy consumption $K_t$ is below D. If $K_t$ exceeds D, agents are penalized proportionally based on their contribution to the total demand, encouraging the reduction of energy usage during high demand periods.

$$K_t={\sum }_{n=1}^N{E}_{n,t}^{ev, in}+E_{n,t}^{hc}$$

(28)

$$r_{n,t}^k=\left\{\begin{array}{c}(1-{\displaystyle \frac{K_t}{D }})\times {\displaystyle \frac{E_{n,t}^{ev, in}+E_{n,t}^{hc}}{K_t}} if E_{n,t}^{ev,in}>0 \\ 0 otherwise \end{array}\right.$$

(29)

Each reward component as designed above is then scaled between [−1, 1], with weight parameters λ applied in (30) to distribute the importance of the reward components within each agent based on the building’s operation prioritizations.

$$r_n^t={\lambda }_{ev}{r}_{n,t}^{ev}+{\lambda }_{hc}{r}_{n,t}^{hc}+{\lambda }_P{r}_{n,t}^P+{\lambda }_k{r}_{n,t}^k$$

(30)

State Transition Functions: After agent n executes an action $a_n^t$ at observation $o_n^t$, the next observation $o_n^{t+1}$ is obtained according to the state transition function. The transition functions of the HVAC and EV are discussed in Section 2 through Equations (8) and (6), but the transition function of EV presence ${cp}_{n,t}$ and peak demand K_t are not fully observable by the agents, making this a Partially Observable Markov Decision Process (POMDP). Additionally, individual agents’ lack of comprehensive knowledge of the system’s global state due to physical limitations realistically reflects real-world community EMS settings, with agents mainly relying on their own sensors and local information for operation.

Multi-Agent Decision Problem: Each agent n obtains rewards as a function of the system state and its own action, represented as $r_n^t$: S × $a_n^t$ → ℝ. These rewards are influenced by other agents’ actions. The agent n aims to maximize its total expected reward R_n=${\sum }_{t=0}^T{\gamma }^t{r}_n^t$ over the time horizon T, where γ is a discount factor in time period t.

4. Proposed LSD-MADDPG Algorithm

This section presents an enhanced MADDPG-based framework, resulting in the LSD-MADDPG algorithm tailored for the smart community EMS, offering a practical, scalable, and cost-effective approach. Since buildings in a community cannot be assumed to be able to observe the operation parameters of other buildings, the community EMS problem as modeled in (1)–(19) cannot be solved directly without a central operator maintaining a global view. To address the major limitations of centralized solutions, we explore a model-free MARL algorithm that enables individual building agents to learn optimal control policies (π) from experience data, guiding their actions to maximize the cumulative reward (R_n), which is designed to align with community EMS objectives. Each agent’s reward, defined in (30), incentivizes cooperation to maximize collective reward and pursuing individual objectives, forming a mixed cooperative–competitive dynamic.

In a conventional MADDPG framework, each agent is modeled using the DDPG configuration, including an actor network and a critic network. The critic network is configured to have full access to the observations and actions of other agents in the environment. During the training process, sharing these observations and actions allows each agent to evaluate its own actions in relation to the overall state of the system and the actions of others, building an understanding of the transition probabilities of the system environment and facilitating more informed decision-making based on the collective behaviors observed.

The proposed LSD-MADDPG differs in that each agent’s critic network is restricted to only have access to the “strategy” of other agents in addition to its own local observations and actions, reflecting the physical observability limitations in the community setting. The “strategy” of each building agent is defined as $s_n^t=[s_{n,hc}^t,s_{n,ev}^t]$, which is further discretized to improve privacy protection as shown in (31)–(32). These agents’ operation strategies are proposed to introduce a simplified, quantifiable indication of each agent’s energy demand priority at a given time, which could be derived from historical operation experiences. The HVAC operation strategy $s_{n,hc}^t$ reflects the urgency for a building to bring temperature into the comfort range, while the EV charging strategy $s_{n,ev}^t$ indicates the urgency of scheduling charging.

$$s_{n,hc}^t=\left\{\begin{array}{c}0.1 if 19.5 °\mathrm{C}<T_{n,t}<23.9 °\mathrm{C}\\ 1 otherwise \end{array}\right.$$

(31)

$$s_{n,ev}^t=\left\{\begin{array}{c}0.1 if{ SoC}_{n,t}\ge 80\% \\ 0.5 if 50\%<{SoC}_{n,t}<80\% \\ 1 if {SoC}_{n,t}\le 50\% \end{array}\right.$$

(32)

The lower half of the LSD-MADDPG architecture shown in Figure 2 depicts where strategies, observations, and actions are processed in the actor and critic neural networks. Each agent’s actor and critic networks are constructed as deep neural networks with dense layers, with ReLU activations in the hidden layers. The actor network maps RL agent’s local observation to its optimal actions, employing a Tanh activation function in its output layer. Meanwhile, the output layer of the critic network produces a Q-value to evaluate the effectiveness of the actor’s actions by using the agent’s action, local observation, and the shared strategies from all other agents. This expanded perspective allows the critic to more accurately calculate the Q-value for evaluating the actor’s performance. As a result, the critic can guide the actor toward more effective decision-making, improving policy optimization and coordination without oversharing private data.

The upper half of Figure 2 illustrates the LSD-MADDPG architecture for a single agent, including key components such as the replay buffer, which stores past experiences, noise to facilitate exploration, and the detailed actor and critic network structures, including their respective target networks. The designed actor and critic network of agent n is denoted as µ_n and Q_n, and their associated weights are ${\theta }_n^{{\mu }_n}$ and ${\theta }_n^{Q_n}$. The target networks, ${\mu }_n^{\prime }$ and $Q_n^{\prime }$, are used to stabilize the training process by being periodically updated with a soft update mechanism, where the weights of the target networks are adjusted as a weighted sum of the current network and the target network, using a small learning rate to ensure gradual changes and prevent large oscillations in weight updates. The critic’s loss function, $L\left({\theta }_n^{Q_n}\right)$, is minimized to improve the accuracy of the Q-value estimates by calculating the temporal difference (TD) error between the target Q-value and the predicted Q-value. This loss is used to adjust the critic’s weights to better estimate future rewards. The policy gradient, ${\nabla }_{{\theta }^{{\mu }_n}} J\left({\theta }_n^{{\mu }_n}\right)$, is computed to update the actor’s weights and ensure the actor learns to select optimal actions over time by maximizing the expected return, achieved by adjusting the actor’s policy to choose actions that yield higher Q-values.

Because MADDPG is a deterministic algorithm, exploration is not inherently introduced through probabilistic actions, so noise must be introduced to the actions space to encourage exploration and prevent premature convergence to suboptimal solutions. Two types of noise are used to tailor to the specific action: Orstein-Uhlenbeck noise [63], which generates temporally correlated noise $N_{hc}^t$, mitigating large deviations in action exploration by selecting temperatures close to one another, and Arcsine noise [64] $N_{ev}^t$, characterized by a probability density function heavily weighted towards the minimum and maximum values, making it suitable for boundary decisions such as charging or not charging. Furthermore, to balance exploration and exploitation, ε-greedy is used, which gradually decreases exploration as training progresses, relying more on the learned policy. When not exploring, each agent takes action $a_n^t={\mu }_n\left(o_n^t\right)$. During exploration, actions are adjusted by adding noise, resulting in $a_{n,t}^{hc}={\mu }_n\left(o_n^t\right)+N_{hc}^t$ or $a_{n,t}^{ev}={\mu }_n\left(o_n^t\right)+N_{ev}^t$, where ${\mu }_n\left(o_n^t\right)$ is the output action of the actor network ${\mu }_n$.

Each agent’s actions $a_n^t$, along with their observations $o_n^t$ at t, are applied to the simulated smart community to calculate building-level energy consumptions, resulting in an aggregated community demand on the distribution system. At the end of t, each agent calculates their reward $r_n^t$ given by (30), reads a new observation $o_n^{t^{\prime }}$, and shares its strategy values $s_n^{t^{\prime }}$ with the community. These shared strategies are then accessed by all agents, and the experience tuple $(o_n^t,a_n^t,r_n^t,o_n^{t^{\prime }},s^t,s^{t^{\prime }})$ is stored in the buffer Ɗ. After every δ number of episodes, all agents’ actor and critic networks are trained by randomly sampling ρ number of transitions from the buffer. The transitions j, j $\in$ρ are used to update the neural network weights for both the actor and critic networks. For a detailed, sequential breakdown of this process, refer to Algorithm 1, which was coded in Python 3.8 using PyTorch 2.4 [65]. During training, both actor and critic networks are actively updated, but upon completion, the critic network and replay buffer are discarded. The final policy ${\mu }_n^{\mathrm{*}}$ is then used for online execution, where the agent selects actions $a_n^t={\mu }_n^{\mathrm{*}}\left(o_n^t\right)$ based on local observations.

Algorithm 1 Offline training phase of LSD-MADDPG

1. Initialize Networks and Replay Buffer:         - Initialize actor network µn and critic network Qn for each agent, n         - Initialize target networks ${\mu }_n^{\prime }$ and $Q_n^{\prime }$ with weights: ${\theta }_n^{{\mu }_n^{\prime }}\leftarrow {\theta }_n^{{\mu }_n}$ and ${\theta }_n^{Q_n^{\prime }}\leftarrow {\theta }_n^{Q_n}$.         - Initialize experience replay buffer Ɗ 2. Training Loop         - For each episode e               - Initialize the environment and get $o_n^0$               - For each timestep t until terminal state:                               A. Observe current state $o_n^t$ and strategies $s^t$                               B. Select action $a_n^t$ using actor network $a_n^t={\mu }_n\left(o_n^t\right)$                               C. Execute action $a_n^t \mathrm{o}$ bserve reward $r_n^t$, next state $o_n^{t^{\prime }}$ and next strategies $s^{t^{\prime }}$                               D. Store transition    $(o_n^t,a_n^t,r_n^t,o_n^{t^{\prime }},s^t,s^{t^{\prime }})$ in Ɗ                               E. If episode e is divisible by δ:                                           i. Sample minibatch of ρ transitions $(o_n^j,a_n^j,r_n^j,o_n^{j^{\prime }},s^j,s^{j^{\prime }})$ from Ɗ                                           ii. Update Critic Network:                                               - Compute target action using target actor network: $a_n^{j^{\prime }}={\mu }_n^{\prime }\left(o_n^{j^{\prime }}\right)$                                               - Compute target Q-value: $y_n^j=r_n^j+\gamma Q_n^{\prime }(o_n^{j^{\prime }},a_n^{j^{\prime }},s^{j^{\prime }}){|}_{a_n^{j^{\prime }}={\mu }_n^{\prime }\left(o_n^{j^{\prime }}\right)}$                                               - Compute current Q-value $Q_n(o_n^j,a_n^j, s^j)$                                               - Compute critic loss: $L\left({\theta }_n^{Q_n}\right)={\displaystyle \frac{1}{\rho }}\sum _{j=1}^{\rho }{(y}_n^j{-Q_n(o_n^j,a_n^j, s^j))}^2$.                                               - Perform gradient descent on $L\left({\theta }_n^{Q_n}\right)$ to update ${\theta }_n^{Q_n}$                                         iii. Update Actor Network                                               - Compute policy gradient: ${\nabla }_{{\theta }^{{\mu }_n}} J\left({\theta }_n^{{\mu }_n}\right)$ = ${\mathbb{E}[\nabla }_{{\theta }^{{\mu }_n}}{\mu }_n\left(o_n^j\right){\nabla }_{a_n^j}{Q}_n(o_n^j,a_n^j,s^j)]$.                                               - Perform gradient ascent on ${\nabla }_{{\theta }^{{\mu }_n}} J\left({\theta }_n^{{\mu }_n}\right)$ to update ${(\theta }_n^{{\mu }_n})$                                         iv. Update Target Networks                                               - For each agent n $$\left\{\begin{array}{c}{\theta }_n^{Q_n^{\prime }}\leftarrow \tau {\theta }_n^{Q_n}+(1-\tau ){\theta }_n^{Q_n^{\prime }}\\ {\theta }_n^{{\mu }_n^{\prime }}\leftarrow \tau {\theta }_n^{{\mu }_n}+(1-\tau ){\theta }_n^{{\mu }_n^{\prime }}\end{array}\right.$$ 3. End Training

5. Case Study

A smart community is simulated to evaluate the effectiveness of the proposed LSD-MADDPG demand-side EMS. Each building within the community is simulated to be equipped with an HVAC system and an EV charging station for ease of implementation, which could be expanded to multi-zone HVAC systems and multiple charging stations. The community EMS’s objectives are designed to minimize energy costs, ensure occupant comfort, and meet EV charging demands at the building level while reducing the peak energy demand at the community level.

The performance of the proposed LSD-MADDPG controller is further evaluated via a comprehensive comparison with four other controllers:

(1)

Naïve controller (NC): Represents the common practice in building energy management, involving setting the thermostat at a constant 72℉ and using a simple plug-in and charge-at-full-capacity charger. This approach does not optimize the community objective or energy consumption, hence the term ‘naïve’.

(2)

DDPG controller: A centralized single-agent RL controller for the entire community EMS operating within the environment and state-action space described in Section 3.

(3)

I-MADDPG controller: Independent Learning MADDPG with decentralized training and decentralized execution.

(4)

CTDE-MADDPG controller: MADDPG with centralized critic training and decentralized execution, with the agent’s critic network accessing the entire global state.

5.1. Simulation Parameters

Each episode was simulated for 48 h, from 12 p.m. to 12 p.m. two days later, to capture extended peak demand periods, including both morning and evening peaks, and to accommodate realistic EV charging scenarios, where vehicles may arrive, depart, and require multiple charging cycles. Episodes were randomly sampled from yearly data to account for temperature variations and introduce variability. Additionally, in our cases, all weights in (30) were set equally for each reward component; all other simulation parameters are summarized in

Table 1. Training hyperparameters.

	Variable	MARL
1	Total Timesteps	260,000
2	Episode Length	48
3	Learning Rate Actor	1 × 10−4
4	Learning Rate Critic	1 × 10−3
5	Noise-rate	0.1
6	Gamma	0.95
7	Tau	0.01
8	Buffer-size	5 × 105
9	Batch-size	256

.

5.2. Performance Analysis with Three Identical Buildings

To demonstrate the effectiveness of the proposed LSD-MADDPG in energy management via consumption shifting, all the EV schedules and building attributes, including HVAC system settings, were purposely set as identical across three buildings in the simulated community. This setting ensured that the observed charging patterns resulted from deliberate load-shifting rather than from differences in EV availability. Hourly weather data and electricity prices, which featured two price points for peak ($0.54/kWh) and off-peak hours ($0.22/kWh), were sourced from CityLearn Challenge 2022 Phase 1 dataset [66]. EV presence was consistently maintained across each day, during the hours from 18:00 to 8:00 each day.

Table 2. Specifications for buildings and energy systems.

Building Attributes	Value
$R_n$	0.0001 min·°/joule
$c$	1005.4 joule/kg·°
$T_{h,n}, T_{c,n}$	50 °C, $10 °\mathrm{C}$
${\psi }_n$	1778.4 kg
$M_n$	60 kg/min
$E_n^{ev,min}, E_n^{ev,max}$	1.5 kW,17 kW
$B_{max,n}$	100 kW
$E_n^{hc,max}$	600 kWm
$f_{hc}^{max}$	5

details the building, charging station, EV, and HVAC systems settings in the smart community EMS simulation.

Figure 3 illustrates the training reward values of four RL based EMS controllers, providing a comprehensive performance comparison. The results demonstrate that the LSD-MADDPG exhibits a robust performance in managing variability, consistently achieving higher cumulative rewards. Specifically, LSD-MADDPG outperformed all the other three RL controllers with the highest mean reward of 0.271 and minimal solution variability, indicated by a standard deviation of 0.0071. Meanwhile, the I-MADDPG and CTDE-MADDPG controllers also showed robust performance, with means of 0.270 and 0.269 and standard deviations of 0.0101 and 0.0122, respectively, while the DDPG controller lagged with the lowest mean of 0.251 and highest variability of 0.047682.

The evaluation was conducted on the CityLearn Challenge 2022 Phase 2 dataset [66], spanning a 48-h period from 12 p.m. on 1 January to 12 p.m. on 3 January.

Table 3. Evaluation results for three buildings.

	NC	DDPG	CTDE	I-MADDPG	LSD-MADDPG
Average Peak (kW)	51.28	● 3    43.20	● 1    27.30	31.96	● 2    31.03
Total Cost ($)	191.92	●    79.59	●    82.40	83.23	●    84.48
Charge (%)	100	●    89.00	90.00	●    92.00	●    91.00
Peak Contribution	31.35	●    36.67	●    43.17	41.68	●    43.46
Total Reward	6.63	●    34.36	●    38.72	38.06	●    38.69
Gini Coeff	0.0	●    0.0372	0.0293	●    0.0264	●    0.0125

¹ Green dots indicate the best value. ² Yellow dots indicate the second best. ³ Red dots indicate the worst value.

presents the 2-day evaluation results of various EMS controllers, detailing their achieved community average daily peak demand, total energy consumption costs, community charging satisfaction, buildings’ peak contribution, total reward, and a Gini Coefficient [67]. Average peak reflects the highest community energy demand recorded within a day, averaged over two days of evaluation. Lower peak demand indicates better community energy load management, to reduce strain on the distribution system. Total cost refers to the total community electricity cost, where lower costs indicate more energy usage during low-price periods. Charge refers to the percentage of required power for the entire community’s EV charging, where a higher value indicates better fulfillment of EV charging demand. Peak contribution sums each building’s contribution in reducing community peak demand, as described in (29), quantifying the reduction in the communities’ hourly peak electricity demand. Total reward represents the controller’s ability to optimize multiple objectives, including cost savings, temperature comfort, peak demand reduction, and efficient EV charging. Higher rewards indicate better overall system performance. Finally, the Gini Coefficient quantifies the equity of reward distribution among agents, where a lower Gini coefficient represents a more equitable allocation of rewards, indicating a fairer distribution of energy cost savings, temperature comfort, peak demand reduction contributions, and EV charging performance across all building agents. Throughout the simulation, building comfort was consistently maintained within predefined temperature requirement bounds; as a result, it is not tabulated.

The multi-objective problem in the demand-side EMS requires balancing various objectives included in (30). When full community information is shared among agents during training, CTDE-MADDPG demonstrates strong cooperation between building agents, achieving the highest community reward of 38.72. This is reflected in its superior ability to balance all objectives, especially peak contribution and community costs. However, LSD-MADDPG follows closely with a reward of 38.69, highlighting its comparable strong overall performance, without compromising data privacy. Both I-MADDPG and LSD-MADDPG, which promote more competitive environments for individual rewards, lead to higher total charge within the community and exhibit lower Gini Coefficients, resulting in a more equitable distribution of rewards. LSD-MADDPG achieves the lowest Gini Coefficient of 0.0125 and best peak contribution. I-MADDPG excels in charge management but lags in peak contribution and overall rewards, due to its lack of information sharing and thus coordination among RL agents. In contrast, DDPG lacks competition and exploits individual buildings, resulting in the highest Gini coefficient and lowest community charging satisfaction, as it prioritizes reducing costs at the expense of other objectives.

Figure 4 further elaborates individual building agents’ performance across four metrics—i.e., building rewards, peak contribution, energy cost, and charging satisfaction—with LSD-MADDPG showing the most balanced contribution, particularly in peak demand reduction, compared to other RL controllers. The closer proximity of the performance lines for each building agent indicates a more equitable load distribution among buildings, highlighting fairer participation across agents. The values are normalized such that the maximum value for each metric is set to 100%, with other values scaled proportionally. Higher values in building rewards, peak contribution, and charging satisfaction indicate better performance, while lower values in energy cost reflect greater energy efficiency. The LSD-MADDPG’s more balanced energy shifting (i.e., peak demand reduction) among buildings demonstrates that it secures a fair participation/coordination among agents, reducing the burden on any single building to manage the community energy demand. This more equitable cost/benefit distribution enhances system reliability by lowering the risk of over-reliance on specific buildings and improving scalability, allowing the system to remain effective as more buildings are included in the community EMS.

Figure 5 compares the individual buildings’ 48-h energy consumption achieved by CTDE-MADDPG and LSD-MADDPG to illustrate how the two algorithms manage the building HVAC and EV charging in response to varying electricity prices. On the first price drop, coinciding with EV presence, buildings initiate charging, leading to the highest energy peaks observed throughout the day. This is a rational behavior, as charging typically benefits from improved charging efficiencies when the SoC is low. While both algorithms reduce energy consumption during high-price hours, CTDE-MADDPG exhibits steeper fluctuations, with only a few buildings responsible for most of the energy reduction. In contrast, LSD-MADDPG smooths out consumption over a broader range of time, suggesting it prioritizes a more balanced distribution of energy shifting across all buildings, which explains the higher costs due to increased consumption during hour 19 of day 1. By involving more buildings in energy management, LSD-MADDPG achieves a steadier and more evenly distributed electric load among the buildings, while CTDE-MADDPG tends to concentrate energy adjustments to fewer buildings. This feature of LSD-MADDPG is better suited for achieving collective community energy management goals, such as peak reduction and load balancing across the entire community. Furthermore, LSD-MADDPG illustrates its advantages in achieving such collaboration while keeping data private. This proposed approach enables agents to both compete and cooperate effectively, striking a balance between equitable reward distribution and community collaboration.

5.3. Scalability Performance Analysis on the Proposed LSD-MADDPG

To evaluate the scalability of the proposed LSD-MADDPG, a smart community including nine buildings with various EV charging and building HVAC configurations was simulated.

Table 4. Building type and attributes.

Buildings	Thermal Characteristics						Charger Characteristics
Attributes	${\mathit{R}}_{\mathit{n}}$	${\mathit{T}}_{\mathit{h},\mathit{n}}$	${\mathit{T}}_{\mathit{c},\mathit{n}}$	${\mathit{M}}_{\mathit{n}}$	${\mathit{\psi }}_{\mathit{n}}$	${\mathit{E}}_{\mathit{n}}^{\mathit{h}\mathit{c},\mathit{m}\mathit{a}\mathit{x}}$	${\mathit{E}}_{\mathit{n}}^{\mathit{e}\mathit{v},\mathit{m}\mathit{a}\mathit{x}}$	${\mathit{B}}_{\mathit{m}\mathit{a}\mathit{x},\mathit{n}}$
Building 1	1 × 10−4	50	14	4.8	1778	300	10	50
Building 2	8.33 × 10−5	55	14	60	2500	420	15	75
Building 3	6.66 × 10−4	60	15	90	3200	600	20	100
Building 4	5.83 × 10−5	65	15	120	4500	900	25	120
Building 5	5 × 10−4	70	16	150	5500	1200	30	150
Building 6	4.16 × 10−4	75	16	180	6500	1500	35	200
Building 7	3.33 × 10−5	80	14	210	7500	1800	40	250
Building 8	6.66 × 10−5	65	15	72	2800	720	18	85
Building 9	5.83 × 10−5	55	15	108	4000	1080	22	125

provides a detailed breakdown of building types and their respective thermal and EV charging station characteristics, capturing the diversity in energy requirements and capacities across various community buildings. These various buildings simulated in the community increase problem complexity and showcase the adaptability of the proposed EMS solution, from smaller residential homes to expansive commercial complexes and their interactions with one another.

Table 5. Peak and off-peak monthly pricing.

Time of Day	Hours	$/kWh	Day Type
Winter (November–February)
Off-Peak	22:00–05:59	$0.09	Every day
On-Peak	17:00–19:59	$0.20	Monday–Friday
On-Peak	06:00–08:59	$0.20	Monday–Friday
Mid-Peak	Other hours	$0.12	Monday–Friday
Mid-Peak	All hours	$0.12	Saturday–Sunday
Summer (May–August)
Off-Peak	22:00–05:59	$0.09	Every day
On-Peak	12:00–17:59	$0.20	Monday–Friday
Mid-Peak	Other hours	$0.12	Monday–Friday
Mid-Peak	All hours	$0.12	Saturday–Sunday
Transitional (March, April, September, October)
Off-Peak	22:00–05:59	$0.09	Every day
Mid-Peak	Other hours	$0.12	Every day

tabulates electricity prices based on the time of day and specific month, aligning with typical winter and summer peak energy consumption patterns. For instance, winter features morning and evening peaks due to high heating demands, while summer exhibits midday peaks driven by air conditioning usage. Transitional months typically experience more moderate weather conditions.

Table 6. Hourly EV presence probability and day classification.

Day	Building 1		Building 2		Building 3		Building 4		Building 5		Building 6		Building 7		Building 8		Building 9
	Hour	%	Hour	%	Hour	%	Hour	%	Hour	%	Hour	%	Hour	%	Hour	%	Hour	%
Weekday	0–6	95	0–6	92	0–6	85	0–8	10	0–8	5	0–8	5	0–23	80	0–8	20	0–6	90
	7–16	10	7–16	20	7–16	30	9–16	80	9–16	85	9–16	90	0–23	80	9–16	80	7–16	15
	17–23	90	17–23	95	17–23	95	17–23	10	17–23	10	17–23	5	0–23	80	17–23	20	17–23	95
Weekend	0–23	95	0–23	95	0–23	90	0–23	100	0–23	5	0–23	20	0–23	80	0–23	10	0–23	95
Holiday	0–23	95	0–23	100	0–23	100	0–23	100	0–23	5	0–23	20	0–23	80	0–23	10	0–23	100

provides a detailed breakdown of the hourly presence patterns of EVs across various building types, segmented by day type—weekday, weekend, and holiday, based upon [68]. For each building, the table lists the probability of EV presence during specific hours, reflecting tailored energy management needs.

The four MARL controllers were trained using outdoor temperature data from [69] and the same number of timesteps, with average training rewards graphed in Figure 6.

Table 7. Controller performance comparison.

Controller	Mean Reward	Standard Deviation
LSD-MADDPG	0.203	0.0184
I-MADDPG	0.188	0.0102
CTDE-MADDPG	0.187	0.0341
DDPG	0.0291	0.308

portrays the way that LSD-MADDPG outperformed all other strategies, achieving the highest mean reward and the lowest variability during training. I-MADDPG and CTDE-MADDPG followed with comparable performance, while DDPG showed significantly lower rewards and higher variability, indicating challenges with scalability and in competitive environments where individual objectives must be balanced.

The main challenge in algorithm scalability lies in scheduling EV charging to balance peak demand reduction without overly sacrificing individual building needs or community objectives. The advantage of MADDPG, i.e., using separate actors and critics for each controller, is that each building can learn its own objectives while still considering the community goal of reducing peak demand. Peak contribution is primarily influenced by EV schedules because buildings with more charging hours (i.e., EVs are present) can spread out their energy use, while those with fewer hours may have to concentrate their charging demands. As the system scales, managing such flexibilities becomes more complex, and varying schedules across buildings create fluctuations in community peak demand, making efficient coordination more challenging. This is reflected in

Table 8. Controller evaluation case II.

	NC	DDPG	CTDE	I-MADDPG	LSD-MADDPG
Total Community Charge	100%	0%	50%	57%	83%
Reward	−101.65	66.31	277.22	340.89	438.82

’s evaluation results, which present the reward and total community charge, i.e., the total kWh provided to the EVs in the community, calculated as the difference between their initial SoCs and the target SoCs over the 10-day period. The naïve approach assumes all EVs are charged towards their full target SoCs without interruptions when present, establishing the maximum charge demand for the community at the cost of all other objectives.

A 10-day evaluation period was chosen to capture the building energy consumption characteristics of different days and schedules, ensuring robustness in various scenarios. LSD-MADDPG outperforms all controllers, which struggle to identify optimal EV charging schedules, demonstrating LSD-MADDPG’s adaptability and scalability in larger scale systems. It achieves the highest reward by meeting 83% of the total community’s EV charging demand. This indicates that LSD-MADDPG scales best, as its trained policy achieved superior results within the same number of timesteps as other algorithms. It demonstrates more efficient adaptability in a complex environment, e.g., competitive settings with diverse buildings, varied equipment capacities, and schedules.

Ultimately, LSD-MADDPG’s crucial strategic insights allow agents to better plan and coordinate their energy usage to optimize community objectives, while satisfying their individual charging needs and maintaining thermal comfort. This proposed approach not only enhances scalability, privacy, and decision-making efficiency, but also supports competition, a critical aspect as the complexity and variability of the community energy system increases.

6. Conclusions

This paper presents an enhanced MARL controller, i.e., LSD-MADDPG, for the demand-side energy management of smart communities. The proposed LSD-MADDPG emphasizes the equivalent priorities of pursuing community-level energy management objectives as well as satisfying individual buildings’ operational requirements in maintaining indoor comfort and optimizing EV charging, two critical elements of energy consumption management in buildings of future smart communities. Through competitive and cooperative interactions among building agents, the proposed LSD-MADDPG EMS controller performs comparably to CTDE-MADDPG in small scale cases while preserving data privacy, but demonstrates better scalability in more complex system environments. This proposed approach, considering the physical limitations of practical EMS systems in data access and privacy preservation, leverages the sharing of energy conservation strategies among agents, avoids the pitfalls of centralized systems—such as single points of failure—and reduces the potential data leakage and communication delays of conventional MADDPG. Future work will aim to incorporate renewable energy sources into the building system, adding complexity but potentially increasing the sustainability and efficacy of demand-side energy management in communities dominated by distributed energy resources (DERs).