A Control-Centric Systematic Review of MARL for EV–Grid Coordination: From Predictive Input to Verifiable Feedback

Abstract

The rapid integration of electric vehicles (EVs) and decentralized renewable energy sources is transforming urban power systems, while simultaneously increasing the complexity of real-time coordination across charging infrastructure, distributed energy resources, and grid-support devices. This systematic review synthesizes recent research on multi-agent reinforcement learning (MARL) for EV–grid coordination, with emphasis on four emerging dimensions: forecasting-informed control, safety-constrained learning, explainability and interpretability, and trustworthy decentralized coordination. A systematic literature search was conducted in IEEE Xplore, Scopus, Web of Science, ScienceDirect, MDPI, and arXiv, covering primarily the period 2016–2025, with selected early-2026 studies retained where relevant, with selected earlier foundational studies retained for context. The review was conducted and reported in accordance with the PRISMA 2020 framework. A total of 412 records were identified through database searching; after duplicate removal and screening, 58 studies were included in the final qualitative synthesis. The reviewed literature shows that MARL is increasingly being applied to EV charging coordination, demand-side management, community energy systems, transactive energy, and ancillary grid services. The evidence further indicates that forecasting integration improves anticipatory control, safety-aware formulations enhance operational reliability, and explainability-oriented designs help address transparency and trust barriers in safety-critical grid environments. However, the literature remains limited by heterogeneous benchmarks, inconsistent evaluation metrics, and a lack of real-world deployment evidence. This review provides a structured synthesis of current methodologies, identifies critical research gaps, and outlines future directions for the development of safe, interpretable, and deployment-ready MARL frameworks for urban energy systems.

1. Introduction

The large-scale integration of electric vehicles, distributed renewable generation, and flexible energy assets is redefining the operational requirements of modern urban power systems. Within this setting, multi-agent reinforcement learning (MARL) has emerged as a promising framework for decentralized coordination under uncertainty, enabling heterogeneous agents to make adaptive decisions while interacting through shared physical and economic constraints. This introduction first presents the EV–grid coordination challenge and the limitations of centralized control, then explains why MARL is well-suited as a decentralized solution, and concludes by outlining the scope, contributions, and organization of this review.

1.1. Background and Motivation

Urban distribution networks are facing an unprecedented coordination challenge. The adoption of electric vehicles and the deployment of renewable energy are accelerating simultaneously—electric-car sales exceeded 17 million in 2024, and public charging points increased by 1.3 million during that year alone [1]. Together, these trends are concentrating highly variable loads on low- and medium-voltage networks that were originally designed for stable and predictable demand. Coincident charging peaks, clusters of fast chargers, and correlated user behavior—such as commuter surges and event-driven demand spikes—now place significant stress on equipment that was sized for passive loads.

This operational stress is inherently spatiotemporal. Charging demand is shaped by mobility patterns, charger availability, and tariff structures, while renewable injections introduce additional volatility and intermittency. High EV penetration intensifies local voltage drops, cable and transformer overloading, and equipment aging, and introduces effects beyond thermal loading—such as harmonics—which further complicate asset-level risk assessment [2,3]. At the same time, extreme events and prolonged outages reveal an important resilience dimension. Under such conditions, charging networks may experience cross-district demand migration and cascading congestion, which can simultaneously degrade quality of service (QoS) and threaten grid stability. These risks make recovery-aware coordination increasingly necessary [4].

These developments have exposed the limitations of traditional control architectures. Centralized control struggles to scale in urban environments where thousands of heterogeneous entities—including EVs, charging stations, DER inverters, storage systems, and feeder devices—must be coordinated. Even when a single operator has theoretical authority over the entire system, centralized decision-making becomes inefficient because it requires extensive metering, continuous system-wide communication, and large volumes of data exchange to support real-time optimization [5].

This burden is further compounded in modern grids by non-stationarity arising from demand variability, price fluctuations, and renewable intermittency. At the same time, hard operational constraints—including voltage limits, thermal limits, transformer capacity, and protection requirements—must be satisfied at all times.

Decentralized control offers improved responsiveness and eliminates single points of failure, yet heuristic decentralized approaches often perform poorly when agents are spatially dispersed and strongly coupled through network physics and shared constraints. Decision rules based only on local information can produce emergent congestion or oscillatory behavior as adoption levels increase and system interactions become denser [6]. Accordingly, there is a clear need for adaptive frameworks that can scale with network size, coordinate under partial observability, and respect safety-critical constraints.

MARL provides a principled framework for decentralized decision-making under uncertainty. In this setting, agents learn cooperative—or in some cases mixed cooperative-competitive—policies through interaction with the environment and with one another. These agents may represent EVs, charging stations, aggregators, microgrid controllers, or feeder-level coordinators. In contrast to static optimization methods, MARL can adapt online to changing operating conditions. It also supports scalability through centralized training with decentralized execution (CTDE), in which agents are trained jointly but execute decisions locally, thereby preserving both coordinated learning and operational scalability.

Recent studies have demonstrated the effectiveness of MARL across a range of EV–grid applications. EVCAR, for example, proposes a multi-district TD3-MADDPG framework for post-outage QoS recovery, jointly managing charging policies and cross-district redistribution to reduce waiting times and improve voltage stability [4]. Kamrani et al. present a hierarchical framework in which the DSO enforces power-flow safety limits while EV aggregators learn purchasing and dispatch strategies that balance efficiency and fairness [7]. Related work further confirms the value of capacity-aware coordination across multiple sites through multi-agent actor–critic algorithms combined with integrated short-horizon forecasts [8].

Figure 1 illustrates the survey scope. The central red hub (MARL for EV–grid coordination) connects four research dimensions: (1) Forecasting-informed MARL (blue): point forecasts, probabilistic forecasts, and scenario/uncertainty-aware methods; (2) Safety-constrained MARL (green): CMDP/Lagrangian methods, safety layers, and robust/risk-aware approaches; (3) Explainability and interpretability (orange): post-hoc methods, intrinsic techniques, and credit assignment; (4) Trustworthy decentralized coordination (pink): audit trails, federated learning, and blockchain-based mechanisms. Five application domains (yellow) span demand-side management, smart communities and DER coordination, peer-to-peer trading, ancillary services, and resilience-oriented recovery.

Figure 1. Taxonomy of the survey scope, showing key application domains and four emerging dimensions of MARL for EV–grid coordination: forecasting-informed learning, safety-constrained control, explainability and interpretability, and trustworthy decentralized coordination.

Despite these advances, reliable field deployment depends on addressing four critical gaps that have become increasingly visible in research published during 2024–2025. These gaps are not purely methodological; they are also closely connected to practical challenges already faced by power-system operators and aggregators. In distribution networks, voltage and thermal constraints must be respected within operationally meaningful timescales, so coordination methods that cannot respond quickly enough, or that rely on unrealistically simplified communication assumptions, have limited practical value. In EV charging and distributed energy scheduling, delayed or unreliable information exchange can reduce both feasibility and coordination quality, especially when decisions rely on timely updates from many spatially distributed agents. At the same time, the growing use of AI in safety-critical infrastructure introduces a verifiability challenge: utilities and regulators require control actions that are auditable, technically explainable, and consistent with established engineering and interoperability standards. These industry-driven constraints motivate the four research gaps identified below and help explain why strong performance in clean simulation alone is insufficient for practical deployment.

• Forecasting-informed MARL.

Forecasting can strengthen agent policies by providing estimates of charging demand, renewable generation, and electricity prices. Recent work has integrated demand and price forecasting directly into RL-based charging decisions [9] and has developed unified forecasting-scheduling pipelines in which neural forecasts inform MARL control loops [10]. Uncertainty-aware transformer forecasting has also been incorporated into MARL-based peer-to-peer trading [11]. However, comparable uncertainty-aware coordination at the city scale remains limited. This gap motivates research on forecast-robust MARL frameworks that can propagate prediction uncertainty through the decision-making process. From an industry perspective, inaccurate forecasts—or forecasts whose uncertainty is not properly propagated—can lead to feeder congestion, degraded charging service, and economically inefficient scheduling under real operational conditions [12].

2.

Safety-constrained coordination.

Grid controllers must operate within physical limits—including voltage bounds, thermal ratings, and transformer capacity—while also avoiding unsafe exploration. Safe RL has developed into a distinct research direction, and recent reviews have documented safety layers and constrained optimization formulations for power-system applications [13]. In EV charging, DuMES illustrates this approach by implementing dual-layer safety modules for DRL-based charging scheduling [14]. Nevertheless, extending these mechanisms to multi-agent settings with coupled constraints and partial observability remains difficult. This gap highlights the need for scalable SafeMARL architectures that can enforce safety guarantees across distributed agents without undermining control performance. In practical utility settings, this challenge is directly related to the requirement to maintain distribution-network voltage and loading limits in real time, even when many EV and DER agents act simultaneously under partial observability and communication delay [15,16].

3.

Explainability and interpretability (X-MARL).

As MARL policies increasingly affect billing, reliability, and regulatory compliance, operators require interpretability for both auditing and debugging. Recent contributions published in 2025 make this need explicit through decision-tree extraction from attention-based multi-agent DRL voltage controllers [17], and through scenario-guided temporal prototypes that provide case-based explanations in low-voltage networks [18]. Even so, explainability remains fragmented in EV–grid MARL relative to the stronger emphasis placed on performance-driven research [19]. This gap also underscores the need to distinguish between offline interpretability for audit and compliance and low-latency interpretability for real-time operational support. The issue is especially important for industrial adoption because operators must be able to verify, justify, and audit AI-assisted control decisions against engineering procedures, interoperability requirements, and emerging governance frameworks for high-risk AI [20,21].

4.

Trustworthy decentralized coordination and P2P markets.

Future EV and DER ecosystems will require peer-to-peer coordination, flexible services, and market-mediated control. Blockchain and smart contracts offer a possible pathway toward transparent, tamper-evident interaction logs and automated settlement. Recent work has combined blockchain with federated forecasting and multi-agent DRL to support decentralized demand response and trading [22]. Blockchain-enhanced mechanisms have also been proposed to improve trust and fairness while reducing the risk of strategic manipulation and collusion in decentralized MARL settings [23]. However, the integration of blockchain-based coordination with safety-constrained and explainable MARL remains underexplored. This limitation motivates further research on verifiable decentralized governance. For real-world deployment, this gap reflects the practical need for coordination mechanisms that remain trustworthy under multi-actor participation, limited mutual visibility, and auditable settlement requirements in decentralized energy and flexibility markets [24].

1.2. Contributions and Organization of the Survey

To address these gaps, this systematic review synthesizes recent research on MARL for EV–grid coordination in smart energy systems. In this review, the term control-centric refers to an emphasis on closed-loop operation, physical constraints, hierarchical controller interaction, and deployment-oriented evaluation, rather than machine-learning performance alone. The review also considers how MARL formulations handle non-stationarity and credit assignment, particularly in cooperative grid settings where individual agent contributions may be obscured by aggregate system rewards. It places particular emphasis on four emerging dimensions: forecasting-informed control, safety-constrained learning, explainability and interpretability, and trustworthy decentralized coordination. Alongside a summary of state-of-the-art methods and applications, this review identifies methodological limitations, benchmarking practices, and open challenges regarding the safe and scalable deployment of MARL in real-world urban energy systems. Table 1 positions this work relative to existing surveys and, where available, reports the number of papers reviewed in each survey to indicate comparative scope; when an original review did not explicitly report the total number of included studies, we conservatively estimated it from the article’s summary tables, study listings, or discussed solution overviews and marked such values accordingly. The specific contributions of this review are as follows:

Table 1. Comparison of recent review papers and the scope of this survey.

Ref.	Year	No. of Papers	Primary Scope	Forecast	Safety	XAI /XRL	Blockchain /P2P	EV– Grid	Key Gap Addressed by This Survey
[5]	2024	∼8 a	MARL for energy networks (edge, operation, markets)	Partial	Partial	Partial	Partial	Partial	Unifies EV-centric MARL with explicit forecasting–safety–XAI–blockchain taxonomy and deployment lenses.
[25]	2025	>50 a	RL for EV charging management (theory + applications)	Partial	Partial	Partial	No	Yes	Extends from RL-for-charging to multi-agent grid-coupled coordination, including market trust and safety-certified operation.
[13]	2025	∼25 a	Safe RL techniques for power-system control (freq/volt/EMS)	No	Yes	Partial	No	Partial	Bridges safe-RL foundations into multi-agent EV–DER coordination and ties safety to forecasting and interpretability.
[26]	2025	∼30 a	RL for microgrid control and management	Partial	Partial	Limited	No	Partial	Re-centres the survey on urban EV–grid integration and multi-agent coordination under distribution constraints.
[27]	2025	∼26 a	RL for grid topology control (benchmarks, L2RPN)	No	Partial	No	No	No	Complements topology-control surveys by focusing on EV charging/DER coordination and the four emerging dimensions.
[2]	2025	∼40 a	Distribution-grid impacts of EV charging (non-RL)	No	No	No	No	Yes	Provides grid-impact background but does not cover learning-based coordination; this survey synthesizes MARL solutions.
This survey	2026	58	MARL for EV–grid integration with four emerging dimensions	Yes	Yes	Yes	Yes	Yes	Unified taxonomy + comparative review + deployment-oriented gaps: prediction-aware coordination, safety guarantees, interpretability, and trustworthy P2P interactions.

^a Estimated for comparison purposes, where the original review did not explicitly report the total number of papers reviewed. Estimates were conservatively reconstructed from summary tables, study listings, or solution overviews reported in each survey.

(i)

We synthesize MARL formulations and coordination mechanisms across EV charging, DER control, and distribution-network operation, highlighting design choices under CTDE and partial observability.

(ii)

We systematize how forecasting—including uncertainty-aware prediction—embeds into MARL state and action design and evaluation protocols.

(iii)

We categorize safety-constrained MARL methods—including CMDPs, safety layers, projection and shielding, barrier functions, and runtime assurance—in the context of EV-driven grid constraints.

(iv)

We review emerging X-MARL methods for operational transparency and regulatory auditing.

(v)

We discuss blockchain-based coordination patterns for secure peer-to-peer interaction and incentive alignment.

The following sections establish the foundational concepts, terminology, and mathematical frameworks underlying MARL for EV–grid coordination. After description of the review methodology, Section 3 presents the foundations and mathematical preliminaries, and examines the principal learning paradigms, coordination structures, and performance metrics used to evaluate MARL in smart energy systems. These foundations contextualize a comprehensive review of MARL applications in urban energy—presented in Section 4—spanning demand-side management, smart communities, transactive energy, and ancillary grid services. The core of the survey then details a control-centric approach across three critical research frontiers: Forecasting-informed MARL (Section 5), which integrates predictive analytics to handle spatiotemporal uncertainty; Safe MARL (Section 6), which enforces physical grid constraints through formal safety mechanisms; and Explainable and interpretable MARL (Section 7), which provides the operational transparency required for industrial deployment. Finally, Section 8 addresses cross-cutting issues and future research directions, including scalability, transfer learning, and blockchain-enabled trust, before providing concluding remarks in Section 9.

2. Review Methodology

This section describes the systematic methodology adopted for this review, including the guiding research questions, the literature search and screening procedures, and the inclusion and exclusion criteria. Our objective is to identify and synthesize recent research on MARL for EV–grid coordination, emphasizing four dimensions: forecasting-informed control, safety-constrained learning, explainability and interpretability, and trustworthy decentralized coordination.

The main research question guiding this review is as follows: How has MARL been applied to EV–grid coordination in contemporary smart energy systems? This question is operationalized through five sub-questions:

• Which MARL paradigms, coordination structures, and learning architectures address EV charging, EV–grid interaction, DER coordination, and transactive energy management?
• How do MARL-based frameworks integrate forecasting and predictive analytics for energy management and charging coordination?
• Which safety-aware mechanisms enforce physical and operational constraints in MARL for power and energy systems?
• How are explainability and interpretability integrated into RL and MARL frameworks for auditing and operational transparency in smart grids?
• Which emerging research directions shape trustworthy decentralized coordination, including blockchain-enabled, privacy-aware, and benchmark-driven MARL?

2.1. Review Design and Reporting Framework

This study was designed as a systematic qualitative review, conducted and reported in accordance with the PRISMA 2020 framework. Since the included literature primarily comprises methodological, simulation-based, and review-oriented studies within engineering and energy systems, the synthesis was performed narratively rather than statistically. The review process encompassed study identification, screening, eligibility assessment, structured data extraction, and thematic qualitative synthesis.

2.2. Inclusion and Exclusion Criteria

Studies were included if they employed reinforcement learning (RL) or MARL for control, coordination, scheduling, or operational decision-making in EV charging, EV–grid interaction, DERs, demand response, local electricity markets, or grid-interactive smart-energy services. Eligible studies focused on power and energy applications within smart grids, distribution networks, microgrids, local energy communities, or EV-integrated environments, and provided sufficient methodological, modeling, or experimental detail via simulation, benchmarks, case studies, or real-world datasets. Although arXiv was searched to identify emerging work and citation leads, only studies meeting the predefined inclusion criteria were retained in the final synthesis. Studies were excluded if they addressed conventional optimization, forecasting, or machine learning without clear RL components, or if they applied RL outside the power and energy domain without relevance to EV–grid coordination. Excluded records included duplicates, inaccessible texts, studies with limited technical detail, and studies misaligned with the review’s thematic scope. For the synthesis, the included studies were categorized by both application context and methodological emphasis. The application-level grouping comprises demand-side management, smart communities and DER coordination, transactive energy and peer-to-peer trading, and ancillary grid services. Meanwhile, the cross-cutting methodological grouping encompasses forecasting-informed MARL, safety-constrained MARL, explainability and interpretability, and trustworthy decentralized coordination.

2.3. Literature Search and Study Selection

The literature search was conducted across major scientific databases and digital libraries, including IEEE Xplore, Scopus, Web of Science, ScienceDirect, MDPI, and arXiv. Additional studies were identified through backward and forward reference chaining. The review prioritized literature published between 2016 and 2025, while also considering selected studies published in early 2026 that were directly relevant to the review scope; though foundational studies were retained where necessary for conceptual or mathematical context. The final search was completed on 28 February 2026.

The search strategy employed Boolean logic to combine three thematic groups:

(1)

MARL and RL terms (multi-agent reinforcement learning, MARL, deep reinforcement learning, MADDPG, PPO, QMIX, CTDE, safe reinforcement learning, and explainable reinforcement learning);

(2)

EV–grid and energy terms (electric vehicle, EV charging, EV–grid coordination, smart grid, distribution network, microgrid, demand response, peer-to-peer energy trading, DER coordination, voltage regulation, and energy management);

(3)

Enabling-dimension terms (forecasting, uncertainty-aware learning, safe MARL, XAI, blockchain, privacy-preserving learning, federated learning, and trustworthy coordination).

An example Boolean search string used in the review was as follows:

• (“multi-agent reinforcement learning” OR MARL OR “deep reinforcement learning” OR MADDPG OR PPO OR QMIX OR CTDE) AND (“electric vehicle” OR “EV charging” OR “EV–grid coordination” OR “smart grid” OR microgrid OR “demand response” OR “peer-to-peer energy trading” OR “DER coordination”) AND (forecasting OR “safe reinforcement learning” OR XAI OR blockchain OR “federated learning”).

Search strings were adapted to fit the specific syntax of individual databases. The search was restricted primarily to English-language publications, including peer-reviewed journal articles, conference papers, and high-relevance review papers. While arXiv was searched to identify emerging work and citation leads, only studies meeting the predefined inclusion criteria were retained for the final synthesis.

Study selection began with title and abstract screening, followed by a full-text eligibility assessment. These stages were performed by a single reviewer, with borderline cases resolved through discussion with a co-author; no automated screening tools were employed.

The identification, screening, and inclusion process followed the PRISMA 2020 framework to ensure transparency and reproducibility. From an initial 412 records, 54 duplicates were removed, leaving 358 records for title and abstract screening. Of these, 266 were excluded, and 92 reports were sought for retrieval. Four reports could not be retrieved; the remaining 88 reports were assessed for eligibility. Ultimately, 30 reports were excluded at the full-text stage due to insufficient relevance, limited methodological detail, or weak alignment with the thematic scope. This resulted in 58 studies for the final qualitative synthesis, as illustrated in the PRISMA flow diagram (Figure 2).

Figure 2. PRISMA flow diagram of the study selection process.

2.4. Data Extraction and Data Items

Data were extracted from included studies using a structured review form developed specifically for this survey. The captured information encompassed publication year, study type, application context, agent type, learning paradigm, MARL/RL algorithm, coordination structure, state and action design, reward formulation, forecasting integration, safety mechanism, explainability technique, blockchain- and trust-related mechanisms, benchmark environment, and key evaluation outcomes. Where relevant, we also recorded whether code availability, data availability, benchmark transparency, scalability assumptions, communication structure, and deployment assumptions were explicitly reported.

Data extraction was performed by one reviewer, with consistency ensured through supervisory checking. No study investigators were contacted for clarification, and no automation tools were employed. In cases where reported information was incomplete or ambiguous, studies were interpreted conservatively, and only clearly supported methodological details were included in the synthesis.

2.5. Quality Appraisal and Risk of Bias Considerations

Because the included studies were primarily methodological, simulation-based, and engineering-focused rather than clinical or intervention-based, a formal medical-style risk-of-bias tool was not applied. Instead, the methodological rigor of the literature was assessed qualitatively during synthesis. Particular attention was paid to the clarity of problem formulation, transparency of algorithmic descriptions, adequacy of benchmark design, completeness of evaluation reporting, and the alignment between reported results and stated conclusions.

This qualitative appraisal informed the narrative interpretation of the literature, specifically by identifying recurring limitations, including restricted benchmark diversity, insufficient real-world validation, inconsistent reporting of constraint violations, and limited reproducibility across studies.

2.6. Synthesis Strategy

Due to the significant heterogeneity in study objectives, system models, benchmark environments, algorithms, and evaluation metrics, a quantitative meta-analysis was deemed inappropriate. Consequently, the findings were synthesized using a structured qualitative narrative approach. Studies were first categorized by application domain—including demand-side management, smart communities and DER coordination, transactive energy and peer-to-peer trading, and ancillary grid services—and subsequently analyzed across four cross-cutting methodological dimensions: forecasting-informed MARL, safety-constrained coordination, explainability and interpretability, and trustworthy decentralized coordination.

Comparative tables are utilized throughout the review to summarize representative studies, contrast methodological paradigms, and organize application-specific findings. No formal effect-size pooling, subgroup meta-analysis, or sensitivity analysis was performed, given the non-comparable nature of the reported outcomes and experimental configurations.

2.7. Protocol and Registration

This review was conducted and reported in accordance with the PRISMA 2020 framework. However, the review was not formally registered, and a standalone review protocol was not prepared prior to the study.

3. Foundations and Mathematical Preliminaries

3.1. From MDP to Dec-POMDP: Formalizing Multi-Agent Environments

MARL in power systems involves interactions among multiple agents—EV chargers, vehicles, DERs, or grid controllers—each observing local state variables (charger load, battery state of charge, local voltage) and selecting actions (charging rate adjustments, energy dispatch, pricing offers). These actions are formulated to maximize long-term cumulative rewards.

Following [28], we begin with the Markov decision process (MDP) $\mathcal{M}=⟨\mathcal{S},\mathcal{A},P,r,\gamma ⟩$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ the action space, $P\left(s^{\prime }\right|s,a)$ the transition kernel, $r(s,a)$ the reward function, and $\gamma \in (0,1)$ the discount factor. A policy $\pi \left(a\right|s)$ induces the expected discounted return

$$J\left(\pi \right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)\right].$$

(1)

In grid applications, $s_t$ encodes network conditions (nodal voltages, line flows), exogenous demand, DER outputs, and device states (e.g., SoC), while $a_t$ represents a control action (an aggregate charging setpoint or a dispatch decision).

With N interacting agents, a Markov game introduces agent-indexed action spaces ${\left\{{\mathcal{A}}_i\right\}}_{i=1}^N$, agent rewards ${\left\{r_i\right\}}_{i=1}^N$, and often partial observability. The joint action is ${\mathbf{a}}_t=(a_t^1,\dots ,a_t^N)\in \mathcal{A}={\prod }_i{\mathcal{A}}_i$. Settings are cooperative (shared rewards), competitive (conflicting rewards), or mixed (partially aligned rewards). In EV charging or distribution control, cooperative formulations minimize total cost or peak demand, whereas market and transactive energy settings employ competitive formulations in which participants optimize private utility.

A decentralized partially observable MDP (Dec-POMDP) is defined as $\mathcal{D}=⟨\mathcal{I},\mathcal{S},\left\{{\mathcal{A}}_i\right\},P,r,\left\{{\mathcal{O}}_i\right\},O,\gamma ⟩$ [28], where $\mathcal{I}=\{1,\dots ,N\}$ is the agent set and each agent receives only its local observation $o_i\in {\mathcal{O}}_i$. The joint observation is $\mathbf{o}\in \mathcal{O}={\prod }_i{\mathcal{O}}_i$ generated by $O\left(\mathbf{o}\right|s,\mathbf{a})$. Each agent conditions on its action-observation history:

$${\tau }_i^t=(o_i^0,a_i^0,o_i^1,a_i^1,\dots ,o_i^t),$$

(2)

and follows a decentralized policy ${\pi }_i\left(a_i^t\right|{\tau }_i^t)$. The cooperative objective under shared reward $r(s,\mathbf{a})$ is

$$J\left(\mathit{\pi }\right)={\mathbb{E}}_{\mathit{\pi }}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,{\mathbf{a}}_t)\right],\mathit{\pi }=({\pi }_1,\dots ,{\pi }_N).$$

(3)

Although Equations (1) and (3) have a similar expected-return form, they represent different optimization problems. Equation (1) describes the objective of a single-agent MDP, where one policy selects actions based on the full state or its available observation. In contrast, Equation (3) is defined over a joint policy $\mathit{\pi }=({\pi }_1,\dots ,{\pi }_N)$ in a cooperative multi-agent setting, where the return depends on the joint action ${\mathbf{a}}_t=(a_t^1,\dots ,a_t^N)$ and on interactions among multiple agents through shared dynamics and rewards. Moreover, under partial observability, each agent selects actions using only its local action-observation history ${\tau }_i^t$, rather than the full system state. As a result, the multi-agent optimization problem is not a simple repetition of the single-agent case, but a decentralized coordination problem with coupled decisions, limited information, and joint credit assignment.

This framework captures realistic grid settings where agents lack full network observability but coordinate through shared rewards, constrained dynamics, or limited information exchange. Recent benchmark environments explicitly account for hard physical constraints and realistic dynamics, enabling reproducible training and evaluation of RL/MARL in power systems [29,30].

Agents’ objectives are cooperative, competitive, or mixed. EV charging agents may cooperate to minimize system costs or peak demand, whereas market settings feature competitive agents optimizing their private profits. Intermediary agents (aggregators or DSO-layer coordinators) can reconcile local and system-level objectives [31]. Recent 2025 EV–grid studies demonstrate hierarchical multi-agent structures allocating grid-feasible power budgets before decentralized agents optimize local decisions [7], and multi-district recovery coordination coupling local charging control with higher-level redistribution to prevent congestion and maintain grid stability [4].

3.2. Coordination Paradigms: Independent Learners vs. CTDE

Centralized Training with Decentralized Execution (CTDE) is a foundational paradigm in MARL because it allows multiple agents to coordinate their actions toward shared objectives. Under CTDE, agents are trained offline using global state information through a centralized critic that observes the states and actions of all agents and provides learning signals during training. During execution, however, each agent acts independently using only its local observations, thereby preserving decentralized operation [32].

A major challenge in MARL is non-stationarity. From the perspective of any individual agent, the environment changes continuously as other agents update their policies. In EV–grid coordination, this challenge is more pronounced because of device heterogeneity, partial observability, and network-coupled constraints. For example, the charging decision of one agent can alter feeder loading, local voltage conditions, or congestion experienced by others. CTDE helps mitigate this issue by allowing training-time critics or mixing networks to access broader joint information, making learning signals more stable even though execution remains decentralized. However, CTDE does not fully resolve the credit-assignment problem, particularly in cooperative settings where global rewards reflect aggregate outcomes such as total load, feeder losses, or system-wide voltage quality [33].

This limitation motivates the use of value-factorization methods. In Value Decomposition Networks (VDNs), the joint action-value function is expressed as the sum of per-agent utilities, which supports decentralized decision-making but assumes a relatively simple additive structure of agent contributions. QMIX extends this approach by learning a monotonic mixing function between individual agent utilities and the global action-value, enabling richer forms of coordination while preserving decentralized argmax consistency. In grid applications, these methods are particularly attractive for cooperative tasks such as coordinated EV charging or DER dispatch because they partly address hidden-contribution settings in which individual responsibility is masked by aggregate load behavior. They are also closely connected to the so-called lazy-agent problem, in which some agents contribute little because a shared global reward can still be optimized through the efforts of others. For this reason, more informative reward shaping, hierarchical decomposition, agent-specific constraints, or explicit credit-assignment mechanisms are often required when local actions have unequal or only weakly observable effects on voltage, congestion, or transformer loading [34,35].

To place these methods in context, it is also useful to contrast them with independent learning. In independent learning, each agent treats the other agents as part of the environment and learns on its own, as in independent Q-learning. This approach is simpler and more scalable. However, it is also more vulnerable to instability and non-stationarity because agents do not account for the learning dynamics of others. In grid environments, this weakness is further amplified by strong coupling through network physics and shared operational constraints.

Building on this foundation, CTDE supports advanced algorithms such as MADDPG, in which centralized critics are used to train decentralized agent policies with joint state-action information. These formulations have been widely applied to coordinate EV charging and DER control [5]. Recent EV–grid frameworks published in 2025 make this structure even more explicit by combining system-level feasibility layers with decentralized learning. For instance, some approaches allocate grid-feasible budgets at upper levels and then optimize decisions locally at lower levels [7]. Hierarchical multi-agent control for post-outage restoration follows a similar hybrid logic: local agents act on local observations, while higher-level coordinators balance congestion and system stress across regions [4].

The choice of application also determines the underlying MARL game structure, which may be fully cooperative, fully competitive, or mixed. In smart grid applications, cooperative formulations are the most common. Examples include charging stations that jointly flatten load profiles or minimize overall cost. In such settings, shared rewards encourage coordination, and methods such as value factorization or centralized critics are often effective. By contrast, electricity markets and transactive energy systems introduce competitive interactions, such as aggregators competing for limited capacity. Mixed settings are also highly relevant in practice: EVs may cooperate within a single fleet while competing with other fleets, and such settings are often implemented through hierarchical agent structures and reward shaping [31]. Recent community-scale deployments reported in 2025 further emphasize cooperative objectives—including cost reduction and energy autonomy—while maintaining decentralized decision-making [36].

3.3. Communication Topologies and Information Sharing in Grids

Coordination quality depends on both learning paradigm (independent vs. CTDE) and information structure—what is observed locally, what is communicated, and how agents couple through cyber-physical links.

Smart grids employ several communication topologies:

(a)

No explicit communication: Agents rely on local measurements and shared rewards.

(b)

Star (aggregator/DSO-mediated): Local agents report summary information to a central coordinator.

(c)

Neighborhood graphs: Aligned with electrical adjacency (e.g., feeder graphs), agents exchange information only with nearby nodes.

(d)

Hierarchical multi-level: Structures like substation–feeder–device, where upper layers coordinate constraints or budgets, and lower layers execute fast control.

These design choices reflect practical constraints related to bandwidth, latency, privacy, and cybersecurity. Reducing inter-agent communication can improve scalability and privacy. However, doing so also requires stronger inductive biases during training, such as CTDE critics, safety layers, and structured reward design. Recent work on safe MARL for voltage/VAR coordination illustrates this balance by using decentralized control together with learned safety layers. In this setting, local inverter actions are permitted, while unsafe commands are projected into feasible regions to preserve voltage and loading limits in active distribution networks [37].

A related set of challenges arises beyond physical control, particularly in transactive energy and peer-to-peer settings. Here, information sharing must support settlement, trust, and resistance to strategic behavior. Recent work published in 2025 combines learning-based methods with secure coordination mechanisms. In particular, blockchain-anchored logs and smart contracts have been used to enforce transparent transactions and improve incentive alignment [22,23]. Privacy-preserving approaches, such as federated forecasting, further reduce data exposure while still enabling decentralized optimization [22]. Together, these developments support the view that communication, including messages, aggregation schemes, and trust mechanisms, should be treated as a primary MARL design variable for grid applications.

These considerations become even more important in operational deployments, where communication overhead can reduce real-time performance. For this reason, MARL frameworks are often designed either to execute solely from local observations after training or to exchange only low-rate information, such as power budgets, price signals, or congestion indicators. Under these conditions, both communication topology, meaning who communicates, and communication semantics, meaning what information is exchanged, must satisfy requirements for reliability, privacy, and computational feasibility.

An additional practical challenge is that real communication channels and measurement streams are inherently imperfect. In operational EV–grid settings, agents may encounter delayed messages, packet loss, asynchronous updates, missing sensor values, corrupted telemetry, and noisy state estimates. These issues can reduce coordination quality, destabilize learned policies that depend on timely shared information, and weaken the reliability of both safety filtering and decentralized execution. Although some studies have acknowledged communication efficiency and partial observability, systematic evaluation of robustness under data loss, noisy measurements, and delay remains limited in the reviewed literature [38].

3.4. Performance Metrics: Balancing AI Convergence with Power System Reliability

Evaluating MARL in power systems requires metrics from both machine learning and power engineering. We organize evaluation criteria into learning metrics (convergence and training robustness) and grid metrics (reliability, efficiency, and cost).

3.4.1. Learning Metrics (Stability, Regret, Sample Efficiency)

Key learning metrics include convergence speed, training stability, and agent scalability. Because MARL is prone to oscillations arising from non-stationarity, stability is commonly evaluated using reward variance across random seeds, policy oscillation, or the frequency of constraint-violating actions during training.

Another important dimension is sample efficiency, which measures the number of environment interactions required to achieve a target level of performance. This issue is especially significant in grid applications, where both high-fidelity simulation and real-world interaction are costly. Recent benchmarks have explicitly emphasized the difficulty of long-horizon control under hard constraints and have introduced standardized tasks and baselines to support more systematic comparison [29,30].

Performance evaluation in time-varying environments requires additional considerations. When demand and renewable generation are uncertain, performance is often assessed using regret-like concepts that compare realized cumulative reward against reference policies. Although exact regret is difficult to compute in realistic simulators, empirical proxies are being used more frequently. These typically measure performance gaps relative to strong baselines over time. Reviews of safe RL further emphasize that deployment depends not only on achieving high reward, but also on maintaining reliable training behavior, avoiding catastrophic exploration, and generalizing robustly beyond the scenarios encountered during training [13].

For deployment-oriented evaluation, robustness should also capture sensitivity to non-ideal data conditions rather than focusing only on reward stability or convergence variance. In practice, MARL controllers may be exposed to missing observations, noisy measurements, delayed communication, stale forecasts, and estimator errors that were not present during training. As a result, a method that performs well under clean simulation conditions may degrade substantially when transferred to realistic cyber–physical environments. This observation suggests that future benchmark protocols should include stress tests under observation noise, packet loss, delayed state updates, and partial data dropout, alongside conventional reward and constraint metrics [39].

3.4.2. Grid Metrics (PAR, Voltage Deviation, Operating Costs)

Power-system metrics depend on the application. In EV charging: peak load reduction, load factor improvement, energy cost savings, and user measures like charging delay and satisfaction. In distribution control, voltage violations are avoided, voltage deviation is reduced, and network losses are reduced.

Peak-to-average ratio (PAR) is standard for demand shaping:

$$\mathrm{PAR}={\displaystyle \frac{{max}_t{P}_{\mathrm{agg}}\left(t\right)}{\frac{1}{T}{\sum }_{t=1}^T{P}_{\mathrm{agg}}\left(t\right)}},$$

(4)

where $P_{\mathrm{agg}}\left(t\right)$ is aggregate power (e.g., total EV charging load) over horizon T. Lower PAR indicates flatter load profiles and reduced stress on upstream assets. Recent 2025 EV frameworks report grid-aware reductions in peak variability and improved congestion management via coordinated charging and allocation [4,7].

Voltage quality measures via maximum deviation, average deviation, or violation counts:

$$\mathsf{\Delta }{V}_{max}=\underset{t,n}{max}|V_n\left(t\right)-V_{\mathrm{ref}}|,N_{\mathrm{viol}}=\sum _{t,n}\mathbb{I}\{V_n\left(t\right)\notin [V_{min},V_{max}]\}.$$

(5)

Because violations are safety-critical, recent 2025 methods incorporate safety layers or constrained formulations to reduce violations while maintaining performance [13,14,37]. When interpretability is required (operator auditing), explainable policy extraction (decision trees) and prototype-based explanations emerge as complementary evaluation dimensions [17].

Operating cost metrics include energy purchase cost, demand charges, or penalty-weighted costs (violations, battery degradation), reported relative to uncoordinated baselines. User-centric metrics include mean charging delay, service-level satisfaction, and QoS. Recent 2025 community-scale studies report cost reductions and autonomy improvements when coordinating EV (V2G) actions with shared storage and local generation [36].

Finally, computational efficiency (wall-clock runtime per control step, training time) and communication overhead (message size/rate, topology) matter for real-time deployment. These considerations are especially important in safety-constrained settings where monitors or projections are applied online [14,40].

MARL offers a robust framework for decentralized decision-making in smart grids. Typically, implementations begin with training (often via simulation), followed by deployment, in which agents operate in real time using local observations. System success is measured by both technical (reliability, constraint compliance) and economic (cost, market outcomes) criteria. Building on these foundations, we examine state-of-the-art MARL applications for urban EV–grid coordination, then explore how forecasting, safety, explainability, and blockchain technologies address current limitations.

Table 2 summarizes these paradigms, highlighting applications in smart grids and citing representative works. This taxonomy organizes the diverse MARL approaches in the literature. With these foundations established, we explore state-of-the-art MARL applications for urban EV–grid coordination and examine how forecasting, safety, explainability, and blockchain technologies address current system challenges.

Table 2. Comparison of MARL paradigms in smart grid coordination.

Paradigm	Description	Smart Grid Use-Case	Notable Work
Independent Learning (Decentralized MARL)	Each agent learns independently, treating others as environment components; simple but prone to non-stationarity.	Decentralized EV charging, where agents minimize individual cost without coordination.	[41]
Centralized Training, Decentralized Execution (CTDE)	Joint training with global critic/state; decentralized execution with local observations.	Coordinated EV charging and grid-aware allocation under joint constraints.	[7,42]
Value Function Factorization	Decomposes joint reward into individual agent value functions (e.g., QMIX-like factorization).	EV coordination to flatten load or maximize collective cost savings.	[43]
Competitive/Game-Theoretic MARL	Agents pursue conflicting objectives and adapt to others (e.g., equilibrium concepts, opponent modeling).	EV aggregators in energy markets; peer-to-peer energy trading.	[44]
Hierarchical & Hybrid Approaches	Combines paradigms with upper-level agents coordinating lower-level MARL agents.	Two-level EV management: system-level allocation and local dispatch; resilience and QoS recovery.	[4,45]
Safety-Constrained MARL/Shielded RL	Adds constraint handling via safety layers, projections, or CMDP-style optimization to ensure safe actions.	Grid-safe EV charging (peak limits) and safe distribution control (voltage/thermal limits).	[13,14,37]
Blockchain-Enhanced/Trust-Aware MARL	Augments decentralized coordination with trusted logging and automated settlement (smart contracts) or secure P2P interaction.	Trustworthy P2P coordination and decentralized demand response with secure transactions.	[22,23]

4. MARL Applications in Urban Energy: The State of the Art

This section reviews how MARL has been applied to coordinate EV charging and energy use in urban power grids. It covers studies published between 2023 and 2025 across four major EV–grid interaction layers: (1) demand-side management, (2) smart community coordination, (3) economic transactive layers, and (4) ancillary grid services. For each layer, the discussion summarizes recent studies with attention to control objectives, system models, agent design, reward formulation, and reported performance. Comparative tables are used throughout the section to highlight algorithmic paradigms, scalability characteristics, and evaluation metrics.

The reviewed studies span a broad range of application settings, including EV charging coordination, community energy management, transactive energy systems, and ancillary grid services. Across this literature, several methodological patterns appear consistently. The most common include CTDE-based actor–critic architectures, simulation-based evaluation on synthetic or benchmark environments, and reward functions that balance economic objectives with grid-support and user-service criteria. The main characteristics of representative studies are summarized in the comparative tables presented throughout this section.

4.1. Demand-Side Management: EV Charging and Load Balancing

Within demand-side management, recent MARL studies typically model each EV or charging point as an individual agent that schedules charging and discharging to reduce energy costs and smooth aggregate demand. In these formulations, agents observe local states, such as remaining charging demand, time windows, local PV generation, and dynamic prices, and then select charging power accordingly.

Yang and Tang (2025) model each public and home charging site as a cooperative agent trained using MADDPG [8]. Their policy reduces peak-hour load variability by approximately 40% and threshold violations by approximately 37%. These results illustrate the ability of coordinated MARL to mitigate stress at the distribution level. Fan et al. (2023) use independent charger agents together with LSTM-enhanced MADDPG critics to capture temporal correlations in PV output and EV arrivals [46]. Their reward function combines energy-cost minimization with a charging-satisfaction penalty to support fairness. Fan et al. (2024) extend this framework by introducing noisy exploration to improve fault tolerance under partial communication or sensing failures [47].

Related work has also explored more decentralized and hierarchical designs. Korkas et al. (2024) propose a fully decentralized DDPG architecture in which each charging spot minimizes individual energy cost and delay while still respecting grid limits [48]. Their framework achieves an energy-cost reduction of approximately 25% and improves user satisfaction relative to centralized baselines. Bachiri et al. (2023) divide the city into regions and assign DDPG agents at the regional level [49]. These agents use ITS, EV, and charging-station information to coordinate local recommendations. This hierarchical structure, shown in Figure 3, improves scalability while also balancing load across districts.

Figure 3. Multi-agent EV charging scheduling architecture in which regional agents coordinate local charging stations and EVs through ITS to balance grid load and reduce system cost, adapted from [49]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

Further advances emphasize spatiotemporal coordination and adaptation to changing urban conditions. Zhang et al. (2023) introduce RLCharge, a spatiotemporal actor–critic MARL framework for recommending public charging stations to EV drivers [50]. Their CTDE formulation uses a graph-based attentive critic to balance station loads and waiting times. Evaluations on real-world data show substantial improvements in both demand satisfaction and station utilization. Jamjuntr et al. (2024) develop an adaptive MADDPG algorithm that continuously fine-tunes policies as stations are deployed or as demand patterns evolve [6]. Field experiments report an approximate 15% reduction in charging cost under dynamic urban conditions.

The comparative characteristics of MARL-based EV charging coordination studies are summarized in Table 3. Most approaches employ CTDE-based actor-critic methods, incorporate fairness or delay penalties into reward design, and emphasize scalability through decentralized execution. Decentralized DDPG variants prioritize privacy and agent independence, while graph-attention critics enhance spatial coordination.

Table 3. Comparison of MARL-based EV-charging coordination studies.

Paper	Agents (Observations)	Algorithm/Paradigm	Control Objective
[46]	EV chargers with PV (SoC, PV output, prices)	MARL (LSTM-MADDPG, CTDE)	Minimize station energy cost and charging delay (fairness)
[47]	EV chargers (same setup)	MARL (MADDPG, CTDE)	Same as above, adding fault tolerance via noisy exploration
[48]	Individual charging spots (local EV SoC, grid price)	Decentralized independent DDPG agents	Minimize per-EV energy cost and delay subject to grid limits
[50]	Public stations (local demand, neighbor loads)	CTDE actor-critic with attentive graph critic	Recommend stations to balance load versus waiting time
[6]	Mobile EV fleet (aggregate network state)	Adaptive MADDPG (online fine-tuning)	Minimize network energy cost and peak load over time

4.2. Smart Community Coordination: Prosumers and DER Integration

In smart communities, MARL agents typically represent buildings or prosumers that manage flexible loads, including HVAC systems, household appliances, EV chargers, and DERs. The main objective in these settings is to smooth community demand profiles while minimizing total operating cost and maintaining occupant comfort.

Recent studies illustrate several ways this objective has been addressed. Wilk et al. (2024) propose LSD-MADDPG, in which agents exchange discretized strategic information during training but operate entirely on local information during execution [51]. In their framework, each building agent optimizes HVAC setpoints and EV charging schedules so that comfort constraints remain satisfied while contributing to a global flat-load objective. Experimental results show reductions in community peak costs and demand variance, while also preserving privacy.

Related work has further explored richer coordination mechanisms across buildings. Xie et al. (2023) introduce an attention-based MADRL approach for multi-building demand response [52]. In this framework, each building agent controls heating, cooling, and flexible appliances, while a shared attention-based critic identifies the most important cross-building interactions. Simulation results indicate an approximate 6% reduction in peak demand relative to independent RL baselines.

This line of research has also been extended to broader community-scale energy systems. Hribar et al. (2025) expand the paradigm to Positive Energy Districts through CoMAD-V2G [36]. In their PED setting, households with local demand, solar generation, and V2G-capable EVs are coordinated together with a shared community ESS. The aim is to improve renewable-energy utilization and strengthen district-level energy autonomy. The integrated community microgrid structure is illustrated in Figure 4. In the experimental evaluation, CoMAD-V2G achieves approximately a 25% reduction in household electricity costs and significantly increases local renewable utilization.

Figure 4. Smart community microgrid integrating DERs, shared storage, EV chargers, and coordinated building agents under a MARL-based energy management system adapted from [53]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

Table 4 summarizes representative MARL applications in smart community energy management. CTDE-based actor-critic formulations dominate, with rewards combining cost minimization, comfort preservation, and load-variance penalties across community scales ranging from five to thirty-two buildings.

Table 4. Comparison of MARL applications in smart community and building energy management.

Paper	Agents	Algorithm/Paradigm	Control Objectives	Reward	Metrics	Scenario
[51]	Buildings (indoor temperature, EV SoC)	LSD-MADDPG (CTDE)	Minimize total cost	Cost + comfort + load variance	Community cost, load flatness	10-home neighborhood
[52]	Buildings (power demand, temperature)	Attention-based MADRL	Minimize cost + DR compliance	Cost + comfort + DR deviation	Peak demand reduction	5-building microgrid
[36]	Houses (PV, shared ESS, V2G EV)	CTDE actor-critic (CoMAD-V2G)	Maximize renewables use + autonomy	Cost + V2G utilization	Energy cost, autonomy	Positive Energy District (32 houses)

4.3. Economic Layers: Transactive Energy and P2P Trading

Transactive energy systems introduce peer-to-peer (P2P) energy markets in which EV owners and prosumers learn bidding and trading strategies. This layer extends MARL beyond physical coordination and into market-based decision-making, where pricing, allocation, and strategic interaction become central design concerns.

Recent studies illustrate several forms of this transition. Han et al. (2024) develop a hierarchical MARL auction framework in which EVs bid for charging slots at either fixed or mobile stations [31]. In their formulation, each EV agent optimizes its charging utility together with its share of SoCial welfare. This design produces higher total welfare and lower latency than traditional auction mechanisms.

Related work has also examined P2P coordination in rural and microgrid settings. Shah et al. (2024) study a rural dairy-farm microgrid that supports peer-to-peer energy trading among prosumers equipped with DERs [54]. Their learning-based coordination framework achieves approximately 43% lower electricity costs and 42% lower peak demand relative to the baseline, while also improving energy-sale performance.

Other studies focus more directly on pricing mechanisms within community markets. May and Huang (2023) apply a CTDE actor–critic framework to dynamic P2P pricing in a simulated community energy market [55]. Their learned real-time pricing policy reduces community electricity costs and increases self-consumption. As a result, net profit improves by approximately 28.6%.

More recent work has begun to incorporate foundation-model guidance into this setting. Lou et al. (2025) integrate large language models (LLMs) into MARL-based P2P trading [56]. In their framework, LLM-generated expert strategies guide policy learning through imitation, while a differential-attention critic improves coordination across large agent populations. Their LLM-augmented MARL method achieves lower total system costs and fewer voltage violations than conventional MARL baselines.

Figure 5 illustrates a representative hierarchical transactive energy architecture in community-based P2P markets. The upper layer represents the P2P market-clearing mechanism, where a global objective is optimized subject to system-level constraints. The lower layer consists of community aggregators coordinating multiple households with demand, PV, and energy storage. Learning agents typically operate at the household or aggregator level, where bidding, pricing, and negotiation policies are optimized via decentralized or CTDE-based MARL frameworks. This hierarchical structure provides the economic foundation for auction-based, Q-learning, actor-critic, and LLM-augmented strategies.

Figure 5. Hierarchical P2P energy trading framework adapted from [57]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

Table 5 summarizes these representative approaches.

Table 5. Comparison of MARL approaches for transactive and P2P energy trading.

Paper	Agents	Paradigm	Objective	Reward/Mechanism	Metrics	Environment
[31]	EV owners (bidding agents)	Hierarchical MARL auction (MADDPG)	Balance supply–demand and minimize waiting time	Auction payoff (cost + welfare share)	SoCial welfare, latency	Simulated urban EV charging network
[54]	Farm prosumers (PV + loads)	Decentralized RL (independent Q-learning)	Maximize profit and peak shaving	Trade profit (energy cost)	Electricity cost, peak load	P2P rural microgrid simulator
[55]	Prosumer community	CTDE actor-critic (dynamic pricing)	Minimize community cost	Community cost, self-sufficiency	Cost, self-consumption	Simulated P2P market
[56]	Prosumer grid (EVs, homes)	LLM-assisted MARL (CTDE)	Minimize total cost and voltage violations	Imitation + attention critic	Economic cost, voltage compliance	Real-time P2P market model

4.4. Ancillary Services: Frequency Regulation and Voltage Support

Ancillary services—including frequency regulation and voltage support—represent an important and emerging application area for MARL in EV–grid coordination. In contrast to demand-side scheduling, these services require responses that are fast, reliable, and constraint-aware when system-level disturbances occur. EV fleets, charging stations, and inverter-based distributed resources are therefore increasingly regarded not only as flexible loads but also as active assets that can contribute to grid stability [58].

In conventional power-system operation, primary frequency control is commonly implemented through droop-based mechanisms, in which power output is adjusted in proportion to frequency deviations. Droop control is simple, physically interpretable, and well-suited to rapid local response. However, it remains inherently rule-based and does not adapt easily to changing user behavior, stochastic EV availability, battery constraints, or the broader objectives of multi-agent coordination. By contrast, RL- and MARL-based approaches can learn adaptive policies that respond to grid signals while jointly accounting for state-of-charge limits, charging requirements, battery degradation, and coordination among distributed resources [59].

This distinction is important from a control-theoretic perspective. MARL should not be viewed as a direct replacement for the fast stabilizing function of conventional primary control. In practical power-system deployment, droop-based and other physics-based local controllers remain responsible for sub-second voltage and frequency response, as they provide a transparent structure and well-established stability behavior. MARL is more appropriately positioned as a supervisory or coordination layer that updates setpoints, charging flexibility, participation factors, or operating envelopes over slower timescales. This layered interpretation preserves timescale separation: fast inner-loop dynamics remain under conventional control, while MARL addresses slower adaptive decision-making under uncertainty [60].

Within this layered framework, recent studies illustrate how MARL can support ancillary-service coordination. For frequency regulation, Wang et al. (2024) propose a fully decentralized MARL framework based on inference-network soft actor–critic (IN-MASAC) with blockchain coordination [58], in which EV agents track frequency-regulation signals without directly sharing private user data, supporting both scalable and privacy-preserving regulation. More broadly, this line of work shows how MARL can move beyond fixed control laws by learning regulation policies that better balance grid-support performance with local operational constraints.

Voltage support presents a related but distinct challenge, particularly in active distribution systems with high penetrations of EV charging and photovoltaics. Unlike frequency regulation, the task is not merely to follow a single global setpoint but to coordinate distributed reactive-power or volt/var actions across spatially coupled nodes. Long et al. (2025) address this problem using multi-agent deep RL for decentralized voltage regulation in active distribution networks with heterogeneous PV inverter modes [61]. By partitioning the network into sub-networks and coordinating local controllers, their method improves voltage regulation under heterogeneous inverter behavior and varying network conditions.

Overall, ancillary-service applications remain at an earlier stage of MARL development than demand-side management or local energy trading. Even so, the literature indicates strong potential for EV fleets and other distributed agents to support grid functions while maintaining user-level feasibility. The key algorithmic distinction is that conventional methods, such as droop control, offer stable and interpretable baseline responses, whereas RL and MARL provide adaptive coordination under uncertainty and competing objectives [62]. Accordingly, future research should emphasize hybrid formulations in which physics-based control structures provide safety and interpretability while MARL improves adaptation, coordination, and economic efficiency. Additional challenges include preserving battery health, achieving sufficiently fast response for ancillary-service markets, handling communication delays, and integrating renewable and load forecasting into multi-agent control design [58].

Table 6 summarizes representative ancillary-service studies and compares MARL-based approaches with a conventional droop-based baseline in terms of agents, control paradigm, objectives, and evaluation setting.

Table 6. Comparison of MARL approaches for ancillary services in EV–grid coordination.

Paper	Agents	Paradigm	Objective	Control/Reward Mechanism	Metrics	Environment
[58]	EVs as distributed regulation agents	Decentralized MARL (IN-MASAC) with blockchain coordination	Track frequency-regulation signals while preserving privacy and scalability	Frequency-tracking reward with decentralized policy learning under EV constraints	Frequency-tracking accuracy, privacy, scalability	EV fleet frequency-regulation setting
[61]	Local volt/var controllers in partitioned subnetworks	Multi-agent deep RL	Maintain voltage within allowable limits under heterogeneous PV inverter modes	Local voltage-regulation reward with decentralized coordination across subnetworks	Voltage deviation, regulation performance	Active distribution network with high PV penetration
Conventional baseline	Inverters/ distributed resources	Droop-based local control	Stabilize frequency or voltage via proportional response	Fixed rule-based droop law	Stability, response speed, simplicity	Primary frequency control/local volt/var support

4.5. Mapping Application Domains to Core MARL Dimensions

Although the four application domains reviewed in this section share common MARL foundations, the relative importance of forecasting, safety, explainability, and trustworthy decentralized coordination varies across deployment scenarios. To make these relationships explicit, Table 7 maps the four major EV–grid application domains to the four core research dimensions of this survey, indicating where each dimension is most strongly demanded and where multi-dimensional integration is especially important for practical deployment.

Table 7. Application-domain–research-dimension mapping across major EV–grid coordination scenarios.

Application Domain	Forecasting-Informed MARL	Safety-Constrained MARL	Explainability/ Interpretability	Trustworthy Decentralized Coordination	Integration Priority/Practical Implication
Demand-side management/EV charging and load balancing	High	Medium–High	Medium	Low–Medium	Forecasting is important for anticipating arrivals, charging demand, and dynamic prices, while safety becomes relevant when feeder or transformer limits are binding. Multi-dimensional integration is especially valuable where coordinated EV charging must remain economically efficient, grid-aware, and operationally feasible under uncertainty.
Smart community coordination/prosumers and DER integration	High	High	Medium	Medium	This domain places strong simultaneous demand on forecasting and safety because local generation, storage, EV flexibility, and community demand must be coordinated under physical network and comfort constraints. Integration strategies often require combining predictive control, safe dispatch, and interpretable coordination across heterogeneous assets.
Economic layers/transactive energy and P2P trading	Medium	Medium	Medium	High	Trustworthy decentralized coordination is especially important because market participation, settlement, and multi-actor interaction require transparency, accountability, and resistance to strategic manipulation. Forecasting and explainability remain relevant for pricing, bidding, and auditability, while safety is still needed when market decisions affect physical network feasibility.
Ancillary services/frequency regulation and voltage support	Medium	Very High	Medium	Medium	This domain places the strongest emphasis on safety and technical suitability because voltage and frequency support are tightly linked to real-time operational limits and control timescales. Multi-dimensional integration is needed when fast response, constraint satisfaction, forecasting support, and operator trust must be balanced in grid-support applications.

As shown in the table, the four MARL dimensions are not equally emphasized across application domains. Demand-side and community-energy settings place particular emphasis on forecasting and safety; transactive energy gives greater prominence to trustworthy, decentralized coordination; and ancillary-service applications place the highest priority on safety and real-time constraint compliance. This comparison also highlights that practical deployments will often require integration across multiple dimensions rather than optimization along a single methodological axis.

5. Enhancing the Input Stage: Forecasting-Informed MARL

Forecasting uncertain variables, including EV charging demand, distributed renewable generation, and electricity prices, is essential for MARL in EV–grid energy management. Forecasts provide anticipatory context, such as expected demand peaks, price spikes, and renewable shortfalls. This enables agents to schedule charging and discharging proactively rather than relying only on present observations [5,63]. Recent findings further support this role. Prediction-informed pricing and control improve economic performance while reducing load volatility, and transformer-based long-horizon temporal modeling has been shown to substantially reduce system-level charging costs [9,64]. At the same time, the value of forecasting depends strongly on forecast quality, particularly for renewable generation, where multi-site spatiotemporal dependencies and weather-bias correction are critical [65]. Importantly, forecasting is not merely a predictive aid; it also serves as a cross-cutting input to the other dimensions of this review. Forecast uncertainty affects safety margins, influences the interpretability of anticipatory actions, and shapes how decentralized agents coordinate when future information is incomplete [13,66].

This broader integration is illustrated in Figure 6. In that framework, forecasted information, including price scenarios, flexibility requests, and ancillary-service needs, feeds into the optimization and risk-management layers of an aggregator. This structure supports proactive EV charging and more effective grid coordination [67].

Figure 6. Forecasting-informed energy management architecture adapted from [67]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

5.1. Mitigating Stochasticity: Integrating Predictive Analytics into the State Space

Integrating predictive analytics into the state space is a practical strategy for EV–grid MARL. In these settings, key disturbances, such as vehicle arrivals, energy requests, PV and wind generation, and electricity prices, are exogenous to the agents. One effective way to mitigate this uncertainty is to embed predictive information directly into the state or belief representation. In practice, agent observations can be augmented with (a) multi-horizon point forecasts, (b) probabilistic forecasts such as quantiles or intervals, and/or (c) scenario sets that represent plausible future trajectories. This transforms uncertainty into structured inputs and allows agents to learn policies conditioned on anticipated near-term congestion as well as longer-term trends [5,63].

The effectiveness of this approach, however, depends strongly on forecast quality. When forecast fidelity is inadequate, the resulting control decisions can be systematically misled. For this reason, improving upstream forecasting models remains essential for reliable forecast-augmented state representations. This includes advances such as multi-site PV forecasting with spatiotemporal learning and the correction of biased weather inputs [65].

5.2. Methodological Approaches: Hybrid Forecast–RL Architectures

Two main integration paradigms dominate this area, although recent systems increasingly combine elements of both. One approach embeds forecasting implicitly within sequential MARL architectures. In this setting, recurrent encoders such as LSTM models, or transformer-based attention modules, learn temporal features directly from historical sequences, so that internal representations become predictive without a separate forecasting stage [46,64]. Transformer-based temporal modeling has shown particular value for EV charging demand prediction [68]. Hierarchical memory-aware transformers further illustrate how predictive sequence modeling can be extended to multi-agent settings [69]. Large-scale multi-station scheduling has also benefited from transformer-based temporal feature extraction and attention-based information exchange, which help capture long-term dependencies [64].

A second approach relies on an explicit forecasting stage, in which a dedicated forecasting model first generates predictions and RL or MARL controllers then use these forecasts—ideally together with uncertainty descriptors—to optimize actions [70]. For example, separate forecasting and control modules can be paired for real-time pricing and charging coordination: short-term demand prediction in Stage 1 is followed by a control policy in Stage 2 that adapts pricing or charging schedules to avoid peaks and improve reliability [9]. Two-stage designs are especially common in aggregator-centric formulations, where long-horizon planning is coordinated with fast short-horizon corrective control and forecast deviations are explicitly accounted for [63,71].

An important design decision concerns how forecast uncertainty is represented before being passed to the MARL controller. Quantile forecasts provide multiple conditional levels of the predictive distribution, allowing agents to distinguish among optimistic, median, and conservative operating conditions. Prediction intervals offer a more compact description of uncertainty width that may be easier to encode in the agent state, but can conceal multimodal structure. Scenario-based forecasts are often better suited to planning-oriented MARL because they preserve multiple plausible future trajectories and allow agents to learn policies that remain robust across alternative system evolutions. Particle-based representations offer an even richer approximation of the predictive distribution, though at the cost of increased state dimensionality and computational burden. In practice, the chosen form of uncertainty representation affects not only state design but also exploration behavior, robustness to forecast error, coordination efficiency, and training stability [66,72].

Forecast horizon selection is similarly application-dependent. Very short horizons are most useful for fast operational tasks—including real-time charging coordination, local voltage regulation, and frequency-related support—where controllers must respond quickly to near-term fluctuations in demand or renewable generation. Medium-range horizons are often more relevant for price-responsive charging, congestion mitigation, and feeder-level coordination, as they allow agents to anticipate demand ramps and renewable variability several steps ahead. Longer horizons become more valuable in hierarchical or aggregator-based settings, where day-ahead planning, flexibility allocation, and multi-timescale scheduling must be coordinated with shorter-term corrective control. No single forecast horizon is therefore optimal across all cases; effective forecast-informed MARL instead depends on aligning the prediction horizon with the temporal scale of the control objective [9,73].

These considerations leave several open design choices at the forecast–policy interface, including forecast-horizon alignment, the encoding of uncertainty through quantiles, intervals, scenarios, or particles, and whether forecasts should remain agent-local or be shared under CTDE. Hybrid designs that combine explicit forecasts with auxiliary predictive losses, safety layers, or hierarchical multi-timescale control are increasingly used to stabilize learning and improve transferability [64,74]. More broadly, future research should examine how uncertainty representation and horizon selection jointly influence policy robustness, sample efficiency, and generalization across EV–grid applications.

5.3. Benefits: Improved Foresight and Proactive Uncertainty Management

Forecast-informed MARL consistently enhances anticipatory decision-making. It enables agents to (i) shift charging toward periods with lower cost and lower grid stress, (ii) hedge against variability in renewable generation and demand, and (iii) coordinate across multiple charging stations or fleets.

These benefits are reflected in both economic and operational performance. From an economic perspective, long-horizon temporal modeling substantially reduces total charging costs in collaborative multi-station settings. Forecast-driven pricing and control also increase revenue while reducing load variance [9,64]. From an operational perspective, safety-aware RL demonstrates robust constraint satisfaction under substantial EV uncertainty, including transformer thermal stress as well as voltage and line limits. This strengthens the case for more reliable real-world deployment [74]. Recent RL scheduling methodologies further emphasize the importance of validation on realistic grid models and explicitly target overload mitigation at critical nodes alongside economic objectives [75].

5.4. Open Issues: Data Quality, Latency, and Forecast Errors

Despite these performance gains, several challenges remain. The first concerns the propagation of forecast errors into control decisions. When predictions are inaccurate, profits can decline, and unsafe actions may occur. This makes robust control essential, whether through uncertainty-aware state augmentation, risk-sensitive objectives, and/or hard safety constraints [63,74]. A second challenge is latency and computational burden. These costs increase when deep forecasting models and attention-based MARL policies must run online, particularly in large-scale systems with limited communication resources [64]. A third challenge relates to data quality and integrity. Forecasting and control depend on telemetry that may be missing, delayed, or maliciously manipulated. Cyber-resilience studies have emphasized that EV charging infrastructure is inherently cyber–physical, and that cyber attacks can produce disruptive load fluctuations that require both assessment and mitigation [76]. Representative examples include false data injection and communication-link hijacking. In false data injection attacks, manipulated measurements—such as EV state of charge, charger demand, local load, renewable generation, or price signals—can distort the information available to forecasting and control modules, potentially leading to incorrect charging schedules, ineffective congestion management, or unsafe operating decisions [77]. In communication-link hijacking, adversaries may delay, block, or alter messages exchanged among chargers, aggregators, and coordination platforms, degrading synchronization, reducing the effectiveness of distributed decision-making, and increasing the risk of infeasible or unstable charging behavior under real-time operating conditions [78].

These issues indicate that forecasting-informed MARL must do more than improve predictive performance alone. It must also (i) quantify and communicate forecast uncertainty, (ii) respect real-time execution constraints, and (iii) incorporate resilience mechanisms for low-quality or compromised data.

6. Safe MARL for Power Systems

Safe operation is a fundamental requirement in modern power grids, where voltage, frequency, thermal, and capacity limits must be maintained at all times [79]. Reinforcement learning offers a data-driven approach to power-system control. However, its trial-and-error learning process can produce unsafe decisions that violate critical operational constraints [13]. This challenge has motivated growing interest in safe reinforcement learning (SafeRL) and its multi-agent extension, SafeMARL. SafeRL explicitly incorporates safety constraints into the learning process so that agents avoid dangerous states and actions [13]. Most existing SafeRL research, however, has focused on single-agent settings, while SafeMARL remains an emerging field with many unresolved problems.

Against this background, this section reviews the primary safety requirements in grid applications and the methodologies for their enforcement in RL and MARL. We examine the inherent complexities of multi-agent settings, survey representative case studies, and identify critical challenges such as scalability, generalization, and formal safety guarantees. As illustrated in Figure 7, da common architecture in recent literature—exemplified by Chen et al. [80]—employs an HOCBF-based safety layer to intercept candidate actions from the RL controller. By filtering or projecting these actions prior to environment execution, operational constraints are systematically enforced. This design prevents unsafe transitions while preserving the integrity of the underlying policy learning process [80]. In practice, safe MARL is inextricably linked to the other dimensions of this review: safety constraints are often conditioned on predictive uncertainty, intervention mechanisms influence the interpretability of agent behavior, and trustworthy deployment requires that safety layers remain auditable across decentralized coordination protocols [13,81].

Figure 7. Safe MARL control loop with an HOCBF-based safety layer, adapted from [80]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

6.1. Defining Physical Boundaries: Grid Operational Constraints

Smart grids operate under strict instantaneous and cumulative constraints on key system variables. For example, bus voltages and line currents must remain within fixed limits at all times [82]. Frequency deviations must also remain small in order to preserve rotor stability, while line thermal ratings and transformer capacity limits cannot be exceeded. The growing presence of distributed resources introduces additional constraints, including battery and ESS state-of-charge limits, charge and discharge rate limits, solar inverter power limits, and EV charger power limits. Even a single violation may damage equipment or initiate cascading failures [83]. Typical safety constraints include:

• Voltage limits: for example, 0.95–1.05 pu on distribution feeders; these limits must hold at every time step.
• Frequency bounds: a tight range around 50/60 Hz is required to maintain stability and prevent load shedding.
• Thermal (current/flow) limits: lines and transformers have maximum current or MVA ratings that must not be exceeded.
• State-of-charge bounds: batteries and EVs have upper and lower limits needed for safety and longevity, including the avoidance of overcharging and deep discharge.
• Power balance and generation limits: each generator, storage unit, or grid interface has minimum and maximum output levels as well as ramp-rate limits.

Many of these requirements are hard, or almost-sure, constraints, meaning that they must be satisfied instantaneously [84]. For instance, bus voltage must remain within the safe range at every time step. This requirement has motivated recent work on safety-aware control in RL settings. Recent algorithms developed in 2025, such as DistFlow Safe RL, address voltage regulation in distribution networks through constrained Markov decision processes (CMDPs). These methods optimize flexibility while still adhering to operational constraints and balancing efficiency with safety. In a related direction, model-based safe RL frameworks incorporate model-based safety modules (MBSMs) into the RL loop. This allows the real and reactive power outputs of controllable DERs to strictly satisfy distribution-network security constraints, without relying on expert knowledge. More broadly, power-system surveys continue to emphasize that safe control must respect constraints at all times [83]. Table 8 summarizes the common smart-grid safety limits used as constraints in RL and MARL formulations.

Table 8. Comparison of SafeRL techniques.

Technique	Formulation	Pros	Cons	References
CMDP/Constrained RL	Constrained MDP with joint state–action and cost constraints	Formal guarantees in expectation; flexible for different safety criteria; grounded in RL theory	Solving constrained problems is complex; convergence may be slow; often centralized; constraints are often satisfied only approximately	[85,86]
Lagrangian/Primal–Dual	Dual-based policy optimization with constraint multipliers	Naturally handles inequality constraints; extendable to MARL through decentralized dual updates	Non-convexity can cause instability; sensitive to dual learning-rate tuning; lacks strict safety guarantees	[87]
Barrier Function/Reward Penalty	Interior-point or penalty-based augmentation in reward design	Simple to implement; encourages constraint satisfaction; relatively low computational cost	No hard guarantees; violations may still occur; performance is sensitive to penalty-weight selection	[84,88]
Safety Layers/Shields	Online action correction using MPC or projection onto a safe set	Can guarantee constraint satisfaction; model-driven; may operate independently of the learned policy	Can override optimal behavior; requires a system model or verification tools; guarantees are sensitive to model mismatch; may require centralized coordination	[89]
Risk-Sensitive/Robust RL	Chance-constrained or minimax CMDP under uncertainty	Protects against worst-case scenarios; can bound risk or violation probability	Often conservative; optimization is more complex; may require modeling adversarial disturbances	[13]

6.2. Taxonomy of Safety Enforcement in MARL

6.2.1. Constrained MDP and Lagrangian Methods

A standard SafeRL formulation models the control problem as a Constrained Markov Decision Process (CMDP). In this setting, the conventional MDP is extended with one or more cost functions, together with bounds on their cumulative expected values. Common solution approaches include Lagrangian or primal–dual optimization, trust-region methods such as CPO, and projection-based updates [86]. More recent work has further strengthened this line of research. In 2025, the Natural Policy Gradient Primal–Dual (NPG-PD) method was shown to achieve global convergence with sublinear rates for constrained MDPs under nonconcave objectives. In parallel, the Multi-Agent Safe Transformer (MAST) introduced the Multi-Agent Total Advantage Decomposition Theorem, thereby linking sequence modeling with safety-critical MARL. MAST allows agents to incorporate both safety- and action-related decisions from preceding team members and achieves up to an 83.58% reduction in safety costs during training.

Beyond formal CMDP-based methods, simpler approaches based on reward shaping and penalty design are also widely used. In these formulations, the learning objective is modified to maximize reward minus $\lambda \times$ constraint violation, where $\lambda$ is either tuned manually or learned. Barrier-function methods, including interior-point formulations, impose increasingly severe penalties as the policy approaches unsafe boundaries [86]. For example, the Interior-Point Optimization (IPO) method augments the reward with a logarithmic barrier term. As the state approaches a constraint boundary, the penalty grows without bound, forcing the learned policy to remain within the safe region. This form of reward shaping is common in power-system applications. Vu et al. (2021) report that barrier-based approaches, such as IPO, have been widely applied in power systems to support constraint satisfaction [90]. In RL settings, barrier losses push policies away from unsafe states, although the choice and tuning of barrier terms still require domain knowledge. Negative rewards or continuous penalty terms applied near operational limits serve a similar purpose [87].

Recent work has also extended these ideas more directly into multi-agent settings. Zhang et al. (2024), for example, address MARL safety through scalable constrained policy optimization, in which safety requirements are enforced as explicit constraints during learning rather than through heuristic reward shaping [91]. In this formulation, agents coordinate under shared safety requirements while accounting for coupled constraints. Such constrained optimization encourages the learning of safer policies. However, practical safety still depends on how accurately the underlying constraints are formulated and how effectively they are enforced.

6.2.2. Safety Layers, Projection, and Shielding

To guarantee safety, many approaches rely on action correction through safety layers. In active voltage control, Shi et al. (2023) insert a safety layer between the MARL policy and the power-system environmen [89]. This layer corrects unsafe actions before execution and preserves voltage constraints during both learning and deployment. Recent reviews of safe RL in power systems classify safety layers and other external safety mechanisms as a major category for enforcing instantaneous constraints [13]. More recent shielding techniques introduced in 2025, such as ADVICE (Adaptive Shielding with Contrastive Autoencoder), distinguish between safe and unsafe state–action features and protect RL agents from hazardous outcomes without modifying the reward structure. Distributive shields operate in parallel with each agent and split or merge dynamically according to agent states in order to prevent unsafe visits.

However, the practical effectiveness of safety layers and shielding mechanisms depends critically on the fidelity of the model, constraint set, or verifier used to define the safe action set. When the shielding mechanism relies on simplified linearized dynamics, reduced-order power-flow approximations, or incomplete representations of coupled network behavior, an action classified as safe by the shield may still violate true system limits after deployment on the nonlinear physical grid. This model-mismatch problem is particularly acute in distribution systems characterized by strong nonlinearities, uncertainty, delayed measurements, and changing operating conditions. Under these conditions, shielding should not be interpreted as a perfect guarantee but rather as a model-contingent safety mechanism whose reliability depends on how closely the internal safety model represents the real environment [92].

6.2.3. Risk-Averse and Robust Optimization

Uncertainty in power-grid operation motivates the use of robust reinforcement learning techniques. Robust RL (RRL) seeks policies that perform effectively under worst-case conditions or uncertain system dynamics [93]. Chance-constrained RL addresses this issue by explicitly limiting the probability of constraint violations through probabilistic bounds [94]. More recent developments have extended these ideas further. In 2025, new algorithms were introduced to identify near-optimal policies in Robust Constrained MDPs (RCMDPs) by using epigraph reformulation to resolve conflicting gradients between objectives and constraints. In parallel, PRIMAL-CVaR emerged as a risk-aware MARL framework that learns full cost distributions and constrains tail-end risks through Conditional Value-at-Risk. These methods improve robustness by producing policies that remain safe under previously unseen disturbances.

The challenge becomes substantially more difficult in multi-agent settings. In these environments, safety constraints often couple the actions of multiple agents [91]. As a result, SafeMARL is considerably harder than single-agent SafeRL because safety is often a joint team property rather than an attribute of any one agent. At the same time, the joint state–action space grows exponentially with the number of agents, creating major scalability barriers [91,95,96]. These difficulties become even more pronounced when global observability is unavailable, since centralized safety mechanisms are often impractical in large systems characterized by local observations and communication limitations [95,96].

To address these issues, recent research has increasingly moved toward factored, local, or neighborhood-based safety enforcement. In these approaches, agents coordinate through structured local interactions rather than relying on monolithic centralized safety modules [91,95,96]. More broadly, MARL already faces core challenges such as non-stationarity, partial observability, credit assignment, and computational complexity. Once hard safety constraints are imposed, each of these difficulties becomes even more severe [97,98].

6.3. Applications: Enforcing Voltage Limits and Transformer Protection

Safe MARL has started to appear across several smart-grid application domains. Table 9 summarizes representative studies.

• Transformer Protection: Current transformer (CT) saturation distorts secondary currents and can lead to relay maloperation. Chothani and Kumar (2025) propose a dynamic differential protection scheme that detects both inrush and CT saturation using time-domain indices, including the waveform distortion measure (WDF) and current polarity index (CPI) [99]. Their method adaptively increases the differential characteristic according to the estimated saturation severity in order to maintain security during external faults. In a complementary approach, Key et al. (2025) propose a dual deep neural network based on stacked denoising autoencoders (SDAEs) [100]. In this framework, a compensation network mitigates CT-saturation distortions, while a discrimination network distinguishes internal faults from inrush conditions, thereby improving dependability under combined events.
• Active Voltage Control: Distributed voltage regulation in low-voltage grids can be formulated as an MARL problem in which each agent adjusts local reactive power. Qu et al. (2024) model this setting as a constrained Markov game and develop a multi-agent primal–dual algorithm in which each agent learns subject to voltage constraints [85]. Similarly, Shi et al. (2023) apply MARL to voltage control with a safety layer that projects actions violating voltage bounds onto the closest safe action [89]. This ensures that bus voltages remain within acceptable limits.
• EV Charging and DR: Safe MARL has also been applied to price-based demand response in residential systems. Markgraf and Althoff (2023) consider multiple household agents responding to dynamic prices and introduce a safety layer based on shrinking-horizon MPC [101]. This layer projects each agent’s action onto its local feasible set in order to enforce operational constraints such as battery and HVAC limits. In EV charging, RL has also been used to account for distribution-grid asset health. Li et al. (2022) propose an evolutionary curriculum learning-based multi-agent DRL framework for EV charging scheduling that jointly considers user requirements and distribution-transformer loss-of-life [102]. Their method improves the balance between customer satisfaction and grid resilience. Recent work continues in this direction through multi-agent deep RL for coordinated residential EV charging and discharging under demand-response and load-balancing objectives [103].
• Microgrid Energy Management: RL is increasingly used to coordinate distributed generators, storage, and coupled energy carriers in microgrids, while safe variants enforce both device-level and network-level operating limits. Wang et al. (2023) propose a physics-informed safe RL framework for multi-energy microgrids [104]. Their method combines a security assessment model with an action-correction mechanism to prevent violations of operational constraints during dispatch.
• Building Energy Systems: Safe RL has also been adopted more widely in building energy management to preserve comfort and reliability while reducing energy consumption. Wang et al. (2025) propose a safe DRL framework that combines deep RL with MPC, where MPC acts as a safety filter by screening candidate actions and enforcing constraints at each control step [105]. Using high-fidelity test cases, the framework improves constraint satisfaction and reduces both the duration and severity of violations while preserving strong overall performance. These characteristics make it well-suited for explicit safety-aware learning in building environments.

Table 9. Applications of SafeRL in power and energy systems.

Task	Safety Mechanism	Algorithm	Evaluation	References
Active Voltage Control	CMDP (voltage bounds) + Lagrange multipliers	Multi-agent primal–dual actor–critic (MACO-MA)	IEEE 33-bus and 123-bus networks; maintain voltages within ±5%; compared with unconstrained MARL	[85,89]
Price-based DR	Safety layer (MPC projection) + reward penalties	Decentralized MARL with safety filter	Residential EV charging; enforce battery SoC and grid limits; evaluate training speed and peak-load reduction	[101]
EV Charging Scheduling	CMDP (SoC and peak limits) or barrier functions	Centralized or decentralized MARL	Simulated EV fleet; enforce SoC and transformer-capacity limits; metrics include constraint violations and energy cost	[84]
Building Energy Management	Policy correction + constraint penalties	Multi-agent DDPG with correction network	Multi-zone commercial building; enforce comfort-zone and power constraints; compared with unconstrained RL	[13]

Table 9. Applications of SafeRL in power and energy systems.

Task	Safety Mechanism	Algorithm	Evaluation	References
Active Voltage Control	CMDP (voltage bounds) + Lagrange multipliers	Multi-agent primal–dual actor–critic (MACO-MA)	IEEE 33-bus and 123-bus networks; maintain voltages within ±5%; compared with unconstrained MARL	[85,89]
Price-based DR	Safety layer (MPC projection) + reward penalties	Decentralized MARL with safety filter	Residential EV charging; enforce battery SoC and grid limits; evaluate training speed and peak-load reduction	[101]
EV Charging Scheduling	CMDP (SoC and peak limits) or barrier functions	Centralized or decentralized MARL	Simulated EV fleet; enforce SoC and transformer-capacity limits; metrics include constraint violations and energy cost	[84]
Building Energy Management	Policy correction + constraint penalties	Multi-agent DDPG with correction network	Multi-zone commercial building; enforce comfort-zone and power constraints; compared with unconstrained RL	[13]

6.4. Challenges: The Trade-Off Between Safety and Optimality

Despite recent progress, SafeMARL continues to face several fundamental challenges. Scalability remains one of the most important issues because the joint state–action space and coupled safety constraints grow exponentially as the number of agents increases. To address this, recent approaches have moved away from fully centralized reasoning and instead use structured local interactions, graph-based aggregation, and constrained decentralized optimization to improve the trade-off between safety and performance in large-scale systems [91,95,106].

At the same time, safety enforcement often introduces conservatism. Shielding mechanisms and constrained optimization can prevent catastrophic actions, but they may also reduce the achievable reward or slow policy improvement when the intervention is overly restrictive [95,107]. Generalization is another open challenge. MARL policies already struggle with non-stationarity, partial observability, and coordination complexity, and these difficulties become even more severe when safety constraints must remain valid under unseen environments or changing agent populations [97,98].

Predictive safety and formal guarantees also remain limited. Although methods such as dynamic model predictive shielding improve the balance between task progress and provable safety, formal verification and shield synthesis for learned policies remain difficult in realistic high-dimensional settings. In many cases, they still rely on simplified models or factored approximations [95,107]. Model mismatch further complicates these guarantees. A shield may successfully block actions that are unsafe under its internal model while still failing to prevent actual violations if the real grid exhibits nonlinear dynamics, unmodeled couplings, parameter drift, topology changes, forecasting errors, or state-estimation uncertainty. Passing the shield, therefore, does not necessarily imply safety on the physical system, and residual violations may still arise in voltage, thermal loading, or transformer constraints despite nominally safe action filtering. This limitation motivates the development of adaptive or data-updated shielding, conservative safety margins, online feasibility re-checking, and post-action monitoring layers that can detect and correct residual risk when model-based safety filters are imperfect [92].

Overall, SafeMARL remains an emerging research area. The central question is how to achieve strong safety guarantees without sacrificing scalability, adaptability, and performance. Addressing this challenge will require a combination of scalable decentralized training, improved transfer and generalization, predictive control, and formal validation [13].

7. Explainable and Interpretable MARL for Smart Grids

The transition toward decentralized, renewable-driven energy systems has created a need for more sophisticated control architectures that can manage the stochastic, distributed, and increasingly bidirectional behavior of modern power grids. Within this context, MARL has emerged as an important paradigm for addressing these challenges. It enables autonomous agents to coordinate DERs, support demand response, and improve grid operation through decentralized decision-making. However, the black-box character of MARL remains a major obstacle to industrial adoption, especially in safety-critical power-system applications where operators must understand the reasoning behind control actions. For this reason, recent peer-reviewed work published in 2024–2025 has increasingly incorporated explainability into learning-based grid-control frameworks. Representative examples include explainable DRL for voltage regulation in active distribution networks with post-hoc policy extraction [17], privacy-aware multi-agent DRL frameworks with explicit explainability modules for decision-making in smart regional grids [108], and explainable reinforcement learning (XRL) for demand-response-enabled HVAC control using intrinsically interpretable policy structures [109].

DDeveloping X-MARL techniques is therefore essential for narrowing the gap between complex AI-based decisions and human understanding in energy systems. This conceptual process is illustrated in Figure 8, where smart-grid applications rely on ML or MARL decisions that are passed through an explainability module to produce interpretable outputs for auditing, validation, and operational analysis. Frameworks of this kind clarify how X-MARL can build trust by providing grid operators with transparent reasoning for automated control actions, thereby helping align algorithmic intelligence with regulatory and safety requirements. Explainability in EV–grid MARL is inherently cross-dimensional, because operator-facing explanations may need to capture not only learned policy preferences but also forecast-driven anticipation, safety-layer intervention, and decentralized coordination logic [17].

An important but often underexamined issue in smart-grid X-MARL is the trade-off between interpretability and the timeliness of explanation. In operational grid settings, the usefulness of an explanation depends not only on how understandable it is but also on whether it can be generated within the decision horizon of the control task. Post-hoc explainability methods—such as SHAP, LIME, counterfactual search, or case-based prototype retrieval—may provide rich diagnostic insight but are often better suited to offline auditing, event reconstruction, or operator training than to millisecond- or sub-second control loops [19]. By contrast, intrinsically interpretable structures—including decision trees, sparse rule sets, monotonic policies, attention summaries, or constrained surrogate policies—are often more suitable for time-critical deployment because explanation is embedded directly within the control architecture rather than computed afterward [19,110].

Figure 8. Explainability workflow for smart grid applications adapted from [111]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

Figure 8. Explainability workflow for smart grid applications adapted from [111]. This figure was generated using Gemini 3.1 Pro (Google AI) based on the original conceptual framework described in the cited source.

7.1. The Trust Gap: Why Black-Box AI Fails Industrial Standards

The “trust gap” in smart-grid operations reflects the distance between the strong empirical performance of autonomous AI controllers and the stringent transparency, interoperability, and compliance requirements of industrial energy systems. Although MARL frameworks can achieve effective decentralized coordination in simulation, their deployment in real-world cyber–physical systems is often limited by weak interpretability and insufficient auditability. In practice, utility engineers and grid operators require not only high-performing controllers, but also models whose decisions can be traced, explained, and aligned with established operational and interoperability standards. This requirement is especially important in modern DER-rich grids. IEEE Std 1547-2018 defines interconnection and interoperability requirements for DERs, while the IEC 61850 series provides a widely adopted communication and information-modeling framework for power-utility automation [112,113,114]. More broadly, recent research on AI governance in smart energy systems has emphasized that limited explainability, governability, and stakeholder visibility remain major barriers to trust and to the operational uptake of AI-based decision-making in the energy sector [115,116]. From this perspective, black-box MARL policies are difficult to certify in safety-critical settings because operators cannot readily determine whether actions taken under abnormal voltage, frequency, or renewable-generation conditions remain consistent with physical constraints and operational rules.

This lack of explainability remains a structural barrier to the large-scale adoption of AI in smart-grid operations. In smart energy systems, stakeholders require more than predictive accuracy or control performance. They also need sufficient transparency and governability to understand how model outputs are generated, assess whether decisions are operationally reasonable, and determine whether automated actions remain aligned with engineering constraints and organizational responsibilities. Recent reviews therefore identify explainability, trustworthiness, and governance as central prerequisites for broader industrial adoption of AI in the energy sector [115,117].

The importance of transparency is increasing further as the regulatory environment becomes more demanding. The EU Artificial Intelligence Act entered into force on 1 August 2024 and will become fully applicable from 2 August 2026, with some provisions taking effect earlier in phased form. Importantly, AI systems used as safety components in the management and operation of critical infrastructure, including electricity, gas, heating, and water supply, fall within the Act’s high-risk framework. For smart-grid applications, this change makes explainability not merely a desirable design feature, but a practical compliance requirement, because high-risk systems are expected to support risk management, documentation, traceability, and human oversight [116,118,119]. From an operational perspective, explainability must also be matched to the control timescale. Explanations that require substantial post-processing may be acceptable for compliance documentation, forensic analysis, and offline model validation, but offer limited direct value for fast protective or inverter-level actions that occur within milliseconds. For this reason, real-time smart-grid deployment is likely to favor explanation mechanisms with low inference overhead, including intrinsically interpretable policies, distilled rule-based surrogates, compact decision structures, and lightweight saliency or attention summaries that can be generated alongside the control action. By contrast, computationally heavier post-hoc methods remain most useful for slower supervisory control, dispatch review, and operator-facing audit workflows [110].

Accordingly, recent literature increasingly argues that smart-grid AI should move beyond post-hoc rationalization and toward architectures that are explainable and governable by design. Such frameworks are attractive because they make model behavior more auditable, support operator understanding during abnormal operating conditions, and provide a more credible basis for accountability in safety-critical cyber–physical systems. The central challenge, therefore, is not only to improve model accuracy, but also to ensure that AI-assisted decisions can be inspected, justified, and trusted in real operational environments [115,117,120]. An overview of the principal industrial standards and regulatory frameworks relevant to MARL interpretability and governance in smart grids is provided in Table 10.

Table 10. Industrial standards and regulatory frameworks relevant to interpretability and governance of MARL in smart grids.

Standard/Framework	Application Scope	Relevance to MARL Systems	Interpretability/Governance Implication	References
IEEE Std 1547-2018	DER interconnection and interoperability with electric power systems	Relevant to MARL-based DER dispatch, inverter control, voltage support, and grid-edge coordination	Implies that learning-based controllers interacting with DER interfaces should remain consistent with standardized interconnection, interoperability, and test requirements; therefore, controller actions should be inspectable and operationally traceable in utility settings	[112,114]
EU AI Act	High-risk AI regulation in critical infrastructure and other regulated domains	Directly relevant when AI is used as a safety component in the management and operation of electricity, gas, heating, or water infrastructure	Requires risk management, technical documentation, record-keeping, transparency-related obligations, human oversight, and post-market monitoring for high-risk systems, making traceability and auditability central design concerns	[116,118,119]
IEC 61850 Series	Communication networks and systems for power utility automation	Relevant to MARL deployment in substations, DER communication, distribution automation, and grid monitoring platforms	Promotes structured information models, interoperable communication, and standardized event and data exchange, which support auditable integration of AI-assisted control into utility environments	[113]
NIST AI RMF 1.0	Organization-level AI risk management across the AI lifecycle	Relevant as a governance framework for designing, deploying, validating, and monitoring MARL-based grid applications	Emphasizes trustworthy AI characteristics and continuous risk management, supporting explainability, validation, accountability, and lifecycle monitoring, although it is voluntary rather than legally binding	[21]
GDPR/NIS2	Data protection and cybersecurity in EU digital and critical-infrastructure systems	Relevant to MARL when smart-grid operation depends on personal, operational, or networked data and cyber-resilient control infrastructure	Reinforces requirements around lawful data handling, accountability, security controls, and incident resilience, which indirectly strengthen the case for transparent and governable AI pipelines in smart-grid applications	[116,121,122]

7.2. Interpretability Techniques for Multi-Agent Systems

In MARL, interpretability is inherently multi-level rather than agent-specific alone. Explanations must account not only for how an individual agent maps local observations to actions, but also for how agents coordinate through shared information, how communication or neighborhood interactions shape collective behavior, and how credit is assigned to individual agents for team-level outcomes. Recent MARL surveys identify communication, coordination, partial observability, and credit assignment as core challenges that complicate transparent policy analysis, while more recent explainability-oriented work has begun to address these issues through explicit credit-assignment mechanisms and explainable decision modules in multi-agent control frameworks [98,108,123].

7.2.1. Post-Hoc Explanations (SHAP, LIME, and Saliency Maps)

Post-hoc interpretability refers to explaining a trained RL policy after the learning process has been completed, without altering either the policy architecture or the training procedure. Recent reviews of explainable reinforcement learning classify SHAP, LIME, saliency-based attribution, and related perturbation-based or surrogate-model approaches as important post-hoc tools because they can be applied to complex deep RL controllers while leaving the original decision pipeline unchanged [117,124].

Among these approaches, SHAP is especially widely used because it provides feature-attribution scores grounded in cooperative game theory. In this way, it quantifies the relative contribution of each input variable to a model output. LIME, by contrast, provides local explanations by fitting an interpretable surrogate model around an individual prediction. This makes it particularly useful for diagnosing specific control decisions or anomalous actions. Saliency maps complement these methods by relying on gradient-based or attention-based visualizations to identify the input dimensions or regions that most strongly influence a policy decision [124,125]. In energy applications more broadly, recent reviews indicate that SHAP and LIME are among the most frequently adopted post-hoc methods for improving transparency and strengthening operator understanding in AI-driven forecasting, diagnosis, and decision-support tasks [126].

Despite these advantages, post-hoc explanation methods also face important operational limitations in safety-critical control. Because they are generated after inference and often rely on repeated perturbation, surrogate-model fitting, or additional backpropagation-based analysis, they can introduce nontrivial computational overhead. As a result, they are often better suited to offline diagnosis, operator training, and forensic analysis than to real-time closed-loop control. Recent work on explainable control systems specifically notes that SHAP- and LIME-based explanations can introduce delays that limit their direct applicability in fast dynamic decision environments. This has strengthened interest in intrinsically interpretable or explainable-by-design control architectures [117,127].

7.2.2. Intrinsic Interpretability: Attention Mechanisms and Rule Extraction

Intrinsic interpretability aims to embed explanation directly within the model architecture or decision pipeline, rather than generating explanations only after training. This approach is particularly attractive in safety-critical energy applications because it can improve transparency while avoiding some of the latency and fidelity limitations asSoCiated with purely post-hoc methods. As a result, recent work on explainable control systems increasingly advocates explainable-by-design architectures, in which the reasoning process is made partially visible to operators through structured intermediate representations, simplified surrogate policies, or symbolic decision pathways [117].

One important route toward intrinsic interpretability is the use of attention mechanisms. These mechanisms can reveal which inputs, neighboring agents, or system components have the greatest influence on a policy decision. In smart-grid control, Zhang et al. (2025) propose an explainable attention-based multi-agent DRL framework for voltage regulation in active distribution networks [17]. Their work shows that multi-agent attention can support not only the coordination of distributed resources, but also the interpretability of learned control logic through policy extraction. In this sense, attention serves a broader role than improved representation learning alone. It also provides a structured view of inter-agent dependencies and grid-state relevance, which is especially valuable when operators need to understand why a controller prioritizes particular resources or locations during corrective action.

Rule extraction offers a complementary path toward intrinsic transparency by translating a trained RL policy into explicit symbolic or tree-based control logic. Instead of requiring operators to trust a nonlinear neural controller directly, extracted rules provide human-readable mappings from operating conditions to actions. This makes verification and validation easier in practice. For example, Razzano et al. (2025) develop a rule-extraction framework that converts a DRL HVAC controller into interpretable regression trees [128]. Their results show that the extracted controller can closely approximate the original policy while remaining much easier to inspect and compare against conventional operating sequences. More broadly, recent reviews of XAI emphasize that rule-based explanations, decision trees, and symbolic reasoning remain among the most important families of interpretable models for high-stakes industrial systems because they provide explicit and auditable decision paths [129]. Taken together, attention-based architectures and rule-extraction methods illustrate how intrinsic interpretability can help bridge the gap between the expressive power of deep RL and the transparency requirements of real-world energy-system operation.

7.2.3. Game-Theoretic Credit Assignment (Shapley Values)

A core challenge in MARL is themulti-agent credit assignment problem, which concerns determining how much each agent contributes to a team-level outcome. In energy communities and other cooperative energy systems, this issue is closely tied to fairness and long-term stability. Collective objectives such as cost reduction, flexibility provision, and load smoothing require transparent mechanisms for attributing value and benefit among participants [123,130,131].

This challenge also connects the different explainability perspectives discussed above. Post-hoc methods help interpret learned behavior after training, intrinsic methods expose decision logic within the model itself, and cooperative credit-assignment methods clarify how team-level outcomes should be distributed across agents. To synthesize this broader methodological landscape, Table 11 provides a comparative overview of recent X-MARL approaches and their relevance to power and energy systems.

Table 11. Representative explainable RL and MARL applications in power and energy systems.

Reference (Year)	Application Context	XAI Techniques Used	Key Interpretability Benefits	Remaining Gaps/Challenges
[17]	Active distribution-network voltage regulation with PV, ESS, SVC, and hydrogen-storage coordination	Attention-based multi-agent DRL; policy extraction to interpretable decision trees	Makes coordinated voltage-control policies more transparent; exposes resource relevance and enables expert verification of extracted control rules	Focused on voltage regulation; tree-extraction fidelity and transferability to other grid tasks remain open
[132]	Real-time demand response in sustainable manufacturing energy systems	Explainable multi-agent DRL; interpretable reward and decision analysis	Improves visibility into multi-agent demand-response decisions and links control behavior to operational objectives in real time	Industrial manufacturing setting rather than utility-grid operation; broader power-system validation is still needed
[133]	Distribution network reconfiguration	Explainer neural network trained on RL decisions; attribution analysis on bus active and reactive power inputs	Provides post-hoc insight into switch-state decisions and clarifies which electrical variables drive reconfiguration actions	Still post-hoc; explanation fidelity and scalability for larger real-time distribution systems require further study
[134]	Power-system emergency control	SHAP-based feature attribution for DRL decisions	Identifies the most influential input variables behind emergency-control actions and helps align policy behavior with engineering intuition	Primarily diagnostic rather than intrinsically interpretable; limited suitability for strict real-time explanation
[135]	Battery-based home energy management	Policy distillation into differentiable decision trees	Produces compact, human-readable control policies with better transparency than black-box RL while retaining strong cost-saving performance	Single-agent setting; interpretability–performance trade-off remains task-dependent
[128]	Multi-zone HVAC control in buildings	Rule extraction from DRL using regression trees; benchmark against ASHRAE sequences	Converts a complex DRL controller into explicit rule-based logic that is easier to inspect, compare, and implement in building automation practice	Building-focused rather than grid-level; extracted rules may need re-tuning under changing operating conditions
[109]	Household multi-zone HVAC demand response	Explainable PPO with interpretable continuous-control trees	Embeds explainability directly into the policy architecture and supports transparent balancing of comfort, demand response, and energy cost	Household-scale case study; extension to larger multi-agent energy communities remains open

As shown in Table 11, recent studies have demonstrated meaningful progress in visualization and reward-attribution mechanisms. However, scalable, real-time, and regulation-aligned explainability for grid-scale MARL remains an open challenge. This gap motivates the empirical investigations presented in the following section.

A central challenge in cooperative MARL is the credit assignment problem, namely, determining how much each agent contributes to a team-level outcome. In this context, Shapley-value-based explanations are especially attractive because they assign credit according to each agent’s marginal contribution to the global reward. This provides a principled alternative to heuristic reward-sharing schemes. Recent peer-reviewed MARL research has used rollout-based local and global Shapley values to explain cooperative policies at both the trajectory and state levels. These methods help distinguish agents that are consistently important from those whose influence is only situational [136]. More broadly, recent work on explainable multi-agent credit assignment continues to emphasize that transparent contribution estimates are essential for understanding coordination quality and for debugging cooperative policies [123].

This perspective is particularly relevant in power and energy systems, where Shapley-value reasoning is already well established as a fairness mechanism for allocating benefits, costs, or revenues among cooperating participants. For example, Cremers et al. (2023) develop practical approximation methods for computing Shapley values in energy communities and show that contribution-based allocation can remain tractable even when exact computation is prohibitively expensive [137]. Similarly, Zhao et al. (2024) apply an improved Shapley-value mechanism for revenue allocation in integrated rural multi-regional energy systems with peer-to-peer trading, while Gasca et al. (2025) show that fairness and cost-sharing design are central to the long-term viability of energy communities [130,131]. Taken together, these findings suggest that Shapley-based credit attribution is especially promising for future X-MARL in smart grids because it provides a common mathematical framework for both interpreting agent contributions and allocating shared value fairly across cooperative energy actors.

7.3. Case Studies: Visualizing Agent Strategies in Grid Scenarios

The practical value of X-MARL is most effectively assessed through benchmark environments and case studies that reflect the physical, operational, and economic constraints of modern energy systems. In this regard, CityLearn v2 has become an important reference platform for grid-interactive communities. It extends earlier building-energy benchmarks by incorporating data-driven thermal dynamics, dynamic occupant feedback, and multi-agent control of DERs such as batteries, heat pumps, and vehicle-to-grid assets [138,139]. Recent peer-reviewed studies have used such environments to move beyond black-box performance metrics and toward more interpretable views of agent coordination. For example, Savino et al. (2025) propose an attention-driven multi-agent DRL framework for large residential districts and evaluate it in the CityLearn environment [140]. Their results show that attention mechanisms can support scalable coordination across 73–100 buildings while also providing a structured indication of which neighboring states and interactions are most relevant to each control decision. In parallel, Yun et al. (2023) develop an explainable multi-agent DRL strategy for real-time demand response in manufacturing energy systems [132]. In their framework, the global control objective is decomposed into agent-level tasks and safe action subsets, making it possible to trace how local decisions contribute to system-level production and energy objectives.

Explainability has also been demonstrated in grid-facing control tasks beyond district energy management. Zhang et al. (2025) present an explainable attention-based multi-agent DRL framework for voltage regulation in renewable-rich active distribution networks [17]. In this work, the learned coordinated policies are further extracted into interpretable decision trees to support expert verification of controller behavior. Similarly, Gholizadeh and Musilek (2024) use an auxiliary explainer network to interpret reinforcement-learning-based distribution network reconfiguration, revealing how electrical variables such as bus active and reactive power influence switching actions [133]. Taken together, these case studies show that explainability in energy-oriented MARL is increasingly achieved through attention maps, task decomposition, feature attribution, and policy extraction. These mechanisms make coordinated agent strategies more auditable and more suitable for real-world deployment. Table 12 summarizes representative peer-reviewed case studies in which explainability has been explicitly integrated into RL or MARL for power and energy applications, highlighting the explanation mechanism used and the corresponding operational outcomes.

Table 12. Case study evaluations of explainable RL and MARL in power and energy systems.

Application Context	RL/MARL Algorithm	Explainability Mechanism	Evaluation Outcomes	References
Large residential district demand-side management	Attention–Actor–Critic Multi-Agent DRL (AAC-MADRL)	Attention mechanism within a CTDE MARL architecture	Evaluated on CityLearn districts with 73 and 100 buildings; reduced energy cost by up to 18% in Texas and 12.5% in California versus a rule-based controller, improved self-sufficiency by 6–10.5%, and achieved faster deployment than decentralized and non-attention cooperative DRL baselines	[140]
Active distribution-network voltage regulation	Explainable attention-based multi-agent DRL	Attention-based coordinated policy learning plus decision-tree policy extraction via imitation learning	Achieved effective coordinated voltage regulation in renewable-rich active distribution networks while extracting the learned policy into interpretable decision trees for expert inspection; validated using real-world traces	[17]
Distribution network reconfiguration	Deep Q-learning/RL-based switching control	Auxiliary explainer neural network with attribution analysis	Trained an explainer network to reproduce RL switching decisions from bus active and reactive power inputs and demonstrated the method on both 33-bus and 136-bus test systems	[133]
Battery-based home energy management	RL policy distilled into differentiable decision trees	Policy distillation to differentiable decision trees	Produced simple, explainable control policies that outperformed rule-based baselines by about 20–25% in the proof-of-concept home energy management setting, while exposing the performance–interpretability trade-off relative to standard RL	[135]
Multi-zone HVAC control in buildings	Deep RL controller with post-training rule extraction	Regression-tree rule extraction from the trained DRL controller	The extracted rule-based controller achieved about 17% electric-energy savings versus conventional ASHRAE sequences and remained within about 3% of the original DRL controller, while being much easier to inspect and implement	[128]
Household multi-zone HVAC demand response	Explainable PPO (XPPO)	Interpretable continuous-control trees used as actor networks	Reduced energy cost by 22.4% compared with operation without demand response and solved the typical 4-zone household optimization problem within 15 min on an i7-12700 CPU, while maintaining an explicitly interpretable policy structure	[109]

7.4. Future Directions: Human-in-the-Loop and Regulatory Auditing

Future research in smart-grid AI is likely to be shaped by two converging priorities: stronger human oversight of autonomous decision-making and closer alignment with emerging regulatory requirements. Recent peer-reviewed reviews emphasize that the next generation of AI-enabled grid control should not only optimize operational objectives, but also provide explanations, support human supervision, and remain governable throughout deployment [117,126,141].

One important direction is the development of human-in-the-loop architectures in which learning-based controllers operate under supervisory interfaces that allow operators to inspect, validate, and constrain automated decisions. In this setting, large language models and other generative AI tools are increasingly being discussed as potential support layers for smart-grid operations. Their role is not to replace real-time control, but rather to summarize system behavior, translate model outputs into operator-readable explanations, and assist with higher-level decision support [141]. A representative application scenario is operator-facing decision translation. In such a setting, a MARL controller may determine coordinated EV charging, DER dispatch, or congestion-mitigation actions on the basis of forecasts, network conditions, and safety constraints, while an LLM functions as an interface layer that translates these outputs into concise natural-language instructions for grid operators. For example, instead of presenting only raw setpoints or action vectors, the system could generate an explanation such as: “Reduce charging power at feeder segment A, defer flexible charging at station B for 15 min, and prioritize local storage dispatch, because forecasted demand and voltage margins indicate elevated congestion risk during the next interval.” This form of translation would not replace the control algorithm itself, but could improve operator awareness, auditability, and human oversight by making learned control decisions easier to interpret in practical utility settings [17,142]. At the same time, hierarchical control structures remain especially attractive in power systems because they allow slower planning and coordination layers to incorporate adaptive intelligence and human oversight, while faster protection and stability layers continue to rely on more tightly verified and operationally constrained control logic [117,141].

A second major direction concerns regulatory auditing. The regulatory environment for AI in critical infrastructure is becoming substantially more consequential, and recent peer-reviewed analyses of smart grids under the EU AI Act argue that compliance will increasingly require traceability, documentation, human oversight, and robust governance mechanisms for high-risk AI applications [116,143]. In this context, explainability is no longer merely a technical preference. It becomes part of a broader compliance and accountability framework in which utilities may need to demonstrate how AI-supported decisions were generated, monitored, and validated.

Finally, the field is moving toward deeper forms of explanation that extend beyond correlation-based feature attribution. Recent peer-reviewed work identifies causality-aware XAI as an important frontier because trustworthy deployment in high-stakes cyber–physical systems ultimately requires understanding not only which variables were influential, but also how interventions and system dependencies shape outcomes [126,144]. For smart grids, this suggests that future X-MARL research will need to combine operator-centered interfaces, auditable governance, and causality-aware reasoning in order to support both technical performance and institutional trust.

8. Cross-Cutting Issues and Future Research Directions

The transition toward increasingly autonomous power and energy systems introduces a set of cross-cutting challenges that extend beyond controller design alone. As learning-based agents take on greater responsibility for coordinating DERs, managing demand flexibility, and participating in local markets, the surrounding digital infrastructure must also support secure decentralization, scalable coordination, reliable transfer across environments, and reproducible benchmarking. Recent peer-reviewed reviews emphasize that these issues are now central to the practical maturation of AI-enabled smart grids [141,145].

Although the four core dimensions of this review—forecasting-informed coordination, safety-constrained learning, explainability, and trustworthy decentralized coordination—are discussed in separate sections for analytical clarity, they should not be viewed as independent design layers. In practical EV–grid MARL systems, these dimensions are tightly interconnected. Forecasting affects safety because uncertainty in demand, renewable generation, prices, and mobility directly influences constraint satisfaction and reserve margins. Safety mechanisms affect explainability because projection, shielding, and constrained optimization determine which actions are admissible and, therefore, which decisions must be interpreted by operators. Explainability in turn influences trustworthy decentralized coordination because auditable policies, traceable decisions, and interpretable agent interactions are often necessary for institutional trust, regulatory acceptance, and dispute resolution in multi-actor energy settings. Trustworthy coordination mechanisms then shape how forecasts, constraints, and explanations are shared, validated, and acted upon across decentralized agents.

The reviewed literature further suggests that integration across these dimensions remains partial rather than mature. Most studies combine no more than two dimensions, such as forecasting with MARL control or safety with decentralized learning. Fully integrated treatment of prediction uncertainty, physical safety, operator interpretability, and trustworthy multi-agent coordination remains uncommon, particularly in utility-oriented settings with real-time operational constraints. A key direction for future research is therefore coordinated optimization across these dimensions rather than the sequential addition of isolated modules. In such a design, predictive uncertainty should inform safe decision envelopes, safety filters should remain compatible with interpretable policy structures, and trustworthy coordination layers should support transparent information exchange, validation, and accountability. This broader perspective on integration is essential if MARL is to progress from promising simulation results to deployable EV–grid control architectures.

From an industry perspective, this means that future MARL architectures will be evaluated not only on control quality in simulation but also on whether they satisfy operational timing, safety verification, communication, and governance requirements in realistic grid environments.

Although this review is control-centric in scope, the surveyed MARL literature provides only limited formal guarantees of closed-loop stability in the Lyapunov sense. Most existing studies evaluate performance using reward convergence, voltage or frequency regulation quality, constraint-violation rates, or empirical robustness in simulation, rather than proving the boundedness of the full cyber–physical closed loop. This limitation is especially important in EV–grid coordination, where learned policies interact with network physics, inverter dynamics, communication delays, and non-stationary disturbances. A more realistic path toward deployment is hierarchical integration, in which conventional droop, volt/var, or other certified local feedback controllers preserve fast local stability while MARL operates at slower supervisory layers for coordination, adaptation, and economic optimization. Under this interpretation, maintaining timescale separation is essential, and formal stability certification remains an important open research direction.

Table 13 provides a compact summary of representative algorithmic innovations across the four core dimensions of this review, complementing the taxonomy in Figure 1 with a method-oriented synthesis.

Table 13. Representative algorithmic innovations mapped to the four core dimensions of this survey.

Dimension	Key Algorithmic Innovations	Representative Methodological Role	Example References
Forecasting-informed MARL	LSTM- and transformer-based temporal encoding; two-stage forecast–control pipelines; uncertainty-aware forecasting via probabilistic or scenario-based prediction	Improves anticipatory decision-making by embedding future demand, price, renewable, or mobility information into MARL state design, planning, and coordination	[9,10,11,64]
Safety-constrained MARL	CMDP-based formulations; Lagrangian constraint handling; safety layers; action projection/shielding; robust and risk-aware learning	Enforces voltage, thermal, transformer, and state-of-charge constraints while reducing unsafe exploration and improving operational reliability	[13,14,74]
Explainability and interpretability	Post-hoc explanation methods; decision-tree extraction; attention-based interpretability; temporal prototypes; Shapley-style credit assignment	Provides transparency for operator auditing, debugging, accountability, and regulatory interpretation of MARL behavior	[17,18,19]
Trustworthy decentralized coordination	Blockchain-enabled coordination; smart contracts; federated/privacy-preserving learning; audit trails; verifiable transaction and settlement layers	Supports secure peer-to-peer coordination, transparent market interaction, privacy-aware learning, and governance-oriented trust in decentralized energy systems	[22,23,146]

8.1. Secure Decentralization: The Role of Blockchain and Smart Contracts

One major research direction is the use of blockchain and smart contracts to create a trusted digital coordination layer for decentralized energy systems. In peer-to-peer energy trading and EV-enabled local markets, blockchain can reduce dependence on a central intermediary while supporting transparent transaction records, automated settlement, tamper-evident logging, and programmable rule enforcement through smart contracts [147,148]. These properties are especially relevant in multi-actor settings in which EVs, prosumers, aggregators, and distributed energy resources must coordinate while maintaining accountability across both market and control interactions.

Trustworthy decentralized coordination is also closely connected to the other dimensions of this review, because reliable market or peer-to-peer interaction depends on the quality of predictive information, the enforceability of safety constraints, and the transparency of agent decisions and interactions [13,145].

Recent peer-reviewed studies illustrate this role in several ways. Saeed et al. (2024) implement a decentralized microgrid trading architecture based on Ethereum smart contracts to automate settlement and record market interactions, while Alam et al. (2024) propose a blockchain-based multilayer framework for virtual power plants that coordinates DER participation under decentralized trading rules [147,148]. More broadly, recent literature increasingly treats blockchain not only as a mechanism for financial transactions but also as an infrastructure for trusted coordination, auditability, and institutional confidence in distributed energy systems [149].

From a MARL perspective, this distinction is important. Although MARL can learn decentralized decision policies under uncertainty, additional mechanisms are required to ensure trust, transparency, and resistance to strategic manipulation in decentralized peer-to-peer environments. Blockchain can complement MARL by providing a verifiable coordination and execution layer: MARL learns adaptive decentralized policies, while blockchain records actions on an immutable ledger, supports transparent transaction validation, and enforces agreed rules through smart contracts [23].

This complementarity is particularly relevant for EV–grid coordination in transactive and community-scale settings. Future peer-to-peer charging or local flexibility markets may involve autonomous agents negotiating charging schedules, energy exchanges, reserve provision, or congestion-aware participation. In such settings, trust cannot be assumed: participants may face conflicting incentives, limited visibility into one another’s behavior, and different ownership structures. A blockchain-based coordination layer can therefore provide auditable interaction histories, automated clearing and settlement, and reduced reliance on a single central platform, thereby strengthening the practical feasibility of decentralized MARL deployment [31].

However, the literature also highlights several important limitations of blockchain in energy applications. Public blockchain platforms may introduce latency, limited throughput, and transaction costs that are difficult to reconcile with real-time control requirements. Privacy also remains a concern because energy transactions and user behavior may reveal sensitive operational patterns even when pseudonymous identifiers are used. In addition, interoperability with grid communication standards, market operators, and utility control systems remains underdeveloped. Finally, blockchain alone cannot guarantee physically safe operation: although smart contracts can enforce market rules, they cannot replace voltage, thermal, and stability constraints, which must still be managed through control-aware optimization or safe MARL mechanisms [146].

For these reasons, the most realistic research direction is not “blockchain instead of control” but rather blockchain-enabled trustworthy MARL. This implies integrating learning-based coordination with secure logging, contract-based settlement, privacy-aware information exchange, and compliance-oriented audit trails. Future work should also connect blockchain layers more explicitly with the other themes of this review, including forecasting-informed coordination, safety-constrained learning, and explainability. For example, forecast uncertainty may influence contractual decisions, safe MARL may be required before contractual execution is approved, and explainable policies may be necessary for dispute resolution or regulatory auditing. Accordingly, trustworthy decentralized coordination should be understood as a cross-layer design problem in which blockchain provides verifiability and governance while MARL provides adaptive control under uncertainty [23,146].

8.2. Scalability and Communication: Handling Large-Scale Heterogeneous Agents

Scalability remains a central challenge because real energy systems include many heterogeneous decision-making entities, such as buildings, batteries, HVAC units, electric vehicles, and prosumers, each with distinct objectives and physical constraints. Recent peer-reviewed research addresses this issue by structuring coordination rather than permitting unrestricted all-to-all interaction. For example, Savino et al. (2025) propose an attention-driven multi-agent DRL framework for large residential districts in which attention mechanisms help agents focus on the most relevant information during coordinated demand-side management [140]. This design improves both scalability and coordination in CityLearn-based districts containing dozens of buildings. Similarly, Yao et al. (2025) develop a hierarchical multi-agent DRL framework for home energy management in which a high-level agent dynamically determines the decision hierarchy of lower-level appliance and storage agents [150]. In doing so, the framework reduces coordination complexity in heterogeneous household systems.

Taken together, these studies suggest that future large-scale MARL in power systems will likely depend on communication-efficient structures such as attention, hierarchy, and topology-aware decomposition, rather than flat coordination across all agents. This perspective is consistent with broader smart-grid AI reviews, which argue that layered control and selective coordination are essential if intelligent energy systems are to remain both scalable and operationally manageable [141].

8.3. Cross-Scenario Applicability of MARL Techniques

The suitability of any given MARL formulation depends strongly on the operating environment in which EV–grid coordination is implemented. Urban distribution networks, rural microgrids, low- and high-penetration EV scenarios, centralized and distributed charging structures, and grid-connected and islanded operating modes differ substantially in terms of network coupling, controllability, observability, communication availability, and safety requirements. Consequently, a MARL technique that performs well in one setting may be less suitable in another. A deployment-oriented comparison, therefore, requires evaluating not only algorithmic performance but also scenario-specific applicability—including whether the control problem is better served by centralized critics, decentralized execution, hierarchical decomposition, explicit safety filtering, communication-efficient coordination, or stronger robustness to sparse and noisy data.

As summarized in Table 14, no single MARL paradigm is universally optimal across all EV–grid environments. Dense urban distribution networks and high-penetration EV scenarios generally favor structured coordination, safety-constrained learning, and scalable communication-aware architectures. By contrast, rural microgrids, distributed charging settings, and islanded operation place relatively greater emphasis on robustness, local autonomy, and resilience under limited data and communication support. This comparison indicates that effective MARL design should begin with scenario characteristics, including network coupling, observability, infrastructure quality, and operational risk, rather than with algorithm choice alone.

8.4. Transfer Learning and Generalization: From Simulation to Real Grids

A persistent obstacle to deployment is the mismatch between simulated training environments and real operating conditions. Recent peer-reviewed research in building energy systems suggests that transfer learning can substantially reduce this gap by shortening warm-up periods, improving early-stage policy performance, and enabling the reuse of knowledge across sites. Genkin and McArthur (2024) propose a transfer-learning-based RL framework for smart buildings that reduces the RL warm-up period by up to 84% and lowers prediction variance by up to 99%, thereby directly addressing the cold-start problem in newly commissioned buildings [151]. Coraci et al. (2025) extend this line of work through the HiLo case study, showing how online transfer learning can support the real-world implementation of DRL controllers in buildings [152]. In parallel, Zangato et al. (2025) combine clustering, forecasting, and constrained RL to enable policy generalization across different building types using real-world data, while also supporting adaptation to stochastic tariff changes without retraining [153].

Although these studies are primarily in building and district energy settings rather than transmission-grid operation, they are highly relevant to future smart-grid MARL. They suggest that robust deployment will require mechanisms for cross-site transfer, adaptation under distribution shift, and safe online refinement, rather than assuming that a policy trained in one simulator will remain valid in a different physical environment [152,153].

8.5. Standardization: Benchmarking MARL Performance in Power Systems

These developments are significant because they illustrate what rigorous benchmarking can look like in energy-oriented MARL. Such benchmarking relies on shared environments, transparent metrics, and progressively richer representations of physical flexibility. At the same time, the benchmark ecosystem remains uneven across the broader power-systems domain. A major priority for future research is therefore the development of comparably rigorous benchmark platforms for distribution-network control, market coordination, and multi-timescale grid operation, so that proposed MARL methods can be evaluated under reproducible and physically meaningful conditions [154].

A final cross-cutting issue is standardization. Without shared benchmark environments, datasets, and evaluation metrics, it remains difficult to compare MARL strategies fairly or to determine whether reported improvements are scientifically meaningful. In the peer-reviewed literature, CityLearn has emerged as one of the most important examples of a common-task framework for multi-agent energy management. The CityLearn v2 journal paper formalizes a richer benchmark for grid-interactive communities with occupant-centric and carbon-aware dynamics, while the CityLearn Challenge paper shows how repeated challenge design can advance the standardized evaluation of advanced control algorithms in smart buildings [138,139]. The EVLearn extension further expands this benchmarking ecosystem by incorporating electric-vehicle flexibility, V1G, and V2G behavior into the CityLearn family of environments [155].

Table 14. Cross-scenario applicability of MARL techniques for EV–grid coordination.

Scenario Type	Typical Operating Characteristics	Main Technical Constraints	More Suitable MARL Approaches	Applicability Note	Representative References
Urban distribution networks	Dense agent populations, strong network coupling, high communication availability, feeder-level operational constraints	Voltage limits, transformer loading, congestion, real-time coordination complexity	CTDE, hierarchical MARL, safety-constrained MARL, graph- or attention-based coordination	Urban settings benefit from structured multi-agent coordination because many heterogeneous agents interact through shared network physics. Safety-aware and communication-efficient methods are especially important under high operational density.	[7,17,89]
Rural microgrids/remote energy communities	Smaller systems, weaker communication infrastructure, higher sensitivity to local generation variability, islanding potential	Sparse data, limited communication, renewable intermittency, resilience and autonomy requirements	Decentralized MARL, robust/risk-aware methods, hierarchical local control, forecast-informed coordination	Rural and remote systems often favor simpler or more communication-efficient MARL structures, with greater emphasis on robustness, local autonomy, and uncertainty-aware coordination.	[54,104,152]
Low EV penetration	Limited charging simultaneity, weaker aggregate network stress, fewer active charging agents	Economic efficiency and scheduling quality dominate; safety constraints are important but less frequently binding	Independent or lightly coordinated MARL, forecast-informed scheduling, simpler CTDE formulations	At lower penetration levels, simpler coordination strategies may remain adequate, and the main value of MARL may lie in adaptive scheduling rather than heavy safety or congestion management.	[9,46,48]
High EV penetration	Frequent simultaneity, stronger feeder coupling, higher probability of congestion and voltage violations	Transformer capacity, voltage deviation, congestion propagation, communication burden	Safety-constrained MARL, CTDE, value factorization, hierarchical coordination, runtime safety layers	As EV penetration increases, coordination becomes more safety-critical and strongly coupled, making scalable multi-agent and constraint-aware methods substantially more important.	[8,14,37]
Centralized charging structures	Aggregator or operator has stronger visibility and coordination authority, often with richer system-wide information	Computation at scale, privacy, dependence on communication reliability	CTDE, centralized critics, hierarchical dispatch, optimization-guided MARL	Centralized or semi-centralized charging settings are often well suited to CTDE and hierarchical methods because broader information is available during training and coordination.	[7,42,63]
Distributed charging structures	Multiple autonomous chargers or prosumers, limited mutual visibility, local decision-making	Partial observability, asynchronous information, local incentive misalignment	Decentralized execution, value factorization, communication-efficient MARL, trustworthy coordination layers	Distributed settings place greater emphasis on coordination under incomplete information, making decentralized execution and trust-aware mechanisms more relevant.	[31,34,48]
Grid-connected operation	Interaction with upstream grid, external price signals, import/export flexibility	Market coordination, feeder limits, dependence on external conditions	Forecast-informed MARL, market-aware MARL, safe coordination with grid constraints	Grid-connected settings often require integration of economic signals with physical feasibility, making forecasting and constraint handling jointly important.	[9,55,74]
Islanded/resilience-oriented operation	Limited local resources, stronger autonomy requirements, no immediate upstream support	Energy adequacy, local stability, resilience, service prioritization under scarcity	Robust MARL, hierarchical local control, safe MARL, recovery-oriented coordination	Islanded operation increases the importance of resilience, robust decision-making, and explicit safety margins, since local errors cannot easily be absorbed by the main grid.	[4,13,104]

Table 14. Cross-scenario applicability of MARL techniques for EV–grid coordination.

Scenario Type	Typical Operating Characteristics	Main Technical Constraints	More Suitable MARL Approaches	Applicability Note	Representative References
Urban distribution networks	Dense agent populations, strong network coupling, high communication availability, feeder-level operational constraints	Voltage limits, transformer loading, congestion, real-time coordination complexity	CTDE, hierarchical MARL, safety-constrained MARL, graph- or attention-based coordination	Urban settings benefit from structured multi-agent coordination because many heterogeneous agents interact through shared network physics. Safety-aware and communication-efficient methods are especially important under high operational density.	[7,17,89]
Rural microgrids/remote energy communities	Smaller systems, weaker communication infrastructure, higher sensitivity to local generation variability, islanding potential	Sparse data, limited communication, renewable intermittency, resilience and autonomy requirements	Decentralized MARL, robust/risk-aware methods, hierarchical local control, forecast-informed coordination	Rural and remote systems often favor simpler or more communication-efficient MARL structures, with greater emphasis on robustness, local autonomy, and uncertainty-aware coordination.	[54,104,152]
Low EV penetration	Limited charging simultaneity, weaker aggregate network stress, fewer active charging agents	Economic efficiency and scheduling quality dominate; safety constraints are important but less frequently binding	Independent or lightly coordinated MARL, forecast-informed scheduling, simpler CTDE formulations	At lower penetration levels, simpler coordination strategies may remain adequate, and the main value of MARL may lie in adaptive scheduling rather than heavy safety or congestion management.	[9,46,48]
High EV penetration	Frequent simultaneity, stronger feeder coupling, higher probability of congestion and voltage violations	Transformer capacity, voltage deviation, congestion propagation, communication burden	Safety-constrained MARL, CTDE, value factorization, hierarchical coordination, runtime safety layers	As EV penetration increases, coordination becomes more safety-critical and strongly coupled, making scalable multi-agent and constraint-aware methods substantially more important.	[8,14,37]
Centralized charging structures	Aggregator or operator has stronger visibility and coordination authority, often with richer system-wide information	Computation at scale, privacy, dependence on communication reliability	CTDE, centralized critics, hierarchical dispatch, optimization-guided MARL	Centralized or semi-centralized charging settings are often well suited to CTDE and hierarchical methods because broader information is available during training and coordination.	[7,42,63]
Distributed charging structures	Multiple autonomous chargers or prosumers, limited mutual visibility, local decision-making	Partial observability, asynchronous information, local incentive misalignment	Decentralized execution, value factorization, communication-efficient MARL, trustworthy coordination layers	Distributed settings place greater emphasis on coordination under incomplete information, making decentralized execution and trust-aware mechanisms more relevant.	[31,34,48]
Grid-connected operation	Interaction with upstream grid, external price signals, import/export flexibility	Market coordination, feeder limits, dependence on external conditions	Forecast-informed MARL, market-aware MARL, safe coordination with grid constraints	Grid-connected settings often require integration of economic signals with physical feasibility, making forecasting and constraint handling jointly important.	[9,55,74]
Islanded/resilience-oriented operation	Limited local resources, stronger autonomy requirements, no immediate upstream support	Energy adequacy, local stability, resilience, service prioritization under scarcity	Robust MARL, hierarchical local control, safe MARL, recovery-oriented coordination	Islanded operation increases the importance of resilience, robust decision-making, and explicit safety margins, since local errors cannot easily be absorbed by the main grid.	[4,13,104]

These developments are important because they illustrate more concretely what rigorous benchmarking in energy-oriented MARL can involve: shared environments, transparent metrics, and increasingly rich representations of physical flexibility. At the same time, benchmarking remains uneven across the wider power-systems domain. Many studies still depend on custom simulators, partially specified test setups, or application-specific evaluation pipelines that are difficult to reproduce across research groups. This heterogeneity limits direct comparison of algorithmic performance and makes it harder to determine whether reported gains result from the learning method itself or from differences in system assumptions, scenario design, or evaluation metrics [154].

To make benchmarking practices more explicit across the reviewed literature, Table 15 summarizes representative benchmark environments used in the included studies and indicates whether code and data availability were explicitly reported. The reviewed studies span three broad categories: (i) shared benchmark platforms such as CityLearn and EVLearn, which provide structured and reusable evaluation settings; (ii) public engineering test systems such as IEEE feeder models, which improve physical comparability but often differ in implementation details and scenario assumptions; and (iii) custom or proprietary simulators, which remain common in peer-to-peer coordination, resilience, and market-oriented studies but offer limited reproducibility. This comparison highlights that benchmark choice is itself a methodological variable in MARL research and should be reported with the same level of care as algorithmic design.

Table 15. Benchmarking Practices and Reproducibility Reporting Across Representative Studies in the Reviewed MARL Literature.

Study/Benchmark Family	Application Area	Benchmark Environment/Testbed	Code Avail. Reported	Data Avail. Reported	Benchmarking Note
[138]	Grid-interactive community energy management	CityLearn v2 shared benchmark platform	Yes	Yes	Common-task benchmark with standardized environment design, reusable data structure, and transparent evaluation logic.
[139]	Comparative controller evaluation in smart buildings	CityLearn Challenge benchmark	Yes	Yes	Challenge-based benchmark designed to promote repeated and comparable evaluation across control methods.
[155]	EV-integrated building/community coordination	EVLearn benchmark (CityLearn extension with EV flexibility, V1G, and V2G)	Yes	Yes	Extends benchmark-driven evaluation toward EV-aware multi-agent control settings.
[156]	Distribution-network operation, voltage regulation, and grid-aware EV/DER coordination	Public engineering test systems such as IEEE feeder models (e.g., 13-bus, 33-bus, 123-bus)	Not explicitly reported	Not explicitly reported	Public feeder definitions improve physical comparability, but implementation details, load models, and scenario design often differ across studies.
[61]	Voltage regulation in active distribution networks	Partitioned active distribution network with heterogeneous PV inverter settings	Not explicitly reported	Not explicitly reported	Physically motivated network-control benchmark, but not presented as a reusable shared benchmark platform in the reviewed discussion.
[17]	Explainable voltage regulation	Renewable-rich active distribution network simulation	Not explicitly reported	Not explicitly reported	Grid-facing benchmark with interpretable policy extraction, but evaluation environment remains study-specific.
[133]	Distribution network reconfiguration	Distribution network simulation for switching and reconfiguration decisions	Not explicitly reported	Not explicitly reported	Useful grid-control case study, though not framed as a standardized common benchmark.
[31]	P2P trading, market coordination, resilience, and EV service coordination	Custom or proprietary simulation environments	Not explicitly public	In-article only	Common across the literature; often application-specific and difficult to reproduce or compare directly across studies.

A major future research need is therefore the creation and broader adoption of rigorous benchmark platforms for distribution-network control, market coordination, and multi-timescale grid operation, so that proposed MARL methods can be evaluated under reproducible and physically meaningful conditions [138,139,155]. More consistent reporting of code availability, data access, and simulation assumptions would also improve transparency and accelerate cumulative progress across the field. Overall, the literature suggests that insufficient benchmark standardization and inconsistent artifact reporting remain major barriers to fair comparison, reproducibility, and deployment-oriented validation in MARL for EV–grid coordination.

9. Conclusions

The rapid growth of electric vehicles, distributed renewable generation, and prosumer-based energy participation is fundamentally reshaping the operation of urban power systems. In this evolving setting, MARL has emerged as a promising decentralized decision-making framework for coordinating heterogeneous agents, including EVs, charging stations, aggregators, DERs, and grid-support devices. Across the literature reviewed in this survey, MARL consistently shows strong potential for improving charging coordination, load balancing, community energy management, peer-to-peer trading, and ancillary grid support under dynamic and uncertain operating conditions. At the same time, the reviewed studies show that reported performance gains are strongly influenced by modeling assumptions, benchmark design, communication structure, and the degree to which physical constraints are incorporated into training and deployment.

This survey also shows that the future value of MARL in EV–grid integration does not depend on control performance alone. Instead, the field is increasingly defined by four interdependent research dimensions: forecasting-informed coordination, safety-constrained learning, explainability, and trustworthy decentralized execution. Forecasting-enhanced MARL provides agents with anticipatory awareness of demand, price, and renewable variability. This improves proactive decision-making and reduces the limitations of purely reactive control. Safe MARL extends this capability by embedding physical constraints, runtime correction, and risk-aware design into learning pipelines so that coordination remains compatible with voltage, thermal, state-of-charge, and transformer-health limits. X-MARL addresses a different but equally important challenge by making agent decisions more transparent, auditable, and aligned with industrial trust, regulatory accountability, and operator supervision. Finally, blockchain-enabled and trust-aware architectures offer a path toward secure and verifiable decentralized interaction, particularly in peer-to-peer trading and distributed market settings where transparency, settlement integrity, and incentive alignment are essential. Taken together, these dimensions indicate that MARL should be understood not as a standalone optimization tool, but as part of a broader operational stack in which prediction, constraint enforcement, interpretability, and institutional trust all shape deployment readiness.

Taken together, these developments suggest that the next generation of MARL-based urban energy systems should be understood not simply as learning controllers, but as components of a broader cyber–physical governance stack. Successful deployment will therefore require simultaneous progress in prediction quality, safety assurance, interpretability, communication architecture, cybersecurity, and benchmarking. The literature also makes clear that important gaps remain. Forecast uncertainty is still not fully propagated into MARL decision processes. Establishing safety guarantees in multi-agent settings remains difficult under coupled constraints and partial observability. Explainability methods are often post-hoc, computationally expensive, or insufficiently aligned with real-time operation. Blockchain-enhanced decentralized frameworks also still require stronger validation under realistic grid-scale conditions. In addition, the lack of unified benchmark platforms across distribution control, transactive coordination, and multi-timescale grid services continues to limit reproducibility and fair comparison across studies.

The review further shows that many published studies still rely on clean-simulation assumptions, limited disturbance modeling, and incomplete reporting of code, data, and benchmark details, which make it difficult to judge how well the reported results would transfer to realistic field conditions. This review has several limitations. First, the included literature is dominated by simulation-based studies, with limited real-world deployment and substantial heterogeneity in benchmarks, case-study assumptions, and evaluation metrics. Second, the lack of standardized terminology across the intersecting fields of MARL, EV coordination, and transactive energy increases the possibility that some relevant studies were not captured despite the systematic search process. Third, because the evidence base is methodologically diverse and largely qualitative, no formal meta-analysis or certainty-of-evidence framework, such as GRADE, was applied. These factors should be considered when interpreting the synthesized trends and identifying research gaps. An additional limitation is that the reviewed literature often reports performance more consistently than robustness, transferability, latency, or reproducibility, which restricts the strength of the conclusions that can currently be drawn about deployment readiness.

Several research directions emerge from this review. First, future MARL frameworks should move toward prediction-aware safe control, in which probabilistic forecasting, uncertainty quantification, and physical safety envelopes are jointly embedded into decentralized policy design. Second, human-centered X-MARL will be essential for practical deployment, especially as regulatory frameworks such as the EU AI Act increase the importance of traceability, auditability, and governance for high-risk AI in critical infrastructure. Third, scalable hierarchical and communication-efficient MARL will be needed to coordinate large populations of heterogeneous agents without prohibitive computational or communication overhead. Fourth, secure and trustworthy coordination layers, potentially supported by blockchain, federated learning, and smart contracts, will become increasingly important as EV–grid ecosystems move toward decentralized market participation and interoperable digital energy platforms. Finally, the field needs standardized and physically grounded benchmark environments that jointly evaluate economic performance, power-system reliability, safety compliance, interpretability, transferability, and robustness to noisy measurements, missing data, communication delays, and packet loss.

In conclusion, MARL provides a compelling foundation for the intelligent coordination of future EV-integrated urban grids, but its transition from simulation to real-world infrastructure will depend on more than algorithmic accuracy alone. The most impactful future systems will be those that are not only adaptive and decentralized, but also forecast-aware, safety-certified, explainable, trustworthy, and benchmarked under realistic operational conditions. More specifically, the main conclusion of this review is that deployment-ready MARL for EV–grid coordination will require tighter integration between learning-based decision-making and established power-system requirements, including hierarchical control design, robust constraint handling, low-latency and operator-usable interpretability, communication-aware scalability, and reproducible benchmark validation. By synthesizing these emerging dimensions into a unified perspective, this survey provides a roadmap for advancing MARL from a promising research paradigm into a deployable technology for resilient, transparent, and sustainable urban energy systems.