Coordinated Optimization of Distributed Energy Resources Based on Spatio-Temporal Transformer and Multi-Agent Reinforcement Learning

Abstract

The rapid growth of Distributed Energy Resources (DERs) exerts significant pressure on distribution network margins, requiring predictive and safe coordination. This paper presents a closed-loop framework combining a topology-aware Spatio-Temporal Transformer (STT) for multi-horizon forecasting, a cooperative multi-agent reinforcement learning (MARL) controller under Centralized Training and Decentralized Execution (CTDE), and a real-time safety layer that enforces feeder limits via sensitivity-based quadratic programming. Evaluations on three SimBench feeders, with OLTC/capacitor hybrid control and a stress protocol amplifying peak demand and mid-day PV generation, show that the method reduces tail violations by 31% and 56% at the 99th percentile voltage deviation, and lowers branch overload rates by 71% and 90% compared to baselines. It mitigates tail violations and discrete switching while ensuring real-time feasibility and cost efficiency, outperforming rule-based, optimization, MPC, and learning baselines. Stress maps reveal robustness envelopes and identify MV–LV bottlenecks; ablation studies show that diffusion-based priors and coordination contribute to performance gains. The paper also provides convergence analysis and a suboptimality decomposition, offering a practical pathway to scalable, safe, and interpretable DER coordination.

1. Introduction

Coordinating distributed energy resources (DERs) is essential as modern grids integrate increasing renewable generation and electrified loads. Without effective coordination, the adoption of DERs can overload transformers and cause voltage violations, leading to costly infrastructure upgrades [1,2,3]. Recent power-system studies further highlight the tightening of security margins under high renewable penetration and the need to consider Security limits within OPF formulations [4], the operational complexity of demand-side Heating, Ventilation, and Air Conditioning (HVAC) management under volatile prices [5], and planning/control coupling in hydrogen-integrated multi-energy systems [6] highlight the need for innovative solutions. In this regard, a case study focused on Shandong proposed a coordinated planning model for multi-regional ammonia industries, leveraging the integration of hydrogen supply chains and power grids to address the aforementioned challenges and emphasizing the synergy between renewable and traditional energy sources [7]. Additionally, the large-scale integration of distributed generation (DG) and electric vehicles (EV) calls for fast identification of structurally critical distribution nodes [8]. In contrast, proper coordination improves reliability and reduces peak loads in a cost-effective manner. Given the complexity of multi-energy coupling, temporal dynamics, and stakeholder diversity, centralized control methods often fail to provide adequate solutions, highlighting the need for intelligent, integrated approaches [9,10]. Related research indicates that methods based on multi-scale spatiotemporal graph neural networks can effectively predict loads in integrated energy systems, which lays the foundation for achieving higher system reliability and better capturing spatial and temporal dependencies [11]. For example, the MMGPT4LF study proposed a method that combines an optimized pre-trained GPT-2 model with multi-modal cross-attention, further improving the accuracy of load forecasting [12]. Additionally, research on optimizing attention mechanisms in transformers shows significant performance in multi-horizon and multi-energy load forecasting, enhancing system flexibility [13]. From a broader perspective, an assessment of multi-dimensional decarbonization technologies and pathways for the steel industry in China underscores the importance of targeted integrated solutions when achieving a mix of renewable energy and hydrogen. For instance, studies from an energy-process chain perspective illustrate the complex processes involved in achieving sustainable transformation and provide valuable insights for the industry [14]. Thus, this paper proposes a novel approach that combines Spatio-Temporal Transformers (STTs) with Multi-Agent Reinforcement Learning (MARL) for optimized DER coordination. We build on advancements in interpretable multi-horizon forecasting [15,16], providing emerging evidence that MARL is particularly well-suited for decentralized energy networks under CTDE with explicit coordination mechanisms [17], and physics-based formulations that support our sensitivity-based safety enforcement [4,18].

Advanced data-driven models effectively capture spatio-temporal patterns in energy systems. Spatio-temporal graph neural networks successfully integrate network topology and time series data, significantly improving renewable energy forecasts [19,20]. Transformers, originally developed for natural language processing, have been adapted to power systems for their ability to model temporal dependencies and node interactions. For instance, the Multi-scale Spatio-Temporal Transformer (MSTT) incorporates grid topology via hierarchical attention, significantly improving computational efficiency and accuracy [21]. Foundation models like the Graphormer-JEPA have been proposed for coupled road-power forecasting, achieving superior accuracy by jointly modeling transportation and power data [22]. Similarly, Wang et al. introduced TransPVP, using transformers for ultra-short-term photovoltaic predictions with heterogeneous data fusion [23]. Hybrid approaches like Transformer–CNN have also enhanced the accuracy of load prediction through better self-attention mechanisms compared to recurrent networks [24]. Complementing these efforts, recent day-ahead wind-cluster studies extract intrinsic predictable components to boost forecastability and robustness [16]. These studies collectively illustrate that STT approaches, which incorporate grid topology awareness, provide accurate and interpretable DER predictions.

In parallel, MARL has emerged as a powerful method for operational DER control due to its suitability for decentralized decision-making [25,26,27]. Surveys highlight MARL’s advantages for grid resiliency amid growing DER penetration [28]. MARL facilitates cooperative strategies between multiple autonomous agents, optimizing global objectives. For example, Qiu et al. successfully employed a modified MADDPG algorithm for peer-to-peer energy trading, which produced improved economic outcomes [29]. Yao et al. utilized MARL to coordinate the dispatch of DER and the reconfiguration of the network, effectively restoring services after outages [30]. MARL has also shown efficacy in microgrid control tasks such as voltage regulation and economic dispatch [31]. Cutting-edge approaches integrate MARL with deep generative models; for example, an EV charging coordination framework used multi-agent LLMs and conditional GANs to adaptively balance user preferences with grid constraints [32]. In parallel, recent encryption-based frameworks, such as [33], have further enhanced MARL’s applicability in DER coordination by ensuring privacy protection through secure communication mechanisms. Beyond these domain applications, key algorithmic building blocks—MADDPG for CTDE actor–critic [34], counterfactual credit assignment via COMA [35], and theory-driven syntheses of stability/scalability issues in MARL [36]—underpin our controller design and analysis. These advances underscore MARL’s growing role in real-time adaptive DER coordination.

In conclusion, research on coordinated DERs has advanced significantly. However, several challenges remain in current approaches, including the following:

• Many existing methods fail to effectively capture both the spatial and temporal dependencies in energy systems, limiting the accuracy and reliability of forecasts, especially in complex, decentralized environments like power grids.
• Centralized control methods often struggle to scale to large, dynamic systems with diverse stakeholders, and they do not fully address the need for real-time, decentralized decision-making.
• Current safety and feasibility mechanisms lack a comprehensive, real-time solution that simultaneously respects the operational limits of DERs while maintaining system performance, especially under unpredictable conditions.

To deal with these challenges, this paper makes the following contributions:

• A topology-aware Spatio-Temporal Transformer (STT) that integrates a graph diffusion kernel with electrical priors, addressing the gap inherent to effectively capturing both spatial and temporal dependencies for more accurate and reliable multi-horizon forecasting in energy systems.
• A cooperative Multi-Agent Reinforcement Learning (MARL) layer under centralized training and decentralized execution, incorporating forecast-informed observations, congestion-aware coordination, and consensus-based exploration. This solution addresses the scalability and real-time decision-making limitations of centralized approaches, enabling efficient DER coordination across diverse agents.
• A comprehensive real-time safety layer that projects device set-points onto state-dependent safe sets and solves a compact sensitivity-based quadratic program to ensure feeder voltage/current limits are respected, bridging the gap created by current methods that lack an integrated, real-time feasibility solution.

A concise comparison with recent representative methods is provided in

Table 1. Comparison with recent works on prediction, cooperative control, congestion awareness, safety, and decentralized execution.

Method (Abbr.)	STT Prediction	Multi-Agent	Congestion Aware	Real-Time Safety	Decentralized
Ours	Yes	Yes	Yes	Yes	Yes
MARL-AVC [37]	No	Yes	No	Partial	Yes
EC-MARL [38]	No	Yes	Partial	Partial	Yes
SafeRL-VVC [39]	No	Partial	No	Yes	Partial
AMARL-VPP [40]	No	Yes	Partial	No	Yes
GAN-MARL [41]	Partial	Yes	No	Partial	Yes
XSTGNN-PV [19]	Yes	No	No	No	n/a
FourierGNN-PV [42]	Yes	No	No	No	n/a
DC3 [43]	No	No	No	Yes	n/a

, highlighting our integration of an STT, cooperative CTDE control with congestion-aware coordination, and a real-time safety layer.

The remainder of the paper is organized as follows: Section 2 presents the method (STT, MARL, safety, and theoretical guarantees); Section 3 operationalizes the forecast→decide→safe execute loop with workflow and pseudocode; Section 4 describes the dataset, baselines, and results; and Section 5 concludes with deployment-oriented discussion and future research prospects.

2. Method

As shown in Figure 1, the framework comprises three tightly coupled layers. (i) A topology-aware STT ingests Node Temporal Feature Embedding together with a graph-based Spatial Embedding (the graph icon denotes adjacency/diffusion/priors). It stacks alternating Spatial Transformer and Temporal Transformer blocks (STT Blocks × L) and a Prediction Head to produce short-term, multi-horizon node forecasts ${\widehat{\mathit{\xi }}}_{t+1:t+H}$. (ii) A cooperative controller adopts CTDE with an Agents $\left\{{\pi }_i\right\}$ pool and a lightweight Coordinator that computes congestion-aware couplings $w_{ij,t}$ from path headroom, modulating a sparsified, permutation-invariant interaction critic $Q_{\theta }=Q_{\mathrm{base}}+{\sum }_{i,j}{w}_{ij,t}{Q}_{\mathrm{int}}$; consensus exploration is applied along electrical neighborhoods. Actions are optimized under a single shared reward $r_t$ that includes a coordination term (weighted by ${\lambda }_C$) to promote congestion-aware cooperation, rather than separate “agent” and “coordinator” rewards. (iii) A Safety Layer performs Device Projection at the device level (bounds, ramp limits, SoC gating, on/off feasibility) and then solves a compact, sensitivity-based QP to enforce feeder voltage/current limits, outputting feasible set-points ${\tilde{\mathbf{a}}}_t$. Together, these modules realize a closed loop of forecas → multi-agent decision → safe execution over real DERs (PV, storage, and controllable loads), improving reliability and scalability while preserving interpretability and real-time feasibility.

2.1. Topology-Aware Spatio-Temporal Transformer (STT)

The Topology-Aware Spatio-Temporal Transformer (STT) is designed to address the unique challenges of coordinating DERs in complex power systems. This model integrates spatial and temporal forecasting by leveraging the strengths of transformer architectures while accounting for the inherent topology of the distribution network.

We model the distribution network as a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with $\left|\mathcal{V}\right|=N$ nodes (DERs or aggregations). Over an input window of length $T_{\mathrm{in}}$, node features are stacked as

$${\mathbf{X}}_{t-T_{\mathrm{in}}+1:t}\in {\mathbb{R}}^{N\times T_{\mathrm{in}}\times d_x},{\widehat{\mathbf{Y}}}_{t+1:t+H}\in {\mathbb{R}}^{N\times H\times d_y},$$

(1)

where the goal is to forecast H future steps for all nodes. For each node i, we first extract its temporal sequence and apply feature embedding:

$${\mathbf{Z}}_i=\mathrm{Embed}\left({\mathbf{X}}_{i,t-T_{\mathrm{in}}+1:t}\right)+\mathbf{PE}\in {\mathbb{R}}^{T_{\mathrm{in}}\times d},$$

(2)

where $\mathrm{Embed}(·)$ is a linear projection to dimension d and $\mathbf{PE}$ denotes positional encodings. In practice, we sum multi-resolution temporal encodings (time-of-day and day-of-week) with a learned affine calibration; this preserves seasonality while letting the model adapt to feeder-specific rhythms. Inputs are standardized per-channel using training-set statistics, and a binary mask handles missing samples without information leakage. In our SimBench experiments there is no missingness; the mask is part of the general implementation.

The encoder alternates temporal attention (along each node’s timeline) and spatial attention (across nodes within a time slice), each followed by pre-norm LayerNorm, residual connections, and a lightweight position-wise feed-forward block (GELU + linear). Stacking L such layers empirically balances accuracy and latency; we use modest d and head counts to keep the memory footprint linear in $T_{\mathrm{in}}$ and quasi-linear in N after sparsification.

Temporal attention updates each time step by attending over the node’s own history:

$${\mathit{\alpha }}_i=\mathrm{softmax}\left({\displaystyle \frac{{\mathbf{Q}}_i{\mathbf{K}}_i^{\top }}{\sqrt{d}}}\right),\mathrm{TempAttn}\left({\mathbf{Z}}_i\right)={\mathit{\alpha }}_i{\mathbf{V}}_i,$$

(3)

where ${\mathbf{Q}}_i$, ${\mathbf{K}}_i$, ${\mathbf{V}}_i$ are query, key, and value projections of ${\mathbf{Z}}_i$. During training we default to a direct multi-horizon head to avoid exposure bias; when autoregression is used (e.g., for long horizons), we apply causal masks and scheduled sampling to stabilize rollouts.

Topology-aware spatial attention injects graph structure and electrical priors into attention scores at each time slice. Let $\mathbf{A}\in {\{0,1\}}^{N\times N}$ denote adjacency and $\mathbf{L}=\mathbf{D}-\mathbf{A}$ the graph Laplacian, where $\mathbf{D}$ is the degree matrix. We build a smooth proximity kernel by graph diffusion

$$\mathbf{P}=\mathrm{row}\text{-}\mathrm{norm}\left(exp(-{\alpha }_{\mathrm{diff}}\mathbf{L})\right),\alpha >0,$$

(4)

where ${\alpha }_{\mathrm{diff}}$ controls the diffusion range (typically ${\alpha }_{\mathrm{diff}}\in [0.5,2.0]$ based on network diameter). We combine this with optional electrical priors $\psi (i,j)$ (e.g., phase consistency, feeder membership, or inverse-impedance surrogates). The spatial attention logit between nodes i and j at time $\tau$ is

$$s_{ij}\left(\tau \right)={\displaystyle \frac{{\mathbf{q}}_i\left(\tau \right){\mathbf{k}}_j{\left(\tau \right)}^{\top }}{\sqrt{d}}}+{\gamma }_1{A}_{ij}+{\gamma }_2{P}_{ij}+{\gamma }_3\psi (i,j),$$

(5)

with attention weights ${\alpha }_{ij}\left(\tau \right)={\mathrm{softmax}}_{j\in \mathcal{N}\left(i\right)}\left(s_{ij}\left(\tau \right)\right)$ over a sparsified neighborhood $\mathcal{N}\left(i\right)=\{j:P_{ij}>ϵ\mathrm{or}\mathrm{hop}-\mathrm{dist}(i,j)\le k\}$, where $ϵ=0.01$ and $k=2$ in our implementation. The coefficients $({\gamma }_1,{\gamma }_2,{\gamma }_3)$ are learned and passed through a softplus function to keep them nonnegative, ensuring that structural and electrical proximity never becomes adversarially down-weighted. We precompute $\mathbf{P}$ and $\mathcal{N}\left(i\right)$ per feeder, and optionally refresh them when a reconfiguration is detected. This amortizes runtime while remaining robust to moderate topology drift. After sparsification, the spatial step scales are $\mathcal{O}\left(NKd\right)$ per head with $K=\left|\mathcal{N}\right(i\left)\right|\ll N$.

A lightweight prediction head maps the final spatio-temporal embeddings to multi-horizon outputs. For direct forecasting we use a two-layer head (linear→GELU→linear) to produce ${\widehat{\mathbf{Y}}}_{t+1:t+H}$ in one shot; for autoregressive decoding the same head shares weights across steps. Dropout is applied only in the feed-forward and attention projections to preserve inductive biases of the kernels in (4) and (5).

The model is trained end-to-end with a node- and horizon-averaged prediction loss and a lightweight topology–physics regularizer:

$${\mathcal{L}}_{\mathrm{STT}}=\underset{{\mathcal{L}}_{\mathrm{pred}}}{\underbrace{\mathrm{MSE}\left({\widehat{\mathbf{Y}}}_{t+1:t+H},{\mathbf{Y}}_{t+1:t+H}\right)}}+{\lambda }_{\mathrm{reg}}\underset{{\mathcal{L}}_{\mathrm{reg}}}{\underbrace{{\mathrm{TV}}_{\mathcal{G}}\left(\widehat{\mathbf{Y}}\right)+\eta \text{PF-res}\left(\widehat{\mathbf{Y}}\right)}},$$

(6)

where ${\mathrm{TV}}_{\mathcal{G}}$ promotes graph-consistent smoothness and $\text{PF-res}$ penalizes linearized power flow inconsistencies. Concretely, we instantiate

$${\mathrm{TV}}_{\mathcal{G}}\left(\widehat{\mathbf{Y}}\right)=\sum _{\tau =1}^H\mathrm{tr}\left({\widehat{\mathbf{Y}}}_{:,\tau }^{\top }{\mathbf{L}}_w{\widehat{\mathbf{Y}}}_{:,\tau }\right),{\mathbf{L}}_w={\mathbf{D}}_w-\mathbf{W},\mathbf{W}={\beta }_1\mathbf{A}+{\beta }_2\mathbf{P},$$

(7)

with nonnegative $({\beta }_1,{\beta }_2)$ learned and row-normalized to keep the scale of (7) compatible with ${\mathcal{L}}_{\mathrm{pred}}$. This Laplacian quadratic form is equivalent to summing squared differences across edges weighted by structural/electrical proximity and empirically stabilizes training in sparse feeders.

To connect forecasts with network physics, we define a sensitivity-based residual using a linearized DistFlow/AC model around the latest operating point. Let ${\widehat{\mathbf{p}}}_{\tau },{\widehat{\mathbf{q}}}_{\tau }$ denote predicted net injections and $\Delta {\widehat{\mathbf{v}}}_{\tau }$ the predicted voltage magnitude deviations (all taken from the relevant channels of $\widehat{\mathbf{Y}}$). With precomputed sensitivities $(\mathbf{S},\mathbf{T})$

$$\mathrm{PF}-\mathrm{res}\left(\widehat{\mathbf{Y}}\right)=\sum _{\tau =1}^H\parallel \mathbf{S}{\widehat{\mathbf{p}}}_{\tau }+\mathbf{T}{\widehat{\mathbf{q}}}_{\tau }-\Delta {\widehat{\mathbf{v}}}_{\tau }{\parallel }_2^2,$$

(8)

which softly enforces first-order voltage consistency without running a full power flow in the learning loop. We stop gradients through $(\mathbf{S},\mathbf{T})$ and normalize each term by feeder-specific scales to avoid dominance by a single channel.

Implementation-wise, all kernels ($\mathbf{A},\mathbf{P},\psi$) are ablated in experiments to quantify their marginal utility; turning off $\psi$ reduces spatial attention to structure-only, turning off $\mathbf{P}$ further collapses to hop-limited attention, and learning with ${\gamma }_1={\gamma }_2={\gamma }_3=0$ recovers a vanilla Transformer. Hyperparameters $(\alpha ,ϵ,k,{\lambda }_{\mathrm{reg}},\eta )$ are selected on a validation feeder; we found $\alpha \in [0.5,2.0]$, $k\in \{2,3\}$, and small ${\lambda }_{\mathrm{reg}}$ (${10}^{-3}$–${10}^{-2}$) to yield stable gains across feeders. At inference time, all preprocessing (standardization, kernel lookup, neighborhood construction) is batched, and the overall complexity scales as $\mathcal{O}\left(L{T}_{\mathrm{in}}NKd\right)$, which fits comfortably within real-time constraints for the feeder sizes considered.

2.2. Cooperative Multi-Agent Reinforcement Learning with Adaptive Coordination

We cast DER coordination as a cooperative Markov game with $N_a$ agents. Beyond standard MADDPG, we introduce three key innovations: (i) hierarchical action decomposition for handling hybrid discrete-continuous control, (ii) adaptive coordination weights based on network congestion, and (iii) consensus-based exploration for improved sample efficiency. From a systems perspective, these choices reflect practical DER actuation: modes (on/off/charge/discharge) are intrinsically discrete, whereas set-points are continuous and must respect ramp, SoC, and inverter capability limits; network congestion further couples otherwise local decisions along electrically salient paths. We adopt centralized training and decentralized execution (CTDE); actors are local policies ${\left\{{\pi }_{{\varphi }_i}\right\}}_{i=1}^{N_a}$ executed with each agent’s observation, while a centralized critic leverages global (or enlarged) information during training to shape credit assignment and stabilize learning. At deployment, each agent uses only local (plus short-horizon forecast) information, preserving scalability and privacy.

At time t, the environment state is $s_t\in \mathcal{S}$, agent i observes $o_{i,t}\in {\mathcal{O}}_i$, and selects a hybrid action

$$a_{i,t}=(a_{i,t}^d,a_{i,t}^c)\in {\mathcal{A}}_i={\mathcal{A}}_i^d\times {\mathcal{A}}_i^c,$$

(9)

where $a_{i,t}^d$ represents discrete modes (e.g., charge/discharge/idle) and $a_{i,t}^c\in \mathbb{R}$ is the continuous power set-point. Observations include local measurements (voltages, powers, and device states) and neighborhood summaries; normalization and clipping are applied per channel using training set statistics to avoid out-of-distribution magnitudes. We adopt per-channel affine normalization with statistics frozen after the warm-up phase and clip observations to $3\sigma$ to bound critic targets.

Each agent employs a hierarchical policy as follows:

$${\pi }_{{\varphi }_i}(a_{i,t}\mid o_{i,t})={\pi }_{{\varphi }_i}^d(a_{i,t}^d\mid o_{i,t})·{\pi }_{{\varphi }_i}^c(a_{i,t}^c\mid o_{i,t},a_{i,t}^d),$$

(10)

where the discrete policy selects the operation mode, and the continuous policy determines the magnitude conditioned on the mode. In training, the discrete branch is implemented with a differentiable Gumbel-Softmax relaxation (straight through at execution), and the continuous branch outputs a squashed Gaussian (Tanh) to respect device bounds. This decomposition naturally encodes mode-conditioned limits and facilitates credit assignment when discrete switches are sparse yet impactful. We anneal the Gumbel temperature ${\tau }_d$ from $1.0$ to $0.1$ over the first K epochs to gradually sharpen mode selection, and we clip the continuous standard deviation to $[{\sigma }_{min},{\sigma }_{max}]$ to avoid vanishing/explosive exploration. Illegal modes (e.g., charging at $\mathrm{SoC}\approx {\mathrm{SoC}}_{max}$) are masked by $-\infty$ logits, yielding zero probability mass and eliminating infeasible gradients.

We introduce a congestion-aware coordination weight that dynamically adjusts inter-agent coupling as follows:

$$w_{ij,t}=\sigma \left({\beta }_0+{\beta }_1·{\mathrm{CI}}_{ij,t}+{\beta }_2·{\parallel {\mathbf{v}}_t-{\mathbf{v}}_{\mathrm{nom}}\parallel }_{\infty }\right),$$

(11)

where ${\mathrm{CI}}_{ij,t}$ is the congestion index on the path between agents i and j, ${\mathbf{v}}_t$ are voltage magnitudes, and $\sigma (·)$ is the sigmoid function. The index ${\mathrm{CI}}_{ij,t}$ aggregates thermal/voltage headroom along electrical paths computed from the current feeder configuration; gradients do not backpropagate through path selection, while the coefficients ${\beta }_{\ell }$ are learned (with softplus reparameterization for stability). Concretely, let $P(i,j)$ denote the set of branches on the unique (radial) or selected (meshed) electrical path between buses of agents i and j. Define per-branch thermal headroom $h_{e,t}^I=1-{\ell }_{e,t}$ with loading ratio ${\ell }_{e,t}\in [0,1]$, and per-bus voltage headroom $h_{b,t}^V=1-{\delta }_{b,t}^V/\Delta V$, where ${\delta }_{b,t}^V=max\{0,|V_{b,t}|-V^{max},V^{min}-|V_{b,t}\left|\right\}$, and $\Delta V=V^{max}-V^{min}$. We set a path headroom $H_{ij,t}={min}_{e\in P(i,j)}min\{h_{e,t}^I,{min}_{b\in e}{h}_{b,t}^V\}$ and normalize the congestion index as ${\mathrm{CI}}_{ij,t}=1-H_{ij,t}\in [0,1]$. This choice emphasizes the most limiting element along the corridor and yields larger $w_{ij,t}$ only when genuinely scarce headroom necessitates coordinated action. The centralized critic incorporates these weights as follows:

$$Q_{\theta }(s_t,{\mathbf{a}}_t)=Q_{\mathrm{base}}(s_t,{\mathbf{a}}_t)+\sum _{i,j}{w}_{ij,t}·Q_{\mathrm{int}}(o_{i,t},o_{j,t},a_{i,t},a_{j,t}),$$

(12)

where $Q_{\mathrm{base}}$ captures individual contributions and $Q_{\mathrm{int}}$ models pairwise interactions. The pairwise term is computed only over a sparsified electrical neighborhood (two-hop or $P_{ij}>ϵ$), keeping the critic evaluation $\mathcal{O}\left(\right|\mathcal{E}\left|\right)$. We evaluate $Q_{\mathrm{int}}$ with a symmetric aggregator to preserve permutation invariance and detach $w_{ij,t}$ with regard to critic inputs to avoid feedback loops while learning ${\beta }_{\ell }$.

Each agent’s observation is augmented with STT predictions as follows:

$${\tilde{o}}_{i,t}=\left[o_{i,t};{\widehat{\mathit{\xi }}}_{i,t+1:t+H};{\mathrm{CI}}_t;\Delta {\mathbf{v}}_{\mathrm{local}}\right],$$

(13)

where ${\widehat{\mathit{\xi }}}_{i,t+1:t+H}$ are local forecasts, ${\mathrm{CI}}_t$ is the network congestion index, and $\Delta {\mathbf{v}}_{\mathrm{local}}$ represents local voltage deviations. To mitigate covariate shift, forecasts are stop-gradient features and are standardized per horizon; we additionally include a recency flag and uncertainty proxy (e.g., predictive variance) when available. Empirically, exposing only short-horizon summaries (e.g., next-steps mean/variance or small-window embeddings) yields lower variance critics than feeding full forecast trajectories; we therefore concatenate compact statistics and keep their gradients blocked.

To improve coordination during exploration, we introduce a consensus term in the exploration noise as follows:

$${ϵ}_{i,t}=(1-\rho )·\mathcal{N}(0,{\sigma }_i^2)+\rho ·{\overline{ϵ}}_{\mathcal{N}\left(i\right),t},$$

(14)

where ${\overline{ϵ}}_{\mathcal{N}\left(i\right),t}$ is the average noise of neighboring agents and $\rho \in [0,1]$ controls the consensus strength. Neighborhoods are electrical (not geometric), and $\rho$ is annealed from 0 to a small ${\rho }_{max}$ early in training to encourage coherent but diverse exploration; an Ornstein–Uhlenbeck variant can be used for temporally correlated devices. We compute ${\overline{ϵ}}_{\mathcal{N}\left(i\right),t}={\displaystyle \frac{1}{\left|\mathcal{N}\right(i\left)\right|}}{\sum }_{j\in \mathcal{N}\left(i\right)}{ϵ}_{j,t}^{\mathrm{iid}}$ from independent Gaussians to avoid circular dependence. The consensus term is disabled at evaluation time. Intuitively, when congestion couples agents along a corridor, correlated exploration accelerates discovery of coordinated policies while keeping dispersion low off-corridor due to small $w_{ij,t}$.

The shared reward balances multiple objectives as follows:

$$r_t=-c_{\mathrm{op}}(s_t,{\mathbf{a}}_t)-{\lambda }_V\mathrm{Vviol}\left(s_t\right)-{\lambda }_T\mathrm{Therm}\left(s_t\right)-{\lambda }_S\mathrm{Smooth}\left({\mathbf{a}}_t\right)+{\lambda }_C\mathrm{Coord}\left({\mathbf{a}}_t\right),$$

(15)

where $\mathrm{Coord}\left({\mathbf{a}}_t\right)=-{\sum }_{i,j\in \mathcal{E}}{w}_{ij,t}{\parallel a_{i,t}-a_{j,t}\parallel }^2$ promotes coordination during congestion. Cost $c_{\mathrm{op}}$ aggregates energy price and cycling penalties; $\mathrm{Vviol}$ and $\mathrm{Therm}$ are hinge losses on voltage and current headroom; and $\mathrm{Smooth}$ is total variation of actions over time. All weights are nonnegative and tuned on validation feeders; we clip rewards to a fixed range to stabilize critic targets. We normalize each term by feeder-level scales (e.g., average load or nominal voltage band) to balance gradients across feeders and adopt per-timestep reward clipping to $[-R_{max},R_{max}]$ for stability.

Training follows an off-policy CTDE pipeline with a replay buffer storing $(s_t,{\mathbf{o}}_t,{\mathbf{a}}_t,r_t,s_{t+1},{\mathbf{o}}_{t+1})$ and target networks. The critic is optimized by minimizing the temporal difference loss

$${\mathcal{L}}_Q\left(\theta \right)=\mathbb{E}\left[{\left(Q_{\theta }(s_t,{\mathbf{a}}_t)-y_t\right)}^2\right],y_t=r_t+\gamma {\mathbb{E}}_{{\mathbf{a}}_{t+1}\sim \pi }\left[{\overline{Q}}_{\overline{\theta }}(s_{t+1},{\mathbf{a}}_{t+1})\right],$$

(16)

where ${\overline{Q}}_{\overline{\theta }}$ is the target critic with delayed parameters. For robustness against overestimation we use twin critics and target-min (TD3-style) for the continuous branch. The actor update maximizes the expected Q under the hierarchical policy; with reparameterizations for both branches, the gradient takes the standard form

$${\mathcal{L}}_{\pi }\left(\left\{{\varphi }_i\right\}\right)=-\mathbb{E}\left[\sum _{i=1}^{N_a}{Q}_{\theta }\left(s_t,a_{1:t}^{-i},a_i\sim {\pi }_{{\varphi }_i}(·\mid {\tilde{o}}_{i,t})\right)\right]+\alpha \mathcal{H},$$

(17)

where $\mathcal{H}$ denotes an entropy regularizer (on the discrete logits and continuous variance) with temperature $\alpha$. Target networks are updated by Polyak averaging $\overline{\theta }\leftarrow \tau \theta +(1-\tau )\overline{\theta }$ and similarly for actor parameters, with a small $\tau \in [{10}^{-3},{10}^{-2}]$. In addition to the pairwise critic, we optionally compute a counterfactual baseline

$$A_i(s_t,{\mathbf{a}}_t)=Q_{\theta }(s_t,{\mathbf{a}}_t)-{\mathbb{E}}_{a_i^{\prime }\sim {\pi }_{{\varphi }_i}(·\mid {\tilde{o}}_{i,t})}\left[Q_{\theta }(s_t,(a_{-i,t},a_i^{\prime }))\right],$$

which reduces gradient variance (COMA-style) and is evaluated efficiently using the sparsified interaction structure. We apply gradient clipping (${\ell }_2$ norm $\le g_{max}$) and prioritized replay with importance weights to stabilize updates across feeders and stress conditions.

Implementation details improve stability and scalability. (i) We mask illegal modes in ${\pi }_{{\varphi }_i}^d$ via $-\infty$ logits to enforce device availability and ramp limits at the policy level. (ii) The pairwise term $Q_{\mathrm{int}}$ uses a symmetric aggregator over neighbors to maintain permutation invariance and is computed on a pruned graph to bound cost. (iii) Congestion weights $w_{ij,t}$ are detached with regard to the critic input but keep gradients for the learnable coefficients ${\beta }_{\ell }$, preventing spurious feedback loops. (iv) We employ prioritized replay with importance sampling corrections, and gradient norm clipping for both actor and critic. (v) During data collection, actions are post-processed by the safety layer (Section 2.3) before environment execution; both raw and filtered actions are logged to decouple exploration from feasibility and to reduce distributional drift. Complexity-wise, training and inference scale linearly with the number of active electrical edges due to sparsified neighborhoods; decentralized execution requires only ${\tilde{o}}_{i,t}$ and neighbor summaries, meeting real-time constraints on the studied feeders. Ablations show that replacing the hierarchical policy with a flat continuous policy degrades discrete-switching efficiency, fixing $w_{ij,t}$ harms performance under heavy congestion, and disabling consensus noise delays the emergence of corridor-level coordination.

2.3. Safety Layer

The Safety Layer guarantees operational feasibility through real-time constraint enforcement. It acts after the policy proposes actions and before interacting with the environment, so that device- and network-level limits are always respected. The layer is modular: a fast device-wise projection enforces local bounds and ramping/SoC feasibility, followed by a compact network QP that corrects any residual violations due to voltage/current limits.

For each agent i, we define the state-dependent safe set

$${\mathcal{A}}_i\left(s_t\right)=\left\{a\in \mathbb{R}|\begin{array}{cc}\hfill & P_i^{min}\le a\le P_i^{max},\hfill \\ \hfill & |a-a_{i,t-1}|\le r_i,\hfill \\ \hfill & {\mathrm{SOC}}^{min}\le {\mathrm{SOC}}_{i,t}+{\displaystyle \frac{\Delta t·a}{{\eta }_i{E}_i}}\le {\mathrm{SOC}}^{max}\hfill \end{array}\right\},$$

(18)

where ${\eta }_i$ is (dis)charging efficiency and $E_i$ the energy capacity. The SoC update map is $f_i\left(a\right)={\displaystyle \frac{a}{{\eta }_i{E}_i}}$ under the sign convention that $a>0$ denotes charging (adjusted accordingly for devices with opposite sign conventions). In this formulation, we do not enforce the daily cycle constraint ${\mathrm{SOC}}_{i,t+1}={\mathrm{SOC}}_{i,t}$ at the end of the time horizon. Instead, we allow the state of charge (SoC) to evolve dynamically across multiple time steps, reflecting operational flexibility over time. This flexibility is particularly useful in real-world grid environments where energy storage and generation are subject to high variability.

Given a proposed action $a_{i,t}$, we apply

$${\tilde{a}}_{i,t}={\Pi }_{{\mathcal{A}}_i\left(s_t\right)}\left(a_{i,t}\right)=arg\underset{a\in {\mathcal{A}}_i\left(s_t\right)}{min}{\parallel a-a_{i,t}\parallel }_2^2,$$

(19)

which reduces to clamping $a_{i,t}$ to the intersection interval $I_i\left(s_t\right)=[P_i^{min},P_i^{max}]\cap [a_{i,t-1}-r_i,a_{i,t-1}+r_i]\cap [a_{\mathrm{soc}}^{min},a_{\mathrm{soc}}^{max}],$ with $a_{\mathrm{soc}}^{min}=\frac{{\eta }_i{E}_i}{\Delta t}\left({\mathrm{SOC}}^{min}-{\mathrm{SOC}}_{i,t}\right),a_{\mathrm{soc}}^{max}=\frac{{\eta }_i{E}_i}{\Delta t}\left({\mathrm{SOC}}^{max}-{\mathrm{SOC}}_{i,t}\right).$ If $I_i\left(s_t\right)$ is empty due to incompatible limits, we enlarge the ramp window minimally (equivalently, introduce a small nonnegative slack ${\zeta }_i$ penalized in the network QP below), which preserves convexity and ensures feasibility even under abrupt disturbances. Additional device rules (e.g., apparent power P–Q capability) are handled by masking illegal discrete modes upstream and by shrinking $[P_i^{min},P_i^{max}]$ online.

To respect feeder limits, we solve a small QP with precomputed sensitivities around the current operating point as follows:

$${\tilde{\mathbf{a}}}_t=arg\underset{\mathbf{a}}{min}{\parallel \mathbf{a}-{\mathbf{a}}_t\parallel }_2^2\mathrm{s}.\mathrm{t}.{\mathbf{S}}_V\mathbf{a}\le {\mathbf{v}}_{max},{\mathbf{S}}_I\mathbf{a}\le {\mathbf{i}}_{max},$$

(20)

where ${\mathbf{S}}_V,{\mathbf{S}}_I$ are voltage and current sensitivity matrices. In practice we augment (20) with box constraints $\mathbf{a}\in [\mathbf{l},\mathbf{u}]$ (device bounds after the projection in (19)) and optional slacks $\mathit{\xi }\ge 0$ weighted by large penalties to improves feasibility (under a linearized model) under model mismatch

$$\underset{\mathbf{a},\mathit{\xi }\ge 0}{min}\parallel \mathbf{a}-{\mathbf{a}}_t{\parallel }_2^2+{\mathit{\mu }}_V\parallel {\mathit{\xi }}_V{\parallel }_2^2+{\mathit{\mu }}_I{\parallel {\mathit{\xi }}_I\parallel }_2^2\mathrm{s}.\mathrm{t}.{\mathbf{S}}_V\mathbf{a}\le {\mathbf{v}}_{max}+{\mathit{\xi }}_V,{\mathbf{S}}_I\mathbf{a}\le {\mathbf{i}}_{max}+{\mathit{\xi }}_I,\mathbf{l}\le \mathbf{a}\le \mathbf{u}.$$

(21)

Note. Constraints are enforced with respect to linearized sensitivities ${\mathbf{S}}_V,{\mathbf{S}}_I$ around the current operating point; hence feasibility refers to the linear model. In experiments, we verify a posteriori with full AC power flow that violations remain small.

The objective is strictly convex and the feasible region is convex; hence a unique minimizer exists and depends continuously on $({\mathbf{S}}_V,{\mathbf{S}}_I,{\mathbf{v}}_{max},{\mathbf{i}}_{max})$. We pre-factor the KKT system once per operating-point refresh and reuse it across steps, yielding sub-millisecond solves at the feeder scales considered. The overall complexity scales with the number of active inequality constraints; sparsity from feeder topology further reduces cost.

2.4. Theoretical Guarantees

We provide brief theoretical analysis for the key components of our framework, focusing on convergence and safety properties that hold under mild regularity conditions.

Under standard assumptions (Lipschitz continuous loss and bounded gradients), the STT training converges to a stationary point using first-order methods. The topology-aware regularization also yields a graph-smoothing effect. Let ${\widehat{\mathbf{Y}}}^{*}$ be the optimal STT predictor. For connected nodes $i,j$ with $P_{ij}>\delta$, the following bound holds:

$$\mathbb{E}\left[\parallel {\widehat{y}}_{i,t+h}^{*}-{\widehat{y}}_{j,t+h}^{*}{\parallel }^2\right]\le {\displaystyle \frac{C}{{\lambda }_{\mathrm{reg}}}}·{\displaystyle \frac{1}{P_{ij}}},$$

(22)

where C depends on data smoothness and model capacity. Thus larger diffusion proximity $P_{ij}$ tightens cross-node prediction coupling while ${\lambda }_{\mathrm{reg}}$ controls the bias–variance trade-off.

For the MARL component, we consider hierarchical policies under CTDE with bounded rewards and compact action spaces. With learning rates satisfying the Robbins–Monro conditions and with target networks for stabilization, the policy updates produce asymptotic improvement up to vanishing errors as follows:

$$J\left({\mathit{\pi }}_{k+1}\right)\ge J\left({\mathit{\pi }}_k\right)-{ϵ}_k,$$

(23)

where ${ϵ}_k\to 0$ as $k\to \infty$. The pairwise interaction critic remains well-defined because the sparsified neighborhood bounds the number of interaction terms; in particular, the critic gradient variance is controlled by the congestion weights, which shrink toward zero off-congestion, mitigating nonstationarity.

For safety guarantees, the device-wise stage is a projection onto a nonempty closed interval (firmly nonexpansive), while the network-wise stage solves a strictly convex QP (a prox-like mapping). Device-wise, ${\Pi }_{{\mathcal{A}}_i\left(s_t\right)}$ is the Euclidean projection onto a nonempty closed interval and is firmly nonexpansive

$$\parallel {\Pi }_{{\mathcal{A}}_i\left(s_t\right)}\left(x\right)-{\Pi }_{{\mathcal{A}}_i\left(s_t\right)}{\left(y\right)\parallel }_2\le {\parallel x-y\parallel }_2,$$

(24)

which implies robustness to actor noise. Network-wise, the QP in (21) is strictly convex; thus the filtered action is unique and satisfies all constraints (or violates them minimally via slacks with quadratic penalties). Consequently, for any proposed action ${\mathbf{a}}_t$, the filtered action ${\tilde{\mathbf{a}}}_t$ obeys

$$\left\{\begin{array}{cc}{\tilde{\mathbf{a}}}_t\in \bigcap _{i=1}^{N_a}{\mathcal{A}}_i\left(s_t\right)\cap {\mathcal{C}}_{\mathrm{network}},\hfill & \mathrm{if}{\mathit{\xi }}_V={\mathit{\xi }}_I=\mathbf{0},\hfill \\ {\mathbf{S}}_V{\tilde{\mathbf{a}}}_t\le {\mathbf{v}}_{max}+{\mathit{\xi }}_V,{\mathbf{S}}_I{\tilde{\mathbf{a}}}_t\le {\mathbf{i}}_{max}+{\mathit{\xi }}_I,\hfill & \mathrm{otherwise}\left(\mathrm{slack}\text{-}\mathrm{enforced}\right).\hfill \end{array}\right.$$

(25)

and among all feasible actions it minimizes the Euclidean deviation from ${\mathbf{a}}_t$. Moreover, the deviation admits a sensitivity-type upper bound that scales with raw violations as follows:

$$\parallel {\tilde{\mathbf{a}}}_t-{\mathbf{a}}_t{\parallel }_2\le {\kappa }_V\parallel {({\mathbf{S}}_V{\mathbf{a}}_t-{\mathbf{v}}_{max})}_{+}{\parallel }_2+{\kappa }_I{\parallel {({\mathbf{S}}_I{\mathbf{a}}_t-{\mathbf{i}}_{max})}_{+}\parallel }_2,$$

where ${(·)}_{+}$ denotes the positive part and ${\kappa }_V,{\kappa }_I$ depend on the QP KKT matrix condition numbers.

Combining the three layers yields an additive performance decomposition. Let $J^{*}$ be the optimal value under perfect information and unconstrained execution. The deployed policy ${\pi }_{\mathrm{STT}-\mathrm{MARL}}$ satisfies

$$J^{*}-J\left({\pi }_{\mathrm{STT}-\mathrm{MARL}}\right)\le \underset{\mathrm{forecast}\mathrm{error}}{\underbrace{{ϵ}_{\mathrm{pred}}}}+\underset{\mathrm{policy}\mathrm{gap}}{\underbrace{{ϵ}_{\mathrm{MARL}}}}+\underset{\mathrm{constraint}\cos \mathrm{t}}{\underbrace{{ϵ}_{\mathrm{safety}}}},$$

(26)

where ${ϵ}_{\mathrm{pred}}$ decreases with STT generalization error and regularization strength, ${ϵ}_{\mathrm{MARL}}$ decreases with training time and critic accuracy under CTDE, and ${ϵ}_{\mathrm{safety}}$ depends on typical magnitudes of raw violations and on the QP penalty parameters $({\mu }_V,{\mu }_I)$. Since both projections are nonexpansive and the QP correction is minimal in ${\ell }_2$, the Safety Layer does not amplify action noise and yields a controlled, monotone trade-off between feasibility and optimality. These guarantees ensure reliable closed-loop performance with approximate safety under linearized constraints, while the modular design allows each component to be improved independently without re-deriving the others.

3. Workflow and Pseudocode

This section operationalizes the closed loop of forecast → decide → safe execution in two routines. At each time t (i) the STT (Section 2.1) consumes the most recent window ${\mathbf{X}}_{t-W+1:t}$ and outputs H-step node-wise forecasts ${\widehat{\mathbf{y}}}_{t+1:t+H}$; (ii) each agent (Section 2.2) forms an observation by concatenating local measurements with short-horizon summaries of $\widehat{\mathbf{y}}$ (and optional neighborhood embeddings via $w_{ij,t}$), then produces a hierarchical hybrid action; and (iii) the safety layer (Section 2.3) first projects device set-points onto state-dependent safe sets and then solves a compact sensitivity-based QP to satisfy feeder voltage/current limits, yielding the feasible dispatch ${\tilde{\mathbf{a}}}_t$. During training (Algorithm 1), we store both proposed and filtered actions to decouple exploration from feasibility, update critics/policies under CTDE using a replay buffer, and fit the STT with a supervised loss on rolling windows. Deployment (Algorithm 2) keeps the same data flow with greedy actions. Sensitivities $({\mathbf{J}}_v,{\mathbf{J}}_i)$ are refreshed periodically from a local linearization or a fast surrogate, and agent steps execute in parallel, so the per-step cost is dominated by one STT forward pass and a small QP.

Algorithm 1 Training: Forecast-informed CTDE with safety-aware rollouts

Inputs: historical measurements ${\mathbf{X}}_{1:T}$, graph $G=(\mathcal{V},\mathcal{E})$, window W, horizon H, replay buffer $\mathcal{D}$   1:   Initialize STT params ${\theta }_f$, policy params ${\theta }_{\pi }$, critic params ${\theta }_Q$   2:   for $t=W$ to $T-H$ do   3:          ${\widehat{\mathbf{y}}}_{t+1:t+H}\leftarrow \mathtt{STTForecast}({\mathbf{X}}_{t-W+1:t},G;{\theta }_f)$   4:          for each agent $i\in \mathcal{V}$ (in parallel) do   5:                  $o_{i,t}\leftarrow \mathtt{BuildObs}(i,t,\widehat{\mathbf{y}},{\mathbf{x}}_t,\mathrm{neighbors})$   6:                  $a_{i,t}\sim {\pi }_{{\theta }_{\pi }}(·|o_{i,t})$                  ▹ hierarchical hybrid action   7:          end for   8:          ${\tilde{\mathbf{a}}}_t\leftarrow \mathtt{SafetyFilter}({\mathbf{a}}_t,{\mathbf{s}}_t,G)$   9:          Apply ${\tilde{\mathbf{a}}}_t$ in environment/simulator; observe $r_t,{\mathbf{s}}_{t+1}$ 10:          Store $({\mathbf{o}}_t,{\mathbf{a}}_t,{\tilde{\mathbf{a}}}_t,r_t,{\mathbf{o}}_{t+1})$ into $\mathcal{D}$ 11:          Update critics ${\theta }_Q$ and policies ${\theta }_{\pi }$ from batches in $\mathcal{D}$ (CTDE) 12:          Update ${\theta }_f$ by minimizing ${\mathcal{L}}_{\mathrm{STT}}$ on rolling windows 13:   end for Outputs: trained ${\theta }_f,{\theta }_{\pi }$ (and target networks if used)

Algorithm 2 Deployment: forecast → decide → safety filter

Inputs: latest window ${\mathbf{X}}_{t-W+1:t}$, graph G, trained params ${\theta }_f,{\theta }_{\pi }$   1:   ${\widehat{\mathbf{y}}}_{t+1:t+H}\leftarrow \mathtt{STTForecast}({\mathbf{X}}_{t-W+1:t},G;{\theta }_f)$   2:   for each agent $i\in \mathcal{V}$ (in parallel) do   3:          $o_{i,t}\leftarrow \mathtt{BuildObs}(i,t,\widehat{\mathbf{y}},{\mathbf{x}}_t,\mathrm{neighbors})$   4:          $a_{i,t}\leftarrow arg{max}_a{\pi }_{{\theta }_{\pi }}\left(a\right|o_{i,t})$   5:   end for   6:   ${\tilde{\mathbf{a}}}_t\leftarrow \mathtt{SafetyFilter}({\mathbf{a}}_t,{\mathbf{s}}_t,G)$   7:   Dispatch ${\tilde{\mathbf{a}}}_t$ to field devices; proceed to $t+1$

4. Case Study

4.1. Dataset and Experimental Environment

We conduct all case studies on the SimBench dataset [44], a publicly available benchmark suite covering low-, medium-, high-, and extra-high-voltage grids with full-year load, generation, and storage time series at a 15-min resolution. Grounded in European distribution networks curated from real operational data, SimBench reflects realistic feeder topologies and year-long profiles (rather than purely synthetic test cases) and is widely adopted for distribution-level studies. SimBench offers a present baseline (Scenario 0) and two future scenarios (Scenarios 1 and 2) and uses canonical codes that specify voltage levels, urbanization class, subordinate grids, scenario index, and switch representation (e.g., 1-MVLV-urban-all-0-sw). Unless otherwise noted, we analyze three representative feeders: (i) an LV rural feeder 1-LV-rural2–0-sw, (ii) an LV urban feeder 1-LV-urban6–0-sw, and (iii) an MV–LV combined urban grid 1-MVLV-urban-all-0-sw. These feeders span rural/urban contexts and MV–LV interfaces, mirroring operating regimes commonly encountered in practice. Networks are loaded via simbench.get_simbench_net(sb_code) and retain the official one-year profiles ($\Delta t=15$ min). Two example topologies from the dataset (one MV feeder and one LV urban feeder) are shown in Figure 2 to provide visual context.

We use the profiles as provided; voltages, powers, and currents are taken from pandapower internal units and converted to per-unit on each voltage level. Timestamps are aligned to UTC and resampled only when explicitly stated. For learning-based components, we apply chronological splits: 70%/15%/15% for training/validation/testing to avoid temporal leakage. Experiments run on Ubuntu 22.04 with 64 GB RAM and an NVIDIA RTX 3090, manufactured by NVIDIA, headquartered in Santa Clara, CA, USA. (24 GB), We use Python 3.10, PyTorch 2.x (CUDA), pandapower 2.x, and simbench 1.x; global seeds (NumPy/PyTorch) are fixed with deterministic CuDNN where applicable. The hyperparameters used in this study can be referred in

Table 2. Hyperparameters and their chosen values with search ranges.

Hyperparameter	Search Range	Chosen Value
Dimensions (d)	[16, 32, 64, 128]	64
Layers (N)	[2, 4, 6, 8]	6
Heads (h)	[4, 8, 16]	8
${\alpha }_{diff}$	[0.5, 1.0, 2.0]	1.0
$ϵ$	[0.01, 0.1, 0.5, 1.0]	0.01
k	[2, 3, 5, 10]	2
$\lambda$	[0.001, 0.01, 0.1]	0.01
$\eta$	[0.001, 0.01, 0.1]	0.01
$\beta$	[0.1, 0.5, 1.0]	0.5
$\gamma$	[0.9, 0.95, 0.99]	0.99
$\rho$	[0.0, 0.1, 0.5]	0.1
Learning Rate	[1 × ${10}^{-4}$, 1 × ${10}^{-3}$, 1 × ${10}^{-2}$]	1 × ${10}^{-3}$
Batch Size	[16, 32, 64, 128]	32
Discount $\gamma$	[0.9, 0.95, 0.99]	0.95
$\tau$	[0.01, 0.1, 0.5]	0.1
Entropy Temperature $\alpha$	[0.1, 0.5, 1.0]	0.5

.

4.2. Baselines

We compare against five literature baselines spanning rule-based, optimization-based (centralized and distributed), forecast scheduling, and learning-based control. All methods use the same SimBench data and evaluation protocol; when forecasts are required, they use the same inputs for fairness.

B1. 

Volt/VAR droop control (IEEE 1547-style)

Each inverter follows standardized Volt/VAR curves with reactive-power priority; local-only measurements drive fast VAR support for voltage regulation [45]. This represents a no communication baseline widely adopted in practice.

B2. 

Centralized convex OPF (SOCP, radial nets). A per-step AC-OPF solved via second-order cone relaxation on radial feeders (branch flow/DistFlow form). Under known conditions the relaxation is exact, yielding physics-feasible setpoints and a strong centralized benchmark [46].

B3. 

Distributed OPF via ADMM. An OPF decomposed across regions/buses and coordinated by ADMM; removes the single point of failure of centralized OPF and reduces solve time through parallelism while enforcing network constraints through consensus [47].

B4. 

Model predictive control (linearized LinDistFlow). A rolling horizon controller over H steps using a linearized network model with device and voltage constraints; captures intertemporal storage dynamics and updates decisions as new measurements/forecasts arrive [48].

B5. 

Multi-agent deep RL for voltage control. A cooperative MARL controller (CTDE) trained to regulate voltages using inverter/storage setpoints, following recent physics-informed or coordination-enhanced designs reported for active distribution networks [49].

4.3. Results

4.3.1. Overall vs. Baselines

We evaluate all methods on three representative SimBench feeders, namely LV-rural2, LV-urban6, and MVLV-urban-all, under Scenario 0 at 15-min resolution. We report operating cost; violation rate, defined as the fraction of buses below 0.95 p.u. or above 1.05 p.u.; maximum line loading; action smoothness measured by total variation; and a tail-risk metric given by ${\mathrm{CVaR}}_{95}$ of voltage-band exceedance beyond 0.95–1.05 p.u. The compact KPI table below summarizes mean ± std across feeders. Full metrics, including Worst-$|V-1|$, peak shaving, and runtime, appear in the Appendix A. Beyond point estimates, we probe three aspects: heterogeneity across feeders and times of day; interactions among cost, smoothness, and tail risk; and links between control activity and reliability outcomes. Unless noted otherwise, statements hold across all feeders.

Reading

Table 3. Overall results on SimBench feeders, mean ± std across feeders. Compact KPI set. Worst-$|V-1|$, peak shaving, and runtime are reported in the Appendix A.

Method	Cost	Viol.	Max Load	Action TV	${\mathrm{CVaR}}_{95}$($\left|\mathit{V}\right|$)
B1-VoltVAR	132.3 ± 16.7	0.167 ± 0.013	0.885 ± 0.041	1.306 ± 0.068	0.0169 ± 0.0028
B2-SOCP-OPF	125.5 ± 9.9	0.008 ± 0.004	0.852 ± 0.010	0.813 ± 0.030	0.0028 ± 0.0014
B3-ADMM-OPF	123.3 ± 9.7	0.022 ± 0.010	0.868 ± 0.025	0.825 ± 0.013	0.0035 ± 0.0002
B4-MPC-LinDF	129.6 ± 7.8	0.024 ± 0.007	0.876 ± 0.011	1.000 ± 0.059	0.0052 ± 0.0008
B5-MARL-lit	134.3 ± 14.8	0.067 ± 0.019	0.866 ± 0.025	1.088 ± 0.059	0.0087 ± 0.0015
Ours	125.4 ± 9.7	0.011 ± 0.007	0.844 ± 0.034	0.739 ± 0.094	0.0022 ± 0.0011

, three patterns stand out and are consistent with later figures. First, convex OPF baselines B2 and B3 push violations close to zero and minimize tail risk, confirming strong feasibility under the given forecasts. Second, our method matches OPF-level feasibility while achieving the lowest Action TV and the lowest ${\mathrm{CVaR}}_{95}$, which indicates smoother actuation and thinner violation tails at almost the same cost. Third, rule-based B1 and literature MARL B5 pay sizeable penalties in smoothness and risk despite occasional cost gains, which foreshadows the heavier tails observed under stress. By feeder, the largest gains of Ours in Action TV and ${\mathrm{CVaR}}_{95}$ occur on MVLV-urban-all, where electrical betweenness concentrates on a few MV–LV corridors. LV-rural2 shows smaller but uniform improvements, which reflects longer electrical distances and more headroom. Time-of-day stratification reveals that Ours suppresses tail risk during ramp-up from 08:00 to 11:00 and during the post-plateau descent from 15:00 to 18:00, namely the periods with rapid re-allocation of set-points. For robustness, we bootstrap over seeds and hours the mean differences between Ours and B2 or B3. The resulting 95% intervals exclude zero for Action TV in all feeders and for ${\mathrm{CVaR}}_{95}$ in MVLV-urban-all, which indicates stable advantages under sampling variability.

Figure 3 makes the trade-off between economy and smoothness explicit. Two knees are visible. B2 and B3 occupy the low-cost region with moderate TV, while Ours pushes the frontier downward in TV without sacrificing cost. When actuator wear, comfort, or secondary regulation effort matters, the frontier shift achieved by Ours translates into fewer adjustments for the same expenditure. Across all methods we observe a positive association between Action TV and ${\mathrm{CVaR}}_{95}$. The points for Ours lie on a lower isorisk contour than B4 or B5 at matched cost, which suggests that coordination plus safety filtering converts a unit of control effort into a larger reduction in risk.

Figure 4 contrasts energy benefits with reliability risk. B4 achieves aggressive peak shaving yet shifts upward in CVaR, which indicates a thicker voltage tail. Ours lies near the efficient frontier with low CVaR for a moderate peak reduction. Taken together with Table 3, the policy achieves a safer balance between economy and reliability than strategies that focus on peak shaving alone. Event-conditioned analysis across ramps, mid-day plateau, and evening peak shows that Ours gains most of its peak-shaving benefit during ramps while keeping ${\mathrm{CVaR}}_{95}$ flat, whereas B4 concentrates gains during the plateau at the cost of a thicker tail. This aligns with the intervention patterns in Section 4.3.4: Ours anticipates ramps with small continuous adjustments and avoids back-and-forth discrete moves.

4.3.2. Ablation Study on Topology Priors and Diffusion Kernel

To isolate the roles of topology priors and the diffusion kernel in forecasting accuracy, we remove each component in turn and assess performance.

Table 4. Ablation study on topology priors and diffusion kernel, mean ± std.

Model	RMSE	MAE	${\mathrm{CVaR}}_{95}$
Full Model	0.042 ± 0.005	0.036 ± 0.004	0.0032 ± 0.0005
No Topology Priors	0.063 ± 0.007	0.051 ± 0.006	0.0041 ± 0.0006
No Diffusion Kernel	0.056 ± 0.006	0.048 ± 0.005	0.0038 ± 0.0004
No Topology Priors and Diffusion Kernel	0.078 ± 0.008	0.065 ± 0.007	0.0050 ± 0.0008

shows that topology priors improve accuracy in complex grid structures, while the diffusion kernel captures spatial dependencies. Removing either component reduces accuracy and increases tail risk.

Two diagnostics connect these modeling choices to downstream control. First, across rollouts the per-step absolute forecast error correlates with the instantaneous safety slack; see Section 4.3.6. Removing either component increases both quantities, which amplifies the need for safety corrections. Second, under identical controllers, the full model generalizes more sharply in space and reduces the frequency of discrete tap or capacitor interventions during ramps. This is consistent with the lower Action TV in Table 3.

4.3.3. Coordination Under Congestion

We amplify peak demand and mid-day PV by a factor of 1.3 and run AC power flow on real SimBench networks. The MVLV feeder serves as the representative stress case. We report complementary cumulative distribution functions with VaR and CVaR annotations for two exceedance types: voltage-band violations and line-loading violations. To better expose rare but severe thermal events, the loading panel aggregates results across a small ensemble of perturbed operating points around the baseline. We also relate tail mass to network structure by computing line betweenness centrality and electrical distance to the substation. Lines with high centrality exhibit heavier CCDF tails. Coordination strengthens along those corridors in our policy through larger inter-agent weights, which yields a measurable left shift of the loading CCDF without a material effect on the bulk.

Figure 5 indicates that reliability risk concentrates in the tails rather than in the bulk. In panel a, the voltage CCDF decays rapidly: most excursions are shallow, and the VaR levels at 95% and 99% remain small. The CVaR markers confirm that, conditional on being in the tail, typical voltage magnitudes only slightly exceed the band, which suggests limited severity even under stress. Panel b exhibits a flatter upper tail for line loading once we aggregate across perturbed operating points. Both VaR and CVaR increase, which shows that a small fraction of periods and lines dominates thermal risk. This pattern is consistent with congestion that concentrates on a few electrically central MV–LV interfaces that become persistent bottlenecks under simultaneous peak demand and strong mid-day PV injections. The ordering of methods in the extreme tail beyond the 99th percentile matches the ordering in Table 3 for Action TV and ${\mathrm{CVaR}}_{95}$. This supports the view that smooth and coordinated continuous control suppresses compounding effects that would otherwise spill into rare but severe overloads.

Under our adaptive coordination mechanism, inter-agent weights strengthen when local headroom vanishes, which encourages coordinated responses along stressed corridors while loosely coupled areas remain largely independent. Empirically, this shifts the loading CCDF leftward and compresses the tail with lower CVaR, without a noticeable change in the well-behaved bulk. From an operational standpoint, the results motivate targeted mitigations such as selective curtailment, local set-point smoothing, or minor topology adjustments, focused on the few MV–LV links identified by high tail mass, where each unit of control effort yields the largest risk reduction.

4.3.4. Hybrid Control with OLTC/Cap

We include OLTCs and capacitor banks together with continuous inverter and storage actions to study discrete–continuous coordination. The comparison follows the unified evaluation protocol used in Section 4.3.3: identical stress profile, identical feeder initialization, and identical measurement cadence. Discrete costs and action magnitudes are normalized across feeders. Bars in Figure 6 aggregate over seeds and operating points for fairness. The two metrics emphasize complementary aspects of practicality. Total discrete switching, computed as the sum of OLTC tap moves and capacitor steps, serves as a proxy for mechanical wear and the tolerance of operators for interventions. Action total variation summarizes how smooth continuous set-points evolve in time. Lower values are preferable for comfort, power quality, and device cycling.

Figure 6 confirms the efficiency of coordinated strategies. B2, B4, and Ours substantially reduce OLTC and capacitor operations while yielding smoother continuous trajectories. B1 and B5 rely on frequent discrete changes yet still exhibit higher TV. Cross-referencing Table 3, the reduction in Action TV by Ours is achieved without extra cost, which implies lower actuator wear and fewer comfort-impacting adjustments for comparable economic performance. Qualitatively, coordinated methods allocate most of the regulation burden to fast and low-amplitude continuous resources and dispatch discrete moves only when the operating regime changes, for instance before or after the mid-day PV plateau or near the evening peak. This reserve-discrete-for-regime-shifts behavior keeps the OLTC latched for longer intervals, reduces capacitor chatter, and avoids back-and-forth reversals. These patterns align with practical operating guidelines. A frequency-domain view of continuous set-points, shows that Ours concentrates most effort at low temporal frequencies and suppresses high-frequency fluctuations that correlate with user discomfort and power-quality variability. Discrete switching events align with slow regime shifts rather than micro-corrections, which explains the joint reduction in panels a and b.

To make the linkage between interventions and system states explicit, we visualize two representative policies.

Figure 7 links intervention to outcome. Both methods concentrate discrete events near stressed hours, yet Ours uses fewer and more targeted actions and produces a tighter voltage band in the heatmap. Compared with B2, the proposed policy engages continuous devices preemptively ahead of anticipated ramps and relies on a small number of decisive tap or capacitor updates that persist through the stress window, rather than frequent micro-corrections. Together with the TV reduction in Figure 6b, this suggests that the proposed policy suppresses dispersion around 1.0 p.u. more effectively per discrete actuation, which is desirable when limiting wear on OLTCs and capacitor banks. This causal chain—anticipatory continuous adjustments → fewer discrete moves → tighter voltage bands—recurs across feeders and seeds and explains why Ours occupies a strictly better region of the cost, smoothness, and risk space; see Figure 3 and Figure 4. Sensitivity to limited forecast accuracy supports this picture. Injecting zero-mean noise into the day-ahead profile preserves the ordering of Ours on Action TV and ${\mathrm{CVaR}}_{95}$, while methods that hinge on precise timing, such as B4, degrade more noticeably.

4.3.5. Stress Testing Matrix

We sweep load scale and PV scale and plot ${\mathrm{CVaR}}_{95}$ heat maps for the three feeders to visualize robustness envelopes. The grid is evaluated under the unified protocol in Section 4.3.3. Initialization, measurement cadence, and stress construction remain identical. Each tile aggregates multiple seeds and perturbed operating points, and the color range is shared across panels for comparability.

Figure 8 delineates the safe domain where ${\mathrm{CVaR}}_{95}$ remains small, shown as bright regions, versus high-risk regimes, shown as dark pockets. LV feeders exhibit broader safe zones, while the MVLV case tightens earlier along high-load and high-PV directions, which is consistent with corridor bottlenecks flagged by Figure 5. The heat maps also reveal ridge-like boundaries where small movements in either axis produce sharp risk changes. These knees indicate operating margins and serve as natural trigger lines for pre-emptive control. In operational terms, horizontal moves correspond to demand-side actions such as peak shifting or shaving, while vertical moves correspond to PV-centric interventions such as curtailment or additional reactive support. The brightest pathways identify high-leverage regions where modest actions yield outsized reductions in tail risk. Dark plateaus indicate saturation where feeder-level measures or minor topology edits are more effective. Cross-validating the stress maps with the ablations in Section 4.3.2 shows that removing spatial inductive bias both shrinks bright regions and steepens ridges. The system then becomes less forgiving to simultaneous load and PV surges.

Taken together with Figure 6 and Figure 7, the stress maps suggest a coherent playbook. Prioritize fast and continuous resources to steer operating points toward brighter tiles. Reserve discrete OLTC and capacitor actions for crossing ridge lines. Where dark pockets persist, target the specific MV–LV links implicated by Figure 5. This provides a practical feedforward guide for planning and real-time operations under simultaneous load growth and PV penetration.

4.3.6. AC Power-Flow Back-Check and Slack–Violation Analysis

Protocol. For each test rollout step, we apply the controller action and recompute a full AC power flow with Newton–Raphson at tolerance ${10}^{-8}$ on the three SimBench MV feeders: urban, semiurban, and rural. We log per-step voltage deviations $|V-1|$ in p.u. and branch loading ratios. We report violation rates, defined as the fraction of steps with $|V-1|>0.05$ or loading $>1.0$, together with 95th and 99th percentiles. For the Safety QP we also record the slack magnitudes $\mathit{\xi }=({\mathit{\xi }}_V,{\mathit{\xi }}_I)$. This back-check verifies feasibility against the nonconvex AC equations and calibrates the informativeness of safety-layer slacks as online risk indicators.

For each step we compute a normalized slack sum $s_t=(\parallel {\mathit{\xi }}_V{\parallel }_1+\parallel {\mathit{\xi }}_I{\parallel }_1)/(N_{\mathrm{bus}}+N_{\mathrm{br}})$ and an AC violation magnitude $v_t={max\left\{\right(\left|V\right|-1-0.05)}_{+},{(L-1)}_{+}\}$. Over the test set we observe a strong Pearson correlation with $r=0.82$ and a stable linear fit $v\approx 0.62s+0.002$. About $95\%$ of samples lie below an affine envelope with slope $\approx 0.90$ and intercept $\approx 0.004$. This gives a practical rule of thumb for online risk monitoring without re-solving AC power flow. Viewed as a classifier for imminent violations, the aggregate slack $\sum \mathit{\xi }$ yields a high area under the ROC curve. Thresholds can be tuned to operator preferences for high recall or low false positives. This calibration closes the loop between the convex safety layer and nonconvex AC feasibility.

Compared with MPC and No-Safety, our method substantially suppresses tail risk across all three feeders. It reduces the 99th percentile voltage deviation by approximately $31\%$ and $56\%$, respectively, and lowers the branch overload rate by approximately $71\%$ and $90\%$; see

Table 5. Closed-loop AC backcheck: Voltage metrics, mean ± std over feeders. Lower is better.

Method	Viol. Rate (%) ↓	${|\mathit{V}-1|}_{95\%}$↓	${|\mathit{V}-1|}_{99\%}$↓
Ours, STT–MARL + Safety QP	$\mathbf{0.9}\pm \mathbf{0.2}$	$\mathbf{0.012}\pm \mathbf{0.002}$	$\mathbf{0.020}\pm \mathbf{0.003}$
MPC, QP–MPC baseline	$2.8\pm 0.6$	$0.018\pm 0.003$	$0.029\pm 0.004$
No–Safety, vanilla MARL	$7.5\pm 1.2$	$0.027\pm 0.005$	$0.045\pm 0.007$

and

Table 6. Closed-loop AC backcheck: Branch loading metrics, mean ± std over feeders. Lower is better.

Method	Overload rate (%) ↓	${\mathit{L}}_{95\%}$↓	${\mathit{L}}_{99\%}$↓
Ours, STT–MARL + Safety QP	$\mathbf{0.6}\pm \mathbf{0.2}$	$\mathbf{0.83}\pm \mathbf{0.03}$	$\mathbf{0.91}\pm \mathbf{0.04}$
MPC, QP–MPC baseline	$2.1\pm 0.5$	$0.88\pm 0.02$	$0.96\pm 0.03$
No–Safety, vanilla MARL	$6.3\pm 0.9$	$0.93\pm 0.03$	$1.05\pm 0.05$

. In addition, the strong and monotone slack–violation relationship indicates that slack magnitudes serve as an informative proxy for residual AC-level risk. This enables lightweight online screening and threshold-based alarms while full AC back-check remains reserved for auditing or recalibration. Residual risk concentrates on a few MV–LV interfaces during synchronized ramps. When telemetry loss persists across adjacent agents, coordination weakens and the safety layer must intervene more frequently. We anticipate further gains from two directions. First, telemetry-robust coordination with imputation-aware weights. Second, selective reinforcement of identified bottlenecks through minor topology edits or local controller upgrades.

5. Conclusions

This paper presents a practical, closed-loop framework for coordinated DER operation that couples a topology-aware Spatio-Temporal Transformer (STT) for multi-horizon forecasting with a cooperative multi-agent reinforcement learning (MARL) controller and a real-time safety layer. The design preserves interpretability and modularity: forecasts inform decentralized decisions; decisions are filtered by device/network-aware safety projections; and all components can be independently improved without re-architecting the system.

Across three representative SimBench feeders, the unified evaluation shows consistent gains along economy, reliability, and actuation quality:

• Overall performance. The compact KPI summary in Table 3 indicates that the proposed method matches OPF-level feasibility while attaining the lowest action smoothness (TV) and the lowest voltage tail-risk (${\mathrm{CVaR}}_{95}$), at costs comparable to B2/B3. This combination—low risk and smooth actuation without a cost premium—establishes a strong baseline for field deployment. Numerically, on the three SimBench feeders our approach reduces tail violations at the 99th percentile voltage deviation by approximately 31 % and 56 %, and lowers branch overload rate by approximately 71 % and 90 % compared to baseline methods.
• Trade-offs made explicit. The Pareto fronts in Figure 3 and Figure 4 visualize key design tensions. Our policy shifts the cost–smoothness frontier downward (fewer adjustments for the same cost) and attains a safer balance between peak shaving and tail risk than aggressively peak-oriented controllers. These frontier shifts are operationally meaningful when actuator wear, comfort, or secondary regulation penalties matter.
• Reliability under congestion. The MV–LV stress test in Figure 5 highlights that bulk metrics can hide risk concentrated in rare, high-impact events. Voltage tails remain thin, but ensemble CCDFs reveal a fatter thermal tail on a small set of periods/lines. This pinpoints MV–LV corridors as reliability bottlenecks and motivates targeted mitigations that deliver the highest risk reduction per unit action. These observations are consistent with the above reductions in voltage tail violations and overload rates achieved by the proposed controller.
• Hybrid actuation quality. The hybrid control summaries in Figure 6 and the intervention → outcome views in Figure 7 show that our controller achieves smoother continuous trajectories with fewer discrete OLTC/cap operations, yet maintains a tighter voltage band during stressed hours. This improves service quality while limiting mechanical wear.
• Stress-tested robustness. StressMaps in Figure 8 delineate safe operating regions across (load and PV) scaling. The MVLV feeder’s safe area shrinks fastest along high-load/high-PV directions, consistent with the tails in Figure 5. Such maps act as where-to-act guides for curtailment, reactive support, or reinforcement planning.

While the present evaluation is simulation-based, the realism and standardization of SimBench ensure reproducibility and practical relevance; we additionally backcheck dispatches with full AC power flow and outline hardware-in-the-loop/field validation as future work. In future work, we aim to incorporate these validation steps to further assess the performance of our framework in real-world feeder environments. While our approach demonstrates promising empirical results, we recognize the necessity of a more robust theoretical analysis, particularly regarding the convergence of multi-agent policies and stability under dynamic network conditions. Future research will focus on deriving formal stability guarantees and assessing scalability in the context of large-scale, real-world distribution grids, enhancing both theoretical and practical performance.

In the future, we can identify three complementary avenues to strengthen the framework and broaden its applicability: first, at the algorithmic level, embed risk awareness directly into learning by optimizing coherent risk measures within the policy objective, and couple them with distributionally robust training that uses principled ambiguity sets and scenario randomization to immunize the controller against nonstationarity and forecast shift; extend this to multi-timescale risk shaping so that fast corrective actions and slower planning signals are jointly optimized under explicit tail objectives; second, on the safety and feasibility side, advance adaptive safety by learning fast physics-informed surrogates for sensitivities to refresh constraints online, by extending projections to mixed-integer actuation with reliable relaxations and repair, and by pursuing formal guarantees via Lyapunov/barrier certificates, contraction analysis, and reachability tools that account for switching, topology/parameter uncertainty, and measurement noise; integrate the safety layer more tightly as a differentiable optimization module to support end-to-end training without sacrificing feasibility; inally, toward deployment at scale, carry out hardware-in-the-loop and real-feeder studies that quantify the impact of telemetry loss and latency, profile the edge-compute/communication footprint, and assess privacy-preserving coordination alongside market co-optimization for ancillary services and demand response; establish open, reproducible evaluation protocols that report extreme-quantile risk, overload frequency, and actuation burden, enabling rigorous comparison and technology transfer to heterogeneous feeders.