A Hierarchical AI Framework for Safe RL-Based Control in Distribution Networks ^†

Abstract

The use of solid-state transformers (SSTs) for intelligent coordination and integration between medium-voltage feeders, dispersed energy sources, fast EV charging, and hybrid AC/DC loads is becoming significantly more prevalent. The control options available with these devices provide better controllability compared with standard low-frequency transformers. However, the development of this high degree of controllability introduces numerous coupling issues associated with voltage, power flow, thermal, and modulation, which are reported in the literature concerning cascaded-based control, especially under renewable variability and parameter drift. This study describes a physics-informed safe reinforcement learning (RL) controller for three-stage SSTs. The methodology consists of combining an actor–critic supervisory controller, a graph-temporal encoder, a power balance and thermal-residual regularizer, and a quadratic-program-based safety shield. The purpose of this safety shield is to project each AI action onto the feasible operating space of the SST before arriving at the inner converter loops. The supervisory controller coordinates the setting of active/reactive power references, DC link targets, and the setting of DAB phase shift before MPC/PI-based inner loops and enabling rapid actuation. A stochastic digital twin was developed for a 1 MVA distribution-grade SST under various operating scenarios, including PV ramps, EV charging pulses, MV sags, islanding transitions, and converter parameter drift. Across 40 stochastic scenarios, the proposed controller achieved a settling time of 8.17 ms (95% CI +/− 0.24), output THD of 1.77% (+/− 0.06), efficiency of 97.43% (+/− 0.05), hotspot temperature of 84.86 °C (+/− 0.44), and a constraint violation rate of only 1.07% (+/− 0.12). Relative to a strong model predictive control baseline, settling time improved from 12.42 to 8.17 ms, efficiency increased from 96.08% to 97.43%, and hotspot temperature reduced from 91.38 °C to 84.86 °C; paired tests were highly significant for all primary metrics (p < 0.001). Compared with DRL without physics regularization, the proposed method remained faster (11.35 ms vs. 8.17 ms) and safer due to a fourfold lower violation rate. The primary contribution of this study is the co-design of AI-based supervisory coordination, physics residual shaping, and hard action projection for SST operation. This integrated design provides a simulation-based foundation for subsequent real-time digital-simulation, controller-hardware-in-the-loop, and experimental validation studies.

1. Introduction

Distribution grids are changing from passive radial designs to more complex structures that are converter-based, cyber–physical networks with real-time and historical data for monitoring system operation. New networks have to accommodate the integration of renewable generation, energy storage, high-speed EV charging, and digitally powered loads, thus creating a need for solutions that can address multiple issues on the same grid. The solid-state transformer (SST) provides numerous benefits over traditional distribution transformers; primary characteristics include: (1) voltage transformation (power quality); (2) galvanic isolation; (3) bidirectional power transfer; (4) interfacing between AC and DC systems; and (5) service intelligence [1,2,3,4].

SSTs’ desirable characteristics also pose significant challenges in controlling them. When the front end of an SST (the inverter) is treated as an isolated DC/DC stage, it can be dynamically coupled through elements such as the DC link, thermal loading, and semiconductor stress; therefore, a controller that helps to improve one performance metric (e.g., response speed) can adversely affect other performance metrics (e.g., circulating current, thermal stress, or zero-voltage-switching margin). These interactions are complicated by weak grids, rapid changes in port power, and changes in component parameters [5,6,7,8,9,10,11,12,13].

There has been limited progress toward using cascaded PI and droop control methods for robust predictive control (e.g., reinforcement learning (RL)-assisted modulation) or multifunctional SST services in the recent literature. However, most existing methods focus on a single converter stage, a single service objective, or an offline optimization problem. This reveals a research gap: the lack of an integrated and AI-driven multi-port SST controller that accounts for the electrical and thermal feasibility constraints of coordinated multi-port SST operation.

The present study proposes and evaluates a new AI model-based SST controller to provide rapid, safe, and physically based SST operation. The controller’s intended application is for feeders with large shares of renewable energy sources, where the SST must control both the DC link and LV-side parameters, provide reactive power during MV disturbances, reduce thermal stress, and maintain modulation feasibility without perfect model knowledge. The proposed framework can be simulated using MATLAB/Simulink R2024a (MathWorks, Natick, MA, USA).

The contributions are fourfold. A novel physics-informed safe RL formulation for controlling three-phase SSTs is provided. An operating hierarchy of deploying SSTs encompasses AI-enabled supervisory commands and conventional fast-loop controls to facilitate prompt transitions. The introduction of a multi-scenario digital twin (representative of three different states), along with a statistically defined protocol for evaluating performance, demonstrates the effectiveness of using this combination of physical and digital approaches to effectively manage the functionality of an SST. Finally, the study reports the evaluation through system-level block diagrams, transient trajectories, performance trade-offs, ablation results, robustness maps, and comparative tables, thereby enabling transparent and reproducible assessment of the proposed framework. Figure 1 shows the system context of AI-driven SST control linking the MV grid, the three SST stages, and hybrid AC/DC loads.

Figure 1. System context of AI-driven SST control linking the MV grid, the three SST stages, and hybrid AC/DC loads.

Recent SST research has moved beyond topology-centered analysis toward a service-oriented understanding of SST deployment in active distribution networks. From this perspective, SSTs are increasingly treated as grid-interactive assets capable of coordinated power-flow control, ancillary-service provision, hybrid AC/DC interfacing, renewable energy integration, and support for electrified transportation. Recent reviews also assess SSTs in terms of distribution-level services, fast-charging integration, lifecycle performance, and clean energy transition impacts. These developments make supervisory coordination and explicit operating-envelope awareness increasingly important because stage-wise control alone cannot capture the electrical, thermal, and service-level interactions of a multi-port SST [14,15,16]. In parallel, recent reinforcement learning research in power systems and power electronic converters indicates that data-driven control is developing rapidly, but unconstrained end-to-end learning remains unsuitable for safety-critical energy infrastructure. Current studies emphasize safe RL mechanisms, hierarchical supervisory deployment, domain-informed state representations, and physics-aware learning so that adaptability can be achieved without sacrificing constraint satisfaction, stability, or implementation realism under renewable and load uncertainty [9,17,18,19,20]. This evidence directly supports the hierarchical, physics-informed, and safety-projected design adopted in the present study [21,22,23].

The main operational drivers that motivate a physics-informed and safety-constrained AI controller for SSTs are summarized in Table 1.

Table 1. Application drivers motivating a physics-informed and safety-constrained AI controller for SSTs.

Driver	Control Implication	Why AI Alone Is Insufficient
High DER volatility	Requires power balancing across ports	May violate converter constraints
Fast EV charging	Causes rapid current and DC link excursions	Can overfit to one scenario
Hybrid AC/DC loads	Needs coordinated AC/DC voltage support	May ignore physics coupling
Weak-grid events	Demands ride-through and reactive injection	Can create unstable actions
Thermal and reliability limits	Needs stress-aware action selection	Needs hard safety envelope

2. Related Work

A focused literature search was conducted on several databases including IEEE Xplore, Science Direct, Wiley Online Library, MDPI, and other indexed academic sources. Sources primarily published from 2019 to 2026 were emphasized while keeping earlier foundational studies that continue to be used for defining the existing architectures, converter types, and control methods of SST [1,2]. The included studies were those that examined some aspect of SST architecture, DAB/multi-active-bridge control, predictive or data-driven control of converters, RL to constrain control for safety, or grid service functionality of power electronics. Any study that did not meet this requirement was excluded from consideration [24,25,26,27].

The studies represented by this literature can be divided into four major themes. The first theme includes reviews of architecture and applications and describes how SSTs fit within smart grids, integration of renewable energy, implementation of active distribution systems, and creation of hybrid AC/DC networks [1,2,28,29].

The second theme covers robust, sliding-mode, and predictive control strategies for DAB converters, multi-stage SSTs, and grid-support functions. The third theme concerns data-driven and reinforcement learning approaches for modulation selection, converter control, and adaptive parameter tuning [5,6,7,13,16]. The fourth theme addresses service-oriented SST control, including multifunctional grid support, frequency regulation, charging-system integration, and coordinated power-flow management [4,13,30,31].

The literature on machine learning for power electronics suggests two primary trends. Model-based approaches retain explanatory power and stability while also measuring success based on a small number of goals or for a single conversion stage. Conversely, reinforcement learning or data-driven systems often have an increased level of adaptability, yet their performance is typically assessed at the individual converter level and does not consider coordination at the electrical, thermal, and safety levels. Therefore, the two bodies of work point to a research void: the development of a hierarchical AI framework capable of providing comprehensive coordination for multiple SST functions, while still adhering to the restrictions of electrical feasibility, modulation restrictions, and thermal stress constraints.

The proposed method addresses this gap by combining graph-temporal state encoding, physics-informed residual regularization, and QP-based action projection within a hierarchical three-stage SST controller. Unlike direct switching-level learning, the proposed RL policy provides supervisory references while deterministic inner loops retain responsibility for high-bandwidth converter regulation [17,20,32]. Table 2 compares the proposed framework with representative SST and converter-control studies.

Table 2. Comparative review highlighting the gap that motivates the proposed work.

Study	Year	Method	Setting	Strength	Gap
[1]	2013	Review	Architectures and applications	Converter topology survey	No AI; no safety-aware real-time control
[2]	2019	Review	SST-based microgrids	Design/control synthesis	Pre-AI control era
[3]	2021	Review	Power electronic transformers	Magnetics and converter design	Limited control intelligence
[10]	2023	DRL	DAB modulation	RL improves wide-range modulation	Single converter stage only
[11]	2023	DRL-PI	DAB voltage control	Data-driven PI tuning	No multi-port SST coordination
[13]	2024	Hybrid control	Hybrid SST for distribution grids	Multifunctional grid support	Conventional supervisory logic
[28]	2025	Review	SST control stages	Architecture-control mapping	No unified AI safety framework
[29]	2025	Review	Hybrid distribution SST control	Compares control-system approaches	Limited stochastic performance metrics
[33]	2025	Predictive control	Frequency support via SSTs	RoCoF-constrained support	Narrow service objective
Present study	2026	Physics-informed safe RL	Three-stage SST	Joint voltage–power–thermal–safe control	Simulation-based only

3. Problem Formulation

The control problem is formulated for a three-stage SST operating in a renewable-rich distribution feeder under stochastic disturbances, coupled electrical dynamics, and hard operating constraints. Let the supervisory control layer operate in discrete time with sampling interval $T_s$. The SST is represented by the state-space model

$$x_{t+1}=f(x_t,u_t,d_t)+w_t,y_t=h\left(x_t\right)+n_t$$

where $x_t\in {\mathbb{R}}^{n_x}$ denotes the aggregated system state, $u_t\in {\mathbb{R}}^{n_u}$ is the applied supervisory command, $d_t\in {\mathbb{R}}^{n_d}$ represents exogenous disturbances, $y_t$ is the measured output vector, and $w_t$ and $n_t$ denote process and measurement uncertainties, respectively. A state vector contains representative electrical, thermal, and grid-interface variables such as DC link voltage, converter currents, output-side voltage, reactive power, and semiconductor hotspot temperature. The disturbance vector includes renewable intermittency, load transients, MV-side voltage sags and parameter drift from operating stress or aging. Unlike direct switching-level learning, the proposed hierarchical formulation of RL enables the agent to generate a supervisory action proposal rather than gate signals.

$$a_t=[P_t^{*},\hspace{0.17em}{Q}_t^{*},\hspace{0.17em}{V}_{dc,t}^{*},\hspace{0.17em}{\varphi }_{dab,t}{]}^{\mathrm{\top }}$$

This design provides specific references for inner MPC/PI loops to help standardize the definitions of high-level active, reactive, direct current link, and DAB phase shifts. The design increases interpretability, avoids dimensionality issues, and is associated with actual certification requirements (more predictable and manageable) for installing medium-voltage SSTs. Safe projection operators are used to generate command signals applied to the plant.

$$u_t={\mathsf{\Pi }}_{\mathsf{\Omega }}\left(a_t\right)$$

where $\mathsf{\Omega }$ denotes the feasible operating set defined by electrical, thermal, and modulation limits. In this way, the learned policy remains expressive, while the deployed controller remains bounded by explicit engineering constraints. The supervisory objective is to minimize a multi-objective finite-horizon cost that balances voltage regulation, power tracking, thermal stress, waveform quality, and control smoothness,

$$J={\displaystyle \sum _{t=0}^{T-1}}\left(w_1\parallel v_{dc,t}-v_{dc}^{ref}{\parallel }_2^2+w_2\parallel p_{o,t}-p_o^{ref}{\parallel }_2^2+w_3{T}_{j,t}+w_4\hspace{0.17em}{\mathrm{T}\mathrm{H}\mathrm{D}}_t+w_5{\parallel \Delta u_t\parallel }_2^2\right)+\lambda L_{\mathrm{p}\mathrm{h}\mathrm{y}\mathrm{s}}$$

where $w_1,\dots ,w_5$ are non-negative weighting coefficients and $\lambda$ controls the influence of the physics-informed regularization term. Electrical regulation and power delivery are enforced through the first two terms, thermal stresses are limited in the third term, distortion is penalized in the fourth term, and the fifth term discourages abrupt supervisory actions that can excite unacceptable converter transients. A residual penalty based on power balance and thermal network constraints is included to maintain physical consistency with the learning problem:

$$L_{\mathrm{p}\mathrm{h}\mathrm{y}\mathrm{s}}={\parallel p_{mv}-p_{dab}-C_{dc}{v}_{dc}{\displaystyle \frac{d{v}_{dc}}{dt}}\parallel }_2^2+{\parallel C_{th}{\displaystyle \frac{d{T}_j}{dt}}-(P_{loss}-{\displaystyle \frac{T_j-T_a}{R_{th}}})\parallel }_2^2$$

The first residual imposes a constraint that provides consistency between input power, transferred power, and energy stored in the DC link; the second residual penalizes the mismatch between the predicted evolution of thermal energy and the thermal RC model’s first-order prediction. These two residuals are intended to loosely guide learning towards actions that both maximize rewards and are physically reasonable given the operational environment change. The feasible set Ω places hard restrictions on converter current, DC link voltage, output voltage quality, DAB phase-shift angle, command slew rate, hotspot temperature, and harmonic distortion. This means that the supervisory decision problem can be expressed as

$$\underset{{\pi }_{\theta }}{\mathrm{m}\mathrm{i}\mathrm{n}}\hspace{0.25em}\mathbb{E}[J]\mathrm{s}.\mathrm{t}.u_t={\mathsf{\Pi }}_{\mathsf{\Omega }}(a_t),\hspace{0.25em} x_{t+1}=f(x_t,u_t,d_t)+w_t,\hspace{0.25em}{u}_t\in \mathsf{\Omega },\hspace{0.25em}{x}_t\in \mathcal{X}$$

where ${\pi }_{\theta }$ denotes the parameterized policy. In practice, ${\mathsf{\Pi }}_{\mathsf{\Omega }}(\cdot )$ is implemented as a small quadratic program that minimally modifies the proposed action whenever constraint violations are predicted. Thus, the control problem can be framed as a hierarchical constrained decision-making process between two levels: at the higher level, RL provides coordination between high-level controls; at the lower level, safety filtering and inner-loop control maintain feasibility and enable fast actuation of converter models. This model is well suited for SSTs because it facilitates the fundamental trade-off inherent in the application, that is, enabling fast, adaptive, multi-objective control while not compromising physical safety or practical deployability.

4. Proposed Method

The proposed controller is a hierarchical physics-informed safe RL architecture. Measured port variables and estimated latent states are passed through a graph-temporal encoder that represents the SST as a small multi-port interaction graph. The actor–critic head outputs supervisory references, while the safety shield enforces feasibility before commands reach the inner loops.

The proposed design retains the physical and functional structure of the SST. The graph branch models interactions among converter stages and electrical ports, while the temporal branch captures the evolution of disturbances, operating regimes, and parameter drift. Physics-informed residuals penalize departures from energy conservation and thermal dynamics, and the QP safety shield prevents infeasible high-level references from reaching the deterministic inner loops. This structured approach is consistent with recent safe RL and power electronic control literature emphasizing domain-informed representations and explicit feasibility handling [12,13]. The overall hierarchical architecture is shown in Figure 2.

Figure 2. Proposed hierarchical SST control architecture with physics estimator, policy network, and safety shield.

4.1. Components, Training Logic, and Deployment

The sensor fusion front end collects synchronized measurements in both RMS and instantaneous form from the MV active front-end converter, DAB stage, LV inverter, DC link, output ports, and thermal estimator. At each time step t of supervisory control, the fused feature vector contains the corresponding voltage (MV & LV side), current (MV & LV side), DC link voltage and tracking error, DAB phase shift and modulation indices, estimated semiconductor hotspot temperature, load demand, grid strength indicators, and disturbance estimates from multiple sensors. All measurement data have been filtered, normalized, time-aligned, assembled into a temporal observation window, and are provided as input to the graph-temporal encoder.

The SST is represented as a multi-port graph, where the set of nodes represents the MV interface, active front-end converter, DC link, DAB stage, LV AC port, LV DC port, and estimated thermal state, and the set of edges represents the power-transfer couplings, shared DC link dependencies among converters, interactions among converters, and common electrical or thermal constraints. The temporal branch captures the variations in renewable-generation conditions, load transition patterns, and parameter drift trends, whereas the graph branch models the relationships between the various SST stages and ports. The final latent representation is concatenated with the power balance and thermal residuals and provided to both the actor network and critic network as shown in Algorithm 1.

The actor proposes the supervisory reference vector as

$$a_t={[P_t^{\mathrm{*}},Q_t^{\mathrm{*}},V_{dc,t}^{\mathrm{*}},{\varphi }_{dab,t}]}^{\mathrm{T}}$$

The DAB’s active power reference, reactive power reference, DC link voltage target, and DAB phase-shift commands are defined by, and can be controlled with, the QP control interface. A safety shield using QP technology is used to interpret the proposed actions and determine which one is feasible before sending it into the inner control loop. The proposed actions are vetted by the QP to ensure that they coordinate all of the SST objectives and do not directly map to semiconductor switching signals.

Algorithm 1. Physics-informed safe RL supervisory step.

$\mathrm{Observe} x_t$$\mathrm{and} \mathrm{disturbance} \mathrm{estimates} d_t$ Compute physics residual features r_t from electrical and thermal models $\mathrm{Encode} \mathrm{port} \mathrm{graph} \mathrm{and} \mathrm{temporal} \mathrm{window} -> z_t$ $\mathrm{Actor} \mathrm{proposes} a_t={\pi }_{\theta }\left(z_t-r_t\right)$ $\mathrm{Project} a_t$$\mathrm{through} \mathrm{safety} \mathrm{shield}: u_t={\pi }_{\Omega }\left(a_t\right)$ $\mathrm{Inner} \mathrm{loops} \mathrm{track} u_t$ on AFE, DAB, and inverter stages Collect reward and physics loss; update critic and actor during training

The training and inference workflow of the proposed controller is summarized in Figure 3.

Figure 3. Training and inference flow of the proposed control framework.

4.2. Complexity and Resource Requirements

The graph-temporal encoder and actor–critic branch contain approximately 0.46 million trainable parameters. The reported 66 μs value represents the mean neural-policy inference time measured in the simulation environment. It should not be interpreted as a worst-case end-to-end execution time on an embedded controller because it excludes sensor acquisition, signal conditioning, communication delay, QP solution time, operating-system overhead, and deterministic inner-loop execution. The 50 μs value reported in Table 3 represents the plant and inner-loop simulation discretization and does not represent the sampling period of the RL supervisory layer.

Table 3. Complexity and deployment footprint of the proposed supervisory controller.

Component	Approximate Parameters/Cost	Function and Limitation
Physics estimator	Analytical; negligible trainable parameters	Generates power balance and thermal residuals
Graph-temporal encoder	0.19 M parameters	Encodes multi-port and temporal dependencies
Actor–critic head	0.27 M parameters	Generates and evaluates supervisory actions
Safety projection	QP with ≤8 decision variables	Enforces constraints; target-hardware WCET not yet measured
Total neural pathway	Approximately 0.46 M parameters	66 μs mean policy inference time in simulation
End-to-end controller	Not yet measured on target hardware	Includes sensing, preprocessing, QP solution, communication, and inner-loop execution

The proposed RL policy is intended to operate as a slower supervisory coordinator, while the high-bandwidth current, voltage, and modulation loops remain under deterministic MPC/PI control [6,7,8,34]. This separation reduces the computational burden relative to direct switching-level learning. Nevertheless, the proposed framework remains more computationally demanding than conventional PI-Droop control. The present simulation therefore demonstrates algorithmic feasibility but does not establish real-time schedulability on an industrial controller. Target-hardware profiling, worst-case execution-time analysis, QP-solver benchmarking, and deadline-miss evaluation remain necessary before deployment. The computational footprint and deployment-related complexity of the supervisory controller are summarized in Table 3.

4.3. Safety, Reliability, and Ethics

SST control integrates safety-critical infrastructure. As a result, the AI layer is limited to the application of predetermined constraints, rather than an unconstrained optimization process, and logs projection events to allow operators to measure how often the learning process would have violated current, voltage, or thermal limits if the shield was not in place.

From a reliability perspective, temperature and violation rate are treated as primary metrics, not secondary. From an ethics and grid operation perspective, operators must have access to the controller’s interpretation so that they can bypass or deactivate it. As a result, this method is better suited for advisory/supervisory implementations than for opaque, autonomous switching at the device level.

Safety-shield intervention must also be treated as a diagnostic quantity. A low post-projection violation rate does not necessarily mean that the unconstrained policy has fully learned the feasible operating envelope. Deployment monitoring should therefore include projection frequency, the correction magnitude $\parallel u_t-a_t{\parallel }_2$, consecutive-intervention count, solver-feasibility status, and the identity of the active constraint. Persistent intervention would indicate excessive policy dependence on the QP shield and should trigger constraint-aware retraining, reward adjustment, or transfer to a deterministic fallback controller. The main safety, reliability, and ethics considerations embedded in the proposed design are summarized in Table 4.

Table 4. Safety and ethics considerations built into the proposed design.

Risk Dimension	Proposed Mitigations
Over-current/device stress	Action projection and rate limits
Thermal runaway	Thermal term in reward + thermal constraint
Unsafe reactive injection	Grid-code-style set-point bounds
Data shift/drift	Scenario randomization + residual features
Cyber misuse	Hierarchical deployment; human override

5. Experimental Setup

A scenario-based stochastic digital twin was developed to evaluate the controller under operating conditions representative of renewable-rich distribution feeders. The simulated plant represents a 1 MVA three-stage SST with averaged converter dynamics, a thermal RC model, measurement noise, bounded uncertainty, and controllable electrical disturbances. The synthetic scenario generator alters renewable resources such as solar and wind energy, grid strength, load composition, temperature, and converter settings within specific ranges. The controlled design of the synthetic scenario generator allows for statistical analysis over a broad range of operating conditions while clearly separating simulation results from experimental results. No hardware-in-the-loop or physical prototype validation is claimed in the present study. The digital-twin structure, scenario generator, and controller-under-test arrangement are illustrated in Figure 4.

Figure 4. Digital twin, scenario generator, and controller-under-test arrangement.

The core simulation and plant parameters used throughout the experimental study are listed in Table 5. The disturbance scenarios and operating conditions used for scenario-based evaluation are summarized in Table 6. The baseline methods included in the comparative study and the rationale for their selection are summarized in Table 7. Metrics utilized consisted of settling time, overshoot, total harmonic distortion, efficiency, temperature at hotspot, control compute time, loss, and violation of constraints. The confidence interval for all metrics reported was 95%. For all dominant metrics, paired t-tests were used because the same stochastic scenario was applied to the methods, along with paired Cohen’s d as the effect size.

Table 5. Core simulation parameters used in the evaluation.

Parameter	Value
MV-side voltage	13.8 kV AC
LV-side voltage	400 V AC/750 V DC
Nominal power	1 MVA
DC link	1.2 kV
Plant and deterministic inner-loop simulation step	50 us
Episode length	1.5–3.0 s
AFE filter L	2.1 mH
DAB nominal phase-shift limit	0.35 rad
Thermal RC time constant	18 s
Training episodes	300
Scenario count	40
Random seed	42

Table 6. Workload and disturbance suite used for scenario-based evaluation.

Scenario	Duration (s)	Main Disturbance	Purpose
Nominal feeder support	1.5	10% step load	Tracking baseline
PV ramp and cloud edge	2.0	35% PV fluctuation	Renewable intermittency
Fast EV charger pulse	1.2	0.6 pu to 1.0 pu LV demand	Port power redistribution
MV voltage sag (0.8 pu)	0.4	20% MV dip + reactive request	Ride-through and current limiting
Parameter drift + thermal rise	3.0	$15\% L_f$ drift + +12 C ambient	Robustness to aging
Resynchronization	1.8	Frequency/phase mismatch	Rid support transition

Table 7. Baselines and comparison rationale.

Baseline	Description	Main Limitation/Advantage
PI-Droop	Classical cascaded controllers with droop services	Simple but limited under coupling
MPC	Prediction-based supervisory control	Good dynamics but parameter-sensitive
DRL	Unconstrained actor–critic supervisory policy	Flexible but unsafe without shield
ANN-TPS	ANN-assisted triple-phase-shift scheduler	Fast inference but weak on physics/thermal coupling
Proposed	Physics-informed safe RL + inner-loop MPC	Balances performance and safety

6. Results and Analysis

The training curve in Figure 5 shows rapid improvement during the early episodes, followed by smoother convergence as the policy learns to reduce actions requiring safety intervention. The increase in cumulative reward is accompanied by a progressive reduction in constraint violations, indicating that the controller does not improve the performance objective by sacrificing operational safety. The cross-method comparison in Figure 6 and Table 8 demonstrates that the proposed controller provides the best overall balance among dynamic response, waveform quality, conversion efficiency, thermal stress, and constraint satisfaction.

Figure 5. Training convergence of the proposed policy showing reward improvement and declining constraint violations.

Figure 6. Comparative performance of the evaluated controllers in terms of settling time, THD, efficiency, and hotspot temperature.

Table 8. Main quantitative results with 95% confidence intervals.

Method	Settling Time (ms)	THD (%)	Efficiency (%)	Hotspot T (°C)	Violation Rate (%)
PI-Droop	18.56 +/− 0.42	3.72 +/− 0.11	95.17 +/− 0.11	97.13 +/− 0.68	8.14 +/− 0.32
MPC	12.42 +/− 0.28	2.74 +/− 0.08	96.08 +/− 0.09	91.38 +/− 0.59	3.88 +/− 0.26
DRL	11.35 +/− 0.38	2.50 +/− 0.09	96.45 +/− 0.06	89.36 +/− 0.67	5.24 +/− 0.27
ANN-TPS	10.83 +/− 0.38	2.28 +/− 0.07	96.81 +/− 0.07	88.52 +/− 0.69	3.16 +/− 0.23
Present study	8.17 +/− 0.24	1.77 +/− 0.06	97.43 +/− 0.05	84.86 +/− 0.44	1.07 +/− 0.12

The proposed method achieved a mean settling time of 8.17 ms compared with 12.42 ms for MPC and 11.35 ms for unconstrained DRL. The lowest measured THD was 1.77%, with the highest recorded efficiency being 97.43%, and the lowest observed hotspot temperature being 84.86 °C, along with the lowest constraint violation rate of 1.07% for this device. The result shows that the improvements are not limited to a particular objective (electrical or thermal) but rather span across all metrics, including efficiency and safety.

The main cross-method comparison of settling time, THD, efficiency, and hotspot temperature is visualized in Figure 6.

The corresponding quantitative results with 95% confidence intervals are reported in Table 8.

6.1. Dynamic Response, Thermal Behavior, and Grid Support

Figure 7 illustrates how the two-way link has a transient response to the simultaneous implementation of an MV voltage sag and a step load. The PI-Droop method produces a slow recovery of the DC link and exhibits larger oscillations than the proposed controller, whereas the proposed controller achieves faster recovery time for the DC link and also provides reactive power support earlier without an excessive number of sinusoidal oscillations. These results support the use of hierarchical AI at the supervisory coordination layer, where competing voltage, power-flow, and thermal objectives must be prioritized online.

Figure 7. Dynamic trajectories under a simultaneous load step and MV voltage sag, including DC link regulation and reactive power support.

The thermal and efficiency results in Figure 8 complement the electrical transient results in Figure 7. Although some controllers provide acceptable short-term dynamic performance, they produce higher hotspot temperatures because they repeatedly select higher-loss modulation patterns or power-sharing actions. The proposed controller mitigates this behavior by including hotspot temperature in the reward and thermal-model inconsistency in the physics-informed residual. Consequently, the controller improves the transient response without transferring the performance cost to semiconductor thermal stress.

Figure 8. Hotspot temperature trajectory and efficiency profile across the evaluated loading range.

The thermal and efficiency behavior of the compared controllers across the loading range is presented in Figure 8.

6.2. Ablation, Robustness, and Statistical Significance

The ablation analysis in Figure 9 and Table 9 identifies the contribution of each major component. Removing the physics-informed loss increases the settling time from 8.17 to 9.46 ms, THD from 1.80% to 2.05%, hotspot temperature from 84.7 °C to 87.6 °C, and violation rate from 1.1% to 2.0%. These changes demonstrate that power balance and thermal residuals improve both dynamic performance and physical consistency.

Figure 9. Ablation analysis showing the contributions of physics regularization, QP safety projection, temporal encoding, and thermal weighting.

Table 9. Ablation results of the proposed controller.

Variant	Settling Time (ms)	THD (%)	Hotspot T (°C)	Violation Rate (%)
Full method	8.17	1.8	84.7	1.1
- Physics loss	9.46	2.05	87.6	2.0
- Safety shield	8.41	1.94	85.9	4.8
- Temporal encoder	9.02	2.01	86.5	1.8
- Thermal term	8.62	1.84	88.8	1.7

Removing the QP safety shield increases the violation rate from 1.1% to 4.8%, despite producing only a moderate change in mean settling time. This result indicates that average tracking performance alone is insufficient for evaluating a safety-critical power electronic controller. Removing the temporal encoder degrades the response to time-dependent disturbances, while removing the thermal reward term increases the hotspot temperature to 88.8 °C.

Figure 10 presents the constraint violation robustness maps under changes in grid strength and converter parameter drift. The proposed framework maintains lower violation rates than MPC under the combined effects of weak-grid operation and model uncertainty. The paired statistical tests reported in Table 10 further show that the improvements over MPC, DRL, and ANN-assisted scheduling are statistically significant for settling time, efficiency, and hotspot temperature.

Figure 10. Constraint violation heatmaps under variations in grid strength and converter parameters.

Table 10. Selected paired significance tests for the proposed method.

Comparison	p (Settling Time)	p (Efficiency)	p (Temperature)
Proposed method vs. MPC	1.84 × 10−23	4.87 × 10−27	6.93 × 10−22
Proposed method vs. DRL	9.03 × 10−16	2.51 × 10−24	5.90 × 10−14
Proposed method vs. ANN-TPS	7.10 × 10−15	1.29 × 10−15	2.37 × 10−10

The robustness of the proposed method under grid strength and parameter drift variations is visualized in Figure 10.

The numerical ablation results for the full method and its reduced variants are summarized in Table 9. The statistical significance of the main pairwise performance comparisons is reported in Table 10.

6.3. Error Analysis

The largest control errors occur during compound disturbances in which weak-grid conditions, a rapid LV-side power pulse, and converter parameter drift occur simultaneously. Under these conditions, the unconstrained policy proposes actions closer to, or outside, the feasible operating boundary, causing the QP safety shield to intervene more frequently. Although the projection prevents infeasible actions from reaching the inner control loops, frequent intervention should not be interpreted as a beneficial outcome by itself.

Persistent QP intervention can indicate a mismatch between the policy’s learned action distribution and the feasible operating set. Excessive dependence on the shield may reduce policy autonomy, generate conservative actions, distort the intended action distribution, or produce repeated corrections near an active constraint. Therefore, the QP shield should function as a final safety layer rather than as a continuous substitute for learning constraint-aware behavior.

Deployment evaluation should report the projection frequency, mean and maximum correction norms, consecutive-intervention count, active-constraint distribution, solver-feasibility rate, and time required to solve the QP. Persistent intervention should trigger retraining with a stronger intervention penalty, modification of constraint-aware exploration, adjustment of the reward weights, or transfer to a deterministic fallback controller. The present study reports the resulting constraint violation rate but does not separately report all raw intervention statistics. This is a limitation of the current evaluation and a required measurement for future HIL experiments.

Computational cost represents a second limitation. The proposed architecture is more demanding than PI-Droop because it combines graph-temporal encoding, actor–critic inference, physics residual computation, and constrained action projection. Although the measured mean policy inference time is compatible with a supervisory application, the current simulation does not establish worst-case real-time execution on industrial hardware. The practical value of the additional computation must therefore be assessed against the achieved improvements in grid-support capability, power quality, thermal stress, and safety.

7. Discussion

The benefits associated with Artificial Intelligence (AI) for the control of SSTs are primarily at the supervisory level of control rather than direct replacement of the converter level of control. In the proposed architecture, the learning process through an AI-based control policy is set up to provide high-level references such as power and voltage that are defined by a policy, and the inner control loops remain model-based and operate at high bandwidth. This division of responsibility may result in greater reliability for grid-connected power electronic assets because it retains the characteristically deterministic nature of conventional control designs, thus requiring very high bandwidth, while exploiting data-driven adaptation when making multi-dimensional decisions for which there is typically no manual means of encoding that data.

Another significant finding is that the utilization of physics-based regularization not only increases the speed of the training algorithms but also improves the operational performance of the resulting controller. The ablative testing shows that without a physics term, there are significant increases in settling time, total harmonic distortion (THD), peak temperature at a hotspot, and the violation rate compared with a system that undergoes training with the addition of the physics term. Another finding is that while the absence of a safety shield in some cases provides only a small average tracking error, the corresponding constraint violation rate increases substantially. The overall significance here is particularly important in the context of power electronic systems, as a controller must not be evaluated on the basis of average tracking performance.

The QP shield should nevertheless be interpreted as a last-line safety mechanism rather than evidence that the unconstrained policy is intrinsically safe. Occasional projection is expected during rare compound disturbances, but persistent projection would indicate that the policy has not adequately internalized the feasible operating envelope. Because the current evaluation reports post-projection violation rates without a complete distribution of projection frequency and magnitude, the results cannot establish that shield dependence has been eliminated. Future evaluations should treat safety-intervention statistics as primary deployment metrics alongside tracking, efficiency, THD, and temperature.

The numerical results also indicate that there are trade-offs at the system level. Compared to MPC, the proposed controller reduces settling time (12.42 ms vs. 8.17 ms), total harmonic distortion (2.74% vs. 1.77%), increases efficiency (96.08% vs. 97.43%), decreases hotspot temperature (91.38 °C vs. 84.86 °C), and reduces the violation rate (3.88% vs. 1.07%). All these improvements are statistically significant and practically significant because the SST in renewable-rich feeders must meet both power quality and efficiency, and thermal reliability at the same time. The improvement in response times alone would not justify the use of an AI-based controller if the response time came at a cost of thermal stress or safety. However, both transient quality and thermal stress metrics improved through the use of the proposed method. These results support the use of hierarchical safe RL as a supervisory control strategy.

Several limitations should be considered when interpreting the results. First, the framework is computationally more demanding than conventional PI-Droop control, and the reported 66 μs value represents mean policy inference rather than worst-case end-to-end execution. Second, the digital twin uses averaged converter and thermal models and does not represent all hardware effects, including ADC quantization, PWM dead time, communication jitter, thermal-sensor delay, device mismatch, protection-system interaction, and real-time QP-solver behavior. Third, the complete controller has not yet been validated through HIL testing or a physical SST prototype.

Future validation should proceed in two stages. The first stage should employ controller-HIL testing using a real-time SST plant emulator to evaluate worst-case execution time, deadline-miss rate, QP-solver feasibility, measurement and communication delays, and safe fallback behavior. The second stage should employ power-HIL or a scaled three-stage SST prototype to examine switching non-idealities, thermal estimation accuracy, protection coordination, semiconductor stress, and long-duration stability. Consequently, the current findings should be interpreted as simulation-based evidence supporting further validation rather than as proof of deployment readiness.

8. Conclusions

This paper presented a physics-informed safe reinforcement learning (RL) framework for supervisory control of three-stage solid-state transformers (SSTs) in renewable-rich distribution grids. The work addressed a clear control gap in the SST literature: most existing approaches either rely on conventional model-based regulation with limited adaptability or apply data-driven control to a narrow converter stage without enforcing system-wide safety and physical consistency. In contrast, the proposed method combined a graph-temporal policy, physics-based residual shaping, and a quadratic-program safety shield within a hierarchical architecture in which conventional inner loops were preserved for fast actuation.

Simulation results demonstrate that the current architecture is co-designed to improve multiple objectives under multiple events. The controller was shown to have a settling time of 8.17 ms, total harmonic distortion (THD) of 1.77%, a 97.43% efficiency, a maximum hotspot temperature of 84.86 °C, and a constraint violation rate of only 1.07% across 40 stochastic scenarios, significantly outperforming other baseline controllers including PI-Droop, MPC, DRL, and ANN-assisted scheduling methods.

The ablation study demonstrated that both the physics-based term and safe projection term are critical components of the architecture: the physics-based term improves dynamics and thermal performance, while the safe projection term is crucial for providing deployable operating safety. This suggests that AI for supervisory control of SSTs should not be designed as an unconstrained black-box controller, but rather as a physics-informed decision layer included in a certified control stack.

The originality of this study comes from the integration of three complementary components: AI-based supervisory coordination for multi-port systems (or “supervised coordination”), residual-based physical regularization, and hard feasible set projection. By combining these three components, the controller can adapt to renewable energy fluctuations, EV charging pulses, MV voltage sags, and converter parameter drift while providing a quality waveform, thermal control, and explicit operating constraints.

The study thus offers a new control methodology along with a documented simulation protocol that can be used to evaluate three-stage SSTs via hierarchical supervised RL control. The current validation was limited to the simulation of the average model; therefore, results should not be interpreted as proof of hardware readiness for deployment.

Future work will involve hardware-in-the-loop (HIL) verification of the controller via a real-time digital SST model, and power-HIL or scaled-prototype verification. These efforts will assess the worst-case scenario for computational latency, QP intervention statistics, measurement and communication latency, protection coordination, switching non-idealities, thermal estimation accuracy, and safe fallback operation. Until this validation is completed, the proposed supervisory control framework will not be ready for deployment within practical renewable-rich distribution networks.