Reinforcement Learning-Driven Control Strategies for DC Flexible Microgrids: Challenges and Future

Abstract

The increasing penetration of photovoltaic (PV) generation, energy storage systems, and flexible loads within modern buildings demands advanced control strategies capable of harnessing dynamic assets while maintaining grid reliability. This Perspective article presents a comprehensive overview of reinforcement learning-driven (RL-driven) control methods for DC flexible microgrids—focusing in particular on building-integrated systems that shift from AC microgrid architectures to true PV–Energy storage–DC flexible (PEDF) systems. We examine the structural evolution from traditional AC microgrids through DC microgrids to PEDF architectures, highlight core system components (PV arrays, battery storage, DC bus networks, and flexible demand interfaces), and elucidate their coupling within building clusters and urban energy networks. We then identify key challenges for RL applications in this domain—including high-dimensional state and action spaces, safety-critical constraints, sample efficiency, and real-time deployment in building energy systems—and propose future research directions, such as multi-agent deep RL, transfer learning across building portfolios, and real-time safety assurance frameworks. By synthesizing recent developments and mapping open research avenues, this work aims to guide researchers and practitioners toward robust, scalable control solutions for next-generation DC flexible microgrids.

1. Introduction

The accelerating penetration of PV generation, battery energy storage systems (BESS), and flexible loads within modern buildings is reshaping the operational paradigm of distribution-level energy systems. Traditional alternating-current (AC) microgrids, although effective in coordinating distributed energy resources (DERs), increasingly reveal structural limitations in scenarios with high DC-native renewable penetration [1]. These constraints—such as repetitive AC/DC conversion stages, synchronization requirements among inverter-dominated sources, and conversion-induced efficiency losses—have stimulated a transition toward direct-current (DC) microgrids, which inherently align with the electrical characteristics of PV modules, electrochemical storage, and an expanding portfolio of DC loads [2].  Building on this technological shift, recent developments have introduced the PEDF system as an integrated building or community-level architecture that unifies PV generation, hierarchical storage, multi-level DC distribution, and flexible demand-side resources within a coordinated cyber–physical framework. Unlike earlier AC or standalone DC microgrids, PEDF systems emphasize intelligent flexibility, fine-grained controllability, and multi-building interoperability, thereby laying the groundwork for DC-based urban energy clusters capable of high renewable self-consumption, resilience enhancement, and coordinated energy sharing.  In parallel with this architectural evolution, RL has emerged as a promising paradigm for managing the increasing operational complexity of PEDF systems. Recent studies across microgrid control, HVAC optimization, and building energy management demonstrate RL’s capability to learn optimal or near-optimal operational strategies under uncertainty [3,4,5]. Yet, despite notable advances, challenges remain in ensuring safety-critical operations, sample efficiency, interoperability across multi-agent environments, and reliable real-world deployment [6,7].  Given this convergence of DC microgrid development and AI-enabled control, a systematic examination of RL-driven strategies for PEDF systems is both timely and necessary. This perspective aims to (i) elucidate the technological evolution from AC microgrids to PEDF architectures, (ii) synthesize recent progress in RL-based control for PEDF systems, (iii) identify the central challenges limiting deployment, and (iv) outline future research directions spanning safety-assured RL, hierarchical and multi-agent control structures, and standardization pathways. Through this analysis, we seek to provide researchers and practitioners with a forward-looking understanding of how RL can unlock scalable, intelligent, and resilient control for next-generation DC energy systems.

2. From AC Microgrids to PEDF Systems—Definition and System Architecture

2.1. Limitations of AC Microgrids Under High Renewable Penetration

The evolution of building-integrated energy systems has followed a gradual yet well-defined trajectory toward improved energy efficiency, enhanced operational flexibility, and deeper penetration of renewable energy resources. Early AC microgrids were developed as decentralized electrical networks that interconnected distributed energy resources (DERs) and end-use loads through conventional AC infrastructure. Such systems enabled islanded operation, enhanced local resilience, and benefited from mature protection and control schemes inherited from legacy power systems [8]. Consequently, AC microgrids long served as the dominant paradigm for integrating distributed generation at building and community scales.

With the rapid proliferation of PV generation and BESS, the inherent limitations of AC-centric architectures have become increasingly evident. The integration of intrinsically DC-based sources and loads into AC microgrids necessitates multiple stages of AC/DC and DC/AC power conversion, which unavoidably introduces additional conversion losses and degrades overall system efficiency [9,10]. Moreover, the extensive use of power electronic interfaces intensifies harmonic distortion and imposes strict synchronization and control requirements on inverter-dominated networks [11,12]. As the proportion of inverter-interfaced distributed energy resources continues to increase, the effective system inertia is progressively reduced, thereby heightening vulnerability to voltage and frequency instability under high renewable penetration conditions [13,14].

In response to these challenges, DC microgrids have gained increasing attention due to their intrinsic compatibility with both generation and consumption subsystems. PV arrays and electrochemical energy storage are inherently DC-based, while a substantial proportion of modern building loads, including LED lighting, information and communication technology equipment, data-center loads, and electric vehicle (EV) charging infrastructure, also rely on internal DC interfaces. By eliminating unnecessary conversion stages, DC microgrids reduce energy losses, mitigate harmonic issues, and simplify local control strategies for integrating renewable energy and managing storage. However, most early DC microgrid implementations remain primarily focused on efficient power conversion and source–load balancing at the electrical layer, offering limited support for system-level coordination of demand-side flexibility, multi-time-scale scheduling, and integrated energy management.

Against this technological backdrop, the PEDF building energy system—first proposed in 2020 and subsequently validated through multiple standardization-oriented demonstration projects—represents a further architectural evolution beyond conventional DC microgrids [15]. As also summarized in

Table 1. Evolution of building-integrated power systems from AC microgrids to DC microgrids and PEDF architectures.

System Type	Advantages	Limitations	Technical Drivers for Evolution	Adaptability to High Renewable Penetration in Buildings
AC Microgrid	Mature protection, strong legacy compatibility; suitable for traditional loads.	Requires repeated AC/DC conversions; synchronization overhead; higher harmonic distortion.	Increasing penetration of DC-native loads and distributed PV; rising conversion losses.	Medium—effective but limited efficiency in high-PV scenarios.
DC Microgrid	Fewer conversions; lower harmonics; natural integration with PV/BESS.	Limited flexibility resources; weak support for demand-side interaction; control standardization lacking.	Proliferation of DC appliances, EVs, and DC chargers; local PV–storage demand.	High—efficient renewable utilization but insufficient flexibility.
PEDF System	Unified PV–storage–DC flexible control; high renewable consumption; supports demand response.	Higher EMS complexity; requires unified DC interface standards.	Building-level carbon neutrality; intelligent demand response; flexibility-centric system coordination.	Very high—supports deep decarbonization and flexible operation.

, PEDF systems integrate PV generation, multi-scale energy storage, DC distribution networks, and flexible demand-side resources within a unified cyber–physical energy management framework rather than a mere DC power delivery architecture. This integrated architecture enables coordinated sensing, communication, and control across generation, storage, and loads, thereby explicitly elevating demand-side flexibility to a first-class system resource, transforming buildings from passive energy consumers into active prosumers capable of adaptive consumption, local balancing, and bidirectional interaction with the wider power system through intelligent control strategies.

2.2. PEDF Architecture

The PEDF architecture integrates PV generation, hierarchical energy storage, DC distribution, and demand-side flexibility into a unified, flexibility-oriented energy supply framework that emphasizes low carbon intensity and high operational stability. Unlike conventional DC or hybrid microgrids that are primarily defined by electrical topology, PEDF systems are functionally characterized by the explicit modeling, aggregation, and coordinated control of demand-side flexibility as a first-class system resource. As illustrated in Figure 1, the overall system is organized around a common DC backbone and can be conceptually decomposed into several tightly coupled functional subsystems that collectively enable efficient renewable energy utilization and flexible power management at the building or community level. From a control perspective, these subsystems are coordinated through a hierarchical framework with explicit time-scale separation, ensuring that learning-based decisions do not interfere with fast electrical dynamics.

Figure 1 presents a representative schematic of a PEDF microgrid structured around a shared DC bus. In this DC-centric configuration, PV generation, battery energy storage, flexible DC loads, and the utility grid are interconnected through multiple power-electronic interfaces, forming a highly coupled energy network with bidirectional power flows. This configuration enables the PEDF system to treat generation, storage, and flexible demand within a unified supervisory control framework, rather than as loosely coordinated subsystems. Specifically, fast converter-level dynamics are governed by local, model-based controllers, while supervisory coordination is performed at slower time scales through the EMS. PV arrays are connected to the DC bus via DC/DC converters equipped with maximum power point tracking (MPPT), ensuring efficient solar energy harvesting under varying irradiance and temperature conditions. Battery energy storage systems are interfaced through bidirectional DC/DC converters, enabling energy balancing, DC voltage regulation, and flexibility provision [16]. Grid interaction is achieved via an AC/DC converter that enables bidirectional power exchange in both grid-connected and islanded operating modes.

At the generation and storage levels, power conditioning is primarily handled by DC/DC conversion stages. The PV subsystem interfaces with the DC bus via a unidirectional DC/DC converter with MPPT functionality, enabling continuous, real-time optimization of renewable energy extraction. The energy storage subsystem consists mainly of electrochemical batteries (most commonly lithium-ion or lead-acid technologies) connected to the DC network via bidirectional DC/DC converters, typically based on Buck/Boost topologies. These converters support multiple operational functions, including charge-discharge regulation, transient power compensation, peak shaving, and DC voltage stabilization, thereby mitigating the inherent variability of PV generation. Within PEDF systems, these functions are further coordinated at the system level to support flexibility-aware scheduling and long-term energy management objectives. Importantly, such coordination is realized through reference-setting and envelope constraints issued by the EMS, rather than direct manipulation of converter switching actions.

The principal power conversion interfaces within the PEDF architecture are further detailed in Figure 2a,b. As shown in Figure 2a, the bidirectional DC/DC converter governs charging and discharging processes of the storage subsystem while stabilizing the DC bus voltage through adaptive duty-cycle modulation. This bidirectional capability enables seamless energy exchange between the storage module and the DC network across different operating conditions. Figure 2b illustrates the bidirectional DC/AC inverter that forms the external interconnection point between the DC network and the utility grid. This interface ensures voltage and frequency compatibility, supports smooth transitions between grid-connected and islanded modes, and enables ancillary service provision when permitted by system and regulatory constraints. Together, these converters constitute the core power-conditioning layer of the PEDF system, facilitating coordinated interaction among PV generation, energy storage, and flexible demand-side resources. It should be noted that the converter topologies illustrated in Figure 2a,b represent typical candidate implementations commonly adopted in PEDF systems, rather than fixed or mandatory designs. These power-electronic interfaces form a dedicated power-conditioning layer, upon which higher-level flexibility coordination and intelligent energy management functions are built. This layered separation ensures that millisecond-level voltage and current regulation remains fully decoupled from learning-based supervisory control decisions.

The DC bus and flexible load subsystem constitute the physical and functional backbone of the PEDF architecture. By interconnecting PV generation, storage units, and controllable loads (such as HVAC systems, lighting, consumer electronics, and EV chargers [17,18,19]), the DC bus enables efficient power routing and coordinated utilization of demand-side flexibility. It should be emphasized that HVAC systems are classified within the flexible load subsystem based on their role as controllable flexibility resources at the EMS level, rather than their underlying AC or DC electrical interfaces. In PEDF systems, flexible loads are not treated as passive consumers but are explicitly incorporated into the energy management decision space as dispatchable and schedulable resources. To accommodate heterogeneous load requirements, the DC distribution platform is typically implemented using standardized voltage levels, including 48 V, 220 V, 375 V, and 750 V, supporting scalable deployment at both building and cluster scales.

A supervisory EMS coordinates the operation of all subsystems by scheduling power flows and control actions to maximize on-site renewable energy consumption while maintaining supply-demand balance under uncertainty. Serving as the functional core of PEDF systems, the EMS enables system-level coordination across generation, storage, and flexible demand, thereby distinguishing PEDF architectures from conventional DC microgrids that rely primarily on local or rule-based control. In practical PEDF deployments, EMS decisions are typically executed at minute-level or multi-minute time resolutions, making them well-suited for optimization- or learning-based scheduling while preserving system stability. Acting as the primary interface between physical components and higher-level optimization or learning-based control strategies, the EMS enables the PEDF architecture to support advanced data-driven energy management paradigms [20]. For clarity, the main subsystems, key components, and their functional roles within the PEDF architecture are summarized in

Table 2. Main components and functional roles of subsystems in a PEDF architecture.

Subsystem	Key Components	Primary Functions	Control Interfaces	Notes
PV Generation Subsystem	PV arrays, unidirectional DC/DC converters with MPPT	Solar energy harvesting and controlled injection into the shared DC backbone	MPPT algorithms, converter duty-cycle control	Power output is intermittent and strongly dependent on irradiance and temperature
Energy Storage Subsystem	Electrochemical batteries, bidirectional DC/DC converters	Energy buffering, DC voltage stabilization, peak shaving, and flexibility provision	SOC regulation, charge–discharge scheduling, voltage support	Buck/Boost-based topologies enable fast bidirectional power exchange
Grid Interface Subsystem	Bidirectional AC/DC inverter, protection, and synchronization units	Bidirectional grid interaction, islanded operation, and ancillary service support	Voltage–frequency control, grid current regulation	Forms the external coupling point between the DC microgrid and the utility grid
DC Bus and Flexible Load Subsystem	Multi-level DC bus (48/220/375/750 V), lighting, ICT loads, EV chargers	DC power distribution, flexible demand integration, and local power routing	Load scheduling, demand response signals, DC voltage coordination	Flexible loads (e.g., HVAC) are grouped based on EMS-level controllability and flexibility abstraction, independent of their physical AC or DC electrical interfaces.
Energy Management System (EMS)	Supervisory controller, sensing, communication, and monitoring modules	System-wide coordination of generation, storage, loads, and grid power flows	Set-point dispatch, optimization, and learning-based control interfaces	Provides the interface between physical assets and advanced data-driven control strategies

. Taken as an integrated whole, the PEDF architecture represents a functional evolution beyond conventional AC and DC microgrids, characterized by bidirectional energy flow, explicit flexibility integration, multi-level DC coordination, and compatibility with intelligent control frameworks, thereby facilitating deeper integration with building clusters and urban energy networks. The explicit separation between fast electrical control layers and slow supervisory decision layers further ensures that advanced learning-based methods can be incorporated without compromising converter-level stability or protection requirements.

2.3. Coupling with Building Energy Systems

In the broader context of urban decarbonisation and the ongoing transition toward digitalised and electrified energy systems, PEDF architectures are increasingly recognised as a key enabling layer for next-generation DC-based buildings. Rather than operating as isolated building-level solutions, PEDF systems provide a scalable, interoperable framework that supports the seamless integration of distributed energy resources, flexible loads, and intelligent control across individual buildings and building clusters.

Figure 3 illustrates a representative application of a PEDF microgrid within a building energy system, organised around a shared DC bus. In this configuration, photovoltaic generation, battery energy storage, grid interfaces, and a wide range of DC loads—including HVAC systems, lighting, office equipment, and EV chargers—are directly interconnected through power electronic converters. By adopting a DC-centric architecture, the PEDF system reduces redundant AC/DC conversion stages, improves overall energy efficiency, and enables direct coupling between renewable generation and storage and end-use devices.

When extended to district- or campus-scale deployments, multiple PEDF-enabled buildings can be interconnected via DC feeders to form multi-building microgrids or local energy-sharing communities. Such configurations enable coordinated operation of distributed PV systems, storage units, and flexible demands, yielding several functional advantages:

• DC intra- and inter-building distribution, which minimizes conversion losses, simplifies system architecture, and enhances compatibility with native DC sources and loads.
• Integrated PV-storage coordination, allowing higher levels of renewable self-consumption, improved operational autonomy, and enhanced resilience through collective energy balancing.
• Hierarchical and flexible control, in which a supervisory EMS orchestrates energy flows across buildings, supports peak shaving and valley filling, and enables participation in demand response and ancillary service markets.

From a system-level perspective, the PEDF paradigm represents an evolution beyond conventional AC microgrids or standalone DC systems. By embedding controllable flexibility, unified DC distribution, and data-driven coordination into the building energy layer, PEDF architectures form a building-centric intelligent energy ecosystem. This ecosystem is inherently compatible with advanced optimisation and learning-based control strategies, such as RL, thereby positioning PEDF systems as a critical bridge between building energy automation, microgrid operation, and future autonomous urban power systems.

3. RL for Energy Management in PEDF Systems

3.1. RL: Definition and Basic Framework

RL is a class of machine learning methodologies tailored for sequential decision-making in stochastic and dynamically evolving environments [21,22,23,24]. Unlike supervised learning, RL does not rely on labelled input-output datasets; instead, an autonomous agent learns optimal control strategies through direct interaction with the environment by continuously observing system states, executing actions, and receiving scalar reward signals that encode long-term performance objectives [25,26]. This trial-and-error learning paradigm renders RL particularly suitable for complex cyber-physical energy systems, where uncertainty, delayed responses, and strong coupling among decision variables are intrinsic [27,28,29].

Formally, an RL problem is commonly modelled as a Markov decision process (MDP), defined by a state space S, an action space A, a state transition kernel $P(s_{t+1}\mid s_t,a_t)$, and a scalar reward function $\rho (s_t,a_t)$. In PEDF applications, this formulation provides a structured abstraction for explicitly mapping system-level flexibility objectives and operational constraints into the reward design, forming the theoretical basis for learning-based control in cyber-physical energy systems.

As illustrated in Figure 4, the agent-environment interaction follows a closed-loop decision framework. At each discrete time step t, the RL agent observes the system state $s_t\in S$, selects a control action $a_t\in A$ according to a parameterized policy ${\pi }_{\theta }(a_t\mid s_t)$, and receives a scalar reward:

$$r_t=\rho (s_t,a_t),$$

(1)

which evaluates the immediate operational consequence of the selected action. In practical PEDF systems, the reward function is designed to operationalize physically meaningful performance metrics rather than serving as an abstract numerical signal. The environment then transitions to a new state $s_{t+1}$ in accordance with the underlying system dynamics, and the interaction process iterates over time.

The resulting agent-environment interaction generates a state-action-reward trajectory $(s_0,a_0,r_0,s_1,a_1,r_1,\dots )$, from which the learning objective is to identify an optimal policy that maximizes the expected cumulative discounted return:

$$J\left(\pi \right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right],$$

(2)

where $\gamma \in (0,1]$ is the discount factor that balances short-term performance against long-term operational objectives.

From an algorithmic perspective, the closed-loop structure depicted in Figure 4 highlights the core computational components of RL-based control: state perception, policy evaluation, action execution, and reward feedback. Value-based methods focus on estimating the state-action value function

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}|s_t=s,a_t=a\right],$$

(3)

where policy-based approaches directly optimize the policy parameters $\theta$ by maximizing the objective in (2). actor–critic algorithms integrate both mechanisms by simultaneously learning a policy (actor) and a value estimator (critic), thereby improving convergence stability and scalability in high-dimensional and continuous control problems.

Together, the MDP formulation and the algorithmic workflow summarized in Figure 4 establish a unified theoretical foundation for applying RL to building-scale microgrids and PEDF systems, where control actions influence system evolution over time and long-term performance emerges from sequential decision-making under uncertainty [30,31,32].

Figure 4. Schematic diagram of RL principles [33].

Figure 4. Schematic diagram of RL principles [33].

3.2. Potential and Advantages of RL for PEDF Systems

In the context of PEDF systems, the state vector $s_t$ typically encapsulates key operational variables such as battery state of charge, DC bus voltage, power flow levels, renewable generation output, and aggregated load demand. Correspondingly, the action space may include continuous control signals for power electronic converters, charging and discharging commands for energy storage systems, and scheduling decisions for flexible loads. To avoid ambiguity across control layers, it should be emphasized that, in practical PEDF implementations, the RL action space is explicitly restricted to supervisory EMS-level variables. In practical PEDF implementations, these action variables are defined at the supervisory energy management level as reference values, operating envelopes, or scheduling decisions, rather than direct switching or inner-loop control commands for power electronic converters.

The reward function is commonly structured as a weighted aggregation of operationalized flexibility objectives, including economic cost minimization, renewable energy utilization, DC bus voltage regulation, battery SOC constraint enforcement, and flexible load comfort bounds [34,35,36]. Reward terms are typically normalized with respect to rated power, voltage limits, or comfort bounds, and the associated weights are selected to encode relative system-level priorities at the EMS layer, reflecting planning and operational preferences rather than controller-level tuning. As a result, reward weighting is treated as a system modeling choice, and sensitivity analysis with respect to reward weights is required in practical implementations to assess robustness and performance trade-offs, rather than to optimize numerical performance alone.

The rapid proliferation of PEDF systems, together with the widespread deployment of power-electronics-interfaced end-use devices, has significantly increased the structural and operational complexity of modern building microgrids. As renewable penetration rises and flexibility resources become more diverse, energy management decisions increasingly involve tightly coupled dynamics, nonlinear interactions, and multi-directional power flows. To manage this complexity without compromising stability, PEDF control architectures explicitly separate fast electrical dynamics from slower energy management decision layers.

Conventional control and optimization approaches, including rule-based heuristics, deterministic optimization, and model predictive control, typically depend on accurate system models and reliable forecasts [37,38,39]. In practice, however, these assumptions are difficult to maintain under stochastic renewable generation, volatile electricity prices, and uncertain occupant-driven load behavior [40,41,42,43]. As a result, model mismatch and forecast errors can degrade control performance and limit scalability as system complexity grows. It should be noted that model predictive control (MPC) remains a strong and widely adopted baseline for EMS-level control in building microgrids and DC systems, particularly when sufficiently accurate system models and short- to medium-term forecasts are available. In the PEDF context, MPC is well-suited for handling explicit constraints and short-horizon optimization problems; however, its performance may degrade when system dynamics, occupant-driven flexible loads, or long-term objectives such as comfort–energy trade-offs and storage degradation are difficult to model accurately. Consequently, RL is not positioned as a replacement for MPC, but rather as a complementary supervisory control option under conditions where model uncertainty, flexibility heterogeneity, or long-term operational adaptation play a dominant role.

Against this background, RL has emerged as an attractive control paradigm for DC flexible microgrids and PEDF systems [44,45,46,47]. As illustrated in Figure 5, an RL agent interacts with the physical PEDF system through a supervisory energy management and safety interface. Within the adopted hierarchical control framework, RL operates exclusively at the EMS layer with minute-level decision intervals [48], while primary (millisecond-level) and secondary (second-level) control functions remain fully model-based and deterministic. System states—including PV generation and forecasts, battery state of charge, DC bus voltage, load demand, and electricity price signals—are continuously collected and used as inputs to a deep RL policy, which generates control actions such as storage charge and discharge setpoints, grid power exchange decisions, and flexible load scheduling commands. The resulting system response is evaluated through a reward function that aggregates operating cost, voltage deviation penalties, hard constraint violation penalties, and renewable energy utilization, with reward weights selected according to system priorities and subjected to sensitivity analysis in practice, enabling adaptive and autonomous energy management through repeated closed-loop interaction.

By reducing reliance on high-fidelity component models, RL alleviates the modeling burden associated with power-electronic-dominated systems while allowing control policies to adapt to changing operating conditions, component ageing, and newly integrated flexibility resources. To mitigate sensitivity to reward weighting, constrained and multi-objective RL formulations are increasingly preferred in PEDF-related studies, providing a principled mechanism for balancing competing objectives without excessive manual tuning. It should be emphasized that such learning-based adaptability is confined to supervisory scheduling and dispatch decisions and does not alter the stability margins ensured by certified low-level controllers. In addition, RL-based control architectures have been explored across multiple temporal and spatial scales in the literature. In PEDF systems, however, RL is typically confined to supervisory energy management and coordination layers, while fast converter-level regulation remains handled by conventional controllers. In the PEDF context considered here, RL is applied at the supervisory and coordination scales, while fast converter-level regulation is intentionally excluded from the RL action space.

A growing body of literature demonstrates that RL-driven controllers can outperform conventional PID, droop, and optimization-based methods in terms of voltage regulation, disturbance rejection, and operational cost reduction, particularly under high renewable penetration and limited system observability [49,50,51,52]. Collectively, these characteristics position RL as a promising enabler for intelligent, resilient, and scalable control of next-generation PEDF infrastructures [53,54].

3.3. Categories of RL Algorithms

Depending on how value functions and control policies are represented and optimized, RL algorithms can be broadly categorized into three classes: value-based methods, policy-based methods, and actor–critic architectures. This classification provides a systematic perspective for understanding algorithmic trade-offs in terms of stability, convergence behavior, and action-space compatibility, and it serves as a practical guideline for selecting suitable RL techniques for different control objectives in DC flexible microgrids [55,56,57,58].

Value-based methods focus on learning the action-value function $Q(s,a)$ and implicitly derive a control policy by maximizing the estimated Q-values. A representative algorithm in this category is the Deep Q-Network (DQN), which employs deep neural networks to approximate the Q-function in high-dimensional and nonlinear state spaces [23,59]. As illustrated in Figure 6, the RL agent observes the PEDF system state, including PV power or forecasts, battery state of charge (SOC), DC bus voltage, load demand, and electricity price signals. Based on these observations, the online Q-network selects discrete control actions—such as battery charging/discharging modes, grid import/export decisions, or load on/off scheduling—according to the greedy action selection rule $arg{max}_{a}Q(s,a)$. To enhance learning stability and data efficiency, experience tuples $(s_t,a_t,r_t,s_{t+1})$ collected from interactions with the PEDF physical system are stored in a replay buffer and randomly sampled in mini-batches for network training. In addition, a target Q-network is employed to decouple target value estimation from online parameter updates, mitigating oscillations during learning. The reward function integrates multiple operational objectives, including minimizing operating costs, penalizing DC voltage deviations, enforcing SOC constraints, and incentivizing renewable energy utilization, thereby guiding the agent toward coordinated and economically efficient control strategies [60,61,62,63].

In contrast to value-based approaches, policy-based methods directly parameterize the control policy and optimize it via gradient-based updates, making them particularly suitable for continuous action spaces commonly encountered in power-electronic and energy-storage control. actor–critic architectures combine the strengths of value-based and policy-based methods by introducing separate function approximators for policy execution (Actor) and value evaluation (Critic), resulting in improved convergence and robustness in complex environments. A prominent actor–critic algorithm is the Deep Deterministic Policy Gradient (DDPG), which extends the framework to deterministic policies in continuous control domains. As depicted in Figure 7, the Actor network maps the observed system state to continuous control actions, such as battery power setpoints or converter control signals, while the Critic network evaluates the corresponding Q-value. The Critic is trained by minimizing the temporal-difference error through the Critic loss, whereas the Actor is updated by maximizing the expected Q-value via the Actor loss. Observations from the PEDF physical system are continuously fed back to update the state representation, forming a closed-loop learning and control process. This architecture enables DDPG to effectively address control tasks such as DC bus voltage regulation, energy storage dispatch, and coordinated power flow management under uncertainty.

Table 3. Representative RL algorithms for DC flexible microgrid control.

Algorithm	Policy Type	Action Space	Key Characteristics	Typical Applications
DQN [64,65,66,67]	Off-policy	Discrete	Value-based learning with deep Q approximation	Mode selection, generator on/off scheduling, basic demand response
DDPG [12,66,68,69]	Off-policy	Continuous	Deterministic actor–critic for continuous control	Battery charge–discharge control, inverter power regulation
PPO [70,71,72,73]	On-policy	Continuous	Stable policy updates via clipped objective	Real-time EMS, adaptive load coordination
SAC [74,75,76]	Off-policy	Continuous	Entropy-regularized stochastic policy	PV–storage coordination, DC bus voltage regulation
A2C/A3C [77,78]	On-policy	Discrete/Continuous	Parallel actor–critic learning	Distributed EMS and multi-agent control
TD3 [79,80,81,82,83,84]	Off-policy	Continuous	Twin critics mitigate value overestimation	Optimal power flow and storage control in AC/DC microgrids

summarizes six representative RL algorithms, along with their action-space characteristics and typical applications in DC flexible microgrid control. This structured overview not only clarifies the algorithmic landscape but also lays the foundation for subsequent discussions on implementation challenges and future research directions in applying RL to PEDF systems.

4. Key Challenges

Despite the rapid progress reported in recent studies, the large-scale deployment of RL-driven control in PEDF microgrids remains constrained by a set of unresolved technical and practical challenges [85]. These challenges stem from the intrinsic safety-critical nature of power systems, the data-intensive characteristics of modern RL algorithms, and the increasing system complexity associated with multi-agent and cyber-physical integration [86,87]. As summarized in

Table 4. Key challenges for scaling RL in PEDF systems.

Challenge Domain	System Context	RL Limitations	Required Advances	Open Gaps
Safety-Critical Exploration	Voltage, SOC, thermal, and power quality constraints	Unsafe trial-and-error; no formal guarantees	Safe RL, constrained learning, hybrid control	Real-time safety certification under fast dynamics
Sample Efficiency	Limited safe operational data	Long training cycles and high data demand	Offline RL, model-based RL, meta-learning	Lack of validated PEDF datasets
Forecast Uncertainty	Stochastic PV, prices, EVs, and loads	Policy brittleness under prediction errors	Uncertainty-aware states and rewards	No standard uncertainty modeling framework
Distributed Coordination	Multi-building and energy community operation	Non-stationarity and communication overhead	CTDE-based MARL, scalable coordination	Scalability and privacy constraints
Real-Time Deployment	Fast control, regulation, and cybersecurity	Black-box policies and cyber vulnerability	Explainable, lightweight, secure RL	Certification and trustworthiness

, the existing limitations can be broadly classified into five interrelated domains: (i) safety-critical exploration and operational risk, (ii) sample efficiency and training cost, (iii) uncertainty propagation and forecasting error coupling, (iv) scalability and interoperability in distributed multi-agent settings, and (v) real-time deployment, interpretability, and cybersecurity. All of these challenges are discussed under the assumption of a hierarchical PEDF control architecture with explicit time-scale separation, where RL is confined to supervisory energy management decisions.

(1)

Safety-critical exploration and constraint enforcement.

Unlike simulated environments, practical PEDF systems operate under stringent voltage, SOC, thermal, and power quality constraints. Any violation of these limits may lead to inverter instability, converter saturation, accelerated battery degradation, or even load interruption. Theoretical frameworks that explicitly incorporate operational constraints into optimal control formulations, such as those developed by Duan et al. [88] and extended in subsequent studies [89], provide valuable references for safe control design. However, most conventional RL algorithms rely on exploratory trial-and-error learning, which renders unconstrained exploration fundamentally incompatible with safety-critical PEDF operation. In practical PEDF architectures, RL is therefore not permitted to directly manipulate safety-critical control variables, and all physical and comfort constraints are enforced by deterministic protection and control layers. These protection and control layers correspond to primary (millisecond-level) and secondary (second-level) controllers, which remain entirely outside the RL action space. Although recent research has proposed safe RL, constrained optimization, and hybrid RL-rule-based architectures, the reliable enforcement of hard operational limits during both training and online execution remains an open challenge [90,91]. To mitigate this risk, exploration is typically conducted offline using high-fidelity digital twins or restricted to hard-constrained action spaces during online deployment, while certified low-level controllers retain full responsibility for fast electrical dynamics and protection functions. This issue is further exacerbated in DC microgrids, where fast converter dynamics and constant-power load characteristics impose millisecond-level stability requirements [92], reinforcing the necessity of strict control separation between learning-based EMS decisions and real-time converter control.

(2)

Sample efficiency and data availability.

State-of-the-art deep RL algorithms such as SAC, PPO, and TD3 typically require extensive interaction data to converge, often ranging from tens of thousands to millions of samples [93,94]. Such data requirements are impractical for real PEDF systems, where prolonged exploratory operation is neither economical nor safe. This limitation is particularly pronounced at the EMS level, where learning episodes correspond to long operational horizons measured in minutes or hours rather than fast electrical transients. While high-fidelity digital twins can partially mitigate this limitation, discrepancies between the simulated and physical environments often lead to performance degradation after deployment. Emerging paradigms, including offline RL, model-based RL, and meta-learning, offer promising avenues to improve sample efficiency, yet their application to PEDF systems remains limited and largely exploratory [95,96].

(3)

Forecast uncertainty and dynamic coupling effects.

PEDF operation is inherently influenced by multiple sources of uncertainty, including stochastic PV generation, volatile electricity prices, building thermal inertia, EV charging behavior, and occupant-driven load variability. Although fast electrical dynamics are regulated by lower-level controllers, forecast uncertainty primarily affects supervisory scheduling and dispatch decisions made at the EMS time scale. RL agents trained on imperfect forecasts may converge to brittle or short-sighted policies, particularly when uncertainty is not explicitly encoded in the state representation or reward structure [97]. Existing studies indicate that temporal window selection, feature engineering, and error-aware forecasting significantly affect policy robustness. Nevertheless, a lack of standardized state representations and uncertainty modeling frameworks continues to hinder reproducibility and cross-study comparison.

(4)

Distributed coordination and communication constraints.

As PEDF systems evolve toward interconnected building clusters and local energy communities, multi-agent RL (MARL) becomes a natural modeling choice for distributed energy management across multiple buildings or subsystems, rather than a monolithic control paradigm. Within the hierarchical PEDF framework, MARL coordination is typically implemented at the EMS level, where agents exchange low-rate scheduling information rather than real-time control signals. Communication latency, bandwidth limitations, and privacy concerns further complicate real-world deployment, highlighting the need for benchmark-driven evaluation and scalable coordination mechanisms in future PEDF research.

(5)

Real-time deployment, interpretability, and cybersecurity.

Practical PEDF deployment requires RL controllers to deliver millisecond-level inference, transparent decision logic, and robust resilience against cyber threats. In the considered PEDF architecture, millisecond-level real-time requirements apply exclusively to the primary control layers, whereas RL-based EMS decisions operate at slower time scales compatible with current inference capabilities. Deep RL policies, however, are typically implemented as opaque neural networks, which complicates certification, debugging, and compliance with building energy regulations. Moreover, adversarial manipulation of sensor data, reward signals, or communication channels may distort learned policies, raising critical concerns regarding trustworthiness and system resilience. Addressing interpretability and cybersecurity, therefore, represents a prerequisite for the safe adoption of RL in real PEDF infrastructures [98].

5. Future Perspectives

RL is widely regarded as a promising enabler of autonomous and adaptive control for next-generation PEDF systems. Although recent studies have reported encouraging proof-of-concept results, the transition from laboratory-scale demonstrations to reliable large-scale deployment remains challenging [99]. Progress beyond experimental settings will require coordinated advances in algorithm design, system integration, and regulatory alignment. Building on the current state of PEDF development [100,101] and broader progress in RL for cyber–physical energy systems [8], several key research directions can be identified, as summarized in

Table 5. Future research directions for RL-driven PEDF systems.

Direction	Core Focus	Key Outcome	Time Horizon	Remarks
Safety-by-Design RL	Constrained learning and safety filters	Certified control under physical limits	Short-term	Fundamental requirement for deployment
Sample-Efficient RL	Offline, model-based, and uncertainty-aware RL	Reduced data demand and improved robustness	Medium-term	Addresses limited real-world data
Hierarchical & Multi-Agent RL	Layered control and CTDE-based coordination	Scalable operation across assets and buildings	Medium-term	Key enabler for PEDF clusters
Real-Time Integration	Lightweight policies and EMS compatibility	Reliable online execution	Medium-term	Focus on latency and computation limits
Standardization & Benchmarking	DC interfaces, datasets, and policy alignment	Interoperable and reproducible deployment	Long-term	Requires socio-technical coordination

.

(1)

Safety-by-design and certifiable RL control.

Operational safety must form the foundation of future RL-based PEDF control. PEDF systems are subject to strict constraints on voltage, battery state of charge, and thermal stress that must be adhered to during learning and operation. Future research should therefore focus on constrained RL formulations, safety filters such as control barrier functions, and hybrid architectures that combine certified low-level controllers with high-level RL agents for supervisory optimization [64]. Ensuring provable constraint satisfaction under fast converter dynamics remains a central challenge [102].

(2)

Sample efficiency and uncertainty-aware optimization.

The data-intensive nature of modern RL algorithms poses a major barrier to real-world PEDF deployment. Approaches such as model-based RL, offline RL using historical data, meta-learning, and uncertainty-aware policy design offer promising pathways to reduce training requirements and improve robustness [103]. These methods are particularly important under stochastic PV generation, volatile electricity prices, and occupant-driven demand variability [104,105].

(3)

Hierarchical and MARL.

PEDF systems operate across heterogeneous temporal and spatial scales, from fast converter-level control to building- and community-level energy management. Hierarchical RL aligns naturally with this layered structure, while multi-agent RL enables coordination among distributed assets and buildings. In practical PEDF deployments, MARL agents are typically defined at coarse granularities, such as the subsystem or building level, in order to limit coordination complexity and communication overhead. Although centralized training with decentralized execution (CTDE) has shown potential, further work is needed to address scalability, communication efficiency, and reward coordination in large systems [106,107]. Moreover, publicly available and standardized benchmark environments specifically tailored to PEDF systems are currently lacking, which limits systematic comparisons of RL strategies and practical guidance for practitioners. The development of open, scalable PEDF benchmark platforms should therefore be regarded as an important future research direction rather than an existing capability.

(4)

Real-time implementation and system integration.

Bridging the gap between algorithm development and operational deployment remains a practical priority. Real-world PEDF applications impose strict requirements on inference latency, computational efficiency, and compatibility with existing energy management systems. Future efforts should emphasize lightweight policy representations and seamless integration of RL controllers into established supervisory control architectures [108].

(5)

Standardization, benchmarking, and socio-technical integration.

Beyond algorithmic advances, large-scale adoption of RL-driven PEDF systems depends on standardized DC interfaces, interoperable data schemas, and openly available benchmark datasets. Regulatory frameworks, tariff structures, and user acceptance must also be incorporated into control design to ensure compliance and long-term viability.

In summary, RL-driven PEDF control is evolving from conceptual exploration toward scalable engineering practice. Addressing safety certification, data efficiency, hierarchical coordination, real-time integration, and standardization—as outlined in Table 5—will be essential for realizing intelligent, resilient, and deployable DC flexible energy systems.

6. Conclusions

The growing convergence of DC-based building energy systems and learning-enabled control is reshaping the technological trajectory of future urban energy infrastructures. The PEDF architecture represents a decisive evolution beyond conventional AC or standalone DC microgrids, providing a unified platform that integrates renewable generation, hierarchical storage, flexible loads, and multi-building energy interaction. As demonstrated in recent pilot projects and modeling studies, PEDF systems offer clear advantages in conversion efficiency, controllability, and renewable self-consumption, laying the groundwork for scalable DC building clusters and energy-sharing communities.

At the same time, RL has emerged as a compelling approach for optimizing PEDF operation under uncertainty, enabling adaptive decision-making across converter-level control, building-level scheduling, and multi-agent coordination. While early results confirm RL’s capability to enhance operational efficiency, respond to stochastic variability, and support distributed flexibility management, widespread deployment remains constrained by several fundamental challenges. These include safety guarantees in power-electronic environments, limitations on sample efficiency, difficulty in coordinating multiple agents across heterogeneous time scales, gaps between simulation and practical deployment, and a lack of unified standards and benchmarking frameworks for DC building systems.

Looking ahead, future research must focus on developing safety-certified RL architectures, uncertainty-aware and data-efficient algorithms, scalable hierarchical and multi-agent structures, and high-fidelity digital twins that enable reliable sim-to-real transfer. Equally important is the advancement of PEDF system standardization—including DC interface protocols, data schemas, and cybersecurity frameworks—as well as the integration of human–building interaction, market design, and regulatory considerations into learning-enabled control strategies.

Taken together, these developments chart a pathway toward autonomous, resilient, and interoperable PEDF systems capable of functioning as the foundational infrastructure for next-generation urban energy ecosystems. By combining the architectural strengths of DC distribution with the adaptive intelligence of RL, PEDF systems hold the potential to deliver transformative gains in efficiency, flexibility, and carbon reduction across buildings and communities.