Voltage Control for DC Microgrids: A Review and Comparative Evaluation of Deep Reinforcement Learning

Abstract

Voltage stability in DC microgrids (DC MG) is crucial for ensuring reliable operation and component safety. This paper surveys voltage control techniques for DC MG, classifying them into model-based, model-free, and hybrid approaches. It analyzes their fundamental principles and evaluates their strengths and limitations. In addition to the survey, the study investigates the voltage control problem in a critical scenario involving a DC/DC buck converter with an input LC filter. Two model-free deep reinforcement learning (DRL) control strategies are proposed: twin-delayed deep deterministic policy gradient (TD3) and proximal policy optimization (PPO) agents. Bayesian optimization (BO) is employed to enhance the performance of the agents by tuning their critical hyperparameters. Simulation results demonstrate the effectiveness of the DRL-based approaches: compared to benchmark methods, BO-TD3 achieves the lowest error metrics, reducing root mean square error (RMSE) by up to 5.6%, and mean absolute percentage error (MAPE) by 7.8%. Lastly, the study outlines future research directions for DRL-based voltage control aimed at improving voltage stability in DC MG.

1. Introduction

The 21st century has witnessed a rapid increase in distributed energy resources (DER), driven by advances in power electronics, the widespread adoption of renewable energy sources (RES), and the growing use of energy storage devices (ESD) [1]. RES such as solar photovoltaic (PV), fuel cells, wind, and tidal energy systems are increasingly favored for their sustainability, abundance, and environmental benefits. They are widely used in residential, industrial, and remote applications where access to conventional grid power is limited or economically impractical. This transition supports global efforts to mitigate climate change and ensure clean, sustainable energy for future generations [2], forming the foundation for the evolution from conventional centralized grids to distributed microgrid (MG) architectures.

MG can be defined as an electrically bounded area of a distributed network that integrates local DER, ESD, and loads to form a self-sufficient energy system [3]. MG are generally classified into three types, based on the current in their main bus: AC, DC, and hybrid. AC MG are widely used to reduce dependence on the main grid and supply power to target loads [4]. Despite challenges, such as grid synchronization, power sharing among coupled DER, and frequency control, AC MG remain prevalent due to their grid compatibility and the maturity of their control and protection technologies [5].

Over the past few decades, AC MG have received primary attention due to the dominance of AC networks. However, the growing integration of RES, ESD, and DC loads has shifted interest toward DC MG [6]. In particular, DC MG offers a simpler structure with zero reactive power flow and provides reduced power losses with improved efficiency. Moreover, they exhibit superior compatibility with the increasing number of modern DC-based consumer devices [7].

Despite numerous advantages, voltage instability remains a significant challenge that affects the performance of DC MG [8]. Voltage and power fluctuations, often exacerbated by the intermittent nature of RES and other uncertainties, can degrade power quality and overall efficiency. Prolonged instability may even cause equipment malfunction or system failure. Consequently, voltage stability has remained a critical area of research, with extensive efforts devoted to developing various model-based, model-free, and hybrid control techniques.

In model-based control, a mathematical model of the system is required to design the control algorithms. These methods rely on accurate system parameters and are commonly applied in optimal and predictive control [9], with examples including sliding mode control, predictive control, passivity-based control, and the classical proportional-integral (PI) controller. They offer analytical stability guarantees and can generate control actions with high accuracy [10], but are often sensitive to parameter variations and unmodeled dynamics. In contrast, model-free techniques do not require an explicit system model and are typically experience-based or rely on data-driven approaches [11], such as fuzzy logic control, artificial neural networks, and reinforcement learning. These methods offer flexibility and adaptability, especially in complex systems. However, their reliance on data can be challenging, particularly when system dynamics change rapidly or training data is insufficient [11]. On the other hand, hybrid control techniques combine aspects of both model-based and model-free schemes, examples include physics informed neural networks and metaheuristic-optimized control techniques.

Several studies have reviewed voltage control methods for DC MG applications. For example, ref. [12] discusses various topologies and control strategies, focusing on centralized, decentralized, and hierarchical control structures. Similarly, ref. [13] analyzes the strengths and weaknesses of various control topologies, with particular emphasis on droop control. In [14], the performance of advanced control technologies is evaluated, specifically for their application in bidirectional DC/DC converters to address instability issues caused by constant power loads (CPL). Another survey, ref. [15], explores voltage control techniques that incorporate different levels of communication for standalone MG applications.

This study provides a comprehensive survey of various advanced voltage control techniques, which are categorized into model-based, model-free, and hybrid approaches. In addition to this, the paper proposes model-free DRL-based control strategies for a DC/DC converter with an input LC filter: value-based TD3 and policy-based PPO. The proposed agents are designed, trained, and tuned using Bayesian optimization (BO) to address instability issues arising in the cascaded system. Furthermore, the proposed approaches are benchmarked against other control strategies through simulations under different operating conditions. Lastly, future research directions for voltage control in DC MG are identified. The main contributions of this work are summarized as follows:

• It provides a survey of various state-of-the-art model-based, model-free, and hybrid voltage control techniques.
• It proposes BO-DRL-based solutions for voltage control of a DC/DC buck converter with an input LC filter.
• It proposes recommendations for future research, particularly those that employ machine learning and data-driven control algorithms.

The organization of the rest of the paper is as follows: Section 2 provides an overview of DC MG control. Detailed reviews of model-based, model-free, and hybrid control techniques are discussed in Section 3, Section 4, and Section 5, respectively. Section 6 presents the design and analysis of a DRL control strategy based on TD3 and PPO, incorporating BO for optimal tuning of the agents’ hyperparameters. Section 7 outlines future research directions, and finally, the conclusion is given in Section 8.

2. Background of the Study

DC MG has gained significant attention for enabling more reliable and sustainable energy systems. In general, their operation can be classified into two modes: grid-connected or standalone (autonomous) [12]. When the DC MG exchanges power with the main grid, it operates in grid-connected mode. On the other hand, when it operates independently, relying on local generation to meet load demand, it is in stand-alone mode. In this mode, ESD must absorb or release the power difference between the generation and the load.

Depending on the application, a DC MG incorporates various types of power electronic converters to integrate distributed energy sources, energy storage systems, and loads. The converters are typically connected to a common DC bus, which serves as an interface for achieving high efficiency and straightforward control. Notably, DC MG has seen widespread adoption across industrial, residential, and military sectors, including electric vehicles, data centers, naval ships, submarines, spacecraft, and net-zero energy buildings. A typical architecture of DC MG is shown in Figure 1.

2.1. Cause of Voltage Instability

The pressing operational challenges of DC MG, particularly in maintaining voltage stability, arise from several key factors, including the following:

• Intermittent generation: The fluctuating nature of RES causes mismatch between generation and demand, leading to voltage instability [16]. For example, PV output depends on weather conditions like temperature and solar irradiance.
• CPL are nonlinear loads characterized by negative incremental impedance [17]. They maintain constant power consumption despite DC bus voltage fluctuates. DC load regulated by DC/DC converters often exhibits CPL behavior, which can destabilize the system. Voltage instability caused by CPL has been extensively studied [18,19]. Figure 2 illustrates this negative incremental impedance behavior.
• Pulse power loads (PPL) draw large current in short durations, potentially causing voltage instability due to their high-power characteristics [20]. They are common in onboard MG of electric ships, particularly for systems like sonar and radar.
• Faults and aging can alter system dynamics, posing the risk of instability and poor performance. Additional challenges related to the operation, control, and protection of DC MGs are reported in [21].
• Filters, such as commonly used LC types, improve power quality, but can reduce the damping ratio of DC MG, increasing the risk of instability [22].

2.2. DC Microgrid Control

To ensure stable and efficient operation, DC MG commonly employs a hierarchical control structure comprising three levels: primary, secondary, and tertiary [23]. The primary controller acts locally to regulate the DC bus voltage and maintain power sharing at the lowest level. It operates at the fastest timescale, and voltage deviations at this level are corrected by the slower secondary controller [24]. The tertiary controller, operating at the highest and slowest level, optimizes the overall system performance and power flow among units. This study specifically focuses on recent advances in primary control, emphasizing model-based, model-free, and hybrid approaches.

3. Model-Based Techniques

Model-based control, which relies on knowledge of the system model, forms the foundation of control theory for designing effective controllers. In DC MG, model-based voltage control employs both conventional and advanced controllers. Traditional linear techniques, such as PI and LQR [25], have been successfully implemented but often struggle with system uncertainties. A review of advanced control techniques is presented in the following sub-sections.

3.1. Sliding Mode Control

Sliding mode control (SMC) is a nonlinear technique well suited for systems with uncertainties and disturbances [26,27]. It operates by driving system states onto a predefined sliding surface and maintaining them there. In DC MG, SMC is widely used for voltage regulation [28,29]. However, it can suffer from chattering, a phenomenon that, if not properly addressed, may cause excessive wear of power components.

Various sliding surfaces have been proposed to reduce chattering. The authors in [30] introduce an approach incorporating a hysteresis band in the switching function. To improve the voltage stability of a boost converter, ref. [31] proposes a fully decentralized second-order SM (SOSM) control capable of handling unknown loads without a state observer. A distinguishing feature of this method is the inclusion of an auxiliary integral control using an appropriate sliding function.

The investigation in [32] presents a particle swarm optimization (PSO)-tuned SMC. The evolutionary algorithm optimizes controller parameters based on stability conditions. To further reduce chattering and provide an adjustable sliding coefficient, ref. [33] introduces an adaptive SMC technique. In [34], a global fast terminal sliding mode controller with hysteresis modulation (HM-GFTSMC) is proposed to enhance voltage stability in a buck converter. The study in [35] proposes a robust higher-order PID sliding mode controller (PID-HOSMC) using a double power reaching law. The PID-HOSMC stabilizes the bus voltage via battery control signals. To achieve proportional load power sharing and DC bus voltage regulation, ref. [36] proposes a decentralized control approach combining SMC based on a higher-order finite-time observer (HOFTO) with droop control.

Moreover, SMC has been combined with a radial basis function neural network (RBFNN) to enhance performance. RBFNNs are three-layer feedforward neural networks with a single hidden layer that can approximate continuous functions with arbitrary precision [37,38]. In [39], an RBFNN adaptively tunes the switching gain of a sliding mode observer, ensuring asymptotic error convergence in the state of charge (SOC) estimation of a lithium-polymer battery. Similarly, ref. [40] employs an adaptive SMC with RBFNN estimation and backstepping control to improve the performance of a proton exchange membrane fuel cell (PEMFC).

Although SMC has been successfully applied for voltage regulation, its implementation faces several challenges. Increasing system complexity can make designing effective SMC difficult. Moreover, its performance strongly depends on model accuracy, sliding surface selection, and control law formulation. Despite efforts to reduce chattering, it remains a persistent challenge in many applications.

3.2. Adaptive Droop Control

Unlike conventional droop control with fixed coefficients, adaptive droop control adapts to variations in power, enabling more precise control [41]. Several droop-based methods have been developed to enhance voltage stability and power sharing accuracy. Reference [42] proposes a hierarchical two-stage controller. The first stage involves an adaptive droop control with voltage feedback compensation, while the second provides supervisory control. The study in [43] presents an adaptive droop and voltage shifting technique for balanced current sharing. Additionally, ref. [44] presents a unified control strategy for supercapacitor systems. The strategy rejects disturbances and regulates terminal voltage, demonstrating satisfactory plug-and-play capability using local measurements.

Another study in [45] presents a nonlinear droop control algorithm for a hybrid islanded MG. The method achieves good tracking performance through the integration of SMC. To further enhance voltage stability, ref. [46] introduces a resiliency-driven droop index control strategy, in which the droop coefficient is tuned using PSO.

Overall, adaptive droop control presents a promising solution for improving both voltage stability and power-sharing accuracy, although its performance may degrade in complex and fast-dynamic systems.

3.3. Model Predictive Control

Model Predictive Control (MPC) uses a system model to predict future behavior and optimize control actions over a finite time horizon [47]. At each sampling instant, an optimization problem is evaluated using a defined cost function while explicitly considering constraints [48].

MPC approaches are generally categorized as continuous control set (CCS-MPC) and finite control set (FCS-MPC) [49]. In CCS-MPC, converter switching is governed by PWM at fixed frequency, whereas FCS-MPC operates with a variable switching frequency [50]. It is worth noting that FCS-MPC offers simple implementation and high dynamic response [51]. However, its online computational burden increases with prediction horizon length. A comprehensive analysis of MPC is discussed in [52].

Recent studies have introduced different schemes of MPC-based control strategies. In [53], an MPC-based scheme for hybrid energy storage systems (HESS) is developed to enhance MG resilience under diverse disturbances. The authors adopt FCS-MPC to control a superconducting magnetic energy storage unit and a backup battery. Reference [54] proposes a fast-distributed MPC method that improves steady-state performance using the alternating-direction method of multipliers. To strengthen dynamic stability, ref. [55] investigates a hybrid MPC capable of operating in both continuous and discontinuous current modes. The simulation results showed improvements in voltage stability and faster response compared to CCS. Moreover, ref. [56] introduces an FCS-MPC algorithm integrated with a Kalman observer. In this framework, the FCS-MPC tracks the reference current, while the Kalman observer provides feedforward compensation.

Despite its advantages, MPC performance remains dependent on model accuracy, and its computational requirements increase with prediction horizon, necessitating a powerful processor.

3.4. Passivity-Based Control

Passivity-based control (PBC) ensures system stability by modeling the system as an energy-conserving (passive) structure [57]. It relies on the concept of passivity to shape the energy of the system and stabilize it [58]. In DC MG, PBC is employed to regulate bus voltage, manage power flows, and maintain stability under varying operating conditions. This approach is particularly effective for controlling DC/DC converters with ESDs, where energy management is critical. The main advantage of PBC lies in its robustness and inherent stability; however, its performance can be limited when dealing with unmodeled disturbances.

Several studies have proposed PBC-based control strategies. For example, ref. [10] presents a passivity-based approach using a Krasovskii-type storage function. In [59], a passivity-based design is introduced for a non-affine, nonlinear system. The study proposes a low-gain controller using a dynamical model described by three inputs, three outputs non-affine system with six states. To improve voltage stability in an isolated DC MG with hydrogen energy storage, ref. [60] proposes an interconnection and damping assignment passivity-based controller (IDA-PBC). A sliding mode reference conditioning (SMRC) loop is complemented to prevent overvoltage. Additionally, ref. [61] proposes an adaptive IDA-PBC for a buck-boost converter supplying unknown CPLs. The authors incorporated an online parameter estimator to track load power, which is challenging to measure in practical applications.

3.5. Active Disturbance Control

Active Disturbance Rejection Control (ADRC) is a model-assisted technique that estimates and compensates for internal and external disturbances in real time [62]. It typically employs an extended state observer to reject disturbances. This makes it effective in situations where the system model is not completely known [63]. Its primary advantage is its ability to handle uncertainties and disturbances. However, the performance depends on the accuracy of the model and the observer.

In recent years, ADRC has been widely applied across various fields. Reference [64] implements ADRC for a flywheel energy storage system. The results showed superior disturbance rejection and dynamic performance compared with PI. Reference [65] proposes a time-scale voltage control technique based on ADRC. This approach offers improved robustness against system uncertainties and external disturbances. By developing a reduced-order model, the authors simplified the controller design and achieved better performance.

The study in [66] evaluates the performance of various ADRC techniques (linear, nonlinear, higher order, and generalized predictive integral). The study also proposes a modified model-assisted ADRC that uses pole information. Results show that incorporating system dynamics (pole information) as a known disturbance enhances stability compared with other model-assisted techniques.

In summary, ADRC offers distinct advantages in handling model uncertainties and suppressing disturbances and can be combined with other control methods to enhance stability. Its design, however, assumes knowledge of the system’s state-space model and order, and its tuning complexity can make implementation challenging, particularly for higher-order systems.

3.6. H-Infinity Control

H-infinity (H_∞) control is used to design controllers that handle external disturbances and model uncertainties by minimizing the worst-case system gain, ensuring stability under adverse conditions. The control problem is formulated as a mathematical optimization task, and the controller is synthesized through a repetitive approach [67].

Reference [68] presents an H_∞ control technique for DC MG stabilization. A robust H_∞-based approach is proposed in [69] to improve voltage regulation under uncertainties. The authors uses Lebesque-measurable matrix to determine the upper and lower bounds of the uncertainties. In [70], a decentralized H_∞ loop-shaping design control is proposed for a small, isolated DC MG. The parametric uncertainties are modeled using an upper linear fractional transformation. This technique shows good dynamic performance; however, it has not been benchmarked against other robust controllers. A summary of model-based voltage control strategies is provided in

Table A1. Summarized review of model-based voltage control techniques.

Control Method	Reference	Year	Proposed Method	Main Contribution	Limitation
SMC	[33]	2023	Adaptive SMC	Improving voltage stability in a buck converter feeding CPLs	Complexity in design
	[34]	2023	HM-GFTSMC	Improving voltage stability in a buck converter	Proving stability can be challenging in a complex MG scenario.
	[35]	2024	HOSMC-PID	Improving large signal stability in a DC MG	Proving stability can be challenging in a complex MG scenario.
	[40]	2022	RBFNN estimation-based Adaptive SMC	Improving voltage stability in a PEMFC.	Increased complexity.
Adaptive Droop Control	[42]	2019	Hierarchical adaptive droop and supervisory control	Improving voltage stability and load power sharing in a DC MG with multi-energy storage devices.	The effectiveness of the proposed technique has not been validated against existing approaches.
	[43]	2020	Adaptive distributed droop	Improving DC bus voltage stability	Stability challenges in a large-scale system
	[44]	2023	Adaptive droop + consensus control	DC MG power smoothing and voltage control	Difficulty in tuning parameters.
	[46]	2022	Droop index control	Improving voltage stability.	Performance may be sensitive to droop index.
MPC	[53]	2021	FCS–MPC	Voltage control and power allocation optimization for DC MG with HESS	Prediction at each control cycle can be computationally intensive.
	[54]	2020	Fast distributed MPC	Improving voltage stability	High computational cost.
	[55]	2021	Hybrid MPC	Improving voltage stability of a boost converter interfaced with CPLs	Prediction at each control cycle can be computationally intensive.
	[56]	2022	MPC combined with Kalman Observer	Enhancing voltage stability of an interleaved boost converter	Increased sensitivity to model accuracy.
PBC	[59]	2019	PBC	DC MG voltage regulation	Performance under varying load conditions has not been investigated.
	[10]	2019	Decentralized PBC	Improving voltage stability	Performance depends on model accuracy
	[60]	2024	IDA-PBC + SMRC	Improving voltage stability	Parameter uncertainties have not been considered.
	[61]	2021	Adaptive PBC	Voltage regulation in a buck-boost converter	Performance depends on the accuracy of the system model.
ADRC	[64]	2015	ADRC	Improving performance of a flywheel energy storage system.	Not specified
	[65]	2017	Time-scale droop control based on ADRC	Time-scale voltage droop control robust to uncertainties and external disturbances.	Performance depends on model accuracy.
	[66]	2019	Modified ADRC	Comparison of ADRC techniques for suppressing disturbances in a boost converter.	Evaluation is based on average model.
H∞	[68]	2019	H∞	Enhancing voltage stability.	Choosing appropriate weighting functions is challenging.
	[70]	2023	Loop-shaping H∞	Robust voltage control of DC MG.	Performance not validated against other methods.

in Appendix A.

4. Model-Free Techniques

While model-based control strategies can achieve high performance when accurate system dynamics are available, their effectiveness is often limited by modeling complexity, parameter uncertainties, and unmodeled nonlinearities. In many practical DC MG applications, deriving an exact model is frequently difficult or impractical. These challenges have motivated the development of model-free techniques, which regulate system behavior without relying on explicit models. This section reviews representative model-free methods, with a focus on their operating principles, practical applications, and inherent challenges.

4.1. Fuzzy Logic Control

Fuzzy Logic Control is a rule-based mathematical concept that extends Boolean logic to a multi-valued framework [71]. It is widely applied in DC MG due to its simple design and strong robustness [72]. An FLC typically comprises three main blocks: fuzzification, rule base lookup table, and defuzzification [73]. The two most commonly used inference engines are Mamdani and Takagi-Sugeno. The unique difference between these models lies in the consequent parts of their rule base, aggregation, and defuzzification procedures [74].

Several studies have applied FLC for voltage regulation. Reference [75] proposes an FLC that determines the reference current of an ultracapacitor converter. An FLC is proposed in the outer voltage control loop, and a K-type regulator is used in the inner current control loop. To improve voltage stability in a DC/DC converter feeding CPLs, ref. [76] presents an intelligence-based, Type II FLC. The proposed method provides a powerful tool for handling systems with uncertainties. To stabilize the voltage in a full-electric ferry boat, ref. [77] proposes an intelligent single-input interval Type-II fuzzy PI controller (iSIT2-FPI). The proposed technique combines SMC with a model-free iSIT2-FPI to compensate for destructive impedance instability.

The study in [78] proposes a voltage stabilization technique using a Fuzzy-PI dual-mode controller. The PI ensures good steady-state response, and the FLC improves transient response. In [79], the stability of an isolated MG is enhanced by tuning fuzzy membership functions using PSO and Cuckoo search algorithms. Simulation results have shown that energy management using PSO outperformed the Cuckoo search algorithm. Another study in [80] applies an adaptive neuro-fuzzy inference system (ANFIS) to mitigate input voltage and load current perturbations. It is worth noting that ANFIS combines the learning ability of artificial neural networks and the structured intuition of FLC. The conclusion drawn from this survey is that its performance is sensitive to membership function selection, input/output tuning, and fuzzy rule base design, which often lacks rigorous mathematical rules.

4.2. Data-Driven Control

Model-free data-driven control (DDC) relies entirely on input/output (I/O) data for training and tuning, eliminating the need for explicit system models. This section surveys recent studies on the application of DDC for voltage regulation in DC MG.

4.2.1. Artificial Neural Network

An artificial neural network (ANN) uses data to learn complex relationships between system parameters and voltage fluctuations, enabling adaptation to changing conditions and improved stability [81]. Neural network-based control is widely adopted due to its ability to learn control strategies without an explicit system model [82]. However, its performance depends on training data and proper selection of network structures and activation functions.

Neural network architectures use distinctive principles to determine rules for diverse applications. Common types include feedforward, convolutional, radial basis function, recurrent, generative adversarial, and hybrid neural networks [83]. Backpropagation is popularly used training algorithm that optimize ANN parameters by minimizing the error between the desired and actual outputs [84].

The research community has proposed numerous ANN-based control strategies. For example, ref. [85] proposes a neural network predictive controller (NNPC) for voltage control in a buck converter. The proposed method combines the advantages of MPC with the high estimation power of ANN. A similar approach was employed in [86] to enhance boost converter stability, where MPC is applied for data generation, and a feedforward network is subsequently trained to generate the converter’s duty cycle. Moreover, ref. [87] integrates CS algorithm with ANN to regulate voltage in a system with a composite energy storage system (CESS).

To regulate voltage in a low-voltage DC MG, ref. [88] proposes a hybrid Bat search and ANN (HBSANN) approach. Similarly, ref. [89] introduces a supervised deep-learning controller that eliminates the need for sensor data. Another study in [90] presents an ANN-based control approach using approximate dynamic programming. The ANN is trained offline to minimize a cost function over a long-time horizon.

Based on the analysis of ANN-based voltage control, several conclusions can be drawn. The application of ANN improves stability in DC MG even with inaccurate model parameters. However, performance depends on the quantity and quality of the training data. Incomplete or inaccurate training data can lead to biased or unpredictable behavior. Furthermore, as system complexity increases, ANN become more susceptible to overfitting, reducing their ability to adapt to real-world disturbances and leading to inaccurate predictions and suboptimal performance.

4.2.2. Local Model Networks

Local model networks (LMNs) are a neural network architecture that combines the advantages of both local and global modeling approaches. They are particularly effective for handling complex, high-dimensional data, which can be challenging for conventional neural networks [91]. The LMN concept involves creating local models (LM) that collectively cover the entire operating range of the controlled process. These LM are then integrated using validity or weighting functions to form a non-linear global model of the plant [92]. Each LM within LMN is responsible for capturing the system dynamics within a specific operating regime [93].

Based on this concept, the study in [94] presents two methods for designing LM structures: the linearization-based approach and the LM-based approach. In the linearization-based method, the LM network is linearized around an initial operating point, and a linear controller is designed accordingly. In contrast, the LM-based approach considers a separate controller for each LM, with the overall control output determined by interpolating the responses of the local controllers according to the current operating point.

To improve the performance of conventional NN, ref. [91] proposes a data-driven strategy based on a LMN for the identification and voltage control of DC/DC converters. The LMN consists of local linear models (LLM), each responsible for modeling the dynamics of the converter within a specific operating regime, as determined by its validity function. The structure of the LMN is determined directly from the input-output measurements using a hierarchical binary tree learning algorithm. The identified LMN, employed in the local linear controller (LLC), is designed based on inverse error dynamics control. The results show that the identified model captures the actual dynamic behavior of the converters.

4.2.3. Model-Free Adaptive Control

Model-free adaptive control (MFAC) is a data-driven control approach for a class of unknown nonaffine, nonlinear, discrete-time systems [95]. This approach only relies on system I/O data, eliminating the need for explicit physical models and traditional stability theory. The controller establishes a dynamic linearization data model of the controlled system at each operating point using pseudo partial derivative, pseudo gradient, or pseudo-Jacobian matrix [96]. It is one of the most promising DDC for its computational efficiency and ease of operation [97]. The application of MFAC has been explored in various areas, including power distribution systems, automobiles, robotics, and MG. Notably, MFAC demonstrates strong potential for maintaining voltage stability in a system where the exact model is unavailable.

Recent studies show that MFAC provides strong control performance with high robustness and effective disturbance rejection. For example, ref. [98] presents an MFAC scheme for active stabilization in DC MG. The control approach based on compact form dynamic linearization shows a satisfactory result. To realize a robust control, ref. [99] presents an intelligent model-free SMC for regulating charging/discharging of ESD. Similarly, ref. [96] proposes a control approach for an ultracapacitor-based three-phase interleaved bidirectional DC/DC converter. The results demonstrate fast transient response and enhanced steady-state performance.

In [100], a model-free composite disturbance rejection control (MFC) is developed to regulate the voltage of a dynamic wireless charging system. A switching-gain-based MFC law is introduced to generate the desired duty cycle without requiring prior model knowledge. An online parameter identification law is then developed to recover the unknown control input gain. To improve the voltage stability, ref. [101] proposes a data-driven model-free disturbance rejection control for a DC/DC converter. In this study, the authors considered a data-driven adaptive extended observer (DAESO) for voltage regulation in the presence of disturbances. DAESO is designed by incorporating a data memory stack and a parameter learning law to estimate the unknown control input gain and the lumped disturbance.

The literature shows that the stability, error convergence, and internal stability of systems controlled by MFAC approaches are theoretically guaranteed through contraction mapping analysis. Design assumptions, such as bounded system states, are typically considered; however, these assumptions may introduce bias into the learning process. Moreover, the performance can be influenced by the quality of the data used for learning.

4.2.4. Deep Reinforcement Learning

DRL is an emerging machine learning technique where an agent learns to perform actions through trial-and-error interaction with a controlled system [24]. In the context of DRL, the controlled system is referred to as an environment. It finds applications in DC MG and many other fields for control and optimization. The study in [102] identified three major factors influencing the growing adoption of DRL: data, computing resources, and learning algorithms. A schematic representation of a typical DRL agent is shown in Figure 3.

Unlike other classes of machine learning, DRL involves sequential decision-making [103], where an agent learns to navigate a potentially stochastic environment, balancing exploration and exploitation. This balancing is crucial, particularly in a system where the agent must decide between exploiting known rewards and exploring new possibilities to maximize long-term reward. A comprehensive review of the applications of DRL is discussed in [92].

DRL models are classified into two main types: model-based and model-free DRL. In this context, the model refers to the agent’s perception of its environment. Model-based DRL explicitly requires a model of system dynamics to find a near-optimal policy [104]. Here, the agent has a clear understanding of the environment in which it operates. The environment can be either partially observable or fully observable. In contrast, a model-free agent has no prior knowledge of the environment it interacts with. The agent learning process mainly relies upon experience.

The Markov decision process (MDP) provides a mathematical framework that describes environment-agent interaction at each sequence of time steps [105]. MDP is used for modeling decisions in uncertain conditions using a trajectory shown in Figure 4 [106].

After formulating voltage control problem as an MDP, it can be trained and solved using various model-free DRL algorithms, either on-policy or off-policy. On-policy agents attempt to enhance the policy they are using to make decisions. In contrast, off-policy agents improve a policy that can be different from the one they use to make decisions. The basic classification of model-free DRL algorithms is shown in

Table 1. Classification of model-free DRL agents.

Learning Policy	Key Features	Agent	Type	Action
On-policy	Learning is based on the current policy. Exploration is limited to the current policy. Suitable when the environment is relatively stable and the agent can explore safely.	State-action-reward-state-action (SARSA)	value-based	discrete
		Policy gradient (PG)	policy-based	discrete or continuous
		Actor critic (AC)	actor-critic	discrete or continuous
		Trust region policy optimization (TRPO)	actor-critic	discrete or continuous
		Proximal Policy optimization (PPO)	actor-critic	discrete or continuous
Off-policy	Learn from different policies. Explore more broadly through behavior policy. Suitable for complex environment	Q-learning	value-based	discrete
		Deep Q-network (DQN)	value-based	discrete
		Double DQN (DDQN)	value-based	discrete
		Deep deterministic policy gradient (DDPG)	actor-critic	continuous
		Twin-delay deep deterministic policy gradient (TD3)	actor-critic	continuous
		Soft actor-critic (SAC)	actor-critic	continuous
		Asynchronous advantage actor-critic (A3C)	actor-critic	discrete or continuous

.

DRL agent optimization is either value, policy, or a combination of both policy and value (using actor and critic). In a value-based approach, a value function is initially selected for each state, and a new value function is then estimated using learning data. This process is iterated until the optimal value function is achieved. While the policy-based method directly optimizes the policy function, mapping states to actions without explicitly estimating value functions [107].

Recent studies have investigated the effectiveness of DRL for voltage control in DC MG. Reference [108] proposes an approach using PID and DRL algorithm. The PID serves as the base controller, while a DDPG agent is employed as an adaptive compensator to improve the system’s dynamic response. In [109], the application of DQN for voltage control of buck converter feeding CPLs is investigated. To overcome the limited exploration capability of DQN, ref. [110] adopts an improved version based on DDPG. The study in [111] presents a nonlinear-based scheme for voltage stabilization of a boost converter. The authors use a regression-based optimization algorithm to improve system efficiency. To overcome the overestimation of value functions that often occurs in DDPG, ref. [112] proposes a TD3 algorithm that uses two separate critic networks and delayed policy updates. Similarly, ref. [113] presents a TD3-based algorithm for a buck converter feeding CPL. The distinctive feature of this study is its framework, which is designed for direct control of the power switches. Moreover, the study in [114] modified and trained a TD3 agent to improve the dynamic stability in a boost converter.

In [115], a DDPG-PI based on a sliding mode observer is proposed to address the effects of unknown and non-ideal CPLs. The proposed scheme eliminates dependence on an accurate model, as the agent learns to accommodate unmodeled dynamics. In [116], an adaptive disturbance rejection controller employing DDPG and Internet of Things (IoT) is proposed for smart grid systems. The DDPG agent offers online tuning through neural network learning. To address the challenges of offline training, ref. [117] presents an integral DRL-based voltage control approach for an interleaved DC/DC boost converter that autonomously approximates the controller gains online.

Overall, DRL-based control algorithms demonstrate strong adaptability in a completely unknown environment; however, they require significant computational effort for agent training. Additionally, sparse rewards and excessive exploration may lead to unstable control actions oscillations. A summary of the reviewed model-free strategies is provided in

Table A2. Summarized Review of Model-Free Voltage Control Techniques.

Control Method	Reference	Year	Proposed Method	Main Contribution	Limitation
FLC	[76]	2020	SCA-HS tuned Type II Fuzzy	Enhancing voltage stability in a boost converter feeding CPLs.	Performance is often sensitive to the choice of optimization parameters and fuzzy rule base.
	[77]	2020	iSIT2-FPI + SMC	The authors proposed an SMC-based model-free FLC.	Relatively complex to implement.
	[78]	2020	Fuzzy-PI dual mode	The authors combined FLC with PI to enhance dynamic response and restrain fluctuations of the bus voltage.	Tuning scaling gains is necessary whenever the system dynamic changes.
	[80]	2019	ANFIS	Improving transient and steady-state responses of a flyback converter using FLC and neural network	Training ANFIS requires high-quality data. High computational cost
ANN	[87]	2022	CCSNN	The authors proposed an EMS to enhance power sharing among CESS, as well as maintain bus voltage stability.	It is computationally intensive to tune CC hyperparameters and train a neural network.
	[88]	2020	HBSANN	Proposed an HBSANN-based power management strategy. Improving the voltage regulation of a DC MG	It requires high quality training data
	[89]	2021	DNN	Proposed a supervised deep learning aided-sensorless controller	Risk of overfitting
	[90]	2021	ANN–approximate dynamic programming	Improving voltage stability under variable load and input voltage conditions.	Requires high-quality data
LMN	[91]	2019	LMN + LLC	Identification of a DC/DC converter’s dynamics directly from measured data. Developed a voltage controller based on the identified model.	Performance was not evaluated against robust control methods.
MFAC	[96]	2023	MFAC	Design a pseudo-gradient estimation algorithm based on I/O data. Improving voltage stability in a BDC.	Pseudo-gradient estimation methods may introduce systematic errors due to the approximation process.
	[99]	2021	Model-free iSIT2-FPI	Improving voltage regulation in a stand-alone shipboard DC MG.	Increased complexity.
DRL	[108]	2023	PID+DDPG	Enhancing the voltage stability of a buck converter	Performance is partially dependent on model accuracy.
	[109]	2022	DQN	Improving the voltage stability of a buck converter	Handles only discrete actions.
	[114]	2024	TD3	Optimizing parameters of a PI controller.	Performance was tested under light load conditions.
	[116]	2020	DDPG	Voltage stabilization of IoT-based buck converter feeding CPLs.	The paper does not discuss training or simulation results.
	[117]	2023	Integral RL	Improving voltage stability in an interleaved boost converter.	The paper does not discuss training or simulation results.

in Appendix A.

5. Hybrid Control Techniques

Hybrid control techniques combine aspects of both model-based and model-free control schemes. This section examines the recent advances in hybrid control techniques for voltage control in DC MG.

5.1. Metaheuristic Optimization Algorithms

Metaheuristic optimization algorithms (MOA) are nature-inspired algorithms designed to find optimal solutions to complex optimization problems. The main advantages of MOAs are their versatility and flexibility [118], as they can be easily modified to meet the specific requirements of a given problem. These algorithms are widely used to optimize the parameters of both classical and advanced control techniques. For example, ref. [119] proposes an intelligent binary and real-code genetic algorithm tuned PID controller for regulating fuel cell voltage. The binary genetic algorithm (GA) is a well-established optimization method for identifying near-optimal solutions. It consists of binary-encoded chromosomes that undergo roulette wheel selection, multipoint crossover, and uniform mutation to produce offspring [119]. Each chromosome in the offspring population is evaluated using a defined objective function.

To improve stability in a buck-boost converter, ref. [120] proposes a PSO-tuned PI controller. The PSO algorithm, inspired by the social behavior of bird flocks searching for food, iteratively adjusts the positions of particles (potential solutions) based on their individual best-known positions and the global best-known position of the swarm. The study in [121] introduces a quasi-oppositional Archimedes optimization algorithm (QOAOA) for tuning a fractional-order PID (FOPID) controller to regulate the output voltage of a cascaded DC/DC boost converter. To enhance voltage stability in PV-powered MGs, ref. [122] combines direct SMC with a hybrid salp swarm algorithm and particle swarm optimization (SSA-PSO). Moreover, ref. [123] proposes a control strategy for a buck converter by optimizing PID parameters using an improved sine-cosine algorithm.

Overall, MOAs offer a powerful framework for controller parameter optimization. They have demonstrated significant improvements in performance across both linear and nonlinear control applications. Nonetheless, their effectiveness may be constrained by sensitivity to local minima and the need for careful parameter tuning.

5.2. Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINN) have recently gained attention as a promising approach for accelerating computations in nonlinear dynamical systems. PINN integrate physics-informed laws (model-based) into neural network methodologies (model-free) to learn complex physical behaviors [124]. This approach ensures that the learned solutions adhere to the underlying physics of the systems. Unlike the conventional neural network paradigm, PINN uses scientific knowledge or physics laws to guide the optimization, design, and implementation of deep neural networks [125]. The four core paradigms of PINN, as outlined in [126], include physics-informed loss functions, physics-informed initialization, physics-informed design of architecture, and hybrid physics-deep learning models.

In recent years, the adoption of PINN in power systems has expanded to applications such as state/parameter estimation, optimal power flow, and dynamics analysis. Reference [127] presents a PINN approach for estimating the state of health (SOH) of lithium-ion batteries. The authors model the attributes influencing battery degradation through empirical degradation and state-space equations and use neural networks to capture system dynamics. The proposed network comprises two components: a solution NN F(.) that builds feature-to-SOH mapping and a NN G(.) that models battery degradation dynamics.

To enhance adaptability and minimize reliance on extensive quality datasets and high computational power, ref. [128] proposes a PINN-based control framework for improving the stability of a buck converter. The framework employs a parallel architecture integrating data-driven, physics, control, and loss function modules. The proposed method performed well in identifying and rejecting unknown load disturbances and internal circuit parameters. Notably, PINN can achieve high accuracy and strong generalization compared to traditional neural networks due to their integration of physical laws. However, they often require high-quality supervised data for training and validation, which can limit their applications.

Table A3. Summarized Review of Hybrid Control Strategies.

Control Method	Reference	Year	Proposed Method	Main Contribution	Limitation
MOAs	[119]	2020	GA-tuned PID	Improving voltage stability and performance of a fuel cell	Poor parameter tuning can influence effectiveness of the algorithm.
	[120]	2021	PSO-tuned PI	Improving voltage stability in a buck-boost converter.	Performance in the presence of disturbances has not been evaluated.
	[121]	2023	QOAOA	Improving efficiency of a cascaded boost converter.	Performance with varying load conditions is not discussed.
	[122]	2023	SSA-PSO	Improving voltage stability in a PV-powered MG.	The proposed method is not compared with other established methods.
PINNs	[127]	2024	PINN	Estimating SOH of a lithium-ion battery.	Difficulty in handling high-dimensional nonlinear models.
	[128]	2024	PINN	Enhancing stability in a buck converter.	Difficulty in handling high-dimensional converter dynamics.

in Appendix A summarises recent research on hybrid voltage control techniques.

The voltage control techniques reviewed in Section 3, Section 4 and Section 5 exhibit distinct characteristics in terms of modeling requirements and performance. A summarized comparison is provided in

Table 2. Comparative summary of voltage control techniques.

Category	Strengths	Limitations
Model-based	High accuracy and fast response when the system model is known. Stability can be guaranteed analytically. Facilitates optimal and predictive control.	Requires system model and parameters. Limited adaptability to uncertainties. Requires re-modeling in the event of system changes.
Model-free	Does not require an explicit system model. High adaptability and flexibility. Robust to parameter variations and uncertainties.	Lacks analytical stability guarantees. High computational cost during training. Requires high-quality data.

. It is important to note that hybrid methods are not included in this comparative summary to maintain a clear distinction between model-based and model-free control techniques. These methods typically combine model knowledge with optimization-based (e.g., MOA) or data-driven (e.g., PINN) adaptation and often function as tuning or augmentation layers rather than standalone controllers. As a result, their performance largely depends on the main controller (model-based or model-free) and may inherit the strengths and limitations of the underlying control strategy.

6. A Case Study of DC/DC Converter with LC Filter

The preceding sections critically reviewed voltage control strategies, highlighting their respective strengths and limitations. While this survey provides a comprehensive view of state-of-the-art, it is equally important to evaluate their effectiveness in a representative and challenging scenario. To this end, this study presents a case study of a DC/DC buck converter cascaded with an input LC. This configuration is particularly relevant, as it captures one of the critical instability issues in DC MG: the adverse interaction between the LC filters and converters, which substantially reduces system damping and compromises voltage stability.

In practical DC MG, as shown in Figure 1, an input LC filter is commonly included at the DC/DC converter stage to suppress harmonics injected into the DC bus and to mitigate instantaneous impulse voltages and electromagnetic interference [129]. However, its inclusion can lower the damping ratio of the system and exacerbate instability risks through resonance with the converter dynamics [22]. When this effect is combined with the negative incremental impedance of CPL, the resulting system stability is severely degraded.

To address such challenges, researchers have proposed a wide range of strategies, including linear feedback control [130], passivity-based control [131], and feed-forward virtual impedance combined with proportional–resonant control [129]. More recently, however, attention has shifted toward model-free approaches, whose flexibility and adaptability to disturbances and parameter variations make them attractive for DC MGs without the need for continuous model updates [95].

The present case study is designed with two objectives: (i) to benchmark conventional and advanced controllers within a unified framework, and (ii) to demonstrate the potential of DRL strategies enhanced through BO for achieving robust voltage regulation under uncertain and nonlinear operating conditions.

It should be noted that hybrid approaches are not included in this comparative study. This decision is motivated by two considerations: first, hybrid methods such as MOA and PINN generally serve as tuning or augmentation layers for existing controllers rather than as standalone real-time control schemes; second, incorporating them would require extensive data or embedding additional optimization modules in the control loop, which is beyond the scope of this benchmark investigation. Instead, this work focuses on a fair comparison between widely adopted conventional controllers and the proposed DRL-based strategies, while recognizing hybrid approaches as a promising direction for future research.

6.1. System Model

To formalize this benchmark scenario, Figure 5 illustrates the studied system. It consists of a buck converter feeding a CPL through an input LC filter connected to a common DC bus. The DC bus voltage and the converter’s output voltage are defined as E and $v_c$, respectively. Similarly, $L_f,L,C_f,$ and C represent the inductances and capacitances of the LC filter and the converter. Additionally, $r_f$ and r are the series resistances of $L_f$ and L, respectively, while S denotes a controllable switch. These parameters are selected such that the converter operates in continuous conduction mode.

The system’s model is represented by the following differential equations:

$$L_f{\displaystyle \frac{d{i}_f\left(t\right)}{dt}}=E-i_f\left(t\right)r_f-v_f\left(t\right)$$

(1)

$$C_f{\displaystyle \frac{d{v}_f\left(t\right)}{dt}}=i_f\left(t\right)-u\left(t\right)i_L\left(t\right)$$

(2)

$$L{\displaystyle \frac{d{i}_L\left(t\right)}{dt}}=u\left(t\right)v_f\left(t\right)-i_L\left(t\right)r_L-v_c\left(t\right)$$

(3)

$$C{\displaystyle \frac{d{v}_c\left(t\right)}{dt}}=i_L\left(t\right)-i_{\mathrm{CPL}}\left(t\right)$$

(4)

The control signal $u\left(t\right)$ takes the value of ‘1’ for the switch ‘ON’ position and ‘0’ for the ‘OFF’ position.

The complete model is a four-state model with a state vector $x\left(t\right)={[i_f,v_f,i_L,v_c]}^T$. The state-space average model is given by

$$\begin{array}{cc}\hfill \dot{x}\left(t\right)& =Ax\left(t\right)+Bu\left(t\right)\hfill \\ \hfill y\left(t\right)& =Cx\left(t\right)\hfill \end{array}$$

(5)

where

$$A=\left[\begin{array}{cccc}-{\displaystyle \frac{r_f}{L_f}}& -{\displaystyle \frac{1}{L_f}}& 0& 0\\ {\displaystyle \frac{1}{C_f}}& 0& -{\displaystyle \frac{S}{C_f}}& 0\\ 0& {\displaystyle \frac{S}{L}}& -{\displaystyle \frac{r}{L}}& -{\displaystyle \frac{1}{L}}\\ 0& 0& {\displaystyle \frac{1}{C}}& -{\displaystyle \frac{1}{RC}}\end{array}\right],B=\left[\begin{array}{c}1\\ 0\\ 0\\ 0\end{array}\right]$$

$$C=\left[\begin{array}{cccc}0& 0& 0& 1\end{array}\right]$$

6.2. DRL Algorithms

Since the objective is to ensure that the output voltage remains fully regulated even under supply and load perturbations, this study proposes a model-free control strategy based on DRL and BO. To provide a balanced evaluation, the article adopts two representative DRL agents: PPO, an on-policy algorithm known for its stable policy updates and robustness across a range of control problems, and TD3, an off-policy algorithm that addresses overestimation bias often found in DDPG.

6.2.1. PPO

PPO is an on-policy DRL algorithm that uses a clipped surrogate function to prevent large policy updates and ensure stable learning [132]. It directly optimizes the policy through interactions with the environment, aiming to maximize the expected cumulative long-term reward. By constraining policy updates from deviating significantly from the current policy, PPO establishes a robust framework for optimizing control policies in a complex dynamic environment [133]. It can be applied in an environment with either continuous or discrete control action [134]. To estimate the policy and value function, PPO is particularly implemented using actor-critic function approximators [135]. The actor estimates the conditional probability of taking an action based on the current state, while the critic evaluates the chosen action and returns the corresponding expectation of the discounted long-term reward.

Policy optimization algorithms maximize the expected return $G\left(\theta \right)$ of a policy ${\pi }_{\theta }(a\mid s)$, parametrized by $\theta$, over MDP trajectories:

$$G\left(\theta \right)={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right]$$

(6)

Applying policy gradient theorem, the gradient of (6) is given by:

$${\nabla }_{\theta }G\left(\theta \right)={\mathbb{E}}_{s_t,a_t\sim {\pi }_{\theta }}\left[{\nabla }_{\theta }log{\pi }_{\theta }(a_t\mid s_t)A_t\right]$$

(7)

where $A_t$ represents an advantage function, defined as the discounted sum of temporal difference error.

In general, PPO optimizes a clipped surrogate objective function using stochastic gradient descent [135]:

$$L^{\mathrm{CLIP}}\left(\theta \right)={\widehat{\mathbb{E}}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right]$$

(8)

where $r_t\left(\theta \right)$ is a probability ratio between the new policy ${\pi }_{\theta }$ and old policy ${\pi }_{{\theta }_{\mathrm{old}}}$, $\mathrm{clip}(·)$ restricts $r_t\left(\theta \right)$ to the range $[1-ϵ,1+ϵ]$, and $ϵ$ is a hyperparameter controlling the allowed policy change.

To encourage exploration and prevent premature convergence of the learned policy, PPO often adds an entropy loss term:

$$L^{\mathrm{ENTROPY}}\left(\theta \right)=\mathbb{E}\left[H\left({\pi }_{\theta }(·|s_t)\right)\right]$$

(9)

Therefore, the final loss is given in (10):

$$L_t\left(\theta \right)={\mathbb{E}}_t\left[L_t^{\mathrm{CLIP}}\left(\theta \right)-c_1{L}_t^{\mathrm{VF}}\left(\theta \right)+c_2{L}_t^{\mathrm{ENTROPY}}\left(\theta \right)\right]$$

(10)

where $L_t^{\mathrm{VF}}$ is a value function loss ${(V\left(s_t\right)-V_t^{\mathrm{target}})}^2$, $c_1$ and $c_2$ are weighting coefficients.

6.2.2. TD3

TD3 is an off-policy actor-critic DRL agent that employs deterministic policies for continuous tasks [136]. It addresses the Q-value overestimation issue commonly observed in DDPG by introducing clipped double Q-learning, delayed policy updates, and target policy smoothing [137]. To estimate the policy and value functions, TD3 uses an actor network with a parametrized policy. The actor takes observation as input and outputs a control action that maximizes long-term reward. TD3 uses two identical critic networks to estimate the value of the state-action pairs.

In general, TD3 trains two critic networks $Q_{{\varphi }_1}(s,a)$, $Q_{{\varphi }_2}(s,a)$, and the minimum of the two values is used by the target. The target for the Q-value is obtained using (11) [137]:

$$y=r+\gamma \underset{k=1,2}{min}{Q}_{{\varphi }_k^{\prime }}(s^{\prime },{\tilde{a}}^{\prime })$$

(11)

where r is the immediate reward, $\gamma$ is the discount factor, $Q_{{\varphi }_k^{\prime }}$ is the target Q-function, and ${\tilde{a}}^{\prime }={\pi }_{{\theta }^{\prime }}\left(s^{\prime }\right)+ϵ$ represents the target action with added noise. It is important to note that, if $s^{\prime }$ is a terminal state, the target Q-value is set to r only.

$$ϵ\sim \mathrm{clip}\left(\mathcal{N}\right(0,\sigma ),-c,c)$$

(12)

where $\sigma$ is the noise standard deviation, and c is the clipping range.

After every training step, the parameters of each critic are updated by minimizing the loss $L_k$ across all sampled experiences:

$$L_k={\displaystyle \frac{1}{2M}}\sum _{i=1}^M{\left(y_i-Q_{{\varphi }_k}(s_i,a_i)\right)}^2$$

(13)

Similarly, the actor parameters are updated using the following sampled policy gradient:

$${\nabla }_{\theta }J={\displaystyle \frac{1}{M}}\sum _{i=1}^M{G}_{{\pi }_i}{G}_{a_i}$$

(14)

where $G_{{\pi }_i}$ is the gradient of the actor output with respect to the actor parameters, $G_{a_i}$ is the gradient of the minimum critic output with respect to the action estimated by the actor network.

TD3 employs a target actor (${\theta }^{\prime }$) and two target critics (${\varphi }_k^{\prime }$). The target networks are updated periodically to avoid too fast convergence using a soft update rule as follows [112]:

$$\begin{array}{cc}\hfill {\theta }^{\prime }& =\tau \theta +(1-\tau ){\theta }^{\prime }\hfill \\ \hfill {\varphi }_k^{\prime }& =\tau {\varphi }_k+(1-\tau ){\varphi }_k^{\prime }\hfill \end{array}$$

(15)

where $\tau \ll 1$ is the smoothing factor.

6.3. DRL Controller Design

The complete block diagram illustrating the interaction between the proposed approach and the system is shown in Figure 6. The design of the state space, action space, reward function, and hyperparameters optimization is described in the following steps:

6.3.1. State Space

The state space contains sufficient information from the environment for the agent to make control decisions. For the objective of voltage regulation, it includes the voltage tracking error, the integral of the error, the delayed output voltage, and the reference voltage. The random variation of the reference voltage during training necessitates its inclusion to facilitate the learning of dynamic control policies. Thus, the state space is defined as

$$s_t=\left[e_v\left(t\right),\int e_v\left(t\right)dt,v_o(t-1),V_{\mathrm{ref}}\right]$$

(16)

where $e_v\left(t\right)=V_{\mathrm{ref}}-v_o$.

6.3.2. Action Space

The action space represents the set of all valid control actions that the agent can execute within the environment. In this study, the converter’s duty cycle serves as the control signal generated by the agent to regulate the voltage. Therefore, the action space is defined as

$$a_t=d\in [0.0,0.99]$$

(17)

6.3.3. Reward Function

The reward function is structured to encourage the agent for favorable actions and penalize it for poor decisions. The reward function used in this study is based on tracking error and is defined in (18).

$$R_t=r_1+r_2+r_3$$

(18)

$r_1=-{\left(e_v\right)}^2,$$r_2=\left\{\begin{array}{cc}1\hfill & \mathrm{if}|e_v|\le 0.01\hfill \\ 0.1\hfill & \mathrm{if}0.1<|e_v|<0.1\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$

$$r_3=\left\{\begin{array}{cc}-100\hfill & \mathrm{if}{v}_o\ge 1.3V_{\mathrm{ref}}\mathrm{or}{v}_o\le 0.7V_{\mathrm{ref}}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

where $r_1$ denotes a parametric reward proportional to the negative squared voltage error, $r_2$ is a non-parametric discrete reward that incentivizes favorable actions, and $r_3$ penalizes violations of voltage limits. The scaling gains of the reward function are tuned using a trial-and-error approach.

6.3.4. Hyperparameter Optimization

DRL algorithms are making significant strides in the field of control engineering; however, their performance is challenged by hyperparameter sensitivity. These hyperparameters, such as discount factor, learning rate, minibatch size, etc., affect the learning process and the learned policy [138]. A popular tuning method involves trial-and-error or manual search. In this method, different hyperparameter values are used, and their impact on the performance is observed. It is noteworthy that manual search requires intuition and sufficient expertise to identify good hyperparameter sets. Furthermore, as the number and range of the hyperparameters increase, optimization becomes increasingly tedious, making it unlikely to find the best possible set.

To overcome the limitations of manual tuning, automatic search methods, such as grid search and random search, have been proposed. Grid search performs an exhaustive search of the entire space [139]. It trains an agent by trying all possible combinations of the hyperparameters. This method is straightforward to implement and widely adopted; however, it suffers from the curse of dimensionality and can be time-consuming, especially as the number and range of hyperparameters increase [139]. To address these drawbacks, random search selects hyperparameters randomly from a defined distributed search space [140]. While this approach reduces training time, it may not fully cover the parameter space.

To reduce computational effort and improve control performance, researchers employ BO to optimize the hyperparameters of DRL agents. BO attempts to minimize a scalar objective function, $f\left(\lambda \right)\mathrm{for}\mathrm{all}\lambda \in \mathrm{\Lambda }$ [141]. It iteratively builds Gaussian process model of the function based on the observed data and uses the model to predict performance. Hyperparameter optimization with BO relies on three key components: a Gaussian model of the objective function, a Bayesian update that modifies the model with each new evaluation of $f\left(\lambda \right)$, and an acquisition function [142].

In this study, BO is employed to optimize the hyperparameters by minimizing the cost function defined in (19).

$$f\left(\lambda \right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\sum _{t=1}^{T_i}r{\left(\lambda \right)}_{i,t}$$

(19)

where $r{\left(\lambda \right)}_{i,t}$ is the reward received with candidate hyperparameters $\lambda$, at time step t in episode i, N is the number of evaluation episodes, and $T_i$ is the maximum number of time steps in the i-th evaluation episode.

The process involves computing four evaluations consecutively after every 20 training episodes. The mean value of the evaluation rewards used in the objective function provides insight into the agent’s performance in the environment, thereby facilitating more effective hyperparameter tuning.

Following the guidelines in [143], the hyperparameters most critical to learning performance, exploration, and stability were selected for the optimization. These hyperparameters and their respective bounds are provided in

Table 3. Agents’ hyperparameter bounds.

Hyperparameter	PPO	TD3
Minibatch size (m)	[50, 400]	[50, 400]
Discount factor ($\gamma$)	[0.9, 1]	[0.9, 1]
Actor learning rate (${\alpha }^a$)	$[1\times {10}^{-6},1\times {10}^{-2}]$	$[1\times {10}^{-6},1\times {10}^{-2}]$
Critic learning rate (${\alpha }^c$)	$[1\times {10}^{-6},1\times {10}^{-2}]$	$[1\times {10}^{-6},1\times {10}^{-2}]$
Number of epochs (k)	[1, 10]	[1, 10]
Target smooth model standard deviation (${\sigma }_{\tau }$)	–	[0.1, 0.5]
Entropy loss function (w)	[0.01, 0.1]	–

and the complete algorithm is presented in pseudocode in Algorithm 1.

Algorithm 1 Bayesian optimization

Input: Objective function $f\left(\lambda \right)$, maximum number of iterations $n_{max}=30$, maximum number of training episodes $p_{max}=300$, hyperparameter sets ${\lambda }_{\mathrm{TD}3}=[m,\gamma ,{\alpha }^a,{\alpha }^c,k,{\tau }_{sd}]$, ${\lambda }_{\mathrm{PPO}}=[m,\gamma ,{\alpha }^a,{\alpha }^c,k,\omega ]$, bounds $\mathrm{\Lambda }$. Initialize: Train the agent using randomly sampled hyperparameters ${\lambda }_0\in \mathrm{\Lambda }$. Collect initial dataset $D_0=\{{\lambda }_0,f\left({\lambda }_0\right)\}$. for $n=1$ to $n_{max}$ do       Fit Gaussian model with observed data $D_{n-1}$.       Select new hyperparameter set ${\lambda }_n\in \mathrm{\Lambda }$ by optimizing acquisition function.       for $p=1$ to $p_{max}$ do             Train the agent and store mean evaluation rewards after every N episodes; $r_i$.       end for       Evaluate $f\left({\lambda }_n\right)=-{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{r}_i$.       Augment dataset $D_n\leftarrow D_{n-1}\cup \{{\lambda }_n,f\left({\lambda }_n\right)\}$. end for Return: Best solution ${\lambda }^{*}$ in observed dataset D.

6.4. Simulation Results

Simulations in MATLAB/Simulink (R2024a) are carried out to verify the performance of the proposed methods. A comparative evaluation with model-based (PI, SMC) and model-free FLC controllers is also presented.

6.4.1. Training and Optimization Results

The control agents’ hyperparameters were optimized through multiple training experiments to minimize the objective function defined in (19). During each experiment, the agent interacted with the system model to track a series of randomly generated reference voltages. Thirty (30) evaluations of BO were performed to identify the optimal hyperparameter combination within the defined search space (Table 3). Each evaluation involved 300 training episodes. It is important to note that high computational power was required only during the training and optimization phases. Once trained, the agent maps system states to corresponding control actions according to the learned policy with negligible computational demand.

The results of hyperparameter optimization are presented in Figure 7. Figure 7a,b illustrate the objective function minimization over 30 iterations for the PPO and TD3 agents, respectively. The results show that the objective function attained its global minimum at the 4th evaluation for the TD3 agent, whereas the PPO agent reached its minimum at the 9th iteration. This observation highlights the faster convergence of TD3 compared to PPO. Moreover, the trajectories demonstrate that the estimated minimum objective closely corresponds to the observed minimum, further validating the effectiveness of the BO process.

Figure 8 illustrates the trend of the episodic and average rewards for the two agents during training. The results were obtained over 1000 training episodes using two hyperparameter configurations: the optimized set obtained through BO and a manually tuned baseline. Figure 8a,b show that the agents trained with optimized hyperparameters converge faster and achieve higher rewards compared to the baseline agents.

6.4.2. Comparative Evaluation Under Dynamic Conditions

To assess the performance of the proposed methods, simulations were performed under different dynamic operating conditions. Three scenarios were considered: (i) varying reference voltage, (ii) supply disturbances, and (iii) load disturbances.

Scenario 1–Varying Reference Voltage

In this scenario, the reference voltage was varied as shown in Figure 9a. Based on the output voltage waveforms in Figure 9b, a detailed transient performance analysis was conducted and summarized in

Table 4. Transient performance metrics.

Metrics	PI	FLC	SMC	BO-PPO	BO-TD3
Rise time (s)	0.0016	0.0021	0.0090	0.0012	0.0016
Settling time (s)	0.0070	0.1398	0.0161	0.0033	0.0032
Overshoot (%)	11.27	5.41	0.03	7.42	5.95

. It can be seen that all the control methods maintained global stability, with the proposed BO-TD3 demonstrating the most balanced response, achieving a fast rise time of 0.0016 s and the lowest settling time of 0.0032 s. Although the BO-PPO achieved the fastest rise time (0.0012 s), it exhibited a relatively high overshoot compared to BO-TD3. It can also be seen that the SMC achieved the lowest overshoot, but exhibited slower response compared to the proposed methods. Furthermore, steady-state performance analysis in terms of root mean square error (RMSE), mean of absolute error (MAE), mean absolute percentage error (MAPE), and integral absolute error (IAE) is presented in

Table 5. Comparison of control methods based on error metrics.

Metrics	PI	FLC	SMC	BO-PPO	BO-TD3
RMSE	0.0826	0.3462	0.1754	0.2798	0.0780
MAE	0.0706	0.2907	0.0889	0.2707	0.0650
MAPE	0.147	0.6057	0.1853	0.5639	0.1355
IAE	0.0095	0.0392	0.0120	0.0365	0.0088

. This analysis clearly indicates that the BO-TD3 consistently achieves the lowest steady-state error.

Scenario 2–Supply Disturbance

Figure 10 represents a scenario with perturbations in supply voltage as illustrated in Figure 10a. The corresponding results, shown in Figure 10b, compare the responses of the proposed and benchmark methods. The results demonstrate that all the methods achieved global stability despite disturbances in the supply. From the analysis presented in

Table 6. Comparison of control methods based on error metrics.

Metrics	PI	FLC	SMC	BO-PPO	BO-TD3
RMSE	0.8363	0.9832	1.2592	0.8676	0.7775
MAE	0.1039	0.3408	0.1038	0.4680	0.0750
MAPE	0.2165	0.7100	0.2163	0.9749	0.1562
IAE	0.3117	1.0223	0.3115	1.4039	0.2248

, it can be concluded that the BO-TD3 achieved the lowest steady state error metrics.

Scenario 3–Load Disturbance

Figure 11 shows the system response when the reference voltage is set to 48 V and the load power varies. Based on the output voltage regulation shown in Figure 11b, a detailed performance analysis is summarized in

Table 7. Comparison of control methods based on error metrics.

Metrics	PI	FLC	SMC	BO-PPO	BO-TD3
RMSE	0.8437	0.9671	1.2679	0.7627	0.7942
MAE	0.1013	0.2980	0.1249	0.2356	0.0620
MAPE	0.2111	0.6208	0.2603	0.4909	0.1292
IAE	0.3040	0.8939	0.3748	0.7069	0.1860

. The results indicate that, among the five control methods, the BO-TD3 achieves the lowest error metrics.

7. Challenges and Future Works

Although the existing literature has focused on developing various advanced voltage control techniques, several challenges still need to be further considered to improve adaptability and robustness. On the one hand, system complexity increases with the integration of more components and loads, along with the potential interaction between DC MG, which amplifies uncertainties and external disturbances. On the other hand, the degradation of ESD needs to be considered. Additionally, validating and implementing the proposed approaches in real-world scenarios, such as using realistic test benches, remain crucial. Therefore, future research directions are summarized below:

• Traditional control approaches often struggle under system disturbances, such as the integration of new DER/ESD or system reconfiguration. These events frequently alter DC MG dynamics. Advanced control strategies like DRL can be trained to continuously adapt their control actions, maintaining stability under high perturbations.
• Data-driven control methods, while offering flexibility and reduced model dependency, often lack interpretability due to their black-box nature. Hybrid control strategies that combine data-driven and physics-based models–such as Model-based reinforcement learning (MBRL) and Physics-informed reinforcement learning (PIRL) should be further explored to balance flexibility, interpretability, and explainability while enhancing voltage stability under uncertainty.
• As MG continue to increase in scale and complexity (multiple DER, diverse loads and interconnection with other MG), the need for more adaptable and flexible control schemes becomes apparent. It is recommended to investigate and develop Multi-agent deep reinforcement learning (MADRL) frameworks. This will enable scalable and coordinated control across large-scale MG systems, allowing for robust autonomous decision-making and enhanced operational resilience.
• To ensure the practical applicability and reliability of the proposed control algorithms, future research should incorporate real-time experimental validation. This could be achieved by establishing Hardware-in-the-loop (HIL) test environments, for initial system integration and controller testing, and Power hardware-in-the-loop (PHIL) to assess performance with actual power-level components.

8. Conclusions

This study conducted a comprehensive review of voltage control techniques in DC MG. The reviewed methods were categorized based on their dependence on explicit models for controller design, resulting in a structured analysis of model-based, model-free, and hybrid strategies. Each category was critically examined, highlighting its respective strengths and limitations.

Based on the survey findings, DRL-based control frameworks using the BO-TD3 and BO-PPO algorithms were proposed for voltage regulation in a buck converter with an input LC filter. Three simulation scenarios were evaluated: reference voltage variation, input voltage disturbance, and load fluctuation. The BO-TD3 achieved the lowest error metrics, with a reduction of up to 5.6% in root mean square error (RMSE), and 7.8% in mean absolute percentage error (MAPE) compared to benchmark methods. These findings highlight the potential of DRL-based methods to enhance the voltage stability of DC MGs, with the main limitation being their high computational cost during training.

Future research should focus on extending the control methods to complex and large-scale DC MG and ensure their practical applicability and reliability through real-time experimental validation.