The Voltage Regulation of Boost Converters via a Hybrid DQN-PI Control Strategy Under Large-Signal Disturbances

Abstract

The DC-DC boost converter plays a crucial role in interfacing low-voltage sources with high-voltage DC buses in DC microgrid systems. To enhance the dynamic response and robustness of the system under large-signal disturbances and time-varying system parameters, this paper proposes a hybrid control strategy that integrates proportional–integral (PI) control with a deep Q-network (DQN). The proposed framework leverages the advantages of PI control in terms of steady-state regulation and a fast transient response, while also exploiting the capabilities of the DQN agent to learn optimal control policies in dynamic and uncertain environments. To validate the effectiveness and robustness of the proposed hybrid control framework, a detailed boost converter model was developed in the MATLAB 2024/Simulink environment. The simulation results demonstrate that the proposed framework exhibits a significantly faster transient response and enhanced robustness against nonlinear disturbances compared to the conventional PI and fuzzy controllers. Moreover, by incorporating PI-based fine-tuning in the steady-state phase, the framework effectively compensates for the control precision limitations caused by the discrete action space of the DQN algorithm, thereby achieving high-accuracy voltage regulation without relying on an explicit system model.

1. Introduction

Recently, DC microgrids have garnered significant attention due to their high energy transmission efficiency, simplified control architecture, and strong compatibility with renewable energy sources and energy storage devices [1]. As a key power conversion unit within DC microgrids, the boost converter plays a vital role in facilitating energy exchange among devices operating at different voltage levels, as shown in Figure 1. This DC bus voltage regulation capability is crucial in ensuring the stable operation of the microgrid, achieving optimal power flow control, and enhancing overall system reliability [2]. However, the highly dynamic nature of the loads and power sources in microgrids presents significant challenges in maintaining control performance under varying conditions [3].

Conventional proportional–integral (PI) controllers with fixed parameters are widely adopted in boost converter voltage regulation due to their simplicity, low implementation costs, and effectiveness under steady-state operating conditions. These controllers are typically designed based on small-signal or state-space models, enabling closed-loop voltage regulation in well-defined scenarios [4]. However, during actual operation, boost converters frequently encounter unpredictable nonlinear behaviors, such as abrupt changes in load resistance or input voltage fluctuations. These nonlinear disturbances can lead to significant variations in system parameters, thereby reducing the accuracy of the PI controller. As a result, conventional PI controllers often struggle to maintain the desired control performance under such conditions, which underscores the need for more robust and adaptive voltage control strategies capable of handling complex and time-varying operating environments.

To address the limitations of conventional PI controllers under nonlinear and time-varying conditions, researchers have proposed a variety of advanced control strategies, such as model predictive control (MPC) [5], sliding mode control (SMC) [6], and fuzzy logic control [7], to enhance the robustness and adaptability of voltage regulation systems. For instance, an MPC-based controller was developed for a versatile buck–boost (VBB) DC-DC converter to enable accurate reference current tracking while maintaining a quasi-constant steady-state switching frequency in [8]. Ref. [9] applied a fuzzy logic controller to a single-switch cascaded DC-DC boost converter, achieving a faster dynamic response and improved steady-state performance. Ref. [10] addressed the burden associated with long prediction horizons in MPC by proposing an improved algorithm with a single prediction horizon, significantly reducing the online computational complexity. Furthermore, ref. [11] presented a dual-loop SMC controller, incorporating both voltage and current control loops for boost converters. The experimental results demonstrated reductions in the steady-state error, overshoot, and settling time compared to conventional lead–lag controllers. While the above advanced control methods demonstrate strong robustness and excellent dynamic performance under sudden changes in load resistance or input voltage, they fundamentally rely on accurate system modeling and prior expert knowledge [12]. The effectiveness of these approaches often depends on precise mathematical models, carefully designed observers, or well-tuned fuzzy rule bases, which are typically derived through extensive analytical efforts or empirical experience. However, in practical industrial environments, the presence of complex load dynamics and measurement uncertainties often makes it difficult to obtain precise system models, thereby posing significant challenges to the implementation of these model-dependent control strategies.

In recent years, reinforcement learning (RL) has emerged as a promising model-free intelligent control method [13]. By interacting continuously with the environment, RL agents are capable of learning optimal control strategies without requiring an accurate mathematical model, aiming to maximize the long-term cumulative rewards. This characteristic makes RL particularly suitable for addressing complex nonlinear control problems. Motivated by this, many researchers have explored the application of deep reinforcement learning (DRL) techniques to the control of power electronic converters. A widely used strategy involves integrating the DRL algorithm as a compensatory mechanism alongside conventional controllers, such as SMC or PI controllers, in order to improve the system stability, robustness, and dynamic response. In [14], an auxiliary deep deterministic policy gradient (DDPG) algorithm was designed to compensate for the observation errors introduced by a sliding mode observer in a PI control system. The proposed approach outperformed traditional PI and MPC controllers by reducing the overshoot and improving the overall control performance. Ref. [15] utilized the RL algorithm to dynamically tune the parameters of a PI controller, significantly enhancing the robustness of the system under nonlinear disturbances. Similarly, ref. [16] proposed an optimal output regulation framework by combining robust stabilization and DRL. This method employed a high-order sliding mode observer (HOSMO) to estimate system uncertainties, followed by a feedforward compensation mechanism to improve dynamic recovery, with the DDPG algorithm adaptively adjusting the control gains in real time. In [17], an adaptive, model-independent control scheme was proposed for a two-coil series–series (SS) compensated wireless power transfer (WPT) system feeding a time-varying constant power load (CPL). This method integrates a deep DDPG-based intelligent feedback controller with a sliding mode observer under an ultra-local model (ULM) framework. By combining the learning capabilities of DDPG and the real-time estimation of system dynamics via SMO, the scheme successfully stabilizes the output voltage and addresses impedance-induced instabilities caused by CPLs. Although these hybrid control strategies incorporating RL as a compensator demonstrate improved control performance compared to traditional methods and promote the application of RL in power electronics, most approaches still depend on partial knowledge of the system dynamics or observer-based state estimation and thus cannot achieve fully model-free control.

Another widely used method involves the direct application of RL algorithms in the control of DC-DC converters, enabling the system to learn optimal control policies from interaction with the environment, without relying on accurate system models. Ref. [18] applied the DDPG algorithm to track the reference voltage of the buck converter, even in a dynamic environment. The twin delayed deep deterministic policy gradient (TD3) algorithm was applied to optimize the efficiency of a dual active bridge converter by minimizing power losses in [19]. The method successfully enabled soft switching for different operating conditions, showcasing the potential of continuous-action DRL algorithms in complex converter control. A well-established DRL algorithm based on proximal policy optimization (PPO) was proposed to achieve near-optimal control of a buck converter operating in both continuous conduction mode (CCM) and discontinuous conduction mode (DCM), under resistive and inductive load conditions, in ref. [20]. The effectiveness and robustness of the PPO-based controller were validated through both simulations and experimental results. The above continuous-action reinforcement learning (RL) algorithms, such as TD3, DDPG, and PPO, offer finer control granularity compared to discrete-action methods. However, this advantage comes at the cost of significantly increased computational complexity. These algorithms typically require a substantially larger number of training episodes to achieve convergence, resulting in prolonged learning times and excessive computational burdens.

Compared to continuous-action reinforcement learning (RL) methods, discrete-action algorithms offer practical advantages in terms of deployment simplicity, training stability, and convergence speeds, making them particularly beneficial for power electronic systems. Among discrete-action algorithms, the deep Q-network (DQN) stands out for its ability to handle high-dimensional, nonlinear states, using deep neural networks to approximate the Q-value function. Ref. [21] applied the DQN algorithm directly to a buck converter with a constant power load (CPL), demonstrating the feasibility of achieving stable control in discrete action spaces. However, due to the inherent discretization of the action space, which selects from a finite set of predefined control actions (e.g., duty cycles), the control precision is limited, making it difficult to achieve precise and wide-range voltage regulation.

To address the issue of limited control precision in the steady state caused by the discrete nature of the action space in DQN-based controllers, this paper introduces a hybrid control strategy that combines the DQN with a conventional PI controller, as shown in Figure 2. In this approach, the DQN is responsible for learning control policies and improving the dynamic response, while the PI controller ensures high steady-state accuracy. This approach effectively overcomes the resolution limitations associated with discrete action spaces and achieves superior dynamic and steady-state performance without requiring a detailed system model. As such, it offers a promising solution for practical applications in power electronics.

The remainder of this manuscript is organized as follows. Section 2 introduces the control target and outlines the principles of the DQN. In Section 3, the proposed hybrid control strategy that combines the DQN with a PI controller is described in detail, including its structure and implementation. Section 4 presents the development of the simulation model in the MATLAB/Simulink environment and discusses the corresponding performance evaluations. Finally, Section 5 concludes the paper.

2. Problem Formulation and DQN Revisit

2.1. Problem Formulation

Figure 1 illustrates the simplified diagram of a boost converter, where energy sources such as photovoltaic panels or batteries are interfaced with the DC bus through the boost converter to regulate the voltage and enable efficient energy transfer. The output voltage of the boost converter is regulated by the duty cycle d of the MOSFET $Q_1$. Traditionally, the average model, shown in Equation (1), is developed to analyze the boost converter and facilitate the design of an effective closed-loop control strategy. Here, $V_{in}$ and $V_{out}$ denote the input and output voltages, respectively. L and C are the inductor and the capacitor, respectively. $R_L$ represents the equivalent resistance. Additionally, the inductor L functions as an essential energy storage element that maintains current continuity and suppresses voltage fluctuations during large-signal disturbances, such as abrupt changes in the load or input voltage.

$$\left\{\begin{array}{c}{\displaystyle \frac{d{i}_L}{dt}}={\displaystyle \frac{1}{L}}(V_{\mathrm{in}}-(1-d)V_{\mathrm{out}})\hfill \\ {\displaystyle \frac{d{V}_{\mathrm{out}}}{dt}}={\displaystyle \frac{1}{C}}\left((1-d)i_L-{\displaystyle \frac{V_{\mathrm{out}}}{R_L}}\right)\hfill \end{array}\right.$$

(1)

However, due to the uncertainty of disturbances, the input voltage and load resistance may change, making the design of a closed-loop PI controller difficult. To address this issue, this paper proposes a hybrid algorithm integrating the DQN and PI to control the output voltage of the boost converter. The control object is to stabilize the output voltage $V_{out}$, even under conditions of sudden load resistance and large input voltage disturbances.

2.2. DQN Algorithm Revisit

RL is an important branch of machine learning that enables agents to learn optimal control strategies through continuous interaction with the environment and feedback in the form of rewards. By iteratively adjusting the policy based on the received rewards, RL demonstrates strong adaptability in the dynamic environment, making it highly promising in a wide range of fields, such as autonomous driving, robotics, and power electronics. Among various RL algorithms, the DQN stands out due to its superior performance in high-dimensional state spaces and excellent generalization capabilities. As a result, it has become one of the most widely applied intelligent control algorithms in the field of power electronics. The flowchart of the DQN algorithm is shown in Figure 3.

The DQN problem can be formulated as a Markov decision process (MDP), which is defined by the tuple $\{S,A,R,P,r\}$, where S is the state space, A represents the action space, R denotes the reward set, P is the state transition probability, and r is the reward function. At each time step t, the agent observes the current state $s_t$, selects and executes an action $a_t$, and then transitions to a new state $s_{t+1}$ with a reward $r_{t+1}$ corresponding to the selected action. The DQN algorithm is a value-based reinforcement learning method aimed at maximizing the accumulated rewards $E\left[{\sum }_{k=0}^T{\gamma }^k{r}_{t+k}\right]$. At each time step, after the agent interacts with the environment, the current Q value $y_i$ is updated according to the following equation:

$$y_i=\left\{\begin{array}{c}{r}_j,\mathrm{if}\mathrm{episode}\mathrm{terminates}\mathrm{at}\mathrm{step}j+1\hfill \\ {\displaystyle r_j+\gamma \underset{a}{max}Q(s_{j+1},a_{t+1},{\theta }^{-}),\mathrm{otherwise}}\hfill \end{array}\right.$$

(2)

where $\gamma$ is the discount factor. During the interaction between the agent and the environment in the DQN algorithm, the tuple consisting of the current state, action, reward, and next state $(s,a,r,s^{\prime })$ is stored in a replay buffer. During the training process, a mini-batch of samples is randomly drawn from the replay buffer to update the neural network. The DQN framework consists of two neural networks with identical architectures, denoted as the evaluation network and the target network, respectively. The evaluation network is used to estimate the action value function $Q(s,a;\theta )$ under the current policy, and its parameters are updated at every training step via backpropagation. The target network, on the other hand, is used to compute the target value in the temporal difference (TD) error. Its parameters are periodically updated by copying the weights from the evaluation network. This mechanism stabilizes the target values and mitigates training instabilities such as oscillations and divergence. A loss function $L\left(\theta \right)$, defined as the mean squared error (MSE) between the predicted Q-value from the evaluation network and the target Q-value from the target network, is introduced to guide the parameter updates of the neural network, as shown in Equation (3):

$$L\left(\theta \right)=E\left[{(r+\gamma \underset{a_{t+1}}{max}Q(s_{t+1},a_{t+1};{\theta }^{-})-Q(s_t,a_t;\theta ))}^2\right]$$

(3)

In the DQN framework, the agent inputs the current state $s_t$ into the evaluation network to obtain the estimated Q-values for all possible actions, denoted as $Q(s_t,a_t;\theta )$. An action $a_t$ is then selected based on the $\epsilon$-greedy policy, which balances exploration and exploitation. Specifically, with probability $\epsilon$, the agent selects a random action to explore potentially better strategies and avoid being trapped in a local optimum. With probability $1-\epsilon$, the agent chooses the action that maximizes the estimated Q-value. Thus, at each time step t, the action $a_t$ chosen by the agent under the $\epsilon$-greedy policy can be described as

$$a_t=\left\{\begin{array}{cc}arg\underset{a}{max}Q(s_t,a;\theta ),\hfill & \mathrm{if}p>\epsilon \hfill \\ a_{\mathrm{random}},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

This exploration–exploitation policy allows the agent to sufficiently explore the environment during the training process while gradually shifting towards exploiting learned knowledge as the policy improves. In this study, the value of $\epsilon$ starts from a high value to encourage exploration in the early stages of training and decays to a lower value to promote more deterministic, reward-maximizing behavior as learning progresses.

3. Control System Design

The control system is composed of a DQN and a PI controller, where the DQN is responsible for learning and adapting to larger-signal disturbances, while the PI controller ensures high-precision regulation in the steady state by compensating for the inherent limitations of the discrete action space in the DQN algorithm. Figure 4 shows the framework of our proposed hybrid algorithm. In this integrated framework, the DQN algorithm continuously interacts with the environment to learn and optimize its control strategy, enabling it to effectively handle nonlinear operating conditions such as large load disturbances or variations in the input voltage. Meanwhile, the PI controller serves to refine and correct the output of the DQN controller, ensuring a fast and stable system response. This cooperative mechanism not only improves the overall steady-state control performance of the system but also enhances its robustness and adaptability to unknown disturbances.

3.1. State Space

In the proposed control strategy, the error between the actual output voltage and the reference voltage, denoted as the voltage error $e\left(t\right)$, is considered one of the most critical feedback signals, directly reflecting the control performance. Therefore, it serves as a key component of the system state representation. Simulations involving sudden changes in load and input voltage are conducted to evaluate the effectiveness and robustness of the proposed hybrid control framework. Accordingly, both the load resistance and input voltage are also included as observed variables in the system state, as their variations significantly affect the system’s dynamic behavior and control requirements. Hence, the system state is defined as a vector composed of multiple features that collectively characterize the current operating condition of the converter. Specifically, the state space is designed as $S_t=\left\{e\left(t\right),v_{in}\left(t\right),R\left(t\right)\right\}$.

3.2. Action Space

The DQN algorithm is inherently designed for control problems with discrete action spaces. Its core principle lies in approximating the optimal policy by selecting actions from a finite set. In this study, the switching duty ratio of the boost converter is selected as the control variable for the construction of a discrete action space. The approximate range of the duty cycle is estimated based on the ratio between the desired output voltage and the actual input voltage under steady-state conditions. To define the action space more systematically, the discrete action space $\mathcal{A}$ is defined as in Equation (5):

$$\mathcal{A}=\{D-\phi +kc|k=0,1,2\dots ,{\displaystyle \frac{2\phi }{c}}\}$$

(5)

where the parameters D, $\phi$, and c, respectively, represent the nominal duty ratio, the fluctuation range, and the minimum step. These can be flexibly tuned according to the system dynamics and the desired control precision, allowing the discrete action space to strike a balance between learning efficiency and control accuracy.

3.3. Reward Function

The control objective is to accurately track the reference voltage. The design of the reward function plays a crucial role in both the learning efficiency and the ultimate performance of the agent. A piecewise reward function, depicted in Equation (6), is adopted in this work to guide the agent by incorporating two subgoals ${\epsilon }_1$ and ${\epsilon }_2$, which represent specific targets related to the control error. These subgoals help to shape the learning process toward accurate and stable voltage regulation. In this study, ${\epsilon }_1$ and ${\epsilon }_2$ are set to 0.2 and 1, respectively, based on empirical testing for complex and dynamic control environments.

$$r\left(t\right)=\left\{\begin{array}{cc}{\beta }_1-{\beta }_3\left|e\left(t\right)\right|,\hfill & \mathrm{if}\left|e\left(t\right)\right|\le {\epsilon }_1\hfill \\ {\beta }_2-{\beta }_3\left|e\left(t\right)\right|,\hfill & \mathrm{if}{\epsilon }_1<\left|e\left(t\right)\right|\le {\epsilon }_2\hfill \\ -{\beta }_3\left|e\left(t\right)\right|,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(6)

3.4. DNN Design

In the traditional Q-learning algorithm, the agent selects an action by consulting a Q-table, choosing the one with the highest Q-value for the current state. However, in the DQN algorithm, Q-values are no longer stored in a tabular format. Instead, a neural network is employed to predict the Q-value corresponding to each possible action, given the current state as input. The agent then selects the action associated with the highest predicted Q-value, effectively integrating value function approximation with policy decision-making.

The architecture of the neural network used in this study is illustrated in Figure 5. It consists of an input layer, three hidden layers, and an output layer. The input layer receives three features that characterize the system state: the tracking error $e\left(t\right)$, the input voltage $V_{\mathrm{in}}$, and the load resistance $R_L$. The output layer produces a vector whose length equals the number of discrete actions; each element of this vector represents the estimated Q-value for the corresponding action under the given state.

Additionally, all three hidden layers are fully connected and comprise 64 neurons each. This network structure strikes a balance between expressive power and computational efficiency, making it suitable for real-time control scenarios in power electronic systems. Moreover, the rectified linear unit (RLU) is used as the activation function in all hidden layers, which provides the advantages of a low computational cost and high training efficiency.

3.5. PI Controller Design

In the proposed hybrid control framework, the PI controller is designed to provide a compensation signal to compensate for the action output generated by the DQN agent. Specifically, after the DQN outputs a preliminary control action $a_t$(k) based on the current system state, the PI controller computes an adjustment term $a_{PI}$ based on the error between the reference voltage and the actual output voltage. The Tustin method is employed for discretization to implement the PI controller in a digital control system with high fidelity to its continuous-time dynamics. The expression of the discrete PI control is shown in Equation (7):

$$u\left(k\right)=u(k-1)+K_p[e\left(k\right)-e(k-1)]+K_i{T}_{s}e\left(k\right)$$

(7)

where $K_p$ denotes the proportional gain and $K_i$ denotes the integral gain. $T_s$ represents the sampling period. In this study, the sampling time of the PI controller is set to $2\times {10}^{-5}$ s. After parameter tuning, $K_p$ is set to 0.000082, and $K_i$ is set to 0.00008. Additionally, the limit of PI controller $\varsigma$ ensures that the PI controller primarily performs fine-tuning around the DQN output, without overriding the learning-based decision-making process. Furthermore, the parameter $\varsigma$ can be flexibly adjusted according to the system requirements, and its impact on the control precision, convergence speed, and robustness can be systematically investigated in subsequent studies.

4. Simulation Verification

4.1. Simulation Configuration

To verify the effectiveness and robustness of the proposed hybrid control strategy, a detailed boost converter simulation model was constructed in the MATLAB/Simulink environment. The key electrical parameters of the boost converter are summarized in

Table 1. The key element parameters of the boost converter.

Parameter	Definition	Value
$V_{in}$	Input voltage	100 V
$V_{ref}$	Reference output voltage	200 V
L	Inductance	10 mH
C	Capacitance	470 $\mathsf{\mu }$F
f	Switching frequency	10 KHz
$R_L$	Resistance	5–20 ohms

. It should be noted that the boost converter operated in continuous conduction mode (CCM) throughout all simulation scenarios. The DQN architecture was developed using the PyTorch 2.2.2 framework, and its associated hyperparameters used for the training of the agent are presented in

Table 2. The hyperparameters of the DQN controller.

Parameter	Definition	Value
$\alpha$	Learning rate	$1\times {10}^{-3}$
$\gamma$	Discount factor	0.98
$ϵ$	Exploration rate	0.05
${\beta }_1$, ${\beta }_2$, ${\beta }_3$	Reward function parameters	10, 5, −15
D	Nominal duty ratio	0.5
$\phi$	Fluctuation range	0.1
c	Minimum step	0.02
B	Mini-batch size	64
M	Replay memory size	5000
N	Training episodes	300

. The DQN network training was accelerated using an NVIDIA RTX 3060 GPU with 12 GB of memory, which significantly improved the computational efficiency and reduced the training time. To enable real-time interaction between the control algorithm and the Simulink-based boost converter model, the MATLAB 2024–Python 3.11 engine API was employed, allowing seamless bidirectional data exchange during the learning and control process.

The simulation system was configured with a fixed-step discrete solver to ensure consistency in time-domain analysis and accurate modeling of the switching behavior. A simulation time step of 1 $\mathsf{\mu }$s was selected to capture high-frequency switching dynamics, while the control interval for the DQN agent was set to 0.01 s, allowing the agent sufficient time to observe the system dynamics and update its control actions accordingly.

4.2. Comparative Performance Evaluation of PI, DQN, and DQN+PI Controllers

To validate the effectiveness and robustness of the proposed hybrid PI+DQN control strategy for the DC-DC boost converter, disturbance experiments were conducted in the MATLAB/Simulink environment. The total simulation time was set to 1.5 s. Two major disturbances were introduced to evaluate the adaptability of the control system under dynamic operating conditions.

• Load Step Disturbance: At $t=0.5$ s, the load resistance was abruptly increased from $10 \Omega$ to $30 \Omega$, corresponding to a 200% increase.
• Input Voltage Drop Disturbance: At $t=1.0$ s, the input voltage was reduced from $100 \mathrm{V}$ to $80 \mathrm{V}$, representing a 20% drop.

The reference output voltage was set to $V_{ref}=200 \mathrm{V}$. The DQN agent was configured with a discrete action space by modulating the switching duty cycle in the range of $[0.4,0.6]$ with a step of 0.02.

Figure 6 illustrates the simulation results of the three control strategies during the system startup phase. It can be seen that, during the initial startup interval (0–0.06 s), the PI controller requires approximately 0.058 s to drive the output voltage to the reference value of 200 V, whereas the DQN controller takes about 0.35 s to achieve the same level of tracking. By contrast, the proposed hybrid DQN+PI strategy accomplishes this task in only 0.02 s, thereby demonstrating its superior capabilities for rapid and accurate voltage regulation during the transient startup phase. Furthermore, inspection of the steady-state waveform between 0.30 s and 0.31 s reveals that, under DQN control, the average output voltage settles at approximately 196 V and fails to precisely reach the reference voltage of 200 V, which is caused by the discretization inherent in the DQN’s action space. In comparison, both the PI controller and the hybrid DQN+PI control strategy maintain the output voltage exactly at the reference voltage of 200 V, confirming that the integration of PI and the DQN not only accelerates convergence during the startup phase but also rectifies the steady-state tracking error introduced by the discrete-action DQN framework.

Figure 7 shows the dynamic response simulation results during the load step disturbance and input voltage drop. It can be seen that, upon a load disturbance, the PI controller requires approximately 0.14 s to return to a steady state of 200 V, with a maximum overshoot of 103%. Under the control of the DQN, the system reaches a steady state of 196 V in just 0.051 s, exhibiting a lower overshoot of 62%. The proposed hybrid DQN+PI control strategy restores steady-state operation of 200 V in 0.085 s with a peak overshoot of 65%.

Table 3. The performance comparison under different controllers.

Controller	Load Step		Voltage Drop
	Settling Time	Overshoot	Settling Time	Overshoot
PI	0.14 s	103%	0.12 s	28%
DQN	0.051 s	62%	0.055 s	10.5%
DQN+PI	0.085 s	65%	0.065 s	10.5%

shows the detailed performance comparison of the different controllers. These results indicate that the DQN controller outperforms the PI controller in terms of the transient response speed and adaptability to nonlinear disturbances, making it highly effective in handling sudden system changes. However, the DQN controller exhibits inferior steady-state performance, as evidenced by the persistent steady-state error. In contrast, both the PI controller and the hybrid DQN+PI strategy demonstrate superior steady-state accuracy. Overall, the proposed hybrid DQN+PI control strategy effectively combines the fast adaptability of the DQN with the steady-state accuracy of PI control. It achieves superior robustness under sudden load and input voltage disturbances while maintaining precise voltage regulation under a steady state, demonstrating strong overall control performance.

Due to the incorporation of an exploration strategy during the training process, the rewards obtained by the agent in each episode exhibit fluctuations. A smoothing technique is applied to better capture the underlying trends in the agent’s performance and reduce the influence of stochastic variability. Specifically, a moving average filter with a window size of five is employed to compute the mean reward over the most recent five episodes to represent the agent’s learning progress. The reward curves of the different control strategies over the training episodes are shown in Figure 8. As shown in Figure 8, the PI controller consistently yields a fixed reward across episodes, as its parameters remain unchanged during training. In contrast, the DQN agent exhibits a gradual improvement in performance with increased training episodes, eventually converging to a stable policy. The steady-state reward attained by the DQN agent surpasses that of the PI controller, indicating enhanced control effectiveness. Moreover, the proposed DQN+PI hybrid control architecture demonstrates a significantly faster convergence rate compared to the DQN approach. In the steady state, it also achieves higher cumulative rewards, highlighting its superior capabilities in terms of both learning efficiency and control performance.

4.3. Impact of PI Output Saturation Limit on System

As illustrated in Figure 4, the control signal of the boost converter is composed of two parts: the action output generated by the DQN agent and the compensatory signal from the PI controller. In this hybrid control architecture, the PI controller primarily serves to fine-tune the output voltage and eliminate steady-state errors introduced by the discrete nature of the DQN’s action space. However, the effectiveness of this compensation is directly influenced by the output saturation limit of the PI controller. Specifically, the saturation bound determines the extent to which the PI controller can contribute to the overall control action. A limit that is too small may restrict the PI controller’s ability to correct errors effectively, while an excessively large limit could introduce instability. Therefore, to investigate the impact of PI output saturation $\varsigma$ on the overall system performance, the impact of the PI output saturation limit on the system was simulated by varying the PI output saturation range [$-\varsigma$, $\varsigma$], where $\varsigma$ represents the maximum output of the PI controller.

Figure 9 illustrates the control performance of the system under various control strategies—PI, DQN, and DQN+PI—with differing PI limit values during the startup process. The simulation results show that the PI controller requires approximately 0.06 s to regulate the output voltage to the reference value. In contrast, both the DQN controller and the DQN+PI hybrid strategy achieve faster voltage regulation across all tested PI limit settings.

Figure 10 shows the simulation results under various control modes—PI, DQN, and DQN+PI—with differing PI limit values during the load changing process (from 10 ohms to 30 ohms). It is evident that, when addressing significant load disturbances, both the DQN and the proposed hybrid control strategy exhibit a faster response speed compared to the PI controller. The DQN algorithm can achieve a steady state within 0.04 s and effectively manage large-scale load changes. In contrast, the proposed DQN+PI hybrid architecture demonstrates a response speed that is superior to that of the PI controller, although it is slightly slower than that of the DQN controller. Furthermore, as the limit value of the PI controller increases, the stability of the system diminishes, resulting in a longer recovery time to reach a steady state.

Figure 11 presents the system response to a significant input voltage drop from 100 V to 80 V. During the input voltage disturbance, the DQN controller exhibits the fastest response, requiring only 0.04 s to re-establish the steady-state voltage. However, it fails to eliminate the steady-state error, as observed in the 0.98–1 s interval. In contrast, both the PI controller and the proposed DQN+PI hybrid strategy successfully maintain accurate steady-state tracking of the reference voltage of 200 V. Furthermore, the simulation results show that the saturation limit of the PI controller significantly affects the system performance. As $\varsigma$ increases, the system has a larger voltage overshoot and this negatively impacts the system’s dynamic stability. When the saturation limit is excessively large, such as 2.5c, the system fails to achieve steady-state voltage tracking.

Thus, from Figure 10 and Figure 11, it can be seen that the output saturation limit of the PI controller significantly influences the overall control performance of the hybrid DQN+PI system. Specifically, increasing the PI limit value enhances the system’s transient response speed during startup and its steady-state tracking performance. However, this improvement comes at the cost of increased voltage overshoot and reduced system stability during large-signal disturbances. When the PI output constraint becomes excessively large (e.g., 2.5c), the system fails to maintain steady-state voltage tracking and exhibits oscillatory behavior. Therefore, there exists a critical trade-off between control responsiveness and system robustness. An appropriately constrained PI output ensures fast convergence while preserving steady-state accuracy and system stability, which is essential for reliable operation under dynamic conditions.

4.4. Performance Comparison with PI and Fuzzy Control

To validate the performance of the proposed control strategy, it is essential to conduct a comparative analysis with conventional control methods known for their robustness. Among these, the fuzzy logic controller (FLC) stands out as a widely used nonlinear control approach capable of effectively handling system uncertainties and nonlinearities. Unlike the classical PI controller, which relies on an accurate mathematical model, the FLC utilizes linguistic rules and reasoning, offering superior adaptability and robustness in dynamic and uncertain environments. In this study, a fuzzy logic controller is designed to regulate the output voltage of the boost converter. The overall control architecture, as illustrated in Figure 12, comprises three main phases.

• Phase 1: Input Signal Preprocessing. The controller receives the instantaneous voltage error $E\left(t\right)$ and its rate of change $\Delta E\left(t\right)$ as input variables. These signals are normalized through gain blocks (Gain1 and Gain2) to ensure compatibility with the fuzzy inference system’s universe of discourse. Two limiters are used to constrain the input ranges with predefined bounds before feeding them into the fuzzy controller.
• Phase 2: Fuzzy Controller. The fuzzy controller, consisting of fuzzification, rule evaluation, and defuzzification stages, was implemented using MATLAB’s Fuzzy Logic Toolbox. Both input and output variables are defined using seven triangular membership functions, covering the linguistic range from Negative Big (NB) to Positive Big (PB). The rule base, constructed based on the system dynamics, is summarized in

  Table 4. The designed fuzzy rule base.

  $\mathit{\Delta }\mathit{E}\backslash \mathit{E}$	NB	NM	NS	ZE	PS	PM	PB
  NB	NB	NB	NM	NM	NS	ZE	ZE
  NM	NB	NM	NM	NS	ZE	ZE	PS
  NS	NM	NM	NS	ZE	PS	PM	PM
  ZE	NM	NS	ZE	ZE	ZE	PS	PM
  PS	NS	ZE	PS	PM	PM	PM	PB
  PM	ZE	PS	PM	PM	PM	PB	PB
  PB	ZE	PS	PM	PM	PB	PB	PB

  .
• Phase 3: Output Signal Processing. The fuzzy controller produces a duty cycle adjustment signal $\Delta d$, which is scaled using a gain block (Gain3) to tailor its magnitude. The final PWM duty cycle applied to the boost converter is determined as

  $$d\left(t\right)=d(t-1)+\Delta d$$

  (8)

After completing the design of the fuzzy logic controller, a comparative analysis was conducted between the proposed hybrid controller, the fuzzy controller, and the conventional PI controller. Figure 13 presents the transient response of the system during the startup phase (0–0.12 s). It can be seen that the PI controller requires approximately 0.06 s to regulate the output voltage to the preset reference voltage. In contrast, the fuzzy controller achieves this in around 0.04 s. Notably, the proposed DQN+PI hybrid controller completes voltage regulation within only 0.02 s, demonstrating the fastest dynamic response among all three controllers. These results highlight the superior startup performance of the proposed controller.

Figure 14 depicts the system behavior under a load step disturbance, where the load resistance abruptly increases from 10 ohms to 30 ohms (a 200% step change). The PI controller exhibits significant degradation in performance under this condition, with the output voltage overshooting to nearly twice the reference level and requiring up to 0.14 s to recover to a steady state. In comparison, both the fuzzy controller and the proposed DQN+PI controller successfully restore the output voltage within approximately 0.04 s, representing a 71.4% reduction in recovery time compared to the PI controller. Moreover, although the fuzzy and proposed controllers exhibit similar settling times, the proposed controller achieves a slightly lower peak overshoot, indicating better transient suppression performance.

Figure 15 shows the system’s response to an input voltage drop disturbance, in which the input voltage suddenly drops from 100 V to 80 V at t = 1 s. The PI controller is significantly affected, requiring more than 0.1 s to stabilize the output voltage and exhibiting a maximum undershoot of 23%. Both the fuzzy controller and the proposed controller respond more promptly, with settling times of around 0.05 s. The proposed DQN+PI controller exhibits only 6.5% overshoot, while the fuzzy controller suffers from 25% undershoot, indicating better transient performance and disturbance rejection.

From Figure 13, Figure 14 and Figure 15, it can be seen that the proposed hybrid control strategy outperforms both the conventional PI controller and the fuzzy logic controller across all tested scenarios. Under challenging conditions, such as load step changes and input voltage disturbances, the proposed hybrid controller exhibits a faster response and improved disturbance rejection capabilities. Specifically, the proposed controller exhibits the fastest transient response during the startup phase. These results validate both the effectiveness and robustness of the proposed control scheme in enhancing the dynamic voltage regulation performance of the boost converter.

5. Conclusions

This paper proposes a hybrid voltage regulation strategy that integrates a DQN with a traditional PI controller to address the challenges of accurate and robust control in boost converters under nonlinear disturbances and model uncertainties. The proposed DQN+PI hybrid control scheme employs the DQN output as feedforward compensation to enhance the dynamic response, while the PI controller ensures steady-state accuracy. The simulation results validate the effectiveness of the proposed approach during the startup process. Additionally, the influence of different PI output limit values on system stability has been anlayzed. The proposed hybrid strategy provides a practical, model-free, and high-performance solution for voltage regulation in power electronic systems, with promising prospects for real-world applications. In further work, the proposed hybrid control scheme can be implemented on a real-time platform such as dSPACE, which is fully compatible with MATLAB/Simulink. This enables the seamless deployment of the control algorithm directly from our simulation environment to real hardware with minimal modification.