MPG-OD-DDPG-Based Optimal Scheduling for Park-Level Multi-Source Synergistic Power Supply System

Abstract

To address the uncertainty of renewable energy output power and load demand in the park-level multi-source synergistic power supply system and achieve economical system operation, an optimal scheduling method based on the mathematical programming guided imitation learning-oscillation-decaying deep deterministic policy gradient (MPG-OD-DDPG) algorithm is proposed. First, a scheduling model of the park-level multi-source synergistic power supply system is established with the objective of minimizing system operating cost. Second, to address the low sample utilization efficiency of the deep deterministic policy gradient (DDPG), a mathematical programming guided imitation learning (MPGIL) method is introduced to obtain demonstration experience and guide the initial training process. Then, targeting insufficient exploration capability of DDPG, an oscillation-decaying Ornstein–Uhlenbeck noise (OD-OU) is designed to enhance the exploratory capability of the algorithm through the oscillation-decaying process of noise intensity. Finally, according to the scheduling model, the reward function is designed, and the proposed algorithm is used to search for the optimal energy regulation strategy, thereby realizing the optimal scheduling of the park-level multi-source synergistic power supply system. Case study results show that compared with the DDPG, the MPG-OD-DDPG reduces the operating cost by 7.41%. It can effectively exploit the energy time-shift capability of the energy storage system (ESS), reduce the operating cost of the park-level multi-source synergistic power supply system.

1. Introduction

Under the global drive to address climate change and pursue sustainable development, the energy transition is steadily advancing [1]. The current predominant trends in the global energy revolution are characterized by a gradual decline in the proportion of fossil fuel consumption and an evolution of energy systems towards cleanliness and intelligence. With the implementation of carbon neutrality targets by various nations, structural adjustments are occurring in energy production and consumption patterns. In this process, renewable energy, owing to its extensive resource availability, environmental friendliness, and continuously decreasing costs, is witnessing a steady increase in its share of energy supply, gradually becoming a critical component of new energy provision.

As the renewable energy system, energy storage system (ESS), and new electricity-consuming device are increasingly integrated, the park is increasingly promoting the local consumption of renewable energy and the construction of new-type power systems. However, this integration has also significantly amplified the impact of uncertainties in renewable energy output power and load demand on park power supply systems, posing challenges to the economic operation of these parks [2,3].

At present, considerable scholarly attention has been devoted worldwide to the optimization of power system scheduling, with particular emphasis on addressing the stochastic characteristics of renewable power and load demand. Stochastic programming is a classical approach for handling optimal scheduling problems with uncertainty. By characterizing uncertain factors probabilistically and performing scenario analysis, it enables the optimal scheduling of microgrids with renewable energy sources [4,5,6]. Robust optimization, by contrast, constructs uncertainty sets and seeks scheduling solutions that remain robust against all possible uncertainty realizations, thereby ensuring feasibility under worst-case conditions [7,8,9,10]. To overcome the computational difficulties caused by nonlinearities in power system optimal scheduling, intelligent optimization algorithms have also been adopted by many researchers. These algorithms simulate swarm intelligence methods to search for optimal solutions in the solution space and are well suited for optimal scheduling problems with complex constraints [11,12,13,14]. However, these methods are all model-based and rely on accurate mathematical models. As renewable sources become increasingly integrated into power supply systems, it becomes difficult for such methods to obtain optimal scheduling strategies rapidly and effectively.

Deep reinforcement learning (DRL) algorithms have been extensively studied and have demonstrated significant potential for application in energy system optimization management and renewable energy consumption [15,16]. As a model-free methodology, DRL derives optimal strategies by continuously engaging environmental dynamics. It can dynamically determine actions according to the environmental state and therefore exhibits stronger adaptability when handling complex sequential decision-making problems [17,18]. In [19], a DRL algorithm based on a flexible Actor–Critic framework was proposed, in which continuous action control of an integrated electricity-gas energy system was achieved through the appropriate design of key methods such as the reward function. In [20], an asynchronous advantage Actor–Critic algorithm was employed to solve the optimal scheduling problem of an integrated energy system with multi-energy coupling, effectively reducing the average daily operating cost. In [21], an improved evolutionary DRL algorithm was proposed for islanded microgrids, where novelty search was introduced and policy novelty was used as a fitness measure to guide the updating direction of the action policy. In [22], to address the high computational complexity of multi-microgrid systems, an improved multi-agent DRL algorithm was proposed, in which the system was partitioned by a modular value function based on communication distance constraints, thereby reducing computational complexity while maintaining the overall stability of the multi-microgrid system. These studies indicate that DRL has certain advantages in solving scheduling problems with continuous action spaces. While the deep reinforcement learning methods mentioned in the above research demonstrate advantages in solving scheduling problems involving continuous action spaces, they fail to consider how to effectively utilize high-quality experiences from other methods. This results in low sample utilization efficiency during the early training stage. Furthermore, they do not address how to maintain exploration capability more persistently to ensure adequate exploration in the later stages of training, which ultimately affects the convergence stability of the algorithm during the training process.

The DDPG algorithm struggles to quickly adapt to dynamically changing environmental characteristics. Meanwhile, the optimal scheduling problem for power supply systems incorporating renewable energy sources exhibits significant uncertainty. This leads to the accumulation of a large number of ineffective experiences during the early training stage, which subsequently interferes with the algorithm’s learning from high-reward experiences, adversely affecting its learning efficiency and final performance. Furthermore, the DDPG algorithm maintains a balance between exploration and exploitation during training by gradually reducing the intensity of the exploration noise. However, this approach may cause the algorithm’s exploration capability to decline too rapidly, resulting in insufficient exploration ability in the later stages of training. This makes the algorithm prone to becoming trapped in local optima, ultimately impacting the final scheduling effectiveness. Therefore, how to guide the DDPG to learn feasible scheduling policies from effective experiences in the early training stage, while maintaining exploration capability to avoid converging to local optima in the later training stage, is key to enhancing its performance in the optimal scheduling of the power supply system.

This study presents an optimal scheduling method for the park-level multi-source synergistic power supply system based on a mathematical programming guided imitation learning–oscillation-decaying deep deterministic policy gradient (MPG-OD-DDPG) algorithm. By searching for the optimal strategy for energy regulation of the ESS, the proposed method enables the ESS to accommodate photovoltaic (PV) output and perform peak shaving and valley filling for loads, thereby achieving economical overall system operation. First, an optimal scheduling model of the park-level multi-source synergistic power supply system is established. Second, to address the problem of excessive ineffective experience in the early training stage of the DDPG, a mathematical programming guided imitation learning (MPGIL) method is proposed to improve learning efficiency. Then, an oscillation-decaying Ornstein–Uhlenbeck noise (OD-OU) is designed to maintain the exploration capability of the algorithm during the later training stage through the oscillation-decaying process of noise intensity, thereby enhancing its global optimization performance. Finally, case studies demonstrate that the presented approach yields superior learning efficiency and optimization performance, and can effectively improve PV consumption capability while reducing the operating cost of the park-level multi-source synergistic power supply system.

2. Problem Formulation

The park-level multi-source synergistic power supply system consists of a PV system, an ESS, the power grid, user loads, and an energy management center. Figure 1 illustrates the internal architecture of the system.

Figure 1. Park-level multi-source synergistic power supply system.

The PV system converts solar energy into electricity, while the ESS provides bidirectional power regulation capability. Through the regulation of the ESS by the energy management center, electrical energy is stored during periods of high PV output and discharged when PV output is insufficient. Meanwhile, considering the peak–valley electricity price difference, electricity is purchased from the grid during low-price periods and the stored energy is released during high-price periods. In this manner, the synergistic power supply of the PV system, the ESS, and the power grid is used to satisfy the user load demand of the park.

2.1. Objective Function

An optimal scheduling model for the park-level multi-source synergistic power supply system is constructed by targeting the minimization of system operating costs. The objective function mainly consists of the grid electricity purchasing cost and the energy storage degradation cost:

$$\min F={\displaystyle \sum _{t=1}^T(C_{\mathrm{grid}}(t)+C_{\mathrm{ess}}(t))}$$

(1)

where $T$ denotes the system scheduling horizon, $C_{\mathrm{grid}}(t)$ (CNY) denotes the grid electricity purchasing cost, and $C_{\mathrm{ess}}(t)$ (CNY) denotes the energy storage degradation cost.

The grid electricity purchasing cost of the power supply system is calculated as shown in Equation (2):

$$C_{\mathrm{grid}}(t)=\rho (t)\cdot P_{\mathrm{grid}}(t)\cdot \Delta t$$

(2)

where $P_{\mathrm{grid}}(t)$ (kW) denotes the grid electricity purchasing power, $\rho (t)$ (CNY/kWh) denotes the time-of-use electricity price, and $\Delta t$ (h) denotes the duration of a single time interval.

During the operation of the park-level multi-source synergistic power supply system, the ESS needs to perform charge–discharge cycles at different depths of discharge (DOD) under rated operating conditions. Therefore, a degradation cost model for the ESS is established based on the DOD-cycle life curve [23].

The fitting function for the DOD-cycle life curve is shown in Equation (3):

$$N_{\mathrm{life}}(k_{\mathrm{DOD}})={\alpha }_0\exp (-{\alpha }_1{k}_{\mathrm{DOD}})$$

(3)

where $N_{\mathrm{life}}$ denotes the natural exponential function that fits the relationship between the cycle life of the ESS and the depth of discharge, $k_{\mathrm{DOD}}\in [0,100]$ (%) denotes the depth of discharge, ${\alpha }_0$ and ${\alpha }_1$ denote the fitting coefficients, whose values are determined based on fitting the measured data.

Equation (3) represents the cycle life of the battery under charge–discharge cycling from the fully charged state at different depths of discharge. In this process, the state of charge (SOC) and the DOD satisfy the following relationship:

$$k_{\mathrm{DOD}}=100\%-k_{\mathrm{SOC}}$$

(4)

where $k_{\mathrm{SOC}}\in [0,100]$ (%) denotes the state of charge.

Assuming that the battery experiences the same life loss at the same DOD and that the life loss is symmetric during the charging and discharging processes, then the life loss when the battery’s SOC is discharged from 100% to $k_{\mathrm{SOC}}$ is given by Equation (5) [24]:

$$L_{\mathrm{ess}}(100\%,k_{\mathrm{SOC}})={\displaystyle \frac{1}{2N_{\mathrm{life}}(100\%-k_{\mathrm{SOC}})}}$$

(5)

where $L_{\mathrm{ess}}(100\%,k_{\mathrm{SOC}})$ denotes the life loss.

Assuming the battery’s charge–discharge state remains constant between two adjacent SOC states, when the SOC changes from $k_{\mathrm{SOC}}(t-1)$ to $k_{\mathrm{SOC}}(t)$, the degradation cost of the ESS is given by Equation (6) [23]:

$$C_{\mathrm{ess}}(t)=C_E\left|L_{\mathrm{ess}}(100\%,k_{\mathrm{SOC}}(t))-L_{\mathrm{ess}}(100\%,k_{\mathrm{SOC}}(t-1))\right|$$

(6)

where $k_{\mathrm{SOC}}(t)\in [0,100]$ (%) denotes the SOC at time t, $C_{\mathrm{ess}}(t)$ (CNY) denotes the battery degradation cost of the ESS, and $C_E$ (CNY) denotes the battery replacement cost of the ESS.

Equation (6) presents a simplified model for the degradation cost of the ESS, which primarily accounts for the impact of DOD on cycle life, without considering factors such as temperature and C-rate. With a relatively concise form, it is easy to embed into the deep reinforcement learning-based optimal scheduling framework for power supply systems, ensuring both accuracy in degradation cost calculation and feasibility in practical application.

2.2. Constraints

2.2.1. Power Balance Constraint

The park-level multi-source synergistic power supply system is not allowed to sell electricity back to the grid. Its power balance constraint is as follows:

$$P_{\mathrm{pv}}(t)-P_{\mathrm{curt}}(t)+P_{\mathrm{ess}}(t)+P_{\mathrm{grid}}(t)=P_{\mathrm{load}}(t)$$

(7)

$$0<P_{\mathrm{curt}}(t)<P_{\mathrm{pv}}(t)$$

(8)

$$P_{\mathrm{grid}}(t)\ge 0$$

(9)

where $P_{\mathrm{pv}}(t)$ (kW) denotes the PV power, $P_{\mathrm{curt}}(t)$ (kW) denotes the curtailed PV power, $P_{\mathrm{load}}(t)$ (kW) denotes the user load demand, and $P_{\mathrm{ess}}(t)$ (kW) denotes the charging/discharging power of the ESS. For $P_{\mathrm{ess}}(t)$, positive values ($P_{\mathrm{ess}}(t)>0$) correspond to discharging operations, while negative values ($P_{\mathrm{ess}}(t)<0$) indicate charging processes.

2.2.2. Operational Constraints of the ESS

$$k_{\mathrm{SOC}}^{\min }\le k_{\mathrm{SOC}}(t)\le k_{\mathrm{SOC}}^{\max }$$

(10)

$$k_{\mathrm{SOC}}(t)=\left\{\begin{array}{cc}{k}_{\mathrm{SOC}}(t-1)-{\displaystyle \frac{P_{\mathrm{ess}}(t)\Delta t}{{\eta }_{\mathrm{dis}}{E}_{\mathrm{ess}}^{\max }}},& P_{\mathrm{ess}}(t)>0\\ k_{\mathrm{SOC}}(t-1),& P_{\mathrm{ess}}(t)=0\\ k_{\mathrm{SOC}}(t-1)-{\displaystyle \frac{{\eta }_{\mathrm{ch}}{P}_{\mathrm{ess}}(t)\Delta t}{E_{\mathrm{ess}}^{\max }}},& P_{\mathrm{ess}}(t)<0\end{array}\right.$$

(11)

$$-P_{\mathrm{ess},\max }^{\mathrm{ch}}\le P_{\mathrm{ess}}(t)\le P_{\mathrm{ess},\max }^{\mathrm{dis}}$$

(12)

where $k_{\mathrm{SOC}}^{\min }$ (%) and $k_{\mathrm{SOC}}^{\max }$ (%) denote the minimum and maximum SOC of the ESS, ${\eta }_{\mathrm{ch}}$ and ${\eta }_{\mathrm{dis}}$ denote the charging and discharging efficiencies of the ESS, $P_{\mathrm{ess},\max }^{\mathrm{dis}}$ (kW) and $P_{\mathrm{ess},\max }^{\mathrm{ch}}$ (kW) denote the maximum discharging and charging powers of the ESS, and $E_{\mathrm{ess}}^{\max }$ (kWh) denotes the maximum energy capacity of the ESS.

3. Improvements to the DDPG

The core of the DDPG algorithm lies in its interaction with the environment. Specifically, the algorithm determines an action $a(t)$ conditioned on the state $s(t)$, and the action is then applied to the environment to obtain the reward $r(t)$ and the next state $s(t+1)$. This process forms an experience tuple $(s(t),a(t),r(t),s(t+1))$, which is stored and randomly sampled to update the neural networks in the algorithm, ultimately seeking the optimal policy function that maximizes the cumulative reward.

3.1. Neural Network Parameter Updates in the DDPG

The neural networks of the DDPG constructed consist of the Actor network $\mu (s(t)|\theta )$, the target-Actor network ${\mu }^{\prime }(s(t)|{\theta }^{\prime })$, the Critic network $Q(s(t),a(t)|\omega )$, and the target-Critic network $Q^{\prime }(s(t),a(t)|{\omega }^{\prime })$, where $\theta$, ${\theta }^{\prime }$, $\omega$ and ${\omega }^{\prime }$ denote the parameters of the corresponding networks. The Actor network approximates the policy function. The Critic network approximates the action-value function, thereby evaluating the current policy function. Its output represents the predicted cumulative reward, i.e., the Q-value, that can be obtained by executing action in state sand subsequently following a specific policy. The target-Actor network and target-Critic network correspond to the Actor network and Critic network, respectively, providing stable targets for their updates and thereby enhancing the training stability of the DDPG algorithm.

Updates to the Critic network’s weights are performed through the minimization of the temporal-difference (TD) error, where the loss function is defined as:

$$L(\omega )={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N{(y^i(t)-Q(s^i(t),a^i(t)|\omega ))}^2}$$

(13)

$$y^i(t)=r^i(t)+\gamma Q^{\prime }(s^i(t+1),{\mu }^{\prime }(s^i(t+1)|{\theta }^{\prime })|{\omega }^{\prime })$$

(14)

where $N$ denotes the batch size of sampled experiences, $Q(s^i(t),a^i(t)|\omega )$ denotes the output of the target Critic network for the i-th experience, $y^i(t)$ denotes the target Q-value, and it is calculated using the immediate reward and the output of the target Critic network at the next time step, providing a closer approximation to the true Q-value, $s^i(t)$, $a^i(t)$, $r^i(t)$ and $s^i(t+1)$ denote the state, action, reward, and next state of the i-th experience, respectively, $\gamma$ denotes the discount factor, representing the importance of future rewards, ${\mu }^{\prime }(s^i(t+1)|{\theta }^{\prime })$ and $Q^{\prime }(s^i(t+1),{\mu }^{\prime }(s^i(t+1)|{\theta }^{\prime })|{\omega }^{\prime })$ denote the outputs of the target Actor network and the target Critic network for the i-th experience, respectively.

The Actor network parameters are optimized by maximizing the action-value function, governed by the following loss formulation:

$$L(\theta )=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^{N}Q(s^i(t),\mu (s^i(t)|\theta )|\omega )}$$

(15)

Target networks are softly updated, with the update formula given as follows:

$${\theta }^{\prime }\leftarrow \tau \theta +(1-\tau ){\theta }^{\prime }$$

(16)

$${\omega }^{\prime }\leftarrow \tau \omega +(1-\tau ){\omega }^{\prime }$$

(17)

where $\tau$ denotes the soft update coefficient, governing how frequently the target networks are updated.

Through this iterative process, the DDPG algorithm continuously interacts with the environment to update the parameters of each network, ultimately searching for the optimal policy function.

3.2. Mathematical Programming Guided Imitation Learning

In the DDPG training phase, exploration is achieved by injecting random noise into the Actor’s actions. The resulting experiences are then archived in a replay buffer to facilitate parameter updates via sampling. However, the uncertainties present in the park-level multi-source synergistic power supply system increase the complexity of the optimal scheduling problem. The exploration based on random noise cannot effectively adapt to the dynamically changing environment, resulting in a replay buffer that accumulates a large number of ineffective experiences during the early training stage, which in turn adversely affects the learning efficiency and final performance of the algorithm.

This paper proposes a mathematical programming guided imitation learning (MPGIL) method. Demonstration experiences are obtained via mathematical programming and sampled together with regular experiences to construct the experience data required for training, thereby guiding the learning process of the algorithm. The specific procedure is as follows:

(1) The mathematical programming method based on the IPOPT solver is employed to solve the optimal scheduling problem of the park-level multi-source synergistic power supply system constructed in Section 2, thereby obtaining action trajectories with high cumulative reward. During the solving process, complete daily historical data of photovoltaic generation and load are used as inputs, and optimal scheduling is performed for each training day with a 24 h scheduling horizon, subject to the constraints specified in Equations (7)–(12). During the training and application of the algorithm, data corresponding to each time point for photovoltaic generation and load are incrementally read as algorithm input. The action trajectories are then augmented, and the final trajectories are expressed as follows:

$${\tilde{A}}_D=\{{\tilde{a}}_D(t)|{\tilde{a}}_D(t)=a_D(t)+\tilde{\epsilon },t=1,2,\dots ,T\}$$

(18)

where $a_D(t)$ denotes the solution obtained by the solver at time step t, ${\tilde{a}}_D(t)$ denotes the data-augmented result, and $\tilde{\epsilon }$ follows a normal distribution with zero mean and variance $\tilde{\sigma }$.

(2) The augmented results are applied to the constructed park-level multi-source synergistic power supply system environment to obtain the demonstration experiences corresponding to the action trajectories, denoted as $(s(t),{\tilde{a}}_D(t),r(t),s(t+1))$. In this process, based on the constraint conditions, explicit clamping is applied to clamp the values of actions and states, thereby ensuring the feasibility of the demonstration experiences.

(3) A dual replay buffer is established to separately store demonstration experiences and regular experiences generated during the algorithm’s exploration, preventing the accumulation of regular experiences from affecting the demonstration experiences. During the algorithm training process, experiences are independently and randomly sampled from the two replay buffers at different ratios. The demonstration experiences and regular experiences are combined to form the experience samples required for training. The loss function for the Critic network is derived with reference to Equations (13) and (14), as shown in Equation (19):

$$L(\omega )={\displaystyle \frac{1}{N}}({\displaystyle \sum _{i=1}^{(1-k)N}{(y^i(t)-Q(s^i(t),a^i(t)|\omega ))}^2}+{\displaystyle \sum _{j=1}^{kN}{(y^j(t)-Q(s^j(t),{\tilde{a}}_D^j(t)|\omega ))}^2})$$

(19)

where $k$ denotes the sampling ratio of demonstration experiences.

During training, the sampling ratio of demonstration experiences is dynamically adjusted to achieve a balance between demonstration-guided learning and regular experience-based exploration. The adjustment is performed as follows:

$$k=k-\Delta k$$

(20)

where $\Delta k$ denotes the decrement step of the demonstration experience sampling ratio.

By setting a relatively high initial sampling ratio for demonstration experiences and gradually decreasing it during training, the algorithm can effectively learn from demonstration experiences in the early stage while maintaining exploration capability in the later stage, preventing over-fitting and the consequent reduction in policy adaptability.

3.3. Oscillation-Decaying Ornstein–Uhlenbeck Noise

The DDPG algorithm uses a deterministic policy, meaning that the Actor network outputs a deterministic value. Therefore, exploration is typically achieved by gradually reducing the intensity of the added random noise to balance exploration and exploitation during training. However, this approach may cause the exploration capability to decline too rapidly, potentially trapping the policy function in local optima.

This paper proposes an oscillation-decaying Ornstein–Uhlenbeck noise (OD-OU). By modulating the noise intensity through an oscillation-decaying process, the exploration and exploitation strengths of the algorithm alternate over time. This allows the algorithm to periodically enhance exploration even in the later stages of training, thereby avoiding convergence to local optima. The OD-OU noise $N(t)$ is defined as:

$$N(t)=N(t-1)+\zeta ({\mu }_{\mathrm{mean}}-N(t-1))+\sigma (n_e){\epsilon }_t$$

(21)

$$\sigma (n_e)=(1+\alpha \sin ({\displaystyle \frac{2\pi \cdot n_e}{\beta }}))\max ({\sigma }_{\max }-\lambda n_e,{\sigma }_{\min })$$

(22)

where ${\mu }_{\mathrm{mean}}$ denotes the mean of noise, $\zeta$ denotes the mean reversion rate, ${\epsilon }_t$ follows a standard normal distribution, $\alpha$ denotes the oscillation amplitude of the noise intensity, $\beta$ denotes the oscillation period of the noise intensity, ${\sigma }_{\max }$ and ${\sigma }_{\min }$ denote the maximum and minimum noise intensities, $\lambda$ denotes the noise intensity decay coefficient, and $n_e$ denotes the training episode number.

3.4. MPG-OD-DDPG Algorithm

The structural framework of the MPG-OD-DDPG algorithm is illustrated in Figure 2.

Figure 2. The structural framework of the MPG-OD-DDPG (⊗ donates the superposition operation).

As shown in Figure 2, the MPG-OD-DDPG algorithm incorporates two main improvements over the standard DDPG algorithm:

The MPGIL method is proposed and integrated. Demonstration experiences are obtained using a mathematical programming approach, and the networks are updated by sampling both demonstration and regular experiences. This guides the training process and improves the learning efficiency of the algorithm.

The OD-OU noise is injected into the actions to facilitate exploration. Through the oscillation-decaying process of the noise intensity, the DDPG algorithm maintains exploration capability even in the later stages of training, enhancing the global optimization ability of the algorithm and preventing the policy function from being trapped in local optima.

The MPG-OD-DDPG algorithm addresses the issues of low learning efficiency in the early training stage and convergence to local optima in the later training stage through the MPGIL method and the OD-OU noise, with its training process illustrated in Algorithm 1.

Algorithm 1. MPG-OD-DDPG algorithm
1	Initialize the Actor network, Critic network, target Actor network and target Critic network
2	Initialize the regular experience replay buffer B1 and the demonstration experience replay buffer B2
3	Obtain the final action trajectories using IPOPT solver and data augmentation
4	Apply the action trajectories, and store the demonstration experiences in B2
5	for i = 1 to max_episode do
6	Initialize the state of the environment
7	Update the demonstration experience sampling ratio according to Equation (20)
8	Update the noise intensity of the OD-OU noise according to Equation (22)
9	for t = 1 to T do
10	Select an action based on the Actor network and the OD-OU noise
11	Execute the action and receive a reward and a new state
12	Store the regular experience in B1
13	Sample from B1 and B2, respectively, according to the demonstration experience sampling ratio
14	Update the Critic network according to Equations (14) and (19)
15	Update the Actor network according to Equation (15)
16	Update the target network according to Equations (16) and (17)
17	end for
18	end for

4. Optimal Scheduling Process of the Park-Level Multi-Source Synergistic Power Supply System

When applying the MPG-OD-DDPG algorithm for optimal scheduling of the park-level multi-source synergistic power supply system, the design of the state space, action space, and reward function is as follows:

(1) State Space

The state space consists of environmental variables that can influence action decisions. For the park-level multi-source synergistic power supply system, the state space includes PV power, user load demand, time-of-use electricity price, and the SOC of the ESS, as expressed in (21):

$$S=\{P_{\mathrm{pv}}(t),P_{\mathrm{load}}(t),\rho (t),k_{\mathrm{SOC}}(t)\}$$

(23)

(2) Action Space

The action space is constructed based on the decision variables in the optimization problem. In the park-level multi-source synergistic power supply system, the charging/discharging power of the ESS is selected as the action variables. The action space can be defined as:

$$A=\{P_{\mathrm{ess}}(t)\}$$

(24)

To ensure that the system always satisfies the power balance constraint, the grid electricity purchasing power and the curtailed PV power are not included as part of the action space and are calculated as follows:

$$P_{\mathrm{grid}}(t)=\max (0,P_{\mathrm{load}}(t)-P_{\mathrm{pv}}(t)-P_{\mathrm{ess}}(t))$$

(25)

$$P_{\mathrm{curt}}(t)=-\min (0,P_{\mathrm{load}}(t)-P_{\mathrm{pv}}(t)-P_{\mathrm{ess}}(t))$$

(26)

(3) Reward Function

The reward function reflects the immediate reward received after the environment takes an action and is crucial for policy optimization in deep reinforcement learning algorithms. To transform the optimal scheduling objective shown in Equation (1) into the immediate reward information required by the deep reinforcement learning algorithm, its negative value is defined as the core part of the reward function. Meanwhile, to provide denser reward signals, two penalty terms are introduced [25], the reward function is defined as follows:

$$r_t=-(C_{\mathrm{grid}}(t)+C_{\mathrm{ess}}(t)+w_1{(E_{\mathrm{ess}}(t)-{\displaystyle \frac{(k_{\mathrm{SOC}}^{\min }+k_{\mathrm{SOC}}^{\max })E_{\mathrm{ess}}^{\max }}{2}})}^2+w_2{P}_{\mathrm{ess}}{(t)}^2)$$

(27)

$$E_{\mathrm{ess}}(t)=k_{\mathrm{SOC}}(t)E_{\mathrm{ess}}^{\max }$$

(28)

where $E_{\mathrm{ess}}(t)$ denotes the energy stored in the ESS at time step t, and $E_{\mathrm{ess}}^{\max }$ denotes the energy capacity of the ESS, the penalty term ${(E_{\mathrm{ess}}(t)-{\displaystyle \frac{(k_{\mathrm{SOC}}^{\min }+k_{\mathrm{SOC}}^{\max })E_{\mathrm{ess}}^{\max }}{2}})}^2$ maintains the operational flexibility of the ESS in the future by constraining the state of charge, the penalty term $P_{\mathrm{ess}}{(t)}^2$ suppresses deep charge–discharge cycles that can accelerate ESS aging by constraining the power of the ESS, $w_1$ and $w_2$ denote weighting coefficients for the penalty terms, used to adjust their impact on the reward function to prevent them from overshadowing the primary cost-related reward signal.

In the DRL framework of this paper, the environment of the park-level multi-source synergistic power supply system is constructed based on historical data. Historical data is read in daily cycles to interact with the DRL algorithm. The state transition of the environment is determined by both an internal model and external historical data: the SOC of the ESS is determined by its previous state and the executed action, while the PV output power and load are obtained by loading historical data. The electricity price is determined by the time-of-use electricity price.

The constraints are enforced through explicit clamping. Specifically, the allowable ranges for the environmental state variables (SOC, grid electricity purchasing power, curtailed PV power) and the action variable (ESS charge/discharge power) are determined according to Equations (7)–(12). If a variable’s value exceeds its constraint range, it is clamped to the boundary of that range, thereby ensuring strict enforcement of the constraints.

At each time step, explicit clamping is performed following these steps: (1) clamp the charging/discharging power, (2) update the SOC, (3) Check whether the SOC exceeds the constraint limits. If so, clamp the SOC and calculate feasible charging/discharging power according to the upper and lower SOC bounds and the previous SOC, (4) calculate the grid purchasing power and curtailed PV power.

The optimal scheduling process of the park-level multi-source synergistic power supply system based on the MPG-OD-DDPG algorithm is illustrated in Figure 3.

Figure 3. Application procedure of optimal scheduling for the park-level multi-source synergistic power supply system based on the MPG-OD-DDPG algorithm.

When applying the MPG-OD-DDPG algorithm to the optimal scheduling problem of the park-level multi-source synergistic power supply system, the process is divided into two stages: the policy training stage and the policy application stage.

During the policy training stage, experiences are collected through interactions between the algorithm and the system. Combined with the MPGIL method and the OD-OU noise, the parameters of the neural networks in the algorithm are updated to search for the optimal policy function.

In the policy application stage, the Actor network parameters corresponding to the policy function obtained during training are loaded, and the system executes actions based on its actual state. No further updates are made to the Actor network parameters during this stage.

5. Simulation Analysis

5.1. Parameter Settings

In this study, the park-level multi-source synergistic power supply system shown in Figure 1 is used for the case study. The scheduling horizon covers 24 h, with a time resolution of 1 h. PV output power and load data from March 2021 of a certain campus are used. The photovoltaic system has an installed capacity of approximately 470 kWp, and the average daily load is about 4.2 MWh. The historical data spans 21 days. The first 14 days serve as the training set, used for obtaining demonstration experiences and training the deep reinforcement learning algorithms. The data from the subsequent 7 days, which is unseen, is used as the test set to evaluate the optimal scheduling results of the algorithm. The parameters of the ESS are listed in Table 1.

Table 1. Parameters of the park-level multi-source synergistic power supply system.

Parameter	Value	Parameter	Value
Maximum SOC	90%	Capacity of the ESS	1000 kWh
Minimum SOC	10%	Battery replacement cost	1,000,000 CNY
Maximum discharging power	500 kW	Charging efficiencies	0.95
Maximum charging power	500 kW	Discharging efficiencies	0.95

According to reference [22], the DOD-cycle life fitting curve for the ESS is obtained as:

$$N_{\mathrm{life}}(k_{\mathrm{DOD}})=21870\exp (-1.957k_{\mathrm{DOD}})$$

(29)

The electricity purchase price follows the time-of-use rates shown in Table 2.

Table 2. Time-of-use electricity price schedule.

Time	Electricity Price/CNY/kWh
10:00–14:00; 18:00–21:00	0.90
07:00–09:00; 15:00–17:00; 22:00–23:00	0.55
00:00–06:00	0.295

The proposed MPG-OD-DDPG algorithm is implemented in Python 3.10 using the PyTorch 2.4.1 framework. The key parameters of the algorithm have a significant impact on its learning efficiency and final performance. The learning rate determines the fitting speed of the Actor network and Critic network in the algorithm. A larger learning rate leads to faster fitting but reduces the stability of the training process. Parameters such as the noise intensity determine the exploration capability during training. Higher noise intensity allows for a wider range of noise variation, enabling more sufficient exploration, but slows down the convergence speed. The algorithm parameters are determined through repeated experiments [26]. The actor and critic networks are configured as three-layer networks with 128 neurons in the hidden layers and ReLU activation functions. Network parameters are updated during training using the Adam optimizer. The detailed algorithm parameter settings are listed in Table 3.

Table 3. Parameters of the MPG-OD-DDPG algorithm.

Parameter	Value	Parameter	Value
Actor learning rate	3 × 10−5	Mean reversion rate of OU noise	0.7
Critic learning rate	3 × 10−4	Maximum noise intensity	0.5
Batch size	256	Minimum noise intensity	0.2
Replay buffer capacity	30,000	Noise intensity decay coefficient	2 × 10−4
Discount factor	0.99	Noise intensity oscillation amplitude	0.3
Soft update coefficient	0.005	Noise intensity oscillation period	20
Mean of OU noise	0	Gaussian noise variance	0.1
Penalty term weighting coefficient w1	1 × 10−4	Penalty term weighting coefficient w2	1 × 10−4

During the demonstration experience construction phase of the MPG-OD-DDPG algorithm, 14 action trajectories are first obtained based on the training dataset. These are then augmented to 1400 trajectories through data augmentation. Finally, the action trajectories are incrementally input into the deep reinforcement learning environment to obtain demonstration experiences, with their feasibility ensured via explicit clamping.

During algorithm training, the sampling ratio k determines the number of samples drawn from the demonstration experiences and the regular experiences. Independent random sampling is performed on the demonstration experiences and regular experiences separately to constitute the training samples. After each training episode, the demonstration experience sampling ratio is updated. Its initial value is 1, with a decrement step size of 0.0004, decreasing to 0 after 2500 training episodes.

The maximum number of training episodes is 5000. Once this maximum is reached, the training concludes and the parameters of the Actor network are saved.

5.2. Training Process Analysis

The variation in cumulative rewards during the training process of the MPG-OD-DDPG algorithm is shown in Figure 4. The reward values are relatively low at the initial stage of training. As training progresses, the network parameters are continuously updated to search for the optimal policy, and the reward values gradually increase. After approximately 4000 training episodes, the reward values stabilize, indicating that the training process has nearly converged. The algorithm consistently converges across multiple runs with a small variance. The converged average daily operating cost is 1138.28 ± 3.73 CNY. Furthermore, the hyperparameters listed in Table 3 were obtained through repeated testing. Slight modifications to them do not significantly impact the algorithm’s performance, which verifies the convergence stability of the algorithm.

Figure 4. Cumulative reward variation curve.

5.3. Analysis of Optimal Scheduling Results

Considering that the PV system in this power supply system is significantly affected by weather conditions, the MPG-OD-DDPG algorithm is tested under three scenarios: sunny, cloudy, and rainy. The optimal scheduling results are shown in Figure 5, Figure 6 and Figure 7 and Table 4.

Figure 5. Optimal scheduling results under sunny conditions.

Figure 6. Optimal scheduling results under cloudy conditions.

Figure 7. Optimal scheduling results under rainy conditions.

Table 4. Results for different scenarios.

Scenario	Total Grid Import Energy/kWh	Total Curtailed PV Energy/kWh	Min/Max SOC/%	Operating Cost/CNY
Sunny	2039.51	292.06	29.52/81.49	1120.52
Cloudy	2925.01	109.37	23.48/63.65	1615.89
Rainy	3047.34	14.56	10/47.35	1838.32

As shown in Figure 5, under sunny conditions, from 00:00 to 06:00, both PV output and the SOC of the ESS are low, and the grid electricity price is at the off-peak level. During this period, the ESS charges while the user load demand is met by the power grid. From 09:00 to 15:00, PV output reaches its peak, and the SOC has not reached its maximum limit. The ESS continues charging, and the park’s user load is primarily supplied by PV output. From 18:00 to 21:00, the grid electricity price is at its peak while PV output declines. The ESS gradually discharges to meet user load demand, thereby reducing grid electricity purchases. During the scheduling period, to maintain the power balance of the power supply system and avoid selling electricity back to the grid, the total curtailed PV energy is 292.06 kWh, accounting for 10.31% of the total PV power generation. The state of charge remains between 29.52% and 81.49%, and no constraint violations occur.

As shown in Figure 6, under cloudy conditions, PV output between 09:00 and 15:00 exhibits higher variability and is lower than in the sunny scenario. In this case, the ESS charges at lower power or even discharges part of its energy to satisfy user load demand, and replenishes energy during subsequent flat-price periods to prepare for high-price periods. During the scheduling period, the total curtailed PV energy is 109.37 kWh, accounting for 5.48% of the total PV power generation. The state of charge remains between 23.48% and 63.65%, and no constraint violations occur.

As shown in Figure 7, under rainy conditions, PV output is almost zero throughout the day, and the park user load demand is primarily supplied by the ESS and the grid. From 00:00 to 06:00, when the grid electricity price is at off-peak levels, the ESS charges. From 10:00 to 14:00, during high-price periods, the ESS discharges in coordination with the grid to meet user load demand. During the scheduling period, the total curtailed PV energy is 14.56 kWh, accounting for 3.28% of the total PV power generation. The state of charge remains between 10% and 47.35%, and no constraint violations occur.

According to the analysis of the scheduling results above, the scheduling policy obtained by the MPG-OD-DDPG algorithm reduces the operating cost of the power supply system by leveraging the time-of-use electricity price and enhancing the PV consumption rate. When the PV generation system is inactive, the algorithm adjusts the charging/discharging power of the ESS based on the time-of-use electricity price: storing energy during periods of low electricity prices and releasing it to replace grid purchases during high-price periods, thereby reducing electricity purchase costs. When the PV generation system is active, the algorithm adjusts the operational state of the ESS according to the PV output power under different weather conditions: charging when PV output is sufficient, thereby increasing the PV consumption rate. This verifies that the MPG-OD-DDPG algorithm can identify a scheduling policy that performs economic optimization based on environmental state information such as electricity prices and PV output power.

5.4. Comparative Experiments and Results Analysis

In order to evaluate the optimization performance of the MPG-OD-DDPG on the operation of the park-level multi-source synergistic power supply system, four comparative algorithms are employed.

The MPG-OD-DDPG algorithm learns from historical data to obtain a scheduling policy—specifically, the Actor network—that can improve the cumulative reward over the entire scheduling horizon, thereby achieving the single-day economically optimal scheduling for the park-level multi-source synergistic power supply system. Therefore, in addition to the deep reinforcement learning method, other classical optimal scheduling methods targeting single-day economic optimization are also adopted for comparison to verify the optimization effectiveness of the proposed algorithm.

First, a comparison is made with real-time online control methods, including the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and the Model Predictive Controller (MPC). The objective function and constraints for MPC are constructed in Section 2, with a rolling horizon of 12 h. The optimization problem within the rolling horizon is solved using IPOPT.

In addition to real-time online control methods, classical day-ahead optimal scheduling methods are used to solve the optimization scheduling problem constructed in Section 2 for a more comprehensive comparison. These include the mathematical programming method based on IPOPT and the Particle Swarm Optimization (PSO) algorithm.

The input data for all the aforementioned algorithms are from the test dataset. The MPG-OD-DDPG and TD3 algorithms use historical data corresponding to each time point as measured values. MPC uses the measured value at the corresponding time point and the forecast values for the subsequent 11 h. IPOPT and PSO use 24 h forecast values as input data. The forecast values are generated by superimposing 5%, 10% and 15% forecast errors onto the historical data from the test dataset. Real-time online control methods and day-ahead optimal scheduling methods differ in their input information: real-time online control methods use measured values at the corresponding time or short-term forecast values, whereas day-ahead optimal scheduling methods employ complete forecast values for the scheduling day. By encompassing different operational paradigms found in the practical optimal scheduling of power supply systems, this study evaluates the performance of the proposed algorithm more comprehensively.

The comparison results of different methods are presented in Table 5, wherein the results for all deep reinforcement learning algorithms are obtained based on five independent runs.

Table 5. Comparison results of different methods.

Method	Average Daily Operating Cost/CNY	Average Daily Energy Storage Degradation Cost/CNY	Average Daily Electricity Purchase Cost/CNY	Average Daily PV Consumption Rate/%
MPG-OD-DDPG	1138.28 ± 3.73	147.15 ± 12.14	991.33 ± 11.09	88.68 ± 0.94
TD3	1176.59 ± 7.51	127.36 ± 17.41	1049.22 ± 11.85	87.23 ± 0.76
MPC (5%)	1242.56	182.99	1059.57	88.24
MPC (10%)	1245.17	182.73	1062.45	88.23
MPC (15%)	1246.65	183.12	1063.53	88.27
IPOPT (5%)	1167.97	160.98	1006.99	87.06
IPOPT (10%)	1178.92	161.82	1017.11	86.55
IPOPT (15%)	1196.19	163.28	1032.91	85.76
PSO (5%)	1283.06	142.37	1140.69	82.54
PSO (10%)	1299.97	135.52	1164.45	79.62
PSO (15%)	1316.66	157.47	1159.19	79.58

PV consumption rate represents, during the scheduling period, the percentage of the total photovoltaic power generation that is actually consumed by the user load demand and the energy storage system, relative to the total available photovoltaic power generation from the PV system. The calculation is as shown in the following equation:

$$\mathrm{PV}\mathrm{consumption}\mathrm{rate}=({\displaystyle \frac{{\displaystyle \sum _{t=1}^T(P_{\mathrm{pv}}(t)-P_{\mathrm{curt}}(t))\Delta t}}{{\displaystyle \sum _{t=1}^T{P}_{\mathrm{pv}}(t)\Delta t}}})\times 100\%$$

(30)

As shown in Table 5, the MPG-OD-DDPG algorithm reduces the average daily operating cost by 3.26% compared with the TD3 algorithm. When considering a 5% prediction error, the average daily operating cost of the MPG-OD-DDPG algorithm is reduced by 8.39% relative to MPC, by 2.54% relative to the IPOPT-based mathematical programming method, and by 11.28% relative to the PSO algorithm. When considering a 10% prediction error, the reductions in average daily operating cost are 8.58%, 3.45%, and 12.44%, respectively. When considering a 15% prediction error, the reductions in average daily operating cost are 8.69%, 4.84%, and 13.55%, respectively.

These results indicate that the MPG-OD-DDPG algorithm achieves a lower average daily operating cost, demonstrating its superior capability to handle the optimal scheduling problem of the park-level multi-source synergistic power supply system under uncertainty, thereby improving the system’s economic performance. The results also show that the IPOPT-based mathematical programming method, MPC, and PSO are sensitive to prediction errors; as the prediction error increases, the discrepancy between the optimized scenario and actual conditions grows, leading to higher average daily operating costs.

In addition, the MPG-OD-DDPG algorithm achieves the highest average daily PV consumption rate among all comparative methods, indicating that it can utilize PV output more effectively, reduce grid electricity purchases, and consequently lower the carbon emissions of the power supply system. Moreover, the algorithm incurs lower average daily energy storage degradation costs, effectively prolonging the lifetime of the ESS.

The superior scheduling results of the MPG-OD-DDPG algorithm are attributed to the guidance capability of the MPGIL for the scheduling policy during the early training stage, and the sustained exploration ability provided by the OD-OU noise during the later training stage. The MPGIL method enables the algorithm to learn economic scheduling strategies for various scenarios more rapidly in the early training phase by constructing and utilizing demonstration experiences. Meanwhile, optimization problems with high uncertainty and nonlinearity may possess multiple local optima. The OD-OU noise, through its oscillation-decaying intensity, allows the algorithm to maintain exploration even in the later stages of training, thereby facilitating the search for a global optimum. In contrast, the baseline DDPG and TD3 algorithms suffer from low sample learning efficiency in the early training stage and insufficient exploration capability in the later stage. While algorithms like MPC, IPOPT, and PSO utilize environmental information over a longer time horizon, they are ultimately limited by their treatment of uncertainty, resulting in suboptimal final scheduling performance.

To evaluate the sensitivity of the MPG-OD-DDPG algorithm to the parameters of the ESS, experiments were conducted by simultaneously adjusting the energy storage capacity and the maximum power. The experimental results are shown in Table 6.

Table 6. Comparison results of different capacities and maximum power.

ESS Capacity/Maximum Power	Average Daily Operating Cost/CNY	Average Daily Energy Storage Degradation Cost/CNY	Average Daily Electricity Purchase Cost/CNY	Average Daily PV Consumption Rate/%
800 kWh/400 kW	1160.58 ± 3.36	133.67 ± 10.74	1026.91 ± 11.67	87.98 ± 0.83
1000 kWh/500 kW	1138.28 ± 3.73	147.15 ± 12.14	991.33 ± 11.09	88.68 ± 0.94
1200 kWh/600 kW	1127.81 ± 4.41	148.82 ± 13.05	978.99 ± 12.26	89.72 ± 0.91

As shown in Table 6, when the capacity and maximum power of the ESS are reduced to 80% of the original settings, although the ESS degradation cost decreases by 9.16%, the grid electricity purchasing cost increases by 3.59%. This is because the energy regulation capability of the ESS is constrained, ultimately leading to a 1.96% increase in the average daily operating cost of the power supply system. Conversely, when the capacity and maximum power of the ESS are increased to 120% of the original settings, the various costs of the power supply system and the PV consumption rate show opposite trends. This is because the additional capacity and charge/discharge power can more effectively utilize the time-of-use electricity price and PV generation to reduce operating costs. This indicates that the MPG-OD-DDPG algorithm can adjust its scheduling policy to address the optimal scheduling problem of power supply systems with different parameters.

In summary, applying the MPG-OD-DDPG algorithm for optimal scheduling of the park-level multi-source synergistic power supply system yields superior overall economic benefits.

5.5. Ablation Experiments and Results Analysis

To verify the effectiveness of the MPGIL and OD-OU noise, ablation experiments are conducted using four methods: DDPG, OD-DDPG, MPG-DDPG, and MPG-OD-DDPG. The variation in reward values during the training process for each method is shown in Figure 8, and the comparison of operating costs is presented in Table 7.

Figure 8. Training reward curves of different methods in the ablation study.

Table 7. Results of ablation studies.

Method	Average Daily Operating Cost/CNY	Total Training Time/s
MPG-OD-DDPG	1138.28 ± 3.73	2006.79 ± 12.37
OD-DDPG	1184.78 ± 5.16	1931.72 ± 10.23
MPG-DDPG	1168.41 ± 8.95	1956.14 ± 9.82
DDPG	1229.32 ± 8.56	1924.61 ± 15.48

As shown in Figure 8, the reward values of the MPG-DDPG algorithm increase most rapidly during the early stage of training, indicating that MPG-DDPG can effectively utilize demonstration experiences to quickly learn action strategies that reduce system operating costs. The OD-DDPG algorithm exhibits a temporary stagnation in reward growth, followed by a subsequent increase, demonstrating its ability to escape local optima and its strong exploration capability. The MPG-OD-DDPG algorithm shows a relatively fast reward increase in the early training stage and achieves the highest and most stable reward in the mid-to-late stages of training, indicating superior learning efficiency and convergence stability. These results validate the efficacy of the suggested enhancements in boosting the algorithm’s learning efficiency and optimizing final performance.

As shown in Table 7, the proposed improvements contribute to enhancing the final performance of the algorithm. Based on the DDPG algorithm, introducing the OD-OU noise reduces the average daily operating cost of the power supply system by 3.62%, indicating that the oscillation-decaying process of the noise intensity enhances the exploration capability of the algorithm, enabling it to learn a policy that yields lower operating costs. Incorporating the MPGIL method decreases the average daily operating cost by 4.95%, demonstrating that demonstration experiences can effectively guide the learning process and positively impact both learning efficiency and final performance.

The proposed MPG-OD-DDPG algorithm integrates both improvements. Although it slightly increases the training time, it achieves the lowest average daily operating cost, reducing it by 7.41% compared with DDPG, thereby validating the effectiveness of the proposed algorithmic enhancements.

6. Conclusions

For the optimal scheduling problem of the park-level multi-source synergistic power supply system, this paper proposes an MPG-OD-DDPG algorithm, which learns the patterns of historical data to construct an optimal operating strategy for the ESS, thereby achieving optimal scheduling of the system. The case study leads to the following conclusions:

The MPG-OD-DDPG algorithm utilizes the MPGIL method to construct and effectively exploit demonstration experiences, addressing the issue of blind learning caused by a large number of ineffective experiences during the early training stage, and thereby significantly improving learning efficiency.

Through the oscillation-decaying process of the OD-OU noise intensity, the MPG-OD-DDPG algorithm maintains strong exploration capability even in the later stages of training, enhancing its ability to search for optimal action policies.

By applying the proposed MPG-OD-DDPG algorithm to the optimal scheduling of the park-level multi-source synergistic power supply system, the bidirectional power regulation capability of the ESS can be effectively utilized for energy regulation, allowing the system to handle uncertainties, improve renewable energy utilization rate, and enhance overall economic performance.

The proposed algorithm, trained based on historical data, is characterized by high sample efficiency and strong optimization capability. It enables real-time scheduling according to system states, laying a foundation for deployment in power supply systems incorporating renewable energy. However, certain limitations remain: how to make the trained actor network quickly adapt to new data and reduce training costs remains a challenge; current research focuses on a single park case, and enhancing its scalability for application to more complex scenarios, such as multiple parks, requires further investigation.