Hybrid Deep Reinforcement Learning Considering Discrete-Continuous Action Spaces for Real-Time Energy Management in More Electric Aircraft

Abstract

The increasing number and functional complexity of power electronics in more electric aircraft (MEA) power systems have led to a high degree of complexity in modelling and computation, making real-time energy management a formidable challenge, and the discrete-continuous action space of the MEA system under consideration also poses a challenge to existing DRL algorithms. Therefore, this paper proposes an optimisation strategy for real-time energy management based on hybrid deep reinforcement learning (HDRL). An energy management model of the MEA power system is constructed for the analysis of generators, buses, loads and energy storage system (ESS) characteristics, and the problem is described as a multi-objective optimisation problem with integer and continuous variables. The problem is solved by combining a duelling double deep Q network (D3QN) algorithm with a deep deterministic policy gradient (DDPG) algorithm, where the D3QN algorithm deals with the discrete action space and the DDPG algorithm with the continuous action space. These two algorithms are alternately trained and interact with each other to maximize the long-term payoff of MEA. Finally, the simulation results show that the effectiveness of the method is verified under different generator operating conditions. For different time lengths T, the method always obtains smaller objective function values compared to previous DRL algorithms, is several orders of magnitude faster than commercial solvers, and is always less than 0.2 s, despite a slight shortfall in solution accuracy. In addition, the method has been validated on a hardware-in-the-loop simulation platform.

1. Introduction

Facing continuous depletion of energy, improving energy efficiency, and reducing energy consumption have become hot discussion topics in the world today. In recent years, similar to efforts to move towards electric vehicles [1], much research has focused on the idea of more electric aircraft (MEA) [2]. The mechanical, hydraulic and pneumatic power systems on traditional aircraft are gradually replaced by electric power systems (EPS), which will be the inevitable trend of future aircraft development [3]. Compared with conventional aircraft, the more electric aircraft not only improves efficiency and reliability, but also reduces the weight, energy consumption and operating costs of the aircraft [4].

Due to the increased size of its power management system, MEA is facing significant challenges in energy management. Currently, extensive research has been carried out by many researchers on the energy management control of MEA. In [5], a decentralised energy management strategy based on improved droop control is proposed for a fuel cell/supercapacitor-based MEA auxiliary power unit. However, it is difficult to achieve accurate power sharing between supercapacitor units of arbitrary capacity due to the effect of the parallel virtual resistors used for state of charge (SoC) recovery. Therefore, [6] proposes a decentralised energy management strategy using a virtual inductor droop strategy and a virtual resistance droop strategy for fuel cell and supercapacitor units respectively, which not only minimises the effect of line impedance on dynamic power distribution but also reduces the order of the system. A novel EPS of MEA containing multiple batteries, PV generators and fuel cells is proposed in [7]. An autonomous energy management strategy with a composite sag scheme is used to achieve dynamic coordination of the three, but the approach does not consider the impact of different scenarios on the energy management of the MEA. To ensure that the MEA operates in different scenarios, [8] uses the optimal droop gain method as a power-sharing method to minimise the losses of EPS of the MEA and a finite state machine to implement the reconfiguration operation.

Although the energy management strategy of MEA is well established in terms of control, less work has been done on its optimization. An adaptive online power management algorithm for hybrid energy storage systems based on a combination of genetic algorithm and Lyapunov optimization method is proposed in [9], aiming at minimizing the power fluctuations of the generator. In [10], the problem is formulated as a multi-objective optimization problem in order to improve the system-level efficiency of the generator and converter. A multi-objective optimization problem similar to [10] was proposed in [11], but with the difference that the lifetime of the energy storage system (ESS) was considered. In the complex MEA power management system, ref. [12] proposed a new distributed framework to reduce the complexity of problem-solving. However, the aforementioned works [9,10,11,12] address only the optimization strategy on the generation side of the MEA, and do not manage the load side in order to prevent overloads. In [13], a mixed integer linear programming solution is used with the objective of load curtailment and the number of generators used, which effectively reduces the number of load curtailment and generator use, while solving the power distribution, contactor switching and battery charging and discharging problems. This decision problem cannot meet the practical requirements of MEA operation due to the possibility of abnormal generator operation. In [14], the energy management problem of MEA is described as a mixed integer quadratic programming problem (MIQP) with the objective of minimizing load shedding and optimal generator power output, and the generator output power schedule, load connection, and battery charging and discharging schedule as decision variables, and is solved using the commercial solver CPLEX. The power allocation and load management optimization strategy of MEA is implemented considering generator and bus damage. In summary, although all of the above methods can achieve higher performance, as the complexity of the system increases, it will lead to increased computational difficulty in the decision-making and control process, creating more uncertainty for real-time energy management. Therefore, it is still a very challenging topic to achieve real-time energy management, considering that most of the current studies on energy management for MEA have paid less attention to the real-time nature of their methods.

With the development of artificial intelligence techniques, reinforcement learning (RL) is gaining attention from researchers and is beginning to be applied in many areas of EPS. Reinforcement learning was introduced in [15] to build the corresponding model for smart island management, and decent results were achieved in terms of computational speed. However, as traditional reinforcement learning methods store Q-values in tabular form, they are prone to dimensional disasters in the face of complex environments and are difficult to apply and promote in practice. As a result, researchers have fused deep learning (DL) with perceptual capabilities and RL with decision-making capabilities to form the deep reinforcement learning (DRL) algorithm [16], which provides solutions for practical energy management scenarios. In [17], a deep Q-learning (DQL) algorithm is developed for real-time energy management in microgrids considering the uncertainty of load demand, renewable energy sources and electricity prices. A new optimization method based on DQL for updating neural networks is designed in [18] and its rapidity and optimality are verified in the energy management of hybrid electric vehicles. However, the DQL algorithm is only suitable for handling discrete actions and needs to be discretized for continuous actions. When the number of controllable devices increases or the granularity of discretization becomes small, the number of action spaces will grow exponentially, making the problem difficult to solve. For the continuous control strategy of a plug-in hybrid electric bus, a deep deterministic policy gradient (DDPG) algorithm was developed in [19], which achieves the optimal energy allocation of the bus on the continuous space. In [20], the energy cost minimization problem for smart homes was studied using the DDPG algorithm. In addition to the above-mentioned energy management, DDPG algorithm is also applied to the energy management of microgrid [21,22] and smart buildings [23,24]. However, current DRL-based methods mainly consider either discrete or continuous action space [25]. For the discrete-continuous hybrid action space, a DDPG-based energy management strategy for residential multi-energy systems is proposed in [26]. When dealing with discrete actions, the continuous outputs are discretized and mapped to discrete actions. However, treating discrete actions as continuous actions may significantly increase the complexity of the action space. Therefore, it is particularly important to design a DRL algorithm that can handle the discrete-continuous hybrid action space.

In this paper, we propose an energy management strategy for MEA based on hybrid deep reinforcement learning (HDRL). The objective function and constraints of the energy management are described based on modelling the characteristics of the generators, buses, loads, and ESS, and the problem is transformed into a Markov decision process (MDP) solution. In this process, the integer and continuous variables of the objective function are considered as discrete action space and continuous action space, respectively. For the hybrid action space, we use the duelling double deep Q network (D3QN) algorithm [27] and the DDPG algorithm to solve the problem and train these two algorithms alternately. The main contributions of this paper are summarized as follows:

(1)

A new MEA energy management model has been developed taking into account the operating state of the generators, the connection priority relationship between the generators and the buses and the shedding priority relationship of the loads.

(2)

For the model of MEA, a hybrid deep reinforcement learning (HDRL) algorithm incorporating D3QN and DDPG is proposed to solve the energy management problem of MEA by taking the generator-bus connection relationship and the load shedding relationship as the discrete action space, and the generator output power and ESS charging and discharging power as the continuous space. The HDRL algorithm proposed in this paper inherits the advantages of DQL in dealing with discrete action space and exploits the advantages of DDPG in dealing with continuous action space. However, in the training process of the HDRL algorithm, we first train on discrete actions and then on continuous actions, alternating between the two until full convergence.

(3)

Simulation studies based on data from the [12] show that the study and numerical analysis of the generator under different operating conditions prove the effectiveness of the proposed HDRL method. The real-time nature of the method is verified by applying it to different time lengths T.

The rest of the paper is organized as follows. Section 2 presents the mathematical formulation of the MEA energy management problem. Section 3 constructs the MDP formulation and proposes an energy management strategy for HDRL. Section 4 describes the case study and simulation results. Section 5 presents the conclusions and future research directions.

2. Mea System Description

2.1. Eps Model

Conventional constant-frequency AC EPS requiring a large number of AC and DC rectifiers to meet power quality requirements results in a significant increase in the weight of the MEA. In [28], a high voltage DC (HVDC) EPS is used, which greatly reduces the number of power conversion units because AC/DC rectifiers are no longer needed, thus achieving a weight reduction effect. In this paper, the EPS structure of MEA consists of an engine, a main generator, an APU, a 270 V HVDC bus, an ESS, various different types of loads and power converters, as shown in Figure 1 [11]. The power generated by the main generator and APU in AC form is fed into the HVDC bus via AC/DC rectifiers and supplied directly to the HVDC load or to the low voltage DC (LVDC) and AC loads via power converters, while access to ESS such as batteries requires bidirectional DC/DC converters to meet peak loads during operation and prevent unessential load shedding.

2.2. Mathematical Formulations

2.2.1. Load and BUS Priority Constraints

Two priority constraints are involved in the considered power system, one is the load connection priority constraint, where the connection priority of critical loads is always higher than that of non-critical loads. During the aircraft flight, critical loads should not be shed in any emergency, while the shed of non-critical loads is related to its priority rules. The second one is the connection priority constraint between each generator and the bus, whose predefined priority relationship is shown in

Table 1. Priority of connection between generator and bus.

Bus	Priority
	1st	2nd	3rd
Bus1	Main Gen1	Main Gen2	APU
Bus2	Main Gen2	Main Gen1	APU

.

2.2.2. Power Balance Constraint

According to EPS of MEA, the generator provides power for the load as the only power source, and the bus plays the role of collecting, distributing and transmitting electrical energy. Considering the power loss, the power balance constraint can be expressed as

$$P_G^k\left(t\right)=\sum _{i\in \mathcal{D}}\frac{P_B^{ki}\left(t\right)}{{\eta }_B^{ki}}$$

(1)

$$P_B^i\left(t\right)=\sum _{k\in \mathcal{G}}{P}_B^{ki}\left(t\right)$$

(2)

$$P_B^i\left(t\right)=P_L^i\left(t\right)+P_{ESS}^i\left(t\right)$$

(3)

$$\left\{\begin{array}{c}\hfill P_L^i\left(t\right)={\displaystyle \sum _{j\in {\mathcal{S}}_C^i}}{y}_C^{ij}\left(t\right)P_C^{ij}\left(t\right)+{\displaystyle \sum _{j\in {\mathcal{S}}_{NC}^i}}{y}_{NC}^{ij}\left(t\right)P_{NC}^{ij}\left(t\right)\hfill \\ \hfill y_C^{ij}\left(t\right)\in \left\{1\right\},y_{NC}^{ij}\left(t\right)\in \{0,1\}\hfill \end{array}\right.$$

(4)

where $P_G^k\left(t\right)$ is the output power of generator k at time t; $P_B^i\left(t\right)$ is the power of bus i at time t; $P_B^{ki}\left(t\right)$ is the transmission power between generator k and bus i at time t; ${\eta }_B^{ki}$ is their transmission efficiency; $P_{ESS}^i\left(t\right)$ is the charging and discharging power of the energy storage system connected to bus i at time t; $P_L^i\left(t\right)$ is the total load demand on bus i at time t; $P_C^{ij}\left(t\right)$, $P_{NC}^{ij}\left(t\right)$ are the critical and non-critical loads; $y_C^{ij}\left(t\right)$, $y_{NC}^{ij}\left(t\right)$ are their connections; $\mathcal{G}$, $\mathcal{D}$ are the sets of generator and bus, respectively; ${\mathcal{S}}_C^i$, ${\mathcal{S}}_{NC}^i$ are the sets of critical and non-critical loads, respectively.

2.2.3. BUS Connection and Generators State Constraints

Since the frequency and phase difference of the power sources are different, only one generator can be connected to each bus at time t. This can be expressed as:

$$\sum _{k\in \mathcal{G}}{\xi }_{ki}\left(t\right)=1$$

(5)

where ${\xi }_{ki}\left(t\right)\in \{0,1\}$ is the state of connection between generator k and bus i at time t.

The current state of the generator directly determines the state of the connection between the generator and the bus, which can be described as

$$U_G^k\left(t\right)=min\{1,\sum _{i\in \mathcal{D}}{\xi }_{ki}\left(t\right)\}$$

(6)

where $U_G^k\left(t\right)$ is the operating state of the generator k at time t, and the operating state of the main generator indirectly affects the connection of non-critical loads.

2.2.4. Generator and BUS Power Capacity Constraints

The output power of each generator should be limited to its power capacity, and the transmission power of each bus should also be no greater than its upper capacity limit.

$$U_G^k\left(t\right)\underline{P_G^k}\le P_G^k\left(t\right)\le U_G^k\left(t\right)\overline{P_G^k},\forall k\in \mathcal{G}$$

(7)

$$0\le P_B^i\left(t\right)\le \overline{P_B^i},\forall i\in \mathcal{D}$$

(8)

$$0\le P_B^{ki}\left(t\right)\le {\xi }_{ki}\left(t\right)\overline{P_B^{ki}}$$

(9)

where $\overline{P_G^k}$, $\underline{P_G^k}$ are the upper and lower limits of the output power of the generator k, $\overline{P_B^i}$ is the upper limit of the power capacity of the bus i, and $\overline{P_B^{ki}}$ is the upper limit of the transmission power.

2.2.5. Generator Optimum Operating Range Constraint

In order to save engine fuel during each phase of flight, the generator needs to be kept in its optimal range, which can be expressed as

$$\underline{P_G^{k,opt}\left(t\right)}\le P_G^{k,opt}\left(t\right)\le \overline{P_G^{k,opt}\left(t\right)}$$

(10)

where $P_G^{k,opt}\left(t\right)$ is the optimal operating point of generator k at time t; $\overline{P_G^{k,opt}\left(t\right)}$, $\underline{P_G^{k,opt}\left(t\right)}$ is the upper and lower limit of the optimal generator power range determined at time t based on the engine state.

2.2.6. ESS Constraints

In this paper, the ESS model considered consists of batteries without internal dynamics. It is a voltage source that depends on the SoC of the battery itself. In energy management of MEA, The ESS can be modelled as a generator that provides positive and negative power to the EPS. Assume that the power $P_{ESS}^i\left(t\right)<0$ of the ESS connected to bus i is in the charging state at time t, and $P_{ESS}^i\left(t\right)\ge 0$ is in the discharging state, but the charging and discharging processes cannot be carried out simultaneously. The amount of change of SoC ($\Delta SoC$) effectively reflects the charging and discharging process of the ESS, and their relationship is as follows.

$$\Delta So{C}_{ESS}^i\left(t\right)=-\frac{{\epsilon }_c{P}_{ESS}^i\left(t\right)\tau }{E_{ESS}^i},P_{ESS}^i\left(t\right)<0$$

(11)

$$\Delta So{C}_{ESS}^i\left(t\right)=-\frac{P_{ESS}^i\left(t\right)\tau }{{\epsilon }_{disc}{E}_{ESS}^i},P_{ESS}^i\left(t\right)\ge 0$$

(12)

where $E_{ESS}^i$ is the total capacity of the ESS; $\tau$ denotes the time interval; ${\epsilon }_c$, ${\epsilon }_{disc}$ are the efficiency factors of the charging and discharging process. The dynamic process of $SoC$ can be described as follows.

$$So{C}_{ESS}^i\left(t+1\right)=So{C}_{ESS}^i\left(t\right)+\Delta So{C}_{ESS}^i\left(t\right)$$

(13)

To avoid overcharging and overdischarging the energy storage system, $P_{ESS}^i\left(t\right)$, $So{C}_{ESS}^i\left(t\right)$ needs to be limited to a predetermined upper and lower limit, i.e.,

$$\underline{P_{ESS}^i}\le P_{ESS}^i\left(t\right)\le \overline{P_{ESS}^i}$$

(14)

$$\underline{So{C}_{ESS}^i}\le So{C}_{ESS}^i\left(t\right)\le \overline{So{C}_{ESS}^i}$$

(15)

2.2.7. MEA Energy Management Optimization Model

In this paper, the main objectives of energy management are to stabilize the generators in the optimal operating range as much as possible; to enforce predefined connection priority rules between each generator and the bus; to minimize non-critical load shedding in case of overload, and to optimize the lifetime of the batteries in the ESS. Thus, the energy management problem can be described as a multi-objective optimization problem.

(1)

The cost function that limits the generator to operate in the optimal range is [29]

$$F_1\left(t\right)=\sum _{k\in \mathcal{G}}{\left(P_G^k\left(t\right)-U{}_G^k\left(t\right)P_G^{k,opt}\left(t\right)\right)}^2$$

(16)

(2)

The cost function for implementing the predefined connection rules between the generator and the bus is

$$F_2\left(t\right)=\sum _{k\in \mathcal{G}}\sum _{i\in \mathcal{D}}{\omega }_{ki}{\xi }_{ki}\left(t\right)$$

(17)

where ${\omega }_{ki}$ is the penalty factor and the connection priority decreases with the increase of the penalty factor.

(3)

The cost function for reducing non-critical load shedding is

$$F_3\left(t\right)=\sum _{i\in \mathcal{D}}\sum _{j\in {\mathcal{S}}_{NC}^i}{\omega }_{ij}(1-y_{NC}^{ij})$$

(18)

where ${\omega }_{ij}$ is the penalty factor and the priority of load shedding decreases with the increase of the penalty factor.

(4)

Considering the cost function of the lifetime of the battery in ESS is

$$F_4\left(t\right)=\sum _{i\in \mathcal{D}}\frac{\left|P_{ESS}^i\left(t\right)\right|}{P_{ESS}^{i,cap}}$$

(19)

where $P_{ESS}^{i,cap}$ is the maximum rechargeable and rechargeable power of ESS.

Based on the above cost function and constraints, the energy management problem of MEA can be formulated as

$$\begin{array}{cc}\hfill & \hfill \underset{{\xi }_{kq}\left(t\right),y_{NC}^{ij}\left(t\right)}{\underset{P_G^k\left(t\right),P_{ESS}^i\left(t\right)}{min}}J=\sum _{t=1}^T\left({\alpha }_1{F}_1\left(t\right)+{\alpha }_2{F}_2\left(t\right)+{\alpha }_3{F}_3\left(t\right)+{\alpha }_4{F}_4\right)\hfill \\ \hfill & \hfill \mathrm{s}.\mathrm{t}.{\alpha }_1+{\alpha }_2+{\alpha }_3+{\alpha }_4=1\hfill \\ \hfill & \hfill Constraints\left(1\right)\sim \left(15\right)\hfill \end{array}$$

(20)

where T is the total time of MEA energy management; ${\alpha }_1$, ${\alpha }_2$, ${\alpha }_3$, ${\alpha }_4$ are the weight share coefficients.

This research solution can obtain the optimal output power of the generator during the flight of MEA and effectively extend the life of the battery, which is economically beneficial to the power system of MEA. It is not difficult to find that the constructed energy management problem is a MIQP problem, which can be solved by many existing mathematical programming solvers such as CPLEX or Gurobi. Although the optimal solution can be found using this method under certain conditions, the computation time increases exponentially as the number of integer variables included in the NP-hard problem increases. Therefore, we propose a DRL-based optimization method for real-time energy management without losing optimality.

3. Hdrl-Based Solution

3.1. Mdp Formulation

In DRL, the agent is constantly interacting with and learning from the external environment. In short, at each time t the agent chooses the appropriate action $a_t$ based on the state $s_t$ of the external environment, which acts on the external environment, whereupon the environment returns a reward $r_t$ and transitions to a new state $s_{t+1}$. By repeating the above process, the agent gains the experience of “trial and error” in interacting with the environment, and finds the optimal strategy for solving the problem in the experience. In essence, this process is called MDP, as shown in Figure 2.

In this paper, we need to construct the energy management problem of MEA as an MDP model with state space, action space and reward function.

(1) State space: For the MEA model, the state information provided to the agent by the external environment at t time generally contains three categories. (1) The operating state $U{}_G^k\left(t\right)$ of the generators; (2) The load demand $P_L^i\left(t\right)$ of the MEA system, which needs to be normalized; (3) The charge state $So{C}_{ESS}^i\left(t\right)$ of ESS. Therefore, the state space of MEA is defined as

$$s_t=\{U_G^k\left(t\right),P_L^i\left(t\right),So{C}_{ESS}^i\left(t\right)\},\forall k\in \mathcal{G},\forall i\in \mathcal{D}$$

(21)

(2) Action space: From the energy management objective of the MEA, the control actions generated by the agent at t moments are (1) the output power $P_G^k\left(t\right)$ of the generator; (2) the charging and discharging power $P_{ESS}^i\left(t\right)$ of the ESS; (3) the connection strategy ${\xi }_{ki}\left(t\right)$ of the generator to the bus; (4) the shedding strategy $P_{NC}^{ij}\left(t\right)$ of non-critical load. Therefore, the action space of the MEA is as follows.

$$a_t=\{a_t^c=\{P_G^k\left(t\right),P_{ESS}^i\left(t\right)\}\cup a_t^d=\{{\xi }_{ki}\left(t\right),y_{NC}^{ij}\left(t\right)\}\},\forall k\in \mathcal{G},\forall i\in \mathcal{D},\forall j\in {\mathcal{S}}_{NC}^i$$

(22)

where $a_t^c$, $a_t^d$ are the continuous action space and discrete action space, respectively.

(3) Reward function: For the discrete-continuous action space, we designed two different reward mechanisms. Combining Equation (20), given the current state $s_t$ and action $a_t$, the reward function from the current state $s_t$ to the next state $s_{t+1}$ is defined as

$$r_t^d(s_t,a_t^d)=-({\alpha }_2{F}_2\left(t\right)+{\alpha }_3{F}_3\left(t\right)+{\rho }^d\left(t\right))$$

(23)

$$r_t^c(s_t,a_t^c)=-({\alpha }_1{F}_1\left(t\right)+{\alpha }_4{F}_4\left(t\right)+{\rho }^c\left(t\right))$$

(24)

Since the objective of RL is to maximize the reward function, and the energy management of MEA in this paper is a minimization problem, they need to be inverted. Equation (23) contains the connection function of the generator to the bus, the shedding function of the uncritical load and the penalty term ${\rho }^d\left(t\right)$. For Equation (24), it includes the quadratic function of the output power of the generator and its optimal operating point, the lifetime function of the battery and the penalty term ${\rho }^c\left(t\right)$. When constraints (1)–(15) are satisfied ${\rho }^c\left(t\right),{\rho }^d\left(t\right)=0$, otherwise a sufficiently large constant will be assigned to ${\rho }^c\left(t\right),{\rho }^d\left(t\right)$. If the value is small, the agent may tolerate behaviour that does not satisfy the constraint.

3.2. HDRL

In the above MDP formulation, the energy management problem of MEA can be viewed as a control problem with discrete-continuous mixed action. However, existing DRL algorithms deal with such problems by either discretizing the continuous space or converting the discrete space into a continuous space, and these methods can significantly increase the complexity of the action space. Therefore, an HDRL algorithm combining D3QN and DDPG is proposed in this paper to solve the MEA energy management problem of hybrid control.

3.2.1. D3QN

General Q-learning is a widely used RL algorithm for solving small state-action space problems. In this paper, the agent determines the action $a{t}^d$ by a policy based on the current state $st$. The expected gain of the Q-value function can be expressed as

$$Q(s_t,a_t^d)=Q(s_t,a_t^d)+\gamma [r(s_t,a_t^d)+\mu \underset{a_{t+1}^d}{max}Q(s_{t+1},a_{t+1}^d)-Q(s_t,a_t^d)]$$

(25)

where $\gamma$ is the learning rate and $\mu$ is the discount factor. Since the algorithm stores state-action values in the form of tables, it is only applicable to the low-dimensional state-action space.

For dealing with the high-dimensional state-action space, a neural network is used to approximate the DQN algorithm of the Q-value function. In DQN, the Q-value function is defined by $Q(s_t,a_t^d;\theta )$ to approximate $Q(s_t,a_t^d)$, where $\theta$ is the training weight of the network. DQN has two networks with the same structure, one is the target network, which is used to calculate the target Q value, and the other is the estimation network, which is used to estimate the Q value of the current state. The goal of DQN is to train suitable weights to minimize the loss of the target Q value with respect to the current state Q value.

$$L\left(\theta \right)=E[{(y_t^d-Q(s_t,a_t^d;\theta ))}^2]$$

(26)

where $y_t^d$ is the Q-value of the target network, which can be expressed as

$$y_t^d=r(s_t,a_t^d)+\mu \underset{a_{t+1}^d}{max}Q(s_{t+1},a_{t+1}^d;{\theta }^{\prime })$$

(27)

where ${\theta }^{\prime }$ is the weight of the target network, which is the parameter replicated from the estimated network at regular intervals.

The DQN algorithm selects and evaluates the Q-value function in terms of the maximum action value, which may lead to the overestimation problem. To solve this problem, the Double DQN algorithm was proposed in [30] to decouple the action selection and evaluation of the target network, i.e., instead of finding the maximum Q value directly in the target network, the Double DQN selects the action with the maximum Q value from the estimated network.

$$y_t^d=r(s_t,a_t^d)+\mu Q(s_{t+1},\underset{a_{t+1}^d}{argmax}Q(s_{t+1},a_{t+1}^d;\theta );{\theta }^{\prime })$$

(28)

In order to guarantee the dominance of action selection in a particular state, the Dueling DQN algorithm was proposed in [31]. The structure of the Dueling network is shown in Figure 3. The main difference with the structure of the DQN network is that the states do not output Q values directly after the hidden layer, but output the state value function $V\left(s_t\right)$ and the action dominance function $A(s_t,a_t^d)$, which is then coupled to each action Q value function and can be expressed as

$$Q(s_t,a_t^d)=V\left(s_t\right)+A(s_t,a_t^d)$$

(29)

where the state value function $V\left(s_t\right)$ is the degree of the state, and the action dominance function $A(s_t,a_t^d)$ is the degree of a given action relative to other actions in that state.

In the constructed network structure, it is difficult to determine the unique $V\left(s_t\right)$, $A(s_t,a_t^d)$ by learning $Q(s_t,a_t^d)$, which will lead to poor training of the network, so Equation (29) can be redescribed as

$$Q(s_t,a_t^d)=V(s_t)+(A(s_t,a_t^d)-\frac{1}{\left|A\right|}\sum _{a_t^{d^{\prime }}\in A}A(s_t,a_t^{d^{\prime }}))$$

(30)

where $\left|A\right|$ denotes the number of all executable actions and A denotes the set containing all executable actions.

In this paper, the D3QN algorithm is constructed on Double DQN by using the framework of Dueling DQN in dealing with discrete action space. It combines the features of Double DQN and Dueling DQN to solve the Q-value overestimation problem while fitting to obtain more accurate Q-values.

3.2.2. DDPG

For continuous motion control, we use the DDPG algorithm based on the actor-critic architecture proposed by Deepmind. In the DDPG network structure, the actor estimation network $\beta$ is used to approximate the policy function, the input is the state, the output is the deterministic action, and the neural network parameters are ${\theta }^{\beta }$. The critic estimation network $\psi$ is used to evaluate the goodness of the current state action, its input is the state and action, the output is the Q value of the state action, and the neural network parameters are ${\theta }^{\psi }$. The critique estimation network is essentially a fit to the Q function, similar to Equation (25).

To improve the stability and convergence of training, the DDPG network structure also introduces the actor target network ${\beta }^{\prime }$ and the critic target network ${\psi }^{\prime }$. They have the same structure as the corresponding estimated network structure, but different parameters, with parameters ${\theta }^{{\beta }^{\prime }}$ and ${\theta }^{{\psi }^{\prime }}$, respectively. The way the parameters of each network in DDPG are updated is shown below.

The parameter ${\theta }^{\beta }$ update of the $\beta$ network is related to its own parameter gradient and the gradient of the generated action-value function of the $\psi$ network. Its expression is given by

$${\nabla }_{{\theta }^{\mu }}J\approx E\left[{\nabla }_{a^c}Q(s,a^c;{\theta }^{\psi })\left|{}_{s=s_t,a^c=\beta \left(s_t\right)}\right.{\nabla }_{{\theta }^{\beta }}\beta (s;{\theta }^{\beta })\left|{}_{s=s_t}\right.\right]$$

(31)

Equation (31) shows that the $\beta$ network has to modify the parameter ${\theta }^{\beta }$ along the direction that will obtain a larger Q value.

The $\psi$ network uses a minimization loss function to update the parameters ${\theta }^{\psi }$.

$$L\left({\theta }^{\psi }\right)=E\left[{(y_t^c-Q(s_t,a_t^c;{\theta }^{\psi }))}^2\right]$$

(32)

where $y_t^c=r_t^c+\mu {\psi }^{\prime }(s_{t+1},{\beta }^{\prime }(s_{t+1};{\theta }^{{\beta }^{\prime }});{\theta }^{{\psi }^{\prime }})$ is the target Q value.

The parameters ${\theta }^{{\beta }^{\prime }}$, ${\theta }^{{\psi }^{\prime }}$ of the ${\beta }^{\prime }$ network and ${\psi }^{\prime }$ network use the soft update method. The expressions are

$$\begin{array}{c}\hfill {\theta }^{{\beta }^{\prime }}\leftarrow \sigma {\theta }^{\beta }+\left(1-\sigma \right){\theta }^{{\beta }^{\prime }}\hfill \\ \hfill {\theta }^{{\psi }^{\prime }}\leftarrow \sigma {\theta }^{\psi }+\left(1-\sigma \right){\theta }^{{\psi }^{\prime }}\hfill \end{array}$$

(33)

where $\sigma \ll 1$ makes the realistic network parameters vary less, training more stable and easy to converge.

3.2.3. HDRL Algorithm Process

The HDL algorithm proposed in this paper uses D3QN to control discrete actions and DDPG to control continuous actions to achieve hybrid control between generators, buses, loads, and ESS. The specific HDRL algorithm process is shown in Figure 4. D3QN and DDPG interact with the environment independently, and they obtain the same state from the environment and update to the next state by their respective actions and rewards. In our design, these two algorithms are trained alternately and interact with each other. In simple terms, if one is training, the other one will be fixed and act as part of the environment.

Considering that the power distribution of the MEA is related to the connection state of the generator, bus and load, we first need to train the D3QN and then the DDPG, the complete algorithm procedure is shown in Figure 5. Here, an empirical replay pool is introduced to store the data and to make effective use of the historical data by taking small batches of samples for training. To deal with the balance between exploration and exploitation, a $\epsilon$-greedy strategy is used, which operates in detail as agent randomly selects the action $a_t$ with probability $\epsilon$ and selects $a_t^d=\underset{a_t^d}{argmax}Q(s_t,a_t^d;\theta )$ and $a_t^c=\beta (s_t;{\theta }^{\beta })+{\mathcal{N}}_t$, where ${\mathcal{N}}_t$ is the Gaussian noise with zero mean.

4. Case Study

The simulation is based on TensorFlow deep learning framework and Python 3.6, with Intel(R) Core(TM) i5-1035G1 CPU and 16 GB RAM.

4.1. System Setup

The MEA system tested consisted of two main generators and one APU, two 270V HVDC buses, and each bus contained three critical loads, three non-critical loads, and an ESS, as shown in Figure 1. The parameter settings of the system are summarized in detail in

Table 2. System parameters.

Parameters	Value	Parameters	Value
$\underline{P_G^k},\overline{P_G^k}$	100, 600	${\eta }_B^{ki}$	0.98
$P_G^{1,opt},P_G^{2,opt},P_G^{3,opt}$	480, 480, 480	${\epsilon }_c,{\epsilon }_{disc}$	0.95
$\overline{P_B^i},\overline{P_B^{ki}}$	600	$So{C}_{ESS}^i\left(0\right)$	0.5
$\underline{P_{ESS}^i},\overline{P_{ESS}^i}$	−100, 100	$E_{ESS}^i$	400
$\underline{So{C}_{ESS}^i},\overline{So{C}_{ESS}^i}$	0.2, 1	$P_{ESS}^{i,cap}$	${10}^9$
$\tau ,T$	1, 20	${\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4$	0.001, 0.2495, 0.2495, 0.5

.

4.2. Model Training

In this paper, the load data are obtained from [11], and in the algorithm training process, for D3QN, the initial exploration constant is set to 1 and reduced to 0.05 with continuous iterations, the learning rate is 0.001, the discount factor is 0.9, the empirical replay pool capacity is 500, the random sampling batch size is 32, the target network parameters are replicated from the estimation network every 300 steps, the number of neurons in the network input layer is the number of state space dimensions. The number of neurons in the input layer of the network is the state space dimension, and the middle is a 3-layer hidden layer then connects the Dueling network structure to the output layer, the number of neurons in the hidden layer is 200, 100, 50, and the number of neurons in the output layer is discrete action space dimension. In the DDPG, both the actor-network and the critic network contain the same hidden layer with 256 and 128 hidden neurons, respectively. the activation function uses the ReLu function, while the output layer of the actor-network uses the tanh function. The learning rate is set to 0.001, the discount factor is 0.9, the soft update factor is 0.01, the empirical replay pool capacity is 10,000, and the random sampling batch size is 32.

As shown in Figure 6, it can be seen from the training process that the proposed algorithm converges faster compared to DDPG, and has more stability and convergence compared to DQN + DDPG. In the initial phase of training, the intelligent body explores the environment and the reward values obtained in this phase are relatively small and accompanied by fluctuations. When the training reaches a certain number of rounds, the algorithm starts to converge gradually and the rewards eventually stabilize, because the intelligent body can learn the optimal strategy and choose the appropriate action based on the experience accumulated in the early stages.

4.3. Simulation Results

A neural network model of the well-trained HDRL was saved and invoked to test the model under three different MEA cruising conditions. The results of this method test are then compared with DQN+DDPG, DDPG and Gurobi respectively.

Condition 1: The main generator 1 and main generator 2 operate normally when the APU is not involved in the EPS of the MEA, this is because the main generator is always prioritised over the APU in connection with the bus during normal operation during the cruise. The variation of the charge/discharge power and SoC of each battery reflects the fact that when the charge/discharge power is less than zero, the battery is charged, otherwise it is discharged. As can be seen from the battery life cost function, the life of the ESS is related to the charge/discharge power. According to Figure 7 it is easy to notice that the output power of both main generator 1 and main generator 2 in the method mentioned above fluctuates above and below the optimal operating point and that the variation in charge and discharge power is not particularly significant within the limits of the charge/discharge power, thus effectively extending the life of the battery.

Condition 2: Main generator 1 is damaged (t = 8 s), and the main generator 2 is normal, as shown in Figure 8. When the main generator 1 fails, according to the principle of load shedding and priority of generators and buses, the main generator 2 should provide all the power and the load is shed in priority at this time. However, in consideration of minimizing load shedding, the APU starts to work to compensate for the impact on the power supply due to the damage to main generator 1.

Condition 3: Damage to main generator 1 (t = 8 s), damage to main generator 2 (t = 13 s). According to Figure 9, at t = 13 s, both main generators fail and only the APU is available in the MEA power system at this time. Due to the output power range limitation, the APU cannot provide power to all the loads. Considering the principle of load shedding priority, part of the load starts to shed, as shown in Figure 10.

In summary, the methods proposed in this paper, DQN + DDPG, DDPG and Gurobi are all capable of good self-adaptation in the face of different environments. To further demonstrate the difference between the proposed HDRL-based method and theirs, we validate it by varying the management length T = 20, 40, 60 and 80, as shown in

Table 3. Comparison of the performance of the proposed method with DQN + DDPG, DDPG and Gurobi.

Management length T (time slots)		20	40	60	80
Number of integer variables		420	840	1260	1680
Number of continuous variables		400	800	1200	1600
Optimal objective value (time slots)	DDPG	11.90	21.69	31.24	38.36
	DQN + DDPG	13.36	22.45	31.56	40.87
	Gurobi	8.32	16.66	24.64	31.34
	Proposed	8.65	18.71	26.41	35.87
Solution time (second)	DDPG	1.73 × 10−1	1.81 × 10−1	1.83 × 10−1	1.90 × 10−1
	DQN + DDPG	1.75 × 10−1	1.78 × 10−1	1.85 × 10−1	1.95 × 10−1
	Gurobi	13.42	109.42	2270.09	7169.95
	Proposed	1.67 × 10−1	1.72 × 10−1	1.86 × 10−1	1.88 × 10−1
Times	Gurobi/Proposed	80.36	636.16	12,204.78	38,138.03

. It is not difficult to find that our proposed method gives better target values compared to DQN+DDPG and DDPG, but slightly larger than the results obtained by the commercial solver Gurobi. The solution time required by Gurobi increases significantly as the management length T increases. At T = 80, Gurobi even takes 7169.95 s to solve this energy management problem, which hardly meets the requirements of MEA in practice. In contrast, our proposed method does not exceed 0.2 s for all four management lengths, which is consistent with the real-time performance of MEA.

4.4. On-Line Hardware-in-the-Loop Test

To further validate the effectiveness of the proposed method experimentally, hardware-in-the-loop (HIL) tests were carried out based on OPAL-RT. In terms of energy management, the OPAL-RT testbed was used to implement power HIL simulations by interfacing the HDRL algorithm part with the energy management part, as shown in Figure 11. In order to demonstrate that the online experiments are also able to effectively implement the power distribution of the MEA, the results of the offline tests and the online experiments were compared when different load requirements were provided, as shown in Figure 12.

In order to assess the adaptive capability of the proposed method under different operating conditions of the main generator, two cases were designed where $G_1$ was not available at t = 7 s and $G_2$ was not available at t = 13 s. The real-time scheduling results were plotted in the laboratory using an oscilloscope. The real-time power dispatch results obtained using this method are shown in Figure 13. We can observe that the output power of all three generators is reasonably dispatched and the capacity of the ESS is fully utilized.

5. Conclusions

In this work, we propose a real-time energy management strategy for MEA based on HDRL using the features that the D3QN algorithm can handle discrete action space and the DDPG algorithm can handle continuous action space, and construct its energy management problem as an MDP to solve it, and verify it through simulation. The simulation results show that the HDRL algorithm is able to obtain an effective energy management strategy for different generator conditions, which not only stabilizes the generator output power around the optimal operating point, but also optimizes the charging and discharging power of the ESS and extends its service life. Compared to previous DRL algorithms and the conventional commercial solver Gurobi, the algorithm achieves an order of magnitude improvement in decision time while maintaining a certain degree of solution accuracy, which meets the practical operational requirements of MEA. In future research, a distributed deep reinforcement learning strategy will be considered to solve the real-time energy management problem in MEA complex power systems from the perspective of a distributed framework.