Secondary Voltage Collaborative Control of Distributed Energy System via Multi-Agent Reinforcement Learning

Abstract

In this paper, a new voltage cooperative control strategy for a distributed power generation system is proposed based on the multi-agent advantage actor-critic (MA2C) algorithm, which realizes flexible management and effective control of distributed energy. The attentional actor-critic message processor (AACMP) is extended into the MA2C method to select the important messages from all communication messages adaptively and process important messages efficiently. The cooperative control strategy trained by centralized training and decentralized execution frame will take over the responsibility of the secondary control level for voltage restoration in a distributed manner. The introduction of the attention mechanism reduces the amount of information exchanged and the requirements of the communication network. Finally, a distributed system with six energy nodes is used to verify the effectiveness of the proposed control strategy.

1. Introduction

With the development of economy and technology, especially the active promotion and application of renewable energy, the energy demand of users has begun to diversify [1]. The development of renewable energy has penetrated into all walks of life, especially in the power industry and energy-efficient buildings [2]. In order to adapt to the new market demand, there are higher requirements for the development and efficient utilization of renewable energy in many countries. At the same time, with the development and maturity of various electricity technologies, more and more technologies can be selected. The hierarchical optimization method has been proposed to realize the complementary nature of different energies [3,4]. The optimal dispatching model with battery energy storage was proposed based on mixed integer programming [5,6]. As a type of open energy system, distributed energy systems have begun to display a multi-functional trend, which not only contains a variety of energy inputs but can also satisfy a variety of energy needs of users at the same time [7,8,9]. The distributed energy system is relative to the traditional centralized energy supply system. The traditional centralized energy supply adopts high-capacity equipment and centralized production. It needs to transmit various energy to a large range of users through special transmission facilities (large power grid, large heat grid, etc.). The distributed energy system is a small and medium-sized energy conversion and utilization system that directly faces users, which can produce and supply energy locally according to users’ needs by designing multiple functions to satisfy multiple objectives [10,11]. The rapid development of various new energy sources has led users to constantly put forward new requirements for energy systems, mainly including high efficiency, reliability, economy, environmental protection, sustainable development, etc. [12,13]. The new distributed energy system can better meet these requirements at the same time and achieve multiple functional objectives by selecting appropriate technologies through system optimization and integration. The rapid development of renewable energy and the deep integration of information and intelligent technologies present a new trend of integrating information and energy and integrating intelligence and materials [14]. Due to the randomness and volatility of renewable energy sources, the stability of voltage is very important for the distributed energy sources system. Currently, there are three control strategies for inverters, which are as follows: constant voltage constant frequency (V/F) control and constant power (PQ) control, and sagging control (droop control) [15]. The droop control simulations of synchronous generator characteristics rapidly gain the stability of voltage and current to “plug and play”. In order to satisfy the different dynamic characteristics of distributed energy, a generalized droop control was applied to promote the rapidity and accuracy of tracking voltage and frequency reference values [16]. Existing literature points out that the multi-input, muti-output control strategy can improve the stability of a distributed energy resources unit [17]. The structure of a distributed energy system with multiple renewable energy sources is similar to that of a multi-agent system [18,19]. Thus, multi-agent system technology has been used to implement load frequency control, and the dynamic performance of the system was effectively improved [20].

In a distributed energy system, there are various coupling modes between different forms of energy, and the energy supply combinations and operation strategies are different. How to design the coordination mechanism of a distributed energy system is an important prerequisite for the collaborative optimization and operation of a distributed energy system [21]. In [22], an intelligent control strategy based on reinforcement learning was proposed to realize the reuse of renewable energy and suppress the frequency fluctuation caused by uncertainty. Each distributed energy node is considered as an agent, and each agent has certain independence and autonomy. A multi-agent-based system can realize the multi-energy cooperative operation of a distributed energy system [23]. Applying the multi-agent reinforcement learning algorithm to the distributed energy system is a research hotspot in the field of control [24]. From this, a frequency control method based on a distributed multi-agent is proposed. Each agent can communicate with its neighbour agents based on power line carrier communication technology. The control objective of a cooperative frequency recovery is realized by adopting the optimized average consensus algorithm [25,26,27]. Considering the inherent trade-off between power sharing and voltage and frequency regulation caused by traditional droop control methods, automatic generation control and automatic voltage control have been proposed based on a multi-agent system to achieve cooperative control of active and reactive power [28,29,30,31]. Multi-agent deep reinforcement learning has been effectively applied in many fields [32,33,34,35]. However, due to the high permeability of renewable energy in distributed information energy systems, the basic architecture of physical information models and effective communication methods in distributed information energy system still need to be analyzed [36,37].

In this paper, the exposition is launched in the following structure. The multi-agent model construction method of distributed energy system is introduced in Section 2. In Section 3, we apply a multi-agent actor-critic algorithm to the secondary voltage control of a distributed energy system. In order to reduce communication pressure and improve the efficiency of processing information from neighboring agents, we add an attentional actor-critic message processor [38]. The simulation results are shown in Section 4. Finally, we conclude with the main thesis of the research and development work of summing up and the further research work in the paper.

2. Distributed Energy System

The distributed energy system is located near the users and directly faces the needs of users. It can effectively simplify the transmission link of the system to provide users with energy, thus reducing the energy loss and transmission cost in the process of energy transmission and increasing the security of users’ energy supply. As a new energy supply mode, a distributed energy system is a powerful supplement to a centralized power supply system. The energy node is the physical node in the energy flow level, and the agent is the virtual agent at the information level. A model of a distributed energy system is shown in Figure 1.

In this paper, we adopt a simple structure for the distributed energy system. It contains a central management center, power generation equipment, electrical load and power electronics. The system is composed of a multi-energy coordinated and complementary energy network, with solar, wind and other renewable energy generation as the primary source and thermal power generation as an auxiliary source. Renewable energy is incorporated into the power grid through the electronic power conversion device, and the inverter is used to convert the electric energy. The central management system is designed to provide real-time information for operation scheduling, carry out scheduling decision-making and realize autonomous management of the distributed energy system.

The control structure of the distributed energy system is shown in Figure 2. The control structure considers each energy node as an agent in a multi-agent system. The secondary voltage control based on the voltage source inverter depends on agents cooperating with each other rather than relying on the centralized controller. In addition to local information, each agent can obtain information from neighboring agents through the communication mechanism. The set points of the primary control of each energy node are obtained by its own distributed secondary controller. The primary control requires each energy node to automatically distribute active and reactive loads using local information while sustaining voltage and frequency stability, which can generally be achieved by $p-\omega$ and $Q-V$ droop control methods.

$$\omega ={\omega }_0-D_{p}P$$

(1)

$$v=V_0-D_{q}Q$$

(2)

where $\omega$ is the frequency generated based on droop control method. ${\omega }_0$ and $V_0$ are the set point of the primary control, which is determined by the secondary control. $D_p$ and $D_Q$ are the corresponding droop coefficients, which are selected based on the active and reactive power rating generated by the coordination of each distributed generator. Considering the randomness and model uncertainty of renewable energy, the multi-agent reinforcement learning method is introduced to achieve the secondary voltage cooperative control strategy of each agent so the node voltage can recover to the reference value. The detailed algorithm design method will be introduced in the next chapter.

3. MA2C-Attention Algorithm

As we all know that the core idea of reinforcement learning is “trial-and-error”, the agent iteratively optimizes based on the feedback it receives through interaction with the environment [28]. In the RL domain, the problem to be resolved is commonly described as a Markov decision process. When multiple agents interact with the environment at the same time, the whole system becomes a multi-agent system, which is able to resolve complex tasks through the collaboration of individual agents. Each agent still follows the goal of reinforcement learning, which is to maximize the cumulative returns that can be obtained, and the change in the global state of the environment is related to the joint actions of all agents. Therefore, the influence of joint action should be considered in the process of agent strategy learning. In the distributed information energy system, the energy node is comparable to an agent with perceptive ability. The cost function and strategy can be optimized by using the multi-agent reinforcement learning method and iterative learning of the reward function.

The distributed energy system is constructed as a multi-agent network, where each energy node is controlled by a local RL agent. The state space of each energy node is designed as:

$$s_{i,t}=(P_i,Q_i,{\delta }_i,i_{odi},i_{oqi},i_{bdi},i_{bqi},v_{bdi},v_{bqi})$$

(3)

where $P_i$ and $Q_i$ are the active power and reactive power of the ith distributed generation unit, respectively. ${\delta }_i$ is the reference frame with respect to the common reference frame. $i_{odi}$ and $i_{oqi}$ are the direct and quadrature output current components of the ith distributed generation. $i_{bdi}$ and $i_{bqi}$ are the direct and quadrature output current components connected to the bus. $v_{bdi}$ and $v_{bqi}$ are the direct and quadrature components of output voltages.

The action of the agent is the adjustable value of the secondary voltage, which ranges from $1.00 pu$ to $1.14 pu$. The action space is designed as a discrete action, and the joint action is $A_t=v_{n1}\times v_{n2}\times \cdots \times v_{nN}$. The design of the reward function is closely related to voltage stability. The higher the reward, the faster it quickly returns to the reference value.

$$r_{i,t}=\left\{\begin{array}{cc}0.05-\left|1-v_i\right|,& {\mathrm{v}}_i\in [0.95,1.05]\\ -\left|1-v_i\right|,& {\mathrm{v}}_i\in [0.8,0.95]\cup v_i\in [1.05,1.25]\\ -10,& \mathrm{v}iolate\end{array}\right.$$

(4)

3.1. Multi-Agent Advantage Actor-Critic

Consider a distributed energy system of N agents operating in a joint environment. Since the energy nodes are not entirely observable, the environment is constructed as a partially observable Markov decision process [29]. It is formally defined as a tuple: $<N,S,A,P,R,O,\gamma >$. The model of the system is denoted as a multi-agent network: $G=(V,ϵ)$, where N is the number of agents, and ${\epsilon }_{ij}\in ϵ:(i,j\in N,i\ne j)$ is the edge of agent i and agent j. Each agent can share information with its neighbors. S, O and $A=[A_1,A_2\cdots A_N]$ are the state space, observation space and action space, respectively. In this paper, the action space is the secondary voltage set point of each energy node, which is discrete. The state transition function is expressed as: $P(s^{{}^{\prime }}\mid s,a):S\times A\times S\to [0,1]$. In our design, each agent shares the global reward $R=O\times A$. $\gamma$ is the discount factor.

An independent Q-learning algorithm (IQL) is a typical multi-agent reinforcement learning algorithm. Other agents are directly regarded as part of the environment; that is, each agent is solving a single agent task [30]. The environment is not stable because of the existence of other agents. This will lead to the calculations not converging, and the agent will easily fall into endless exploration. Considering these problems, the IQL algorithm was combined with the actor-critic method to adaptive traffic signal control, which is called the multi-agent advantage actor-critic(MA2C) algorithm [29,30]. The framework of the training method is shown in Figure 3.

Assume that ${\pi }_{i,t}={\pi }_{{\theta }_i^{-}}(·|s_t,{\pi }_{t-1,j})$, where ${\pi }_{i,t}$ is the local policy of agent i, and ${\pi }_{t-1,j}$ is the latest strategy of the neighbor agent. The global reward of agent i is defined as

$${\tilde{R}}_{i,t}={\widehat{R}}_{i,t}+{\gamma }^{t_T-t}{V}_{{\omega }_i}^{-}({\tilde{s}}_{t_B},V_i,{\pi }_{t_T-1},j|{\pi }_{{\theta }_{-i}^{-}})$$

(5)

where ${\widehat{R}}_{i,t}=\gamma r_i$ is the estimated reward of each energy node. $V_{{\omega }_i}^{-}$ is the value regressor. Then, the local advantage function is defined as

$$A_{\theta }^i(s,a)=Q_{\pi }(s,a)-V_{{\omega }_i}(s,a^{-})$$

(6)

$$V_{{\omega }_i}(s,a^{-})=\sum _{a^i\in A^i}{\pi }_{{\theta }_i}^i(s,a^i)·Q_{\theta }(s,a^i,a^{-i})$$

(7)

The actor-critic algorithm contains two steps in the training process: strategy evaluation and strategy promotion. The action-value TD-learning method is used to update the parameter of policy evaluation in the critic step. Each energy node updates its policy via the policy gradient algorithm in the actor step [21]. The loss function of value and policy for updating is illustrated as

$$L\left({\omega }_i\right)=-\frac{1}{2T}\sum _{t\in T}{(R_{i,t}-V_{{\omega }_i}\left({\tilde{s}}_t,V_i,{\pi }_{t-1,j}\right))}^2$$

(8)

$$L\left({\theta }_i\right)=-\frac{1}{T}\sum _{t\in T}(log{\pi }_{{\theta }_i}\left(u_{i,t}\right|{\tilde{s}}_t,V_i,{\pi }_{t-1},N_i){\tilde{A}}_{i,t}-\beta \sum _{u_i\in U_i}log{\pi }_{{\theta }_i}\left(u_{i,t}\right|{\tilde{s}}_t,V_i,{\pi }_{t-1,j}))$$

(9)

3.2. MA2C-Attention Algorithm

The secondary voltage control of energy nodes based on multi-agent reinforcement learning requires good coordination and communication among energy nodes. There are usually two training frameworks for solving multi-agent problems based on reinforcement learning, namely centralized and decentralized training frameworks. The centralized training and decentralized execution scheme has become the mainstream alternative in recent years. During training, the centralized mode is adopted. Each agent only makes decisions by applying the trained strategy network according to its own local observations.

Attention mechanism was initially proposed in the field of visual images and has been widely used in many artificial intelligence algorithms [38]. In this paper, soft attention is extended to the multi-agent advantage actor-critic algorithm. The normalized importance score $W^k$ is defined as

$$W^k=\frac{exp\left(f(T,S_k)\right)}{{\sum }_1^{K}exp\left(f(T,S_i)\right)},\sum _1^K{W}^k\equiv 1$$

(10)

where $f(T,S_k)$ is the defined function to judge the importance score. The input vector $\left[S_1,S_2\cdots S_k,\cdots S_K\right]$ contains the important information, and T is the target vector.

The motivation for introducing attention mechanisms is to obtain a better reward by selectively paying attention to the actions of other energy node. The structure of a multi-agent A2C algorithm with an attentional mechanism is shown in Figure 3. As shown in the figure, each agent contains three types of networks: an actor network, a critic network and a communication network. h is the hidden layer of the deep neural network, $m_i$ is the local message and $M_i$ is the global message. Due to the fully connected network treating all the messages equally, the attention is embedded in the actor network to select more important messages from $m_1,m_2\cdots m_N$ adaptively. The updating method of the message coordinator network is described as follows. Firstly, we need to set a “query” feature space $m_i^q=m_i\times w_i^q$ and a “key” feature space $m_{ij}^k=m_j\times w_i^k$. The importance score is defined as

$$W_i^j={{(}_i^q)}^T\left(m_{ij}^k\right),i\ne j$$

(11)

$$W_i=[W_i^1,\cdots .,W_i^j\cdots ,W_i^N]=softmax(W_i^1,\cdots .,W_i^j\cdots ,W_i^N)$$

(12)

$$\sum _{j=1}^N{W}_i^j=1,i\ne j$$

(13)

Then the global message is generated based on the importance score

$$M_i=\sum _{j=1}^N{W}_i^j{m}_j$$

(14)

where $W_i^j$ is the weight of $m_j$ for $m_i$. The more important local messages will be selected, and the unimportant messages will be ignored according to the weights.

The attention mechanism is combined with the original network; therefore, the training method inherits the above method according to Equations (8) and (9). The detailed illustration of the proposed algorithm is shown in Algorithm 1. The input hyperparameters include $\alpha ,\gamma ,{\eta }_{\omega },{\eta }_{\theta }$, and $L,S$, where $\alpha$ is the temperature parameter of entropy, $\gamma$ is the time-discount factor, ${\eta }_{\omega }$ is the learning rate of the actor network, ${\eta }_{\theta }$ is the learning rate of the critic network, L is the time step of every episode, and S is the total training episodes. The training process of the algorithm circulates thousands of episodes (Lines 2–21). In every episode, each agent picks up information according to the observation information and sends their message to the message coordinator network (Lines 4–6). By setting a “query” feature space and a “key” feature space, we can calculate the importance score. All agents interact with the environment by selecting and executing actions based on the current state and global message (Lines 7–10). Then, the agents move to the next state and receive an immediate reward. The weights of the actor network and the critic network are updated using a policy gradient algorithm (Lines 18–19).

Algorithm 1 MA2C-Attention algorithm

Input:$\alpha$, $\gamma$, ${\eta }_{\omega }$, ${\eta }_{\theta }$, L, S Output:${\theta }_i$, ${\omega }_i$ 1: initialize $s_0$, ${\pi }_{-1}$, $t\leftarrow 0$, $D=⌀$; 2: for$j=0$ to $S-1$ do 3:       for $t=0$ to $L-1$ do 4:            for $i\in V$ do 5:                 send $m_{i,t}=o_{i,t}$ 6:                 send $M_{i,t}=m_{i,t}$             end 7:             for $i\in V$ do 8:                   let $m_i^q=m_i\times w_i^q$, $m_{ij}^k=m_j\times w_i^k$, 9:                   calculating the importance score: Equation (11) 10:                  send $M_{i,t}$ to $m_{i,t}$ 11:                  update $a_{i,t}={\pi }_{i,t}$             end 12:         update $v_{i,t}=V_{{\omega }_t}(o_{i,t},a_{N_i,t})$ 13:         simulate $s_{t+1},r_{i,t}$ 14:         update $t\leftarrow t+1$, $j\leftarrow j+1$ 15:         update 16:         if $DONE$ then 17:              for $i\in V$ do 18:                   update ${\theta }_i\leftarrow {\theta }_i+{\eta }_{{\theta }_i}{\eta }_{{\theta }_i}{▽}_{{\theta }_i}J\left({\theta }_i\right)$ 19:                   update ${\omega }_i\leftarrow {\omega }_i+{\eta }_{{\theta }_i}{\eta }_{{\omega }_i}{▽}_{{\omega }_i}J\left({\omega }_i\right)$                   end            end 20:         initialize $D\leftarrow ⌀$        end 21:     update $s_0,{\pi }_{-1},t\leftarrow 0$    end

4. Simulation

In this section, an easy distributed energy system is built around a distribution network consisting of six buses connected to three renewable energy generators, one distributed energy storage unit, one thermal generator, and one balancing generator. The topology is shown in Figure 4. Each energy was regarded as an agent, and each node can communicate with its neighbor nodes. The secondary control corrects voltage deviations caused by droop control based on the proposed method. The simulation was performed on a Windows 10 server with an Intel(R) Core i5-10400F CPU @ 2.90 GHz processor and 64 GB memory.

In order to illustrate the effectiveness of the proposed algorithm, we compared the algorithm with the independent advantage actor-critic algorithm and multi-agent advantage actor-critic algorithm. We set 4000 training episodes, and one episode lasts 20 steps. During the training process, each episode generated different random seeds. To ensure the fairness of comparison, the different algorithms shared the same random seed. The average reward value of different algorithms is shown in Figure 5.

The red curve is the average reward value trained based on the proposed method. The blue and green curve is the average reward value trained based on MA2C algorithm and IA2C algorithm, respectively. Obiviously, the multi-agent advantage actor-critic algorithm with attention mechanism has a higher episode reward and faster convergence speed compared to other algorithms.

In order to further verify the effectiveness of the proposed method, we conducted random tests on the trained model. The test results in different random seeds are shown in

Table 1. Average reward value under random test.

Random Seeds	MA2C-Attention	MA2C	IA2C
Case 1	0.27	0.26	0.178
Case 2	0.25	0.25	0.112
Case 3	0.26	0.25	0.194
Case 4	0.27	0.26	0.177
Case 5	0.27	0.25	0.175
Average reward	0.264	0.25	0.167

. The average reward of the MA2C-attention algorithm is $0.264$. The average reward of five groups was tested, and the results obtained are 1.056 and 1.58 times compared with the MA2C and IA2C algorithm.

We also compared the voltage control results between the secondary controller and only the primary controller. The voltages of the six distributed generators under different algorithms are shown in Figure 6. The abscissa value 0–5 represents the terminal voltages of the six distributed generators. The ordinate represents the unit value of the voltage. The specific voltage value of Figure 6 is shown in

Table 2. Voltage value under different algorithms.

Method	MA2C-Attention	MA2C	IA2C	Droop Control
Agent 1	1.00975192	1.00467732	0.97759587	0.9
Agent 2	1.00788861	0.9908174	0.98287053	0.89002
Agent 3	0.99718829	0.9887299	0.97389743	0.90125
Agent 4	0.99951272	0.9894690	0.97161282	0.89234
Agent 5	0.99088914	0.9894690	0.98344116	0.91232
Agent 6	0.99591974	0.9862523	0.9800902	0.90023

. Agent 1–6 denotes the six energy nodes, respectively. According to the chart, we can clearly see that the voltage value of each node is closer to the reference value under the proposed method. The advantages of the method proposed in this paper are more clear winners than the typical droop control. Due to the randomness of renewable energy, under primary control, the voltage cannot recover to the reference value rapidly. The proposed algorithm can quickly promote the node voltage to the reference value. The voltage fluctuations within the allowable range and remain relatively stable.

5. Conclusions

This paper proposes a multi-agent advantage actor-critic algorithm to solve the collaborative control of secondary voltage in a distributed energy system. The attentional mechanism is extended to the multi-agent advantage actor-critic algorithm to improve communication efficiency, so that all agents cooperate more effectively with each other. The simulation results of a simple distributed energy system demonstrate that the proposed algorithm achieves the cooperative control of secondary voltage. Compared with the independent advantage actor-critic algorithm and the multi-agent advantage actor-critic algorithm, the introduction of the attentional mechanism facilitates the promotion of global reward. This study has potential limitations. The construction of the distributed energy system lacks the consideration of an actual power system. We took into account the effect of voltage offset in the design of the algorithm’s model, but the frequency offset is also an important influencing factor for the stability of the distributed energy system. We will build a realistic simulation environment through the collection and analysis of the actual operating system. In addition, we will consider the design of the three-level control scheme of the distributed energy system. The objective function will be designed according to the needs of the user to achieve optimal control of the distributed energy system. The dispatch scheme will be designed based on the multi-agent reinforcement learning algorithm to provide the reference value of the optimal power generation of each distributed energy. The reference values transmitted to the lower layer through the communication network control the output power of energy nodes to realize the safe and economic operation of the distributed energy system.