A Sparse Neural Network Based Control Structure Optimization Game under DoS Attacks for DES Frequency Regulation of Power Grid

Abstract

With the rapid growth of distributed energy sources, power grid has become a flexible and complex networked control system. However, it increases the chances of being a denial-of-service attack, which degrades the performance of the power grid, even causing cascading failures. To mitigate negative effects from denial-of-service attack and enhance the reliability of the power grid, we propose a networked control system structure based optimization scheme that is derived from a Stackelberg game model for the frequency regulation of a power grid with distributed energy sources. In the proposed game model, both denial-of-service attacker and control system designer as a defender are considered without using any analytical model. For defenders, we propose a sparse neural network based DES control and system structure design scheme. The neural network is used to approximate the desired control output and reinforce signals for the improvements of short- and long-term performance. It also introduces the sparse regulation of column grouping in the neural network learning process to explore the structure of control system that involves the placement of sensor, distributed energy sources actuator, and communication topology. For denial-of-service attackers, the related attack constraints and attack rewards are established. The solution of game equilibrium is considered as an optimal solution for both denial-of-service attack strategy and control structure. An offline optimization algorithm is proposed to solve the game equilibrium. The effectiveness of proposed scheme is verified by two cases, which illustrate the optimal solutions of both control structure and denial-of-service attack strategy.

1. Introduction

With the development of network techniques, electricity supply via the modern power grid is increasingly depending on networked control systems (NCSs). In the power grid and communication integrated network, the efficiency and reliability of power grid are gradually enhanced [1,2]. However, new network control techniques generate related vulnerabilities in the control system of power grids. As the connection of virtual and physical worlds, cyber attacks against NCSs can render the large disturbances of power grid that have been confirmed during the past few years [3].

As a common type of cyber attack, denial-of-service (DoS) attack occupies communication resources to prohibit the transmission of measurement and control signals in NCSs [1]. Compared with deception attacks [4], DoS attacks not only require little prior knowledge about NCSs, but also destroy control operations in real time [5]. In particular, blocking real-time control signals can cause the instability of the power grid [6]. A lot of security control approaches, such as stochastic time delay system, triggering strategy, and game theory, have been applied to the prevention of DoS attacks in power grid.

Stochastic time delay system: In a stochastic time delay system, DoS is modeled as a stochastic process with a delay in the signal. Subject to intermittent DoS attacks, An et al. proposed a decentralized solution of adaptive output feedback control for a power grid [7]. A switching-type state estimator is presented to estimate the state of power grid by using discontinuous output measurement. Sun et al. modeled DoS attacks as a Markov process that converted DoS attacks to stochastic noises of NCSs first [8]. Then, a resilient control model was used to process the converted noises.

Triggering strategy: Event-triggering time-sequence of control signal is adopted to reduce communication costs in the system. The triggering time sequence is able to defend against DoS attacks. Peng et al. proposed a resilient event-triggering based frequency control method for power system control in energy-limited DoS attacks [1]. The proposed event-triggered communication scheme can tolerate a certain degree of data loss in the open communication induced by energy-limited DoS attacks. Hu et al. designed a periodically resilient event-triggering communication scheme to identify DoS attacks initiated by power-constrained pulse-width-modulated jammers [9].

Game theory: In game theory based cyber security model, a game model is constructed for both DoS attacker and defender to obtain a Nash equilibrium as an optimal solution. Li et al. modeled the interactions between the transmission power of sensors and the interference power of DoS attacker by a signal-to-interference-and-noise-ratio (SINR) based network [10]. A modified Nash Q-learning algorithm was proposed to analyze the related interactions as well. Yuan et al. analyzed the resilient control issues of NCS under DoS attack via a unified game approach [5]. Yuan et al. also built a multi-stage hierarchical game with a corresponding hierarchy of decisions that was implemented to achieve a resilient control system [11]. According to optimal control structures, optimal criteria were constructed for DoS attackers and cyber defenders. Ding et al. modeled the remote estimation under DoS attacks by using the strategy of zero-sum stochastic games and presented a monotone structure to solve the proposed model optimistically [12]. At the same time, Ding et al. also formulated the decision-making process of a target channel as a two-player zero-sum stochastic game framework and proposed a Nash Q-learning algorithm to obtain the optimal strategy [13]. Zhu et al. built a hybrid game-theoretic framework to improve the robustness of power system security [14]. Since game theory based control strategy can process strategic interactions among multiple decision makers, it is often used in large-scale system security control.

Despite the game theory based method being able to be efficient against conventional DoS threats, only using control methods to defense DoS attacks is not sufficient to counter potential and increasingly sophisticated attacks. In large-scale control systems, structure factors, such as distributed energy sources (DES) placements, sensors, and communication link topologies, can critically affect the performance in DoS attacks. Existing work of energy storage systems [15], DES placement [16], sensor scheduling [17] and coverage aims at minimizing the electrical and computational costs [18]. Upon existing research in the optimization of control system structure for cyber security, there are several critical challenges. One critical challenge is the transient and dynamic issues of DES and cyber resources allocation. Most existing research focused on the characteristics of steady state [19]. They ignored the transient and dynamic issues in control systems [20]. Another critical challenge is that most existing algorithms used a precisely analytical model as a premise [21]. However, most large-scale power systems are unable to be precisely modeled. In this case, a model free method is required for control-system-structure optimization.

With the consideration of DoS [22,23] attacks and the control-system-structure optimization problem, we propose a sparse neural network based NCS optimization method that is derived from the Stackelberg game model for the frequency regulation of DES in power systems. A neural controller is trained by reinforcement learning of offline simulation and online processes to improve the system performance. The system performance is also improved under limited cyber resources by optimizing the control structure, which involves the placement of DES, RTU sensors and communication topology. It is optimized by imposing a group sparse regulation on neural network weights. The Stackelberg game model is used, which derives a minimax optimization of system performance, so that the optimized control structure is robust to DoS attacks under the consideration of worst case attacks. The structure consists of the placement of DES, RTU sensors and communication topologies. The contributions of this paper are summarized as follows:

• Sparse neural network based reinforcement learning is proposed to improve the frequency regulation of DES in control systems without using a power system analytical model, which involves adaptiveness, performance, and structure.
• The Stackelberg game model is used to derive the optimal control scheme and structure, so that the proposed frequency regulation system is robust to the worst case of DoS attacks. In addition, the reliability of proposed frequency regulation system is enhanced.

The remainder of this paper is organized as follows: the system model and related problems are formulated in Section 2, which introduces the power system and DoS attack model, and describes the frequency regulation as well; Section 3 elaborates control structure and control law design by sparse neural network based reinforcement learning; Stackelberg game model and the optimization scheme of control structure are derived under DoS attacks in Section 4; and Section 5 demonstrates the simulation results to verify the proposed algorithm.

2. Problem Formulation

The formulation of power grid and control objective will be introduced in this section. We consider a multi-area system that is integrated with Distributed Energy resources. The control objective is to mitigate the frequency regulation. The DoS attack may degrade the control system performance and even lead to failures. Thus, a design problem involving controller design and structure design under the cyber attack is also introduced.

2.1. Power Grid Frequency Dynamic Model

We consider the interconnected multi-area power system. Each one of n areas is connected to each other by tie-line (also called transmission line). As shown in Figure 1, each area equips a turbine generator and DES, such as wind power, solar power, battery, etc. [20].

It also contains a load frequency controller (LFC) and tie-line bias controller (TBC) for frequency synchronization. Even if the synchronization measures regulate the frequency, an auxiliary control offered by DES may be necessary to enhance the system performance, when the power system encounters a severe disturbance, such as system fault, and sudden large load drop. Considering the auxiliary control, the dynamic model of area i can be formulated as a discrete linear difference equation [24]:

$$x_i(k+1)=A_i{x}_i\left(k\right)+B_i{u}_i\left(k\right)+\sum _{j\in N_p\left(i\right)}{B}_{ji}{x}_j\left(k\right)+E_i{w}_i\left(k\right),$$

(1)

where $i\in N^{+}\cap [1,n]$, $x_i={\left[\begin{array}{ccccc}\Delta f_i& \Delta P_{mi}& \Delta P_{vi}& \Delta P_{tie-i}& \int AC{E}_i\end{array}\right]}^T$ is area state. $\Delta f_i$ is the deviation related to synchronized frequency; $\Delta P_{mi}$ is the mechanical power deviation of generator; $\Delta P_{vi}$ is the valve position deviation of turbine; $\Delta P_{tie-i}$ is the deviation of tie-line power injection from other physical neighbored areas; $AC{E}_i$ is the $ACE$ signal of area i and $AC{E}_i={\alpha }_i\Delta f_i+\Delta P_{tie-i}$. $u_i$ as the auxiliary control output of DES for frequency regulation is the sum of all the powers generated from power-electronic interfaced DES; $w_i$ is the disturbance caused by model error or other time-varying factors; $A_i$ is the system transition matrix; $B_i$ and $B_{ji}$ are the gains of control effect and other physical neighbored systems; $E_i$ is the disturbance gain. $N_p\left(i\right)$ denotes the physical neighbored areas of area i.

In this linear model, loads are assumed to be constant because the variation of loads is slow relative to the dynamic frequency regulation. Therefore, $A_i$, $B_i$, and $B_{ji}$ can be modeled in time invariant [24]. The system is linear time invariant (LTI) as well as an NCS. The DES controller of area i for frequency regulation are written as

$$u_i={\phi }_i(x_i,x_{N_c\left(i\right)}),$$

(2)

where the time stage k is neglected. $N_c\left(i\right)$ denotes the cyber connected areas of area i. The controller calculates DES control outputs by the received state $x_i$ and $x_j$ as well as $j\in N_c\left(i\right)$ from local and remote areas, respectively. The control objective is to mitigate the frequency deviation and reduce the overall costs defined in Section 3, which consider the least quadratic of state $x_i$ and the control output $u_i$.

2.2. DoS Attack Model

For the previously mentioned NCS, we consider attacker launch attacks, when the power system requires the emergency auxiliary control offered by DES. DoS attack blocks communication channels to degrade the control performance, even causing system failures. The blocks of communication channels probably result in the absence of some remote states $x_j$ and $j\in N_c\left(i\right)$ of the current time stage k [25]. When controller cannot obtain the remote state from cyber connected area j, “zero-control” strategy is applied to assigning $x_j=0$. $u_i$ becomes

$$u_i=\phi \left[\left\{{\alpha }_{ji}{x}_j\right|j\in N_c\left(i\right)\cup i\}\right],$$

(3)

where ${\alpha }_{ji}\in \{0,1\}$ is distributed according to Bernoulli distribution, which is

$$\left\{\begin{array}{c}P({\alpha }_{ji}\left(k\right)=0)={\delta }_{ji},\hfill \\ P({\alpha }_{ji}\left(k\right)=1)=1-{\delta }_{ji},\hfill \end{array}\right.$$

(4)

where ${\delta }_{ji}\in [0,1]$ is the probability of packet drop in the communication from area j to i, $k\in N^{+}$. The probability ${\delta }_{ji}$ increases the intensity of DoS attack. DoS attacker has the limited cyber resources, which constrains the intensity of DoS attack at a certain degree. The constraint in our model is subject to

$$\sum _{i=1}^n\sum _{j\in N_c\left(i\right)}{\delta }_{ji}\le C.$$

(5)

The probability ${\delta }_{ji}$ is limited to the intensity of DoS attack. The intensity of DoS attack is under the constraints of cyber cost, which is also limited. Therefore, the sum of probability ${\delta }_{ji}$ must be less than a maximal real number C.

2.3. Control, Structure Design and Optimization Problem

Even though the NCS of power grid suffers DoS attacks, one of the NCS objectives is to enhance the system performance by mitigating the frequency deviation $\Delta f_i$ and minimizing the control cost. To achieve the control objective, the control law ${\phi }_i$ should be investigated. Moreover, the design of control structure and its optimization also need to be considered. Since the control structure associates the placement of sensors (in remote terminal unit (RTU)), actuators (in DES) and communication topology, these placements seriously affect the control performance in a large-scale multi-area power system. Generally, the placed resources (RTU, DES, etc.) are expensive.

In mathematical form, the control design gives a ${\phi }_i$ to achieve the system performance. We define Q shown in Equation (6) as the control cost to measure system performance.

$${{\phi }_{}}^{*}=arg\underset{{\phi }_{}}{min}Q,$$

(6)

where “argmin” here also denotes the functional minimization; the control structure design aims to select cyber connected area set $N_c\left(i\right)$ for $i=1,\dots ,n$, which is

$${N_c}^{*}=arg\underset{N_c}{min}Q.$$

(7)

Therefore, as an optimization, it searches a combination of ${\phi }_i$ and $N_c\left(i\right)$ for $N_c\left(i\right)$, $i=1,\dots ,n$ to minimize the control cost in the worst case of DoS attack with constraint Equation (5) as shown in Equation (8):

$$\left(\begin{array}{ccc}{\phi }^{*},& {N_c}^{*},& {\delta }^{*}\end{array}\right)=arg\underset{\delta }{max}\underset{\phi ,N_c}{min}Q,$$

(8)

where $\phi =\left(\begin{array}{ccc}{\phi }_1& \cdots & {\phi }_n\end{array}\right)$, $N_c=\left[\begin{array}{ccc}{N}_c\left(1\right)& \cdots & N_c\left(n\right)\end{array}\right]$, $\delta =\left[\begin{array}{ccc}{\delta }_{1N_c\left(1\right)}& \cdots & {\delta }_{n{N}_c\left(n\right)}\end{array}\right]$.

We assume the attacker knows the information of the designed power system. The attacker can figure out the minimal system performance, when the controller reaches its maximal performance. The details of optimization in a Stackelberg game are discussed in Section 4.

3. Control and Structure Design

3.1. Control Design by Reinforcement Learning

Before introducing control design, a cost function is defined to measure the control cost Q. Control performance usually involves the quadratic of state and control output. Thus, we define the control cost Q by a control strategy utility function used in our previous work [24]:

$$Q_i\left(k\right)=\underset{u_i(k+j),j\in [0,\infty ]}{min}\sum _{t=0}^{\infty }{\alpha }^{N-t}{p}_i(k+t),$$

(9)

where $0<\alpha <1$ is the discount rate, and N is a positive integer. $p_i$ as a binary performance index is defined as:

$$p_i\left(k\right)=\left\{\begin{array}{cc}1,& a_1∥x_i\left(k\right)∥+a_2∥u_i\left(k\right)∥\le c,\\ 0,& \mathrm{otherwise},\end{array}\right.$$

(10)

where $∥·∥$ denotes 2-norm, $a_1$ and $a_2$ are the weights of state and control cost respectively, and c is the threshold of performance indication. If the current cost is in an allowed range (less than c), the binary performance index shown in Equation (10) is 1; otherwise, it is 0. Binary performance index makes the strategy utility function limited. When time horizon is limited, even the system diverges, and it can avoid a numerical crash in the learning process.

To prevent the system diverge, we construct a desired control output $u_{di}$ to apply a damping rate $L_i\in R^{5\times 5}$, $∥L_i∥<1$ to the system, so that the power grid frequency dynamic becomes

$$x_i(k+1)=L_i{x}_i\left(k\right)+\sum _{j\in N_p\left(i\right)\backslash N_c\left(i\right)}{B}_{ji}{x}_j\left(k\right)+E_i{w}_i\left(k\right).$$

(11)

If we approximate $u_{di}$ by a neural network output ${\widehat{u}}_{di}$, the neural network is defined as

$${\widehat{u}}_{di}\left(k\right)=M_a{W}_i\varphi \left(k\right),$$

(12)

where $\varphi \in R^{nl\times 1}$ is the radial basis of neural network calculated from state $x_1,x_2,\dots ,x_n$, $W_i\in R^{nl\times nl}$ is the trainable weight, and $M_a\in R^{1\times nl}$ is a given constant matrix. As shown in Figure 2, the proposed neural network structure is based on a radial basis function (RBF). An RBFb-based neural network can identify nonlinear dynamical systems [26]. l is the dimension of radius basis for one area. Under the control of ${\widehat{u}}_{di}=u_i$, the system of area i becomes

$$x_i(k+1)=L_i{x}_i\left(k\right)+B_i[{\widehat{u}}_{di}\left(k\right)-u_{di}\left(k\right)]+\sum _{j\in N_p\left(i\right)\backslash N_c\left(i\right)}{B}_{ji}{x}_j\left(k\right)+E_i{w}_i\left(k\right).$$

(13)

According to the control theory of stability, if $B_i[{\widehat{u}}_{di}\left(k\right)-u_{di}\left(k\right)]+E_i{w}_i\left(k\right)$ and ${\sum }_{j\in N_p\left(i\right)\backslash N_c\left(i\right)}{B}_{ji}{x}_j\left(k\right)$ in the right part of Equation (13) are bounded, the system is ultimately uniformly bound (UUB) [27]. We assume the disturbance $w_i\left(k\right)$ is bounded as $∥w_i\left(k\right)∥\le \epsilon$, where $\epsilon$ is a small real number, so learning $W_i$ should involve the approximation of $u_{di}$.

The control cost involving system performance is also approximated by a neural network as shown in Equation (14):

$${\widehat{Q}}_i\left(k\right)=M_c{W}_i\varphi \left(k\right),$$

(14)

where $M_c\in R^{1\times nl}$ is a given constant matrix.

We train the neural network weight $W_i$ to approximate the desired control output $u_{di}$ as well as the performance measurement for the improvement of system performance. Therefore, we define a loss function for training as    

$$\begin{array}{cc}\hfill V_i\left(k\right)=& \frac{1}{2}{∥B_i{\widehat{u}}_{di}\left(k\right)-B_i{u}_{di}\left(k\right)∥}^2+\hfill \\ & \frac{1}{2}{∥{\alpha }^{N+1}{p}_i\left(k\right)+{\alpha }^{-1}{\widehat{Q}}_i(k+1)-{\widehat{Q}}_i\left(k\right)∥}^2+\frac{1}{2}{∥{\widehat{Q}}_i\left(k\right)∥}^2.\hfill \end{array}$$

(15)

The first term denotes the approximation error of the desired control output, the second term denotes the approximation error of system performance, and the third term is used to improve the long-term system performance. As shown in Equation (16), the desired control output $u_{di}$ and system performance $Q_i$ can be expressed by the neural network:

$$\left\{\begin{array}{c}{u}_{di}\left(k\right)=M_a{W}_i^{*}\varphi \left(k\right)+{\upsilon }_{ai}\left(k\right),\hfill \\ Q_i\left(k\right)=M_c{W}_i^{*}\varphi \left(k\right)+{\upsilon }_{ci}\left(k\right),\hfill \end{array}\right.$$

(16)

where $∥{\upsilon }_{ai}\left(k\right)∥\le v$ and $∥{\upsilon }_{ci}\left(k\right)∥\le v$ are the optimal approximation errors, v is is a small number, and $W_i^{*}$ is the optimal approximation of a constant matrix. We derive the defined loss function $V_i\left(k\right)$ subject to $W_i\left(k\right)$ and neglect $v_{ai}$ and $v_{ci}$, so the converted online iteration formula is shown as

$$\left\{\begin{array}{c}{W}_i(k+1)=W_i\left(k\right)-{\beta }_i\frac{\partial L_i\left(k\right)}{\partial W_i\left(k\right)},\hfill \\ \frac{\partial L_i\left(k\right)}{\partial W_i\left(k\right)}=\left\{M_a^T\left[x_i(k+1)-L_i{x}_i\left(k\right)\right]+M_c^T\left[{\alpha }^{N+1}{p}_i\left(k\right)+{\alpha }^{-1}{\widehat{Q}}_i(k+1)\right]\right\}\varphi {\left(k\right)}^T.\hfill \end{array}\right.$$

(17)

3.2. Structure Design by Sparse Neural Networks

The structure design involves the placement of sensor, actuator, and communication topology. It is formulated by the neural network weight $W_i$. The weight matrix is separated into column groups, so we can define the radius basis $\varphi$ as

$$\varphi ={\left(\begin{array}{cccc}{{\varphi }_1}^T& {{\varphi }_2}^T& \cdots & {{\varphi }_n}^T\end{array}\right)}^T,$$

(18)

where ${\varphi }_i={\left(\begin{array}{cccc}{e}^{-\frac{{∥x_i-x_{c1}∥}^2}{{\sigma }^2}}& e^{-\frac{{∥x_i-x_{c2}∥}^2}{{\sigma }^2}}& \cdots & e^{-\frac{{∥x_i-x_{cl}∥}^2}{{\sigma }^2}}\end{array}\right)}^T$, $x_{cj}$ is a given radius center, $j\in N^{+}\cap [1,l]$, and $\sigma$ is a given radius width. According to the definition of $\varphi$, we know that the neural weight is separated into n column groups. Each group that has l columns corresponds to the radius basis ${\varphi }_i$ of one area. If the weight matrix $W_i$ is group sparse [28] with respect to these columns, the structure of control system can be figured out. For instance, if the column group j of $W_i$ corresponding to the gain of ${\varphi }_i$ is zero, the controller of area i does not need the information from area j, and the communication channel between area j and i is also not necessary. If all the controllers do not require the information from area j, any sensor is not necessary for area j. If $W_i=0$, it means that the actuator in area i is not required. The key is to force $W_i$ to be column group sparse. Thus, a group sparse regulation term is added to the loss function. Thus, the loss function becomes

$$\begin{array}{c}{V}_i\left(k\right)=\frac{1}{2}\left[{∥B_i{\widehat{u}}_i\left(k\right)-B_i{u}_{di}\left(k\right)∥}^2+{∥{\widehat{Q}}_i\left(k\right)∥}^2\right]+\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\frac{1}{2}{∥{\alpha }^{N+1}{p}_i\left(k\right)+{\alpha }^{-1}{\widehat{Q}}_i(k+1)-{\widehat{Q}}_i\left(k\right)∥}^2+{\gamma }_i\sum _{j=1}^n∥vec\left(W_{iG\left(j\right)}\right)∥,\hfill \end{array}$$

(19)

where $W_{iG\left(j\right)}$ denotes the column group j or the gain of radius basis ${\varphi }_j$. ${\gamma }_i$ is the regulation weight. The sparse regulated learning iteration is demonstrated in Equation (20), which can be also derived by Equation (17):

$$\left\{\begin{array}{c}{W}_i(k+1)=W_i\left(k\right)-{\beta }_i\frac{\partial V_i\left(k\right)}{\partial W_i\left(k\right)}\hfill \\ \frac{\partial V_i\left(k\right)}{\partial W_i\left(k\right)}=\left\{M_a^T\left[x_i(k+1)-L_i{x}_i\left(k\right)\right]+M_c^T\left[{\alpha }^{N+1}{p}_i\left(k\right)+{\alpha }^{-1}{\widehat{Q}}_i(k+1)\right]\right\}\varphi {\left(k\right)}^T\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}+{\gamma }_i{W}_i\left(k\right)D_i\left(k\right),\hfill \end{array}\right.$$

(20)

where $D_i=diag\left[\begin{array}{cccc}\mathbf{1}{∥vec\left(W_{iG\left(1\right)}\right)∥}^{-1}& \mathbf{1}{∥vec\left(W_{iG\left(2\right)}\right)∥}^{-1}& \cdots & \mathbf{1}{∥vec\left(W_{iG\left(n\right)}\right)∥}^{-1}\end{array}\right]$, and $\mathbf{1}=[1,1,\dots ,1]\in {\mathrm{R}}^l$. $W_{ij}$ means the jth weight block in $W_i$, which involves ${\varphi }_j$. Thus, we have

$${\widehat{u}}_{di}=W_{i1}{\varphi }_1+W_{i2}{\varphi }_2+\dots +W_{in}{\varphi }_n.$$

(21)

The stability analysis of iteration in Equation (20) can be taken in a similar way used in [24]. We know that if ${\beta }_i$ and ${\gamma }_i$ are small enough, the iteration in Equation (20) is stable and the errors in results are limited.

4. Structure Optimization under DoS Attacks

In this section, the control system structure design is solved by a structure optimization problem. Considering DoS attack, a Stackelberg game is formulated in the structure optimization. In addition, the optimization algorithm is also presented.

4.1. Stackelberg Game Formulation

As shown in Equation (20), the iteration searches a system structure and control scheme in simulation, and then a DoS attacker observes the system and takes attack actions. It is a leader–follower sequence, therefore a Stackelberg game model is proposed to obtain the optimal solution of control structure optimization and DoS attacks.

There are two actors in the proposed model. One is a defender that is the system designer, and the other one is an attacker that is a DoS attacker. The defender’s action is $N_c=\left\{N_c\left(1\right),N_c\left(2\right),\dots ,N_c\left(n\right)\right\}$. The reward function of defender is defined as

$$r\left(N_c,\delta \right)=\sum _i^n{\left.Q_i\right|}_{N_c\left(i\right),\delta }.$$

(22)

The attacker’s action is $\delta =\left\{\begin{array}{cccc}{\delta }_{N_c\left(1\right)1}& {\delta }_{N_c\left(2\right)2}& \cdots & {\delta }_{N_c\left(n\right)n}\end{array}\right\}$. The reward of attacker is $-r\left(N_c,\delta \right)$. Therefore, it is a zero-sum game. The structure optimization becomes a min-max optimization problem as follows:

$$\left({N_c}^{*},{\delta }^{*}\right)=arg\underset{\delta }{min}\underset{N_c}{max}r(N_c,\delta ),$$

(23)

where ${N_c}^{*}$ and ${\delta }^{*}$ are the equilibrium of game model, which can also be treated as the optimal solution for both defender and attacker. Thus, it can obtain $r(N_c^{},{\delta }^{*})\le r(N_c^{*},{\delta }^{*})\le r(N_c^{*},{\delta }^{})$.

4.2. Structure Optimization under Dos Attacks

For a given DoS attack $\delta$, we know that the optimal structure and control law can be figured out by the iteration of Equation (20): The iteration derives as follows.

$${\left.N_c^{*}\right|}_{\delta }=arg\underset{N_c}{max}r(N_c,\delta ).$$

(24)

If $j\notin N_c^{*}\left(i\right)$, the iteration of Equation (20) usually obtains a small $∥vec\left(W_{ij}\right)∥$ instead of $∥vec\left(W_{ij}\right)∥=0$ because it is an numerical method. Therefore, a threshold method is used in our algorithm to obtain ${\left.N_c^{*}\right|}_{\delta }$. We can obtain the estimated ${\left.N_c^{*}\right|}_{\delta }$ by Equation (25):

$${\left.{\widehat{N}}_c^{*}\right|}_{\delta }\left(i\right)=\left\{\left.j\right|∥vec\left(W_{ji}\right)∥\le {\rho }_i\right\},i\in {\mathbf{N}}^{+}\cap [1,n],$$

(25)

where ${\mathbf{N}}^{+}$ is the integer set, and ${\rho }_i$ is a given threshold that is a small positive number. The equilibrium of Stackelberg game can be reached by solving the optimization problem of Equation (23), or an optimization problem of minimization with constraints as follows:

$$\begin{array}{c}{\delta }^{*}=arg\underset{\delta }{min}r\left({\left.N_c^{*}\right|}_{\delta },\delta \right),\hfill \\ \begin{array}{ccc}s.t.& {\displaystyle \sum _{i=1}^n}{\displaystyle \sum _{j\in N_c\left(i\right)}}{\delta }_{ji}\le C,{\delta }_{ji}\in [0,1]& j,i\in {\mathbf{N}}^{+}\cap [1,n].\end{array}\hfill \end{array}$$

(26)

In summary, the algorithm of structure optimization under DoS attacks is described in Algorithm 1.

Algorithm 1 Algorithm of Structure Optimization under DoS Attacks

Input: : the radial basis dimension l, area number n, learning rate ${\beta }_i$ and ${\gamma }_i$. Output: : system structure $N_c^{*}$ and optimal attack strategy ${\delta }^{*}$. Initialize $M_a$, $M_c$, $W_i$, $i=1,2,\dots ,n$ and $\delta \left(0\right)$, each element is subject to Gaussian distribution with small average value 0.01 and variance 0.14. Set $t=0$ and maximal iteration number as MAXITER; For given $\delta \left(t\right)$, find ${\left.{\widehat{N}}_c^{*}\right|}_{\delta \left(t\right)}$ by the iteration of Equation (20) and (25); Measure the gradient of $r\left[{\left.N_c^{*}\right|}_{\delta },\delta \left(t\right)\right]$ that is subject to $\delta$; Update $\delta$ to obtain $\delta (t+1)$ based on the gradient obtained in step c, and handle the constraints of Equation (26) by Lagrange method [29] or barrier function method [30]; Check whether the updated variance is less than a threshold or t equals to the maximal iteration number MAXITER. If yes, end the algorithm and go to step f, otherwise $t=t+1$ and return to step b; Obtain $N_c^{*}={\left.{\widehat{N}}_c^{*}\right|}_{\delta \left(t\right)}$ and ${\delta }^{*}=\delta \left(t\right)$ .

The obtained $N_c^{*}$ by Algorithm 1 is the optimized control structure of power system. The worst case of system performance with our controller is also known by Algorithm 1. However, we do not further consider the unstable control system caused by a large C [5] or the attacker has a lot of cyber resources to launch DoS attacks. These issues will be investigated in future.

5. Experiments and Analysis

This section illustrates two cases of IEEE 14 bus system and 24 bus system [31] to show the effectiveness and advantages of the proposed scheme. The scheme takes DoS attacks into account and considers the worst case of attacks under the constraints in the Stackelberg game model mentioned above. In the Stackelberg game model, the scheme uses the optimal structure design to enhance the system performance by reinforcement learning. We assume sub-system parameters are as follows:

$$\begin{array}{c}{A}_i=\left(\begin{array}{ccccc}0& -\sum _{j\in D_i}{T}_{ji}& 0& 0& 0\\ 1/M_i& -D_i/M_i& 1/M_i& 0& 0\\ 0& 0& -1/T_{di}& 1/T_{di}& 0\\ 0& -1/\left(T_{gi}{R}_{gi}\right)& 0& 1/T_{gi}& K_i/T_{gi}\\ 1& -b_i& 0& 0& 0\end{array}\right),B_{ji}=\left(\begin{array}{ccccc}0& 0& 0& 0& 0\\ \frac{T_{ji}}{M_i{\sum }_{h\in D_i}{T}_{ih}}& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\end{array}\right),\hfill \\ B_i={\left(\begin{array}{ccccc}0& 1/M_i& 0& 0& 0\end{array}\right)}^T,\hfill \end{array}$$

where $M_i$ and $D_i$ are inertia and damping constant respectively, $T_{gi}$ and $T_{di}$ are the governor and gas turbine constant respectively, and $R_{gi}$ and $T_{ij}$ are the regulation and synchronizing constant respectively [20]. The related parameters of sub-system are listed in

Table 1. Parameters of sub-system.

Parameter Name	Description	Value
$M_i$	inertia constant	0.2
$D_i$	damping constant	0.26
$T_{ji}$	synchronizing constant	0.5
$T_{di}$	governor constant	5
$R_{gi}$	regulation constant	0.5
$b_i$	frequency bias gain	1
$T_{gi}$	gas turbine constant	0.2
$K_i$	tie-line bias control gain	0.1

[24].

The constraints of DoS attacks are set as follows:

$$\sum _{i=1}^{14}\sum _{j\in N_c\left(i\right)}{\delta }_{ji}\le 1.4.$$

The subsystems in grid communicate with each other by communication channels. The above constraint means that the sum of all communication channel jamming possibilities is 1.4, or the average jamming possibility is 0.1 for each communication channel. The larger C in Equation (26) is, the more cyber resources for attackers would be taken. In the following simulations, we set the sampling period to 0.1 s.

5.1. Case I: IEEE 14 Bus Test System

The IEEE 14 bus test system is carried out under reinforcement learning control and DoS attacks. The initial state of power grid simulation is assumed to be affected by a large disturbance. The simulation time lasts 5 s. Without any control, we can see that the system collapses under serious DoS attacks after a large power grid disturbance occurs as shown in Figure 3. The frequency increases to a large value, which reaches to a maximal value about 40 Hz in 5 s. Therefore, a controller is required to maintain the stability. The parameters of reinforcement learning algorithm are listed in

Table 2. Parameter of the controller in the IEEE 14 bus test system.

Parameter Name	Description	Value
$\alpha$	damping factor for cost $p_i$	0.9
N	Horizon length for cost	10
${\beta }_i$	Learning rate of neural network	0.1
${\gamma }_i$	Weight of group sparse regulation term	0.012
$\left|Li\right|$	Norm of damping parameters $L_i$	0.1

. The detail of parameter selection principle can be referred to [2].

The parameter ${\beta }_i$ decides the result of IEEE 14 bus test system. If ${\beta }_i$ is too large, the online learning of Equation (20) diverges; otherwise, too small ${\beta }_i$ may cause the slow convergence rate. The selection of ${\gamma }_i$ is another key factor. Too large ${\gamma }_i$ may result in the poor control performance and high sparsity of $W_i$.

Figure 4 and Figure 5 show the results that use the proposed control scheme and the optimal DoS attack strategy. The optimal DoS attack strategy is obtained by solving Equation (26), and the optimal controllers are learned by the online iteration of Equation (20). According to the results in Figure 4, we can see the frequency deviation curves of all sub-systems converge to a steady value around 0. The maximal fluctuant magnitude of these curves is a small value about 1.2 Hz. The swings end in about 4 s.

For the optimal DoS attack strategy shown in Figure 5a, the attacks focus on the communication from buses 3, 6, 10, 11, and 13. For the optimal control structure about placement of sensor, actuator, and communication topology shown in Figure 5b, it shows the F-norm for block $W_{ij}$ in $W_i$, $i,j=1,2,\dots ,n$. If the norm value is large, it means sub-system j needs information from sub-system i. Thus, sub-system i needs to install sensors, and sub-system j requires actuators as well as the communication topology from sub-system i to j. The sensors are mainly installed in buses 2, 3, 6, 10, 11, 13, and 14. According to the solution, those communication lines that are not attacked maintain the system stable. Thus, the obtained solution is optimal for both attacker and defender (control system designer) as Nash equilibrium.

5.2. Case II: IEEE 24 Bus Test System

This section verifies the effectiveness of the proposed scheme in a relatively large system—IEEE 24 bus test system. The simulation time lasts 10 s. Both reinforcement learning control and DoS attacks are applied. The optimal attack strategy and the optimized control structure are obtained by solving the optimization of Equation (26). Specifically, the optimal attack strategy is solved by the offline optimization algorithm described in Section 4.2, and the optimized control structure is obtained by the online learning of Equation (20).

The parameters of reinforcement learning algorithm are listed in

Table 3. Parameter of controller in IEEE 24 bus test system.

Parameter Name	Description	Value
$\alpha$	damping factor for cost $p_i$	0.8
N	Horizon length for cost	10
${\beta }_i$	Learning rate of neural network	1
${\gamma }_i$	Weight of group sparse regulation term	0.005
$\left|Li\right|$	Norm of damping parameters $L_i$	0.1

. The detail of parameter selection can be referred to [2].

The simulation results are shown in Figure 6 and Figure 7. Figure 6 shows the frequency deviation curves of power grid under the worst case of DoS attacks. The results illustrate the effectiveness of propose solution under the large-size disturbance and DoS attacks. The swings of all the frequency deviation curves end in about 5 s. The magnitude of swings is small, and its maximal value is 1.5 Hz.

From the optimal attack strategy results shown in Figure 7a, we know the attacker should focus on the communications from buses 2, 3, 4, 14, 19, 21, and 22 with the attack possibility of 0.2–0.35. The optimal control structure under the worst case of DoS attacks is shown in Figure 7b. The sensor should be installed in buses 2, 3, 4, 6, 14, 15, 19, 22, 23, and 24. The weight of bus 19 is high, which means the information obtained from bus 19 has high importance. Thus, DoS attacks have such an attack strategy that attacks the communication lines of bus 19 with a relatively high attack rate.

6. Conclusions

This paper proposes a novel optimization method of control structure as well as a reinforcement learning method in an integrated neural network under DoS attacks. The reinforcement learning, which involves the frequency damping and control performance, is approximated by the integrated neural network. The frequency damping includes the desired control input holding the system stable and accelerating the convergence rate in online learning. The approximation of control performance is adopted as the reinforcement signal to enhance the long-term system performance. To obtain a reasonable control scheme and structure under DoS attacks, a Stackelberg game model is also proposed. The worst case of DoS attacks is considered and solved by the proposed optimization solution. The optimization solution also derives the optimal control structure, which involves the placement of sensors, actuators (DES), and the communication topology. The simulation results illustrate the effectiveness of the proposed scheme. The optimal DoS attacks and the control structure are obtained in the simulation. In future, the convexity of the game and the existence of game equilibrium will be analyzed. The constraints of DER considering the environment and variation will also be considered. In addition, the error of the proposed algorithm will be analyzed theoretically and numerically in the next steps.