Optimal Security Protection Strategy Selection Model Based on Q-Learning Particle Swarm Optimization

Abstract

With the rapid development of Industrial Internet of Things technology, the industrial control system (ICS) faces more and more security threats, which may lead to serious risks and extensive damage. Naturally, it is particularly important to construct efficient, robust, and low-cost protection strategies for ICS. However, how to construct an objective function of optimal security protection strategy considering both the security risk and protection cost, and to find the optimal solution, are all significant challenges. In this paper, we propose an optimal security protection strategy selection model and develop an optimization framework based on Q-Learning particle swarm optimization (QLPSO). The model performs security risk assessment of ICS by introducing the protection strategy into the Bayesian attack graph. The QLPSO adopts the Q-Learning to improve the local optimum, insufficient diversity, and low precision of the PSO algorithm. Simulations are performed on a water distribution ICS, and the results verify the validity and feasibility of our proposed model and the QLPSO algorithm.

1. Introduction

An industrial control system refers to the equipment, system, network, and controller applied in industrial production, whose main functions are to operate, control, and assist industrial automation production [1,2,3]. ICS is mainly divided into four parts: Supervisor Control And Data Acquisition (SCADA) [4], Distributed Control System (DCS) [5], Programmable Logic Controller (PLC) [6], and Process Control System (PCS) [7]. ICS is widely used in electric power, nuclear power plants, petroleum, and other industries [8,9]. Due to the increasingly close connection between ICS and the Internet, ICS often faces complex network attacks, which pose a huge threat to the social economy and people’s security. In recent years, cyber-attack incidents against ICS have occurred frequently, such as the Iranian “Stuxnet” incident in 2010 [10] and the Ukraine power grid cyber-attack in 2015 [11], causing serious property damage. However, there are essential differences between ICS and IT, which results in the unavailability of the traditional IT security protection strategies, such as access control, firewall, and so on, for the protection of ICS [12,13,14,15].

Recently, a hot research topic in the protection strategies for ICS has been to focus on the active protection, which implements active protection strategies based on the risk assessment on the current security risk of the ICS. Shameli-Sendi et al. proposed a retrospective burst response method based on an adaptive and cost-sensitive model [16]. This method takes into account the effectiveness of the application response. However, this approach does not meet the security requirements of ICS well. K. Deb et al. proposed the non-dominated sorting GA algorithm, which can keep the diversity of solutions at a high level [17]. Genetic algorithms generate defense and attack strategies, while fitness functions are used to infer dominant strategies. This method can maintain a high accuracy, but it will fall into the local optimal and cannot solve the optimal solution. Granadillo et al. proposed a geometric model to select a best-response combination based on the response Return on Investment (RORI) index [18]. However, this approach ignores attack modeling. Attack modeling is critical because cyber attacks are becoming more destructive these days, and the response time is the determining factor. Miehling et al. used the concept of POMDP to model defense issues [19]. Their goal is to provide the best dynamic defense strategy against ongoing cyber attacks against protected systems. However, this approach uses only a single metric: the cost of deployment costs of attack or defense operations, and cannot quantify the problem well.

In the stage of implementing protection strategies, each protection strategy is accompanied with costs. How to balance the benefits and costs of protection under the constraint of limited resources is a typical optimization problem. Constructing such a reasonable optimization problem and finding its optimal solution are the main challenges.

In this paper, we propose an optimal security protection strategy selection model based on QLPSO. In order to evaluate the security situation of ICS, we introduce the protection strategy into the Bayesian attack graph, and calculate the probability of each attribute node being attacked to obtain the risk value. When choosing the optimal protection strategy, the general PSO algorithm often falls into a local optimum. To solve this problem, QLPSO is proposed, which updates the parameters of PSO algorithm through Q-Learning. Finally, we verify the validity and feasibility of our model and the QLPSO algorithm for a water distribution ICS.

2. Related Work

A lot of research has been done on constructing the optimal security protection strategy for various complex systems. Jaquith proposed security metrics, such as attack cost, defense implementation cost, attack impact, operation cost, and other indicators to define the factors of optimal solution [20]. However, this approach lacks specific and commonly used measurement systems to reliably evaluate countermeasures. S. Bandyopadhyay et al. proposed the single-objective optimization problem and multi-objective optimization problem to determine the optimal strategy [21]. Nevertheless, they did not discuss how to find the optimal strategy. Poolsappasit et al. proposed a multi-index quantitative analysis method based on cost and benefit, and calculated the optimal security protection strategy through genetic algorithm [22], while it is also easy to fall into the local optimum. Yigit et al. developed a network defense algorithm under limited budget conditions [23]. However, the algorithm only considers the minimum cost, does not consider the attack benefit, and lacks comprehensive measurement. Lei et al. developed a Markov game-based strategy selection mechanism for a balance between the defensive revenue and network service quality [24]. However, the method has high time cost. Herold et al. defined the response selection approach based on user-defined cost metrics to counteract security incidents in complex network environments [25]. However, this method does not meet the ICS security requirements well due to the ignorance of the balance between security risks and protection costs. S. A. Butler proposed a multi-attribute risk assessment framework, in which several complex metrics are introduced, such as total security control cost, attack strategy cost, and so on [26,27]. Roy et al. proposed a cost-effective countermeasure system for cyber attacks using various functions to form the objective function [28]. Viduto et al. proposed a new risk assessment and optimization model to solve the problem of security countermeasure selection, in which the total cost of security control and the total risk level of the system together constitute the objective function [29]. In this approach, historical databases are used without consideration for the first experienced or zero-day attacks, exposing the system to undue risk. R. Dewri proposed Genetic Algorithm (GA) and S. Wang et al. proposed Ant Colony Optimization (ACO), which utilizes single-objective and multi-objective optimization cost functions to select the optimal set of strategies [30,31]. Similarly, the indicators used in the methodology are inconsistent with globally accepted standards. B. Kordy et al. proposed a game selection method combining ADTree modeling with integer linear programming [32,33]. The only response considered in this approach is to fix the vulnerabilities, and the many aspects (risk, cost, budget constraints, etc.) that define the optimal response plan are not considered. Speicher et al. proposed a Stackelberg programming algorithm that models the game choice problem as a two-person game in which the defender applies mitigation strategies (to minimize the probability of a successful attack) and the attacker tries to counter and maximize his chances of successfully executing the attack [34]. However, in order to improve the efficiency, this method only considers the critical attack path. Zenitani et al. proposed a multi-objective cost-benefit optimization algorithm, which can replace the use of multi-objective optimization algorithms and obtain the optimal solution through a series of iterations [35]. However, this method has only been tested in some networks, and its performance in practice cannot be guaranteed.

The above-mentioned research focuses mainly on balancing benefits and costs, and on constructing the objective function, and the research on the algorithm for selecting the optimal protection strategy has not been discussed in-depth. In addition, there are some other problems, such as the use of indicators that are not consistent with globally accepted standards, and so on. Based on the above research, this paper proposes an optimal security protection strategy selection model based on QLPSO. The model mainly answers the following questions: (1) how to evaluate the security risk of the ICS; (2) how to determine the objective function according to the security risk of the ICS and protection cost; (3) how to choose the optimal protection strategy.

3. Model Framework

Our model can be divided into three modules: ICS security risk assessment, construction of objective function, and optimal protection strategy selection. The framework of the model is shown in Figure 1.

(1)

ICS security risk assessment: First, we build a Bayesian attack graph based on the network configuration and asset information of the ICS [36], then calculate the exploit success rate of each edge of the attack graph, and construct the local conditional probability distribution (LCPD) table according to the exploit success rate. Finally, the prior probabilities of all attribute nodes being attacked are calculated [37];

(2)

Construction of the objective function: Firstly, we construct all possible protection strategies based on the network configuration and asset information of the ICS, then quantify the attack benefit and protection cost, and finally construct the objective function based on both the attack benefit and protection cost;

(3)

Optimal protection strategy selection: First, we design the Q-Learning particle swarm optimization algorithm (QLPSO), then solve the objective function using the QLPSO, and finally find the optimal protection strategy.

4. ICS Security Risk Assessment

4.1. Definition of Bayesian Attack Graph

The Bayesian attack graph is a directed acyclic graph, defined as: $BAG=(S,E,A,P)$, where

(1)

$S=\{S_1,S_2,\dots ,S_n\}$ is the set of all attribute nodes of the attack graph.

(2)

$E=\left\{\dots ,E_{ij},\dots \right\}$ is the set of all directed edges of the attack graph, where $E_{ij}$ has two end nodes $S_i$ and $S_j$, and $S_i$ is the parent node and $S_j$ the child node.

(3)

$A=\left\{A_1,A_2,\dots ,A_n\right\}$ means atomic attack. $A_i=1$ means that the attack has been launched, otherwise $A_i=0$.

(4)

$P=\{P\left(S_1\right),P\left(S_2\right),\dots ,P\left(S_n\right)\}$ is the set of the probabilities that the attribute nodes can be attacked. $P\left(S_i\right)$ indicates the success probability of attribute node $S_i$ of being attacked.

The Bayesian attack graph is established by combining the network configuration and asset information, and can be directly constructed using the MulVAL tool [38].

4.2. Calculation of Success Probability of Vulnerability Exploitation

We adopt the vulnerability scoring system CVSS (common vulnerability scoring system [39]) to calculate the probability of successful exploitation of each vulnerability, which is given by

$$p\left(S_i\right)=2\times AV\times AC\times AU$$

(1)

where $AV$, $AC$, and $AU$ are the CVSS availability indicators. The specific scores of CVSS indicators are shown in

Table 1. CVSS score.

Index	Rank	Score
	local access	0.395
Access Vector (AV)	adjacent network access	0.646
	network accessible	1.0
	high	0.35
Access complexity (AC)	medium	0.61
	low	0.71
	multiple instances of authentication	0.45
Authentication (AU)	single instance of authentication	0.56
	no authentication	0.704

. $AV$ is the attack route value. The higher the $AV$, the farther the attacker can launch an attack. $AC$ is the attack complexity value. The higher the $AC$ score, the lower the attack complexity of the attacker. $AU$ is the authentication value. The higher the $AU$ score, the less authentication times the attacker launches attacks. $v_i$ represents the exploit between the current node and its parent node.

4.3. Calculation of Local Conditional Probability Distribution (LCPD)

Conditional probability [40] represents the successful exploitation probability of an attribute node under the influence of its parent node set, denoted by $p(S_i\mid Par\left(S_i\right))$. $Par\left(S_i\right)$ refers to the set of parent nodes of the $S_i$. The relationship between the attribute node and its parent node is shown in Figure 2. $d_j$ represents the respective type of each attribute node, which is divided into two types: AND and OR.

When $d_{\mathrm{j}}=\mathrm{AND}$, attribute node $S_i$ is only exploited when all its parent nodes are exploited,

$$p\left(S_i\mid Par\left(S_i\right)\right)=\prod _{S_j\in Par\left(S_i\right)}p\left(S_j\right)$$

(2)

When $d_{\mathrm{j}}=\mathrm{OR}$, attribute node $S_i$ can be exploited when any of its parent nodes are exploited,

$$p\left(S_i\mid Par\left(S_i\right)\right)=1-\prod _{S_j\in Par\left(S_i\right)}\left[1-p\left(S_j\right)\right]$$

(3)

4.4. Prior Probability Calculation

The prior probability of an attribute node $S_i$ refers to the probability that the attribute node $S_i$ can be reached under static conditions. It is expressed as the joint probability of the attribute node $S_i$ and all attribute nodes in the path to reach the attribute node $S_i$.

The prior probability of attribute node $S_i$ is calculated as follows. When $d_{\mathrm{j}}=\mathrm{AND}$, attribute node $S_i$ is only exploited when all its parent nodes are exploited,

$$P\left(S_i\right)=\prod _{S_j\in Par\left(S_i\right)}P\left(S_j\right)p\left(S_i\right)$$

(4)

When $d_{\mathrm{j}}=\mathrm{OR}$, attribute node $S_i$ can be exploited when any of its parent nodes are exploited,

$$P\left(S_i\right)=1-\prod _{S_j\in Par\left(S_i\right)}\left[1-P\left(S_j\right)p\left(S_i\right)\right]$$

(5)

When the LCPD tables are calculated, the prior probability of each attribute node is equal to the conditional probability of the attribute node multiplied by the conditional probability of its parent node.

5. Construction of Objective Function

5.1. Protection Strategy

The M is a set of protection strategies, denoted as $M=\{M_1,M_2,\dots ,M_n\}$, where $M_i$ is a protection strategy that can be operated on the attribute node $S_i$ to reduce its risk of being attacked. $M_i=1$ represents that the protection strategy is enabled, otherwise $M_i=0$. When the protection strategy is enabled, the exploit success probability of attribute nodes will be affected, followed by a certain reduction in the exploitation probability. That is

$$p\left(S_i\mid M_i=1\right)<p\left(S_i\mid M_i=0\right).$$

(6)

As a consequence, we have

$$P\left(S_i\mid M\ne \mathbf{0}\right)\le P\left(S_i\mid M=\mathbf{0}\right).$$

(7)

Implementing protection strategies will inevitably incur protection costs. The protection cost is represented by $COST=\left\{COS{T}_1,COS{T}_2,\dots ,COS{T}_n\right\}$, where $COS{T}_i$ represents the cost of implementing the protection strategy $M_i$. $COS{T}_i$ is defined as [41]:

$$COS{T}_i={\omega }_i\times value\times 100$$

(8)

where ${\omega }_i$ is the normalized weight of the protection strategies, and $value$ represents the value of the asset. Therefore, the total protection cost of implementing the protection strategies $M=\{M_1,M_2,\dots ,M_n\}$ is given by.

$$C\left(M\right)=\sum _{i=1}^n{M}_{i}COS{T}_i.$$

(9)

5.2. Attack Benefit

The attack benefit of attribute node $S_i$ is expressed as $AG(S_i$), which refers to the attack benefit obtained by successfully attacking attribute node $S_i$. $AG(S_i$) can be calculated as

$$AG\left(S_i\right)=P\left(S_i\right)\times value.$$

(10)

In addition, the attack benefit of attribute node $S_i$ under the protection strategies M is given by

$$AG\left(S_i|M\right)=P\left(S_i|M\right)\times value.$$

(11)

Therefore, the total attack benefit under the protection strategies M can be obtained by summing the benefits of all attribute nodes, that is

$$AG\left(M\right)=\sum _{S_i\in S}AG\left(S_i|M\right).$$

(12)

5.3. Attack Benefit-Protection Cost Objective Function

Under the above definitions of attack benefit and protection cost, it is easier to illustrate our objective as being to minimize both the attack benefit and the protection cost. Therefore, the objective function can be expressed as

$$min\left(\delta AG\right(M)+(1-\delta \left)C\right(M\left)\right)$$

(13)

subject to

$$C\left(M\right)<B$$

(14)

where $\delta$ and $1-\delta$ are the preference weights of attack benefit and defense cost, respectively, and $0\le \delta \le 1$. B is the constraint of total protection cost.

6. Q-Learning Particle Swarm Optimization Algorithm

6.1. Particle Swarm Optimization

In 1995, Kennedy and Eberhardt proposed the PSO algorithm, which is motivated by imitating the foraging activities of birds for solving single-objective optimization problems [42]. The state of each particle i in the particle swarm has two fundamental properties: velocity $V_i$ and position $X_i$, where:

$$V_i=\left(V_{i1},V_{i2},\dots ,V_{iD}\right),i=1,2,\dots ,\mathrm{N}$$

(15)

$$X_i=\left(X_{i1},X_{i2},\dots ,\dots ,X_{iD}\right),i=1,2,\dots ,\mathrm{N}$$

(16)

Among them, $V_{ij}$ represents the velocity of the ith particle in the jth dimension, and $X_{ij}$ represents the position of the ith particle in the jth dimension. D is the dimension.

All particles have memory, which enables them to remember their local optimal position P. At the same time, all P of different particles will be shared in the whole population, and the optimal P is regarded as the global optimal position $Gp$ reached by the population. Based on the obtained P and $Gp$, all particles in the population are updated in position and velocity using the following equations [43].

$$\begin{array}{c}\begin{array}{c}{V}_{id}^{t+1}=weight\times V_{id}^t+Lp\times rand(0,1)\times \left(P_{id}^t-X_{id}^t\right)\\ +Lb\times rand(0,1)\times \left(G{p}_d^t-X_{id}^t\right)\end{array}\hfill \end{array}$$

(17)

$$\begin{array}{c}\begin{array}{c}{X}_{id}^{t+1}=X_{id}^t+V_{id}^t\end{array}\hfill \end{array}$$

(18)

Among them, $weight$ is the inertia weight, $Lp$ is the self-learning factor, $Lb$ is the global learning factor, rand (0, 1) is a random number in [0, 1]. $V_{id}^t$ represents the dth dimensional velocity of the ith particle in the tth generation. $P_{id}^t$ indicates the local optimal particle of the ith particle in the dth dimension of the tth generation. $G{p}_d^t$ represents the globally optimal particle in the dth dimension of the tth generation. $X_{id}^t$ represents the dth dimensional position of the ith particle in the tth generation.

6.2. Q-Learning

Q-Learning is a learning method proposed by Watkins in 1989 [44]. In Q-Learning, an agent learns in a ‘trial and error’ way, and the reward guides behavior obtained by interacting with the environment, and the goal is to make the agent obtain the maximum reward.

Q-Learning has four key elements: Q table, state, action, and reward. The process of Q-Learning is [45]: the agent selects the action with the largest Q value from the Q table according to the state. After the action is completed, the state changes, and the reward is determined according to the quality of the state change, and then according to the current state, the next state, actions, and reward values update the Q table, in turn.

The updated method of the Q table is as follows [46].

$$\begin{array}{c}Q\left(s{t}_{t+1},a{t}_{t+1}\right)=(1-\alpha )Q\left(s{t}_t,a{t}_t\right)+\alpha \left[R\left(s{t}_t,a{t}_t\right)+\gamma \underset{at}{max}Q\left(s{t}_{t+1},at\right)\right]\end{array}$$

(19)

where, $s{t}_t$ is the current state, $s{t}_{t+1}$ is the next state. $a{t}_t$ is the action taken for the current state, and $a{t}_{t+1}$ is the action taken for the next state. $\alpha$ and $\gamma$ are the learning rate and discount factor, respectively, R($s{t}_t$,$a{t}_t$) is the reward generated by the action $a{t}_t$ taken by the state $s{t}_t$. $ma{x}_{at}Q(s{t}_{t+1},at)$ refers to the maximum Q value in the case of state $s{t}_{t+1}$.

The model of Q-Learning is shown in Figure 3.

6.3. Q-Learning Particle Swarm Optimization (QLPSO)

The PSO algorithm can be directly used to solve the optimization objective function Equation (13). However, PSO often falls into local optimum due to its fixed parameters. In this paper, we consider the QLPSO algorithm, which could update the parameters of PSO algorithm through the Q-Learning to avoid the local optimum problem [47].

The state, action, Q table, and reward are also the core elements of the QLPSO algorithm, which are shown in Figure 4.

(1)

States

Unlike PSO, which has only one state, QLPSO has two states: objective space state and decision space state. The objective space state needs to consider the relationship between the particle and the global optimal particle position. The decision space needs to consider the relationship between the fitness of the particle and the fitness of the global optimal particle.

The decision space state has four sub-states: DFarthest, DFarther, DNearer, and DNearest. They respectively represent the relative state of the Euclidean distance between the particle and the global optimal position $Gp$ compared with the size of the search space. The objective space state also has four sub-states: the largest fitness difference, the larger fitness difference, the smaller fitness difference, and the smallest fitness difference. They represent the relative state of the particle’s fitness compared to the global optimal fitness and the global worst fitness difference, respectively. In this article, we only need to consider the fitness value between the two solutions.

The specific information of decision space state and objective space state is shown in

Table 2. Decision space state.

Relative Distance (Rd)	Decision Space State
0$\le Rd\le 0.25△R$	DNearest
0.25$△R\le Rd\le 0.5△R$	DNearer
0.5$△R\le Rd\le 0.75△R$	DFarther
0.75$△R\le Rd\le △R$	DFarthest

and

Table 3. Objective space state.

Single-Objective Problem Relative Fitness (Rf )	Objective Space State
0$\le Rf\le 0.25△F$	FSmallest
0.25$△F\le Rf\le 0.5△F$	FSmaller
0.5$△F\le Rf\le 0.75△F$	FLarger
0.75$△F\le Rf\le △F$	FLargest

[48].

In Table 2, $Rd$ is the Euclidean distance between a certain particle and the global optimal particle $Gp$. $△R$ is the range of decision space search. In Table 3, $Rf$ is the fitness, which refers to the fitness difference between a certain particle and the global optimal particle $Gp$. $△F$ is the difference between the fitness of the global best particle and the fitness of the global worst particle.

(2)

Action

There are also four types of actions, which correspond to different parameters of the particle swarm: $weight$, $Lp$, and $Lb$. Different values of $weight$, $Lp$ and $Lb$ will affect the exploration form of the particle. The larger the $weight$ is, the stronger the global exploration ability and the weaker the local exploration ability. On the contrary, the smaller the $weight$ is, the weaker the global exploration ability and the stronger the local exploration ability. The larger the $Lp$, the stronger the global exploration ability. The larger the $Lb$, the stronger the particle convergence ability [49]. The detailed parameter settings are shown in

Table 4. Detailed parameter settings of the actions.

Search Status	Weight	Lp	Lb
Large scale search	1	0.5	0.5
Small scale search	0.8	2	1
Slow search	0.6	1	2
Fast search	0.4	0.5	0.5

.

(3)

Q table

Since there are four types of objective space states, decision space states, and actions, the Q table of QLPSO is different from the two-dimensional Q table used in general Q-Learning. The three-dimensional Q table used here is a $4\times 4\times 4$ three-dimensional Q table. The three-dimensional Q table is shown in Figure 5. As shown in the figure, we first determine the state of the objective space and the state of the decision space, such as (the closest distance, the smallest fitness difference). Then, according to the state of the objective space and the state of the decision space, the action with the largest Q value corresponding to the state is selected.

(4)

Rewards

After selecting a certain action, if the fitness value becomes worse, it should be punished. Otherwise, if the fitness value becomes better, it should be rewarded. The reward function defined in this paper is as follows.

$$\begin{array}{c}R=Rf(state+1)-Rf\left(state\right)\end{array}$$

(20)

where $Rf\left(state\right)$ and $Rf(state+1)$ represent the fitness values of the current state and the next state, respectively.

The flowchart of the QLPSO algorithm is shown in Figure 6 and described as follows.

(1)

Initialize the population and Q table;

(2)

The state of each particle is determined according to its position in the objective space and decision space;

(3)

The action (parameter) of the particle is determined using the Q table;

(4)

Update the particles according to the parameters determined in the previous step;

(5)

Update the Q table according to the reward function;

(6)

These steps are repeated for all particles in each generation until the number of iterations is reached.

7. Experimental

7.1. Experimental Scenario

The experimental scenario in this paper is a water distribution system, as shown in Figure 7. There are three PLCs in the water distribution system, which respectively control the water pump, the pipeline, and the water tanker in the virtual system. The platform can be used for industrial control security attack and defense drills, security risk assessment, and other related experiments [50].

7.2. Generating a Bayesian Attack Graph

Asset and vulnerability information is obtained by scanning assets as shown in Figure 8. The asset consists of three PLCS, each of which has the same vulnerability CVE-1999-0517. At the same time, there is some additional information for each PLC. The attack graph of the water distribution system is generated by using the MulVal. It is shown in Figure 9 and

Table 5. Prior probabilities of each node of the Bayesian attack graph.

Attribute Nodes	Prior Probability	Attribute Nodes	Prior Probability	Attribute Nodes	Priori Probability
$S_1$	0.5000	$S_2$	0.1812	$S_3$	0.5000
$S_4$	0.5000	$S_5$	1.0000	$S_6$	1.0000
$S_7$	0.3967	$S_8$	0.5000	$S_9$	1.0000
$S_{10}$	1.0000	$S_{11}$	1.0000	$S_{12}$	1.0000
$S_{13}$	0.8372	$S_{14}$	0.3257	$S_{15}$	0.5000
$S_{16}$	1.0000	$S_{17}$	1.0000	$S_{18}$	0.5000
$S_{19}$	1.0000	$S_{20}$	0.6793	$S_{21}$	1.0000
$S_{22}$	1.0000	$S_{23}$	1.0000	$S_{24}$	0.9863
$S_{25}$	0.8000	$S_{26}$	1.0000	$S_{27}$	1.0000
$S_{28}$	0.9960	$S_{29}$	0.5814	$S_{30}$	1.0000
$S_{31}$	0.5641	$S_{32}$	1.0000	$S_{33}$	1.0000
$S_{34}$	0.9530

.

7.3. Attack Benefits and Protection Costs

The prior probabilities of all attribute nodes are calculated through the LCPD table, as shown in

Table 6. Details of the attack graph.

Node	Node Information
1	execCode(192.168.0.1,someUser):0.3623
2	RULE 2 (remote exploit of a server program):0.3623
3	netAccess(192.168.0.1,udp,161):0.906
4	RULE 5 (multi-hop access):0.3173
5	hacl(192.168.0.10,192.168.0.1,udp,161):1.0
6	execCode(192.168.0.10,someUser):0.3967
7	RULE 2 (remote exploit of a server program):0.3967
8	netAccess(192.168.0.10,udp,161):0.992
9	RULE 5 (multi-hop access):0.8
10	hacl(192.168.0.1,192.168.0.10,udp,161):1.0
11	RULE 5 (multi-hop access):0.8
12	hacl(192.168.0.2,192.168.0.10,udp,161):1.0
13	execCode(192.168.0.2,someUser):0.389
14	RULE 2 (remote exploit of a server program):0.389
15	netAccess(192.168.0.2,udp,161):0.9727
16	RULE 5 (multi-hop access):0.8
17	hacl(192.168.0.1,192.168.0.2,udp,161):1.0
18	RULE 5 (multi-hop access):0.3173
19	hacl(192.168.0.10,192.168.0.2,udp,161):1.0
20	RULE 6 (direct network access):0.8
21	hacl(internet,192.168.0.2,udp,161):1.0
22	attackerLocated(internet):1.0
23	networkServiceInfo(192.168.0.2,sun sunos,udp,161,someUser):1.0
24	vulExists(192.168.0.2,CVE-1999-0517,sun sunos,remoteExploit,privEscalation):0.4998
25	RULE 6 (direct network access):0.8
26	hacl(internet,192.168.0.10,udp,161):1.0
27	networkServiceInfo(192.168.0.10, sun sunos,udp,161,someUser):1.0
28	vulExists(192.168.0.10,CVE-1999-0517,sun sunos,remoteExploit,privEscalation):0.4998
29	RULE 5 (multi-hop access):0.3112
30	hacl(192.168.0.2,192.168.0.1,udp,161):1.0
31	RULE 6 (direct network access):0.8
32	hacl(internet,192.168.0.1,udp,161):1.0
33	networkServiceInfo(192.168.0.1,sun sunos,udp,161,someUser):1.0
34	vulExists(192.168.0.1,CVE-1999-0517,sun sunos,remoteExploit,privEscalation):0.4998

. The quantification standard of cost index of protection operation is shown in

Table 7. Protection cost quantification standard.

Index	Weight (${\mathit{\omega }}_{\mathit{i}}$)
Disable cost	0.357
Disconnect cost	0.286
Patch cost	0.214
Install cost	0.143

[34]. In Table 7, $Pi$ refers to the impact of the implementation strategy on vulnerability utilization. The specific operation, cost and other details of the protection strategies are shown in

Table 8. Detailed parameters of the protective strategies.

Protective Strategies	Protective Action	COST	Pi
$M_1$	Disconnect 192.168.0.1–192.168.0.10	14	0.25
$M_2$	Disconnect Disconnect 192.168.0.2–192.168.0.10	14	0.25
$M_3$	Disconnect Internet 192.168.0.10	15	0.30
$M_4$	Disconnect Internet 192.168.0.2	15	0.30
$M_5$	Disconnect Internet	22	0.05
$M_6$	Disable Internet	22	0.05
$M_7$	Disable Internet direct network access	22	0.05
$M_8$	Disconnect Internet 192.168.0.1	15	0.30
$M_9$	Disable multi-hop access 192.168.0.10	12	0.10
$M_{10}$	Disable udp	12	0.10
$M_{11}$	Disable direct network access 192.168.0.10	12	0.10
$M_{12}$	Disable direct network access 192.168.0.2	12	0.10
$M_{13}$	Disable direct network access 192.168.0.1	12	0.10
$M_{14}$	Disable netAccess 192.168.0.10	12	0.10
$M_{15}$	Disable networkService 192.168.0.10	18	0.45
$M_{16}$	Patch CVE-1999-0517 192.168.0.10	18	0.45
$M_{17}$	Disable service programs	25	0.20
$M_{18}$	Disconnect 192.168.0.10–192.168.0.1	14	0.25
$M_{19}$	Disable execCode 192.168.0.10	20	0.30
$M_{20}$	Disconnect 192.168.0.10	20	0.30
$M_{21}$	Disconnect 192.168.0.10–192.168.0.2	14	0.25
$M_{22}$	Disconnect 192.168.0.1–192.168.0.2	14	0.25
$M_{23}$	Disable multi-hop access 192.168.0.1	20	0.30
$M_{24}$	Disable multi-hop access 192.168.0.2	20	0.30
$M_{25}$	Disable netAccess 192.168.0.2	20	0.30
$M_{26}$	Disable netAccess	12	0.10
$M_{27}$	Disable networkService 192.168.0.2	14	0.25
$M_{28}$	Patch CVE-1999-0517 192.168.0.2	18	0.45
$M_{29}$	Disable execCode 192.168.0.2	25	0.20
$M_{30}$	Disable multi-hop access	20	0.30
$M_{31}$	Disconnect 192.168.0.2–192.168.0.1	14	0.25
$M_{32}$	Disable netAccess 192.168.0.1	12	0.10
$M_{33}$	Disable service program 192.168.0.1	12	0.10
$M_{34}$	Disable networkService 192.168.0.1	18	0.45
$M_{35}$	Patch CVE-1999-0517 192.168.0.1	18	0.45
$M_{36}$	Install ids	30	0.20

.

7.4. Experimental Results

In this paper, assuming the benefits of attack are as important as the costs of protection. The fitness function parameter $\delta$ is set to 0.5, and there is no upper limit to the set-up cost B.

The experimental results are shown in Figure 10. The blue curve is the experimental result without protection strategies. The fitness value is always at 546.0160, and the attack benefit is very large, which means that without protection strategies, the attacker can very easily launch attacks to obtain attack benefit. The red curve is the experimental result of the ordinary particle swarm. It can be seen that the fitness value is always 430.0326, and the attack benefit plus protection cost is still quite large, indicating that the optimal protection strategy has not been obtained. The yellow curve is the experimental result of the Q-Learning particle swarm. It can be seen that the fitness value tends to be stable at 327.9708, and the attack benefit plus protection cost reaches the minimum value. The experiments show that the Q-Learning particle swarm algorithm found the optimal protection strategy [0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0], and the attack benefit plus protection cost reached a minimum of 332.4039.

It is proved that the optimal protection strategy obtained by QLPSO algorithm has a good protection effect and minimizes the attack benefit and protection cost. Therefore, the water distribution system is well protected by the strategy selected through QLPSO.

8. Conclusions

Constructing the optimal security protection strategy is of great significance to the ICS. Since the traditional PSO algorithm tends to fall into local optimum when choosing the optimal protection strategy, in this paper, we propose an optimal security protection strategy selection model and Q-Learning particle swarm optimization algorithm. The algorithm can easily select the optimal protection strategy and the experimental results verify the feasibility and effectiveness of our model and the QLPSO algorithm.