False Data Injection Attack Detection in Smart Grid Using Energy Consumption Forecasting

Abstract

Supervisory Control and Data Acquisition (SCADA) systems are essential for reliable communication and control of smart grids. However, in the cyber-physical realm, it becomes highly vulnerable to cyber-attacks like False Data Injection (FDI) into the measurement signal which can circumvent the conventional detection methods and interfere with the normal operation of grids, which in turn could potentially lead to huge financial losses and can have a large impact on public safety. It is imperative to have an accurate state estimation of power consumption for further operational decision-making.This work presents novel forecasting-aided anomaly detection using an CNN-LSTM based auto-encoder sequence to sequence architecture to combat against false data injection attacks. We further present an adaptive optimal threshold based on the consumption patterns to identify abnormal behaviour. Evaluation is performed on real-time energy demand consumption data collected from the Australian Energy Market Operator. An extensive experiment shows that the proposed model outperforms other benchmark algorithms in not only improving the data injection attack (95.43%) but also significantly reducing the false positive rate.

1. Introduction

While electricity consumption growth has slowed in recent years, it is predicted to increase significantly in the future. Peak electricity demand (times when electricity demand is at its highest) is growing at a much faster rate than overall consumption and is increasingly a challenge for the grid. Smart Grid consists of a network of measurement units or sensors which are coordinated through remote control and automation to balance the supply and demand by ensuring a reliable operation of electricity grid and benefits the environment, the community, and the consumer. It is incorporated with information and communication technologies which makes the interactions between devices more efficient [1]. It is a complex cyber-physical system which has huge advantages like identifying and isolating faulted area(s) and restoring unaffected area(s) [2]. Utilization of computing and communications effectively improves the quality of monitoring and control in the smart grid. However, it also has vulnerabilities to cyber-attacks [3].

Quick development of smart grids coupled with IoT devices has brought the issue of cyber-attacks; thus, energy companies and electricity systems face a substantial and growing threat of cyber attacks. Because the power grid is a critical infrastructure, it is a tempting target for sophisticated and well-equipped attackers. Cyber attacks are usually based on Malicious Software (malware) that must communicate with a controlling entity over the network to coordinate and propagate. From small viruses built by isolated groups to massive and complex cyber-attacks done by states and bigger entities, electrical power stations and nuclear facilities have shown breaches that governments must deal with. For example, Ukraine Blackout compromised three electric power distribution companies affecting approximately 225,000 people for several hours. The attack was done using spearphishing and a Trojan horse called BlackEnergy which was able to delete the data, destroy the hard drives, and take control of infected computers. Not only attacking utilities equipment, it went further and launched a coordinated denial-of-service attack on the support phone number of companies who manage the power station. Consequently, users could not reach the support centers to inquire about the breakdown. The severity of a coordinated cyber attack is shown in [4]. It was the first publicly acknowledged incident, and it exposed the weakness of highly automated cyber-physical systems like smart grids. There was a significant increase of cyber-attacks in 2017 which grew even more in 2018 as described by the United States Department of Homeland Security—from Russian infiltration into the American electrical network to more than 4300 cyber-attacks of the French network (Electricity Transmission Network known as RTE). In September 2019, Dtack malware was found in India’s Kudankulam Nuclear Power Plant, which was used to collect and harvest data from the victim computer [5]. One of the captured Dtrack samples contained information about the power plant’s internal network. Since these cyber-attacks may lead to catastrophic events by misleading the control system, it is crucial to detect and mitigate these attacks.

According to the 2019 report of Kaspersky ICS CERT [6], injection is one of the most common vulnerabilities in an Industrial Control system (ICS). The recent literature shows that False Data Injection (FDI), a data integrity attack similar to the man-in-the-middle or spoofing attack, can severely damage the reliability of the system [3]. As illustrated in [7], malicious attacks can affect the operational decision in Supervisory Control and Data Acquisition (SCADA) of the grid by hampering correspondences between devices. A spiteful attack can create disturbances in different segments of a smart grid by blocking communication or by adding falsified information [8]. A well organised simultaneous cyber-attack on multiple substations can lead to whole system breakdown [9]. False data on power demand may lead unpleasant circumstances such as additional energy generation or lack of power, which may become a public safety issue [10]. Anomaly detection is one of the most effective ways of countering these FDI attacks [11].

The SCADA continuously monitors and controls the power flow to ensure the operation of uninterrupted operation of the grid. It contains a large amount of connectivity among the field devices which provides information for decision-making in the center for remote maintenance of the system [12]. This communication infrastructure is critical to the SCADA, but it also increases the vulnerability of the system to cyber attacks. If a FDI attack is successful in compromising data from the field devices, the SCADA will seriously be crippled. A fundamental requirement of a smart grid is to have accurate state estimation to operate properly that completely relies on the SCADA measurements. The data integrity between the field devices and the center has to be maintained all the time. However, the communication between the field devices and the center is one of the most vulnerable places for FDI attacks [13]. To neutralize cyber-attacks, a combination of Firewalls and Intrusion Detection Systems (IDS) is recommended by the Department of Homeland security [14]. The most effective measurement against FDI is to detect, prevent, and detach irregular activity of a particular system by continuous monitoring [15]. In order to overcome this challenge, two kinds of detection systems were introduced which were model-based detection algorithms and data-driven detection algorithms [16]. In model-based detection algorithms, the whole grid system is modeled with real-time measurements and system parameters, whereas the data-driven detection algorithms do not require any system parameters or models. The main disadvantage of model-based detection algorithms is that it requires system parameters, and, if there are uncertainties in these parameters, then the performance of the algorithms will be affected. Model-based algorithms are also eminently computationally complex [16].

In this paper, a data-driven two step anomaly detection engine based on a deep learning model CNN-LSTM Auto-encoder is proposed. The anomaly detection technique requires accurate forecasting on the real-time data to efficiently detect the presence of anomalous behaviours [17]. Different machine learning models and a statistical models are compared in this work to benchmark the proposed forecasting model. Experimental results show that, out of these considered machine learning algorithms, the CNN-LSTM Auto-encoder had the most accurate forecasting on the dataset, which motivated us to utilize this method for developing the anomaly detection engine. The CNN-LSTM Auto-encoder recognizes the behavior patterns of the real-time data using the historical data. Then, an optimum threshold boundary $\sigma$ was empirically selected depending on the false positive rate, false negative rate, and accuracy of an anomaly detection engine considering a wide range of attack scenarios. The key contributions made in this paper are as per the following:

• proposes a real-time forecasting aided anomaly detection technique to identify a specific type of cyber attack in the SCADA system known as false data injection attack.
• present CNN-LSTM auto-encoder architecture and determine the existence of the intrusions if the field measurements deviate significantly from the forecasts.
• experiment is conducted on a real-time dataset (Australian Energy Market Operator) that shows a significant gain in performance in comparison to benchmark methods.

The rest of the paper is organized as follows: in Section 2, literature related to forecasting based anomaly detection engine is discussed. In Section 3, the proposed two step approach is described. The performances of different forecasting models and a comparison among them are discussed in Section 4 followed by the performance of the anomaly detection. In Section 5, the concluding remarks are presented.

2. Related Work

For decades, forecasting has been used as a powerful tool in weather pattern prediction, economic events, affair outcome prediction, supply chain, and more. Forecasting has been proposed to exercise in verity sectors for its efficiency, low risk management, cost reduction, and improved optimization. In this section, existing work is classified in two categories: (i) literature based on forecasting models and (ii) literature based on forecasting based anomaly detection.

2.1. Forecasting Models

Economic forecasting is becoming more widely used due to its better decision-making capabilities. Subsequently, forecasting has been used in supply chain management [18]. The goal of supply forecasting assists customers by supplying the right products at the right place, time and price. Another approach of forecasting is prediction of financial time series using support vector machine (SVMs) [19]. Kyoung-jae Kim clarifies the feasibility of SVM application by comparing with a back propagation neural network. Similarly, for seasonal and trend time series forecasting, a neural network has been widely used. In the paper, Peter Zhang elaborately examines ANN and ARIMA model effectiveness in business and economic sectors [20]. A quasi-linear auto regressive model, vector regressive model, ANN, and ARIMA based model are also used in a similar manner [21,22,23,24]. Weather forecasting has been a concern for its low accuracy. In most cases, linear relations between input and output weather data were established, which is primarily incorrect. Abhishek et al. aim to establish a nonlinear relation for weather analysis [25]. The authors improved a statistical model based of ANN which also indicates the behavior of an increased hidden layer on accuracy and performance. Several works related to energy forecasting are also reported. Solar energy forecasting based on a hybrid neural network is proposed in [26]. Similarly, wind power forecasting and multi-step wind speed forecasting are discussed in [27,28], respectively. These statistical models establish a mathematical relation between time series data but lack potential features such as data-mining and feature extracting. For some time series forecasting, it is not feasible to use statistical models due to the intermittent and chaotic nature of those time series like renewable energy forecasting [29]. Deep learning-based models are ideal for such time series forecasting. An artificial intelligence-based model such as deep leaning can discover abstract feature and invariant structure in data. Consequently, deep leaning is practiced in probabilistic wind power forecasting with high-accuracy and consistency [30]. In [31], a multi-modality graph neural network is proposed addressing the major issues in price prediction in the financial industry. The proposed model was used to learn the lead-lag effects of financial time series. Six deep learning models were employed in [32] for forecasting of cases and death rate for COVID-19 in Australia and Iran. These models include LSTM, Convolutional LSTM, GRU, and their bidirectional extensions. It was found that the bidirectional extension models perform better than the non-bidirectional models. An architecture of LSTM involving Conv1d is proposed in [33] to forecast Web Traffic.

2.2. Forecasting Based Anomaly Detection

Forecasting towards cyber data injection attack detection is a comparatively unexplored region. A comprehensive literature review is done in [34] on an FDI attack. The world is gradually depending more and more on cyber networks. Cyber security is vital because it encompasses everything that pertains to protecting our sensitive data, personally identifiable information, protected health information, personal information, etc. Among a few works, Alfeld et al. elaborated a mathematical framework against a data poisoning attack based on a linear autoregressive model [35]. A similar approach was taken in [36]. In their work, the authors developed an algorithm against anomalies in a power load system. The algorithm, Machine Learning based Anomaly Detection (MLAD), uses neural networks to reconstruct the benchmark and scaling data. Bayes classification, which is based on the cumulative distribution function, is used to estimate a cyber attack template. The authors also prosecute dynamic programming and numerical simulation to calculate load parameters. For smart grid security, a handful of models had been proposed. For example, the vector auto-regressive model and dynamic factor model show promising results against FDI detection [37]. Different approaches were also taken like the use of nonlinear models [38]. Another approach was taken in temperature anomaly detection for electric load [39]. In their work, the authors enhance the accuracy of the final load forecasts by removing the fabricated altered data from the original input data. In supply chain management, a huge amount of information can lead to an overload of data and generate unexpected patterns or anomalies. A machine learning based method such as LSTM and LSTM Auto-encoder techniques can optimally detect anomalies [40]. An LSTM variational Auto-Encoder was implemented in [41] for detecting anomolies in water treatment facilities and water distribution facilities. In [42], several prediction models including LSTM Auto-Encoders were investigated for anomaly detection on a time series of Quebec Metro ridership. In [43], a CNN-LSTM framework was established for anomaly detection for solar power forecasting under cyber-threat. A deep neural network was used in [44] for state prediction and detection of FDI attacks. Most of these detection methods are vulnerable to contamination attacks, FDI attack, or SQL injection. Those models are generalized using an ideal condition as a consequence of a falsified data-set can make the entire system malfunction as well as cause detection delay. On the contrary, in this paper, we took a different approach to sort out detection delay.

3. Proposed Method

The proposed method contains two stages: (i) in the first stage, the deep learning model gives a forecast on real-time data, and (ii) in the second stage, the anomaly detection engine detects anomalies based on the forecast. The proposed method contains two stages: (i) in the first stage, the deep learning model gives a forecast on real-time data, and (ii) in the second stage, the anomaly detection engine detects anomalies based on the forecast.

3.1. Energy Consumption Forecasting Using CNN-LSTM Auto-Encoder Sequence to Sequence Architecture

The CNN-LSTM Auto-encoder architecture for forecasting energy consumption consists of a series connection of CNN and LSTM. The Encoder consists of an input layer, CNN layer, and several hidden layers of pooling layers and flatten layers. Equation (1) is the result of the vector from a convolutional layer, $y_{ij}$ is calculated by input vector $x_{ij}$ to the convolutional layer, b represents bias, w is the weight of the kernel, n is the index value of the filter, and $\alpha$ is the activation function:

$$y_{ij}=\alpha (b_j+\sum _{n=1}^N{w}_{m,j}{x}_{i+m-1,j}^0)$$

(1)

CNN reads across the sequence input and automatically learns the salient features. It projects the results into feature maps. A pooling layer is used to combine the output of a neuron cluster in one layer and convert it into a single neuron in the next layer. It causes reduction in the number of parameters and simplifies the feature maps. It also helps adjusting over fitting. In Equation (2) for a max pooling layer, S is the stride and R is the pooling size, which is less than the size of the input y.

$$p_{ij}=\underset{r\in R}{max}{y}_{i\times S+rj}$$

(2)

The output of the pooling layer is flattening into one long vector, which is later interpreted by the decoder. The Decoder consists of only LSTM layer and an output layer. An LSTM layer stores information about the characteristic of the sequence extracted through the CNN Encoder. LSTMs are a special kind on the RNN network that consists of a set of gates that controls the output. LSTMs are explicitly designed to avoid the long-term dependency problem. It also shares the same variables across all time steps, which reduces the number of unknowns significantly. LSTM has a strong ability to capture the dynamic features via cycles in the graph. As the structure of LSTM is shown in Figure 1, given input sequence X = ($x_1$,$x_2$,......x${}_N$), the LSTM computes a two vector sequence, h = ($h_1$,$h_2$,......h${}_N$) and Y = $y_1$,y${}_2$,......y${}_N$). LSTM can remove or add information to the cell state by the gates where cell state ${\mathrm{c}}_t$ is the LSTM structure that stores information. LSTM consists of three sigmoid gates (1) one is for forget cell, f${}_t$ that removes old information, (2) the second one adds new information in the cell state, and (3) the third one is used to update old cell state, c${}_{t-1}$ into a new cell state c${}_t$. Here, $\sigma$ is the sigmoid function that has an output range [0,1], tanh is the hyperbolic tangent function that has an output range [−1,1], and the purple dotted circles represent pointwise operations. Therefore, LSTM was chosen after the CNN layer to learn dependencies in the sequence of time series data.

The LSTM transition functions are defined as follows:

$$f_t=\sigma (W_f\times [C_{t-1},h_{t-1},x_t]+b_f$$

(3)

$$i_t=\sigma (W_f\times [C_{t-1},h_{t-1},x_t]+b_i$$

(4)

$$C_t^{/}=tanh(W_C\times [h_{t-1},x_t]+b_C$$

(5)

$$o_t=\sigma (W_f\times [C_t,h_{t-1},x_t]+b_o$$

(6)

$$C_t=f_t\times C_t+i_t\ast C_t^{/}$$

(7)

$$h_t=o_t\times tanh\left(c_t\right)$$

(8)

LSTM learns the temporal relationship on a long-term sequence. It is well suited for forecasting time series. The encoder converts input sequences of variable length into a fixed length vector that is used as the input state for the decoder and the decoder generates an output sequence of l length. In training, back-propagation signals flow from a decoder to encoder network. As a result, weights for the network are updated in order to minimize the following error Equation (9). The whole structure of the deep learning network is shown in Figure 2:

$$L_E=\sum {(y_t-\widehat{y_t})}^2$$

(9)

3.2. Hyperparameter Tuning

For optimizing the hyperparameters of proposed model, we first listed the different combination of hyperparameters such as the filter number and kernel size of conv1d, number of units in LSTM layer, etc. A training loop was created where all the different combinations of hyperparameters of our forecasting model were implemented, and we found out the Mean Absolute Error (MAE) in each combination. From a log of MAE of each model, we obtained the optimal one. MAE is described in Section 4.2. The process is summed up in Figure 3.

Table 1. Optimized hyperparameters of the proposed model.

Hyperparameter Type	Hyperparameter Value
Conv1d Filter number	32
Conv1d Kernel size	3
Maxpooling1d pool size	2
LSTM unit number	220

shows the optimal set of hyperparameters of the model.

3.3. Real-Time Anomaly Detection Based on Accurate Forecasting

The anomaly detection algorithm attempts to find abnormal behavior in the collected data. Historical data of previous time periods are stored in SCADA historians, which are used to forecast current time steps with the CNN-LSTM Auto-encoder model. Based on the forecasted data of a certain time period ‘t’, irregularities in the measured real-time data of that period can be detected.

In this step, the algorithm compares the forecasted value of an instance ‘i’ of time period ’t’ based on previous time period ‘t-1’ with a real-time measured value. The range [$d_t^{lb}$,d${}_t^{ub}$ ] where $d_t^{lb}$ and d${}_t^{ub}$ is given by Equations (10) and (11); then, there is no anomaly, and, if it does not, then there is an anomaly in the system shown in Equation (12):

$$d_t^{lb}=\mu -\sigma \times \mu$$

(10)

$$d_t^{ub}=\mu +\sigma \times \mu$$

(11)

$$d_t=\left\{\begin{array}{cc}NoAnomaly\hfill & \mathrm{if}{d}_t^{lb}\le d_t\le d_t^{ub}\hfill \\ Anomaly\hfill & \mathrm{if}{d}_t<d_t^{lb},d_t>d_t^{ub}\hfill \end{array}\right.$$

(12)

Here, $\mu$ is the forested value of an interval ’i’ and $\sigma$ is the threshold parameter, which is empirically determined. The algorithm checks all the instances of in the time period and determines whether or not if there is an anomaly like in Figure 4. The flowchart of the algorithm is given in Figure 5.

The CNN-LSTM Auto-encoder is trained with a sliding approach as shown in Figure 6 [45]. The first few intervals are counted as input, and the forecasting model predicts the energy consumption of the next intervals. Then, it moves next to the window of the same time period and at the same time the anomaly detection tries to find the irregularities of the system by the above-mentioned process.

The heart of the proposed framework is illustrated in Figure 7, where “Data Archived” which contain all historical data plays a critical role. It is assumed that the historical data are attack free which is basically used as a ground truth for training the model towards building the profiles of normal behaviours. Data treatment is vital for training of the CNN-LSTM based model which may contain data sheet preparation, scaling, feature extraction, etc. Gain and allowance for particular time period ‘t’ is the terminal outcome of the trained model. In this framework, raw data input from the energy grid is sensed by the Phasor Measurement Units (PMU) units and then relocated to SCADA; any type of injection attack may materialize concurrently. Real-time data and Forecasted data are then fed to the logic loop that inspect for irregularities, disturbance, and contortion of features. Anomaly is recognized when any type of disturbance is detected. Any type of disturbance within a bound is assumed to be a negligible system error. If anomaly is perceived in any loop of interval ‘i’ a alarm is raised, and the system is brought to a halt; if not, PMU reading is considered as interruption free and stored in the “Data archive” for use of the next loop. This process reiterates for the next time period.

4. Results and Evaluation

The performance of proposed method is evaluated on the basis of a False Positive (FP) rate, False Negative (FN) rate, and Accuracy (ACC) Value. The FP rate, FN rate, and ACC value vary with the variation of $\sigma$ value of the proposed method. A test dataset was created to find out the False Positive (FP) rate, False Negative (FN) rate, and Accuracy (ACC) Value of the proposed method.

4.1. Experimental Dataset

The dataset consisting of three months of data on energy demand consumption was taken from Australian Energy Market Operator (AEMO) [46]. In addition, 65% of the dataset was used for training the model and rest was used for testing.

4.2. Error Matrix

Several error measures are used to evaluate the forecasting models, the Mean Absolute Error (MAE) as shown in Equation (13), Mean Absolute Percentage Error (MAPE) shown in Equation (14), Mean Squared Error (MSE) shown in Equation (15), and Root Mean Squared Error (RMSE) as shown in Equation (16).

Table 2. Comparison among different forecasting models.

Models	Epoch	MAE	MAPE (%)	MSE	RMSE
ARIMA	-	0.2147 $\pm$ 0.0118	32.68 $\pm$ 2.69	0.0682 $\pm$0.0014	0.2612 $\pm$ 0.0339
CNN	500	0.1374 $\pm$ 0.0171	22.98 $\pm$ 0.86	0.0249 $\pm$ 0.0003	0.1581 $\pm$ 0.0181
LSTM	50	0.1610 $\pm$ 0.0274	27.24 $\pm$ 1.32	0.0370 $\pm$ 0.0004	0.1924 $\pm$ 0.0209
	100	0.1563 $\pm$ 0.0159	24.39 $\pm$ 0.85	0.0369 $\pm$ 0.0006	0.1922 $\pm$ 0.0249
	200	0.1502 $\pm$ 0.0194	23.09 $\pm$ 0.92	0.0338 $\pm$ 0.0006	0.1840 $\pm$ 0.0257
	500	0.1314 $\pm$ 0.0208	21.52 $\pm$ 0.97	0.2461 $\pm$ 0.0005	0.1569 $\pm$ 0.0238
CNN-CNN	50	0.1668 $\pm$ 0.0165	26.15 $\pm$ 0.87	0.0355 $\pm$ 0.0002	0.1886 $\pm$ 0.0165
Auto-encoder	100	0.1335 $\pm$ 0.0150	22.18 $\pm$ 0.72	0.0246 $\pm$0.0003	0.1570 $\pm$ 0.0179
	200	0.1313 $\pm$ 0.0128	21.69 $\pm$ 0.65	0.0240 $\pm$0.0003	0.1552 $\pm$ 0.0174
	500	0.1152 $\pm$ 0.0145	18.08 $\pm$ 0.69	0.0186 $\pm$0.0002	0.1366 $\pm$ 0.0159
LSTM-LSTM	50	0.1395 $\pm$ 0.0275	22.49 $\pm$ 2.86	0.0334 $\pm$ 0.0013	0.1830 $\pm$ 0.0362
Auto-encoder	100	0.1117 $\pm$ 0.0264	17.65 $\pm$ 1.16	0.0246 $\pm$ 0.0019	0.1570 $\pm$ 0.0443
	200	0.1098 $\pm$ 0.0319	13.07 $\pm$ 1.82	0.0201 $\pm$ 0.0016	0.1420 $\pm$ 0.0412
	500	0.1053 $\pm$ 0.0144	12.09 $\pm$ 0.74	0.0184 $\pm$ 0.0003	0.1357 $\pm$ 0.0175
CNN-LSTM	50	0.1668 $\pm$ 0.0291	25.91 $\pm$ 2.95	0.0357 $\pm$ 0.0012	0.1886 $\pm$ 0.0347
Auto-encoder	100	0.1016 $\pm$ 0.0250	15.96 $\pm$ 1.45	0.0170 $\pm$ 0.0002	0.1305 $\pm$ 0.0165
	200	0.0947 $\pm$ 0.0196	10.87 $\pm$ 1.22	0.0148 $\pm$ 0.0007	0.1217 $\pm$ 0.0281
	500	0.0799 $\pm$ 0.0128	8.07 $\pm$ 0.62	0.0106 $\pm$ 0.0003	0.1033 $\pm$ 0.0176

shows the comparison between the forecasting models in all the mentioned error matrixes. MAE evaluates the absolute deviation between the measured energy consumption and predicted energy consumption. It also takes into account the negative errors. MAPE is a statistical measure which is also used to explicate accuracy of forecasting models. Another approach is to square the deviations, which is done in MSE. This weighs the greater deviations more than the lower one. However, the units are squared, which could be unwanted. To fix this, RMSE is used, which returns the original unit. All these error matrixes together give a clear picture about the accuracy of forecasting models. The equations use predicted energy consumption value ‘$\widehat{Y}$’ and measured energy consumption value ‘Y’. The number of samples are ‘N’:

$$MAE=\frac{1}{N}\sum |\widehat{Y}-Y|$$

(13)

$$MAPE=\frac{1}{N}\sum \left|\frac{\widehat{Y}-Y}{Y}\right|\times 100\%$$

(14)

$$MSE=\frac{1}{N}\sum {(\widehat{Y}-Y)}^2$$

(15)

$$RMSE=\sqrt{\frac{1}{N}\sum {(\widehat{Y}-Y)}^2}$$

(16)

Bar charts of RMSE and MAE of the models are shown, in Figure 7 and Figure 8. Table 2, Figure 7 and Figure 8 are discussed in Section 4.7.

4.3. Results

The performance of the proposed method is evaluated on the basis of False Positive (FP) rate, False Negative (FN) rate, and Accuracy(ACC) Value. The FP rate, FN rate, and ACC value vary with the variation of $\sigma$ value of the proposed method. A test dataset was created to find out the False Positive FP rate, False Negative FN rate, and Accuracy ACC Value of the proposed method. Thus, to obtain the highest performance from the Anomaly Detection Engine, we have to find the optimum $\sigma$ value of the method, which yields low FP rate, low FN rate, and high ACC value.

Three types of attack scenarios were included in the the test dataset. These attack scenarios are modeled by particular adversary models, which are described in [47].

Attack Types

In a scaling attack, true measurements are multiplied by a scaling parameter ${\gamma }_S$ to a higher or lower value for a time period. It is given by

$$P_t^F=(1+{\gamma }_S)\times P_t,for{t}_s<t<t_e$$

(17)

Here, $P_t$ is the measured true value, $t_s$ is the starting time and $t_e$ is the ending time for the cyber-attack duration, and $P_t^F$ is the modified value after cyber-attack.

In a ramp attack, true measurements are modified in manner where the value increases or decreases for a time period by a ramping parameter ${\gamma }_R$. There are two types of Ramp attack. In Type I, only Up-ramping is considered, and, in Type II, both Up-ramping and Down-ramping are included. Type I is given by Equation (18), and Type II is given by Equations (19) and (20):

$$P_t^F={\gamma }_R\times (t-ts)\times P_t,for{t}_s<t<t_e$$

(18)

$$P_t^F=[1+{\gamma }_R\times (t-ts)]\times P_t,for{t}_s<t<(t_s+t_e)/2$$

(19)

$$P_t^F=[1+{\gamma }_R\times (t-ts)]\times P_t,for(t_s+t_e)/2<t<t_e$$

(20)

Here, t${}_s$ is the starting time, and t${}_e$ is the ending time for the cyber-attack duration.

Pulse attacks are opposite to scaling attack in kind where the true measurements are modified to higher or lower values for a specified short time period by a parameter ${\gamma }_P$. It is given as below:

$$P_t^F=(1+{\gamma }_P)\times P_t,fort=t_P$$

(21)

Here, t${}_P$ is the occurrence time of the cyber-attack.

These attacks were randomly distributed, and the attack magnitude was scaled from 2 to 25% of the measured true value.

4.4. False Positive Rate

False Positive is the case where the algorithm accounts a true measurement value as an irregularity. If the False Positive rate is excessively high, many of the true values will be accounted as anomalies, and there will be less confidence in the framework. FP Rate is calculated from Equation (22), where TN refers to True Negative. A 0% False Positive rate means that the proposed method can classify every secure measurement value as secure:

$$FPRate=\frac{FPCounts}{FPCounts+TNCounts}$$

(22)

Figure 9 shows the variation of a false negative rate for different values of $\sigma$. With the increase of $\sigma$, the False positive Rate decreases. On the x-axis, the $\sigma$ value is given from 0.00 to 0.12, and the y-axis indicates percentage of False Positive or False Negative. For the $\sigma$ value beyond 0.105, the False positive becomes zero. Below the 0.06 $\sigma$ value, the FP Rate is significantly high.

4.5. False Negative Rate

False Negative is the case where the algorithm wrongly classifies an abnormality as a true value. A high False Negative rate implies that the algorithm fails to detect many cases of anomalous behavior in the system. FN Rate is calculated from Equation (23), where TN refers to True Positive. A 0% False Negative rate means the Anomaly Detection Algorithm can detect every attack:

$$FNRate=\frac{FNCounts}{FNCounts+TPCounts}$$

(23)

Figure 9 presents the variation of FN Rate for different values of $\sigma$ value. On the x-axis, the $\sigma$ value is given from 0.00 to 0.12, and the y-axis indicates a percentage of False Positive or False Negative. For $\sigma$ value above 0.09, the FN Rate becomes significantly high, and for $\sigma$ value below 0.025, the FN Rate is zero. Above the 0.095 $\sigma$ value, the False Negative rate begins to have significantly high value.

4.6. Accuracy Value

Accuracy value can be defined as how many cases where the anomaly detection algorithm can classify an attack as an attack and a secure measurement value as secure. Accuracy Value is calculated from Equation (24). A 100% accuracy value means that the proposed method can detect every attack and also classify every secure measurement value. It is as follows:

$$ACC=\frac{TPCounts+TNCounts}{FPCounts+TPCounts+TNCounts+FNCounts}$$

(24)

Here, ACC is accuracy value. TP, TN, FP, and FN are True Positive Counts, True Negative Counts, False Positive Counts and False Negative Counts. Figure 10 presents the variation of ACC value for $\sigma$ values from 0.05 to 0.12. On the x-axis, the $\sigma$ value is given from 0.00 to 0.12, and the y-axis indicates Accuracy Value from 88% to 96%. At .075 $\sigma$ value, the ACC value is 95.58%, which is the highest. From 0.075 to 0.09 $\sigma$ value, the ACC value is above 95%. After 0.09 $\sigma$ value, the ACC value begins to drop.

4.7. Discussion

A comparison was made between different forecasting models like statistical model ARIMA, Deep Learning models like LSTM, CNN, CNN-CNN Auto-encoder, LSTM-LSTM Auto-encoder, and CNN-LSTM Auto-encoder. In Table 2, MAE, MAPE, MSE and RMSE of the mentioned models are shown. The first column represents the forecasting models, the second column holds the Epoch number, the third, the fourth, the fifth and the sixth ones hold MAE and MAPE, MSE, RMSE, respectively. The color coding gives a good overview with a deep red color indicating large error and a deep green color indicating small error. Table 2 shows that statistical model ARIMA has larger MAE and RMSE than all the deep learning models. However, all deep learning models trained with a small epoch number also have large MAE and RMSE. As the models are trained with more epoch numbers, the MAE and RMSE become smaller and smaller. However with larger number of epochs, the models require more time and, up to a certain number of epochs, the MAE and RMSE remain more or less the same. Among all the deep learning models, training the LSTM-LSTM Auto-encoder requires more time than others. The CNN-LSTM Auto-encoder with 500 epochs has the smallest MAE, MAPE, MSE and RMSE among all the mentioned models. The CNN-LSTM Auto-encoder outperforms all the other models. The MAE and RMSE of CNN-LSTM Auto-encoder of 500 epochs are 0.0799 $\pm$0.0128 and 0.1033 $\pm$0.0176, respectfully. In Figure 7 and Figure 8, bar charts of RMSE and MAE of the models are shown. The top one shows the MAE, and the bottom one shows RMSE. Here, the blue bar, orange bar, green bar and yellow bar indicate 50, 100, 200 and 500 epochs, respectfully.

The models do not only have to have a small MAE and RMSE, but also have to perform well over a long period of time as the effectiveness of the anomaly detection algorithm depends on it. Otherwise, the Anomaly Detection Algorithm will utterly fail to detect anomalies during some period of time, and other times it will give a false alarm. In Figure 7 and Figure 8, a comparison is made among all the mentioned models over a sample time period. The x-axis indicates time in hour:minute and the y-axis indicates total power demand. Here, the black line is actual measurement of energy consumption, and the purple line is the predictions of CNN-LSTM Auto-encoder model in Figure 11.

Here, we can see that the predictions are relatively close to the actual measurement of energy consumption value. In Figure 12, the performance of the CNN-LSTM Auto-encoder in two days is shown. Here, the x-axis also indicates time in hour:minute and the y-axis indicates total power demand. Here, the allowance is 10% of the measured value in every instance. It is shown in Figure 12 that the CNN-LSTM Auto-encoder is performing consistently well for two-day time periods and is under 10% margin of the actual measurement of energy consumption value.

5. Conclusions

In this paper, a multi-stage anomaly detection framework is developed to address the data integrity issues of the meter data management. The Proposed Method has high accuracy, a low false positive rate and a low false negative rate. However, the accuracy of the forecasting model is essential for the anomaly detection engine to perform well. Thus, the forecasting result of the CNN-LSTM Auto-encoder model is a very critical step for the anomaly detection engine to perform properly; otherwise, it will have a high false positive rate. In this paper, the CNN-LSTM Auto-encoder model performs significantly better compared to other benchmark algorithms in terms of forecasting accuracy considering the real-world dataset from AEMO. The proposed model with better forecasting identifies anomalies with high detection accuracy with low false positives and shows its effectiveness towards PMU data integrity.