Trustworthiness of Deep Learning Under Adversarial Attacks in Power Systems

Abstract

Advanced as they are, DL models in cyber-physical systems remain vulnerable to attacks like the Fast Gradient Sign Method, DeepFool, and Jacobian-Based Saliency Map Attacks, rendering system trustworthiness impeccable in applications with high stakes like power systems. In power grids, DL models such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks are commonly utilized for tasks like state estimation, load forecasting, and fault detection, depending on their ability to learn complex, non-linear patterns in high-dimensional data such as voltage, current, and frequency measurements. Nevertheless, these models are susceptible to adversarial attacks, which could lead to inaccurate predictions and system failure. In this paper, the impact of these attacks on DL models is analyzed by employing the use of defensive countermeasures such as Adversarial Training, Gaussian Augmentation, and Feature Squeezing, to investigate vulnerabilities in industrial control systems with potentially disastrous real-world impacts. Emphasizing the inherent requirement of robust defense, this initiative lays the groundwork for follow-on initiatives to incorporate security and resilience into ML and DL algorithms and ensure mission-critical AI system dependability.

1. Introduction

Deep learning, a subset of artificial intelligence (AI), is extensively utilized in power systems for tasks such as demand forecasting, fault detection, and grid optimization. These models help in achieving unprecedented efficiencies in energy production, distribution, and consumption. However, these models are susceptible to adversarial attacks—minor perturbations to inputs that cause the model to make incorrect predictions. For instance, small changes to a traffic light sign could lead an autonomous vehicle to misinterpret it, demonstrating the potential risks.

Innumerable industries have undergone irreversible transformation because of AI and machine learning (ML), which has enabled the deployment of sophisticated cyber-physical systems in a variety of businesses. The importance of this issue cannot be overstressed given the growing reliance of our world on deep learning (DL)-based power systems. Adversarial attacks, a group of expertly crafted online threats intended to trick deep learning algorithms, have the power to undermine the foundations of these complex systems. These assaults cause inaccurate predictions and judgments by taking advantage of faults in AI models, endangering the trustworthiness of deep learning-based power systems.

The substantial effects of AI and ML are felt throughout many industries, radically changing how these sectors operate. DL-based power systems have become a leading example in this transformational process, enabling previously unheard-of levels of efficiency, adaptability, and reliability. “AI in energy is like the discovery of electricity itself”, said Andrew Ng, co-founder of Google Brain, “it will impact every industry”. These DL-based power systems have a wide range of uses, from improving the production of renewable energy to boosting grid resilience and delivering accurate demand projections. However, the susceptibility of these deep learning models to hostile attacks is a sobering truth that comes with this era of technological revival. These carefully designed cyberattacks can wreak havoc in DL-based power systems. They raise serious safety concerns due to their vulnerability to adversarial threats. Such threats could lead to catastrophic consequences.

Adversarial attacks in DL models have been widely studied in computer vision and natural language processing contexts. For example, Eykholt et al. showed that small stickers on a stop sign could mislead autonomous vehicle classifiers [1]. Ebrahimi et al. demonstrated that distracting sentences can reduce the accuracy of reading comprehension models [2]. However, there has been limited research applying adversarial methods to power system DL models. Jiang et al. performed blind spoofing attacks to mislead phasor measurement unit placement algorithms but did not directly attack the ML inputs [3]. This gap highlights the need to investigate adversarial threats in the unique context of power grids.

The capacity to protect DL-powered power systems against hostile threats is critical. These systems, which are responsible for supplying uninterrupted energy, must be reliable and secure. Failure in this arena has ramifications that go beyond basic inconvenience; they have economic, environmental, and societal implications. “AI is only as secure as the data and algorithms it relies on”, says Tiscareno. A substantial disruption in the energy sector caused by a cyberattack might be disastrous [4].

Consider the potential consequences of a DL-driven power grid that lacks proper security safeguards in the face of an adversarial attack. The consequences could include extensive power outages, economic losses, environmental harm, and even public safety issues. As reliance on DL-powered power systems grows, ensuring their resilience against adversarial attacks becomes a critical imperative, as stated by the International Energy Agency (IEA): “The electrification of end-use sectors and the use of AI in the energy sector require significant investments in cybersecurity” [5].

Protecting DL models within power systems from adversarial threats becomes a mission-critical task against the backdrop of technological revolution and vulnerability. The capability to defend these systems from malicious attacks becomes a cornerstone of resilience and reliability as we advance deeper into an AI-powered future. The ramifications of inadequate security in DL-driven power systems are extensive. If an adversarial attack is successful, it may result in cascading failures, extended power outages, environmental damage, financial losses, and even hazards to public safety. As a result, protecting DL models against hostile threats goes beyond a technological problem to become both an ethical and a practical necessity. The International Energy Agency (IEA) agrees, highlighting the necessity of “significant investments in cybersecurity for the electrification of end-use sectors and AI integration in the energy sector”.

To comprehend the scale of the challenge, it is imperative to explore three specific types of adversarial attacks that pose significant threats to AI-driven power systems. We implement three well-known attack methods that have been extensively explored in computer vision: DeepFool, Jacobian-based Saliency Map Attack (JSMA), and the Fast Gradient Sign Method (FGSM). DeepFool iteratively computes linear approximations of the decision boundary to find small perturbations that cause misclassification. JSMA identifies the most salient input features and alters them to push the model toward a specific target class. The FGSM utilizes gradient information to introduce perturbations that maximize the loss and minimize model accuracy. Here, we generate adversarial examples using these techniques and measure their impacts on ML model performance for power system datasets. The objective is to corrupt the decision-making process of the model, leading to compromised functionality. In the context of DL-driven power systems, these attacks could introduce subtle distortions during the model training phase, resulting in long-term vulnerabilities that may be exploited during real-world operation.

The purpose of this research study is to delve into the heart of this vital issue by investigating the vulnerabilities of DL-based power system to adversarial attacks, providing insights into potential threats, and recommending security strategies. We strive to protect the dependability and reliability of these systems that support the modern world through rigorous study. This research extends our previous work [6]. Our previous work reports preliminary results on optimizing the neural network models under adversarial attacks. Our main contributions are as follows:

• Investigated vulnerabilities of deep learning models in cyber-physical power systems under adversarial attacks.
• Analyzed and implemented three important adversarial methods: the Fast Gradient Sign Method, DeepFool, and Jacobian-Based Saliency Map Attacks.
• Demonstrated the consequences of adversarial attacks on power system trustworthiness and functionality.
• Proposed a new framework to assess and model adversarial risks in power systems.

Through this study, we advanced the knowledge in the field and pave the way for the implementation of robust security measures in deep learning-based power systems. Ultimately, this endeavor is crucial to ensure the resilience and reliability of these systems in the face of potential adversarial attacks.

2. Related Work

In recent years, academics have focused on the susceptibility of DL in power systems under adversarial assaults. Understanding possible risks and developing mitigation techniques is crucial for guaranteeing the reliability and security of vital infrastructure. This section examines prior research initiatives that explored the effects of adversarial assaults on power systems and identifies areas where more improvements are required to properly defend against these attacks.

Several researchers have carried out studies to evaluate the impact of adversarial attacks on DL-based power systems. Hug et al. conducted a major study on the effects of adversarial attacks on the state estimation process in power grids [7]. They demonstrated that attacks utilizing the Jacobian-Based Saliency Map Attack (JSMA) could result in erroneous state predictions, potentially causing power system instability. This research emphasizes the importance of strong defenses against adversarial attacks, particularly in critical infrastructure.

Recent research has achieved significant advances in the understanding of adversarial attacks against machine learning (ML) models in power systems, focusing on detection, mitigation, and vulnerability analysis for enhancing the security of critical infrastructures [8]. Ding et al.refer to increasing vulnerabilities of power grids due to increased reliance on communication infrastructure and demand robust defense mechanisms against attacks like False Data Injection Attacks (FDIAs) and Denial of Service (DoS) attacks, which can destabilize grid stability. Their research demands the utilization of AI-based detection mechanisms to counter such threats, particularly in smart grids where digitalization has offered a wider attack surface [9].

Meanwhile, a study addressed the pressing issue of power system stability in the face of adversarial attacks. The researchers tested the ability of a Multilayer Perceptron (MLP) to detect False Data Injection Attacks (FDIAs) and applied it by observing manipulated measurements and perturbations in power systems to evaluate the vulnerability [9]. It goes beyond detection, allowing for the identification of the most vulnerable components in the power grid.

On the detection front, Liberati et al.provide a system-theoretic perspective on cyber-physical smart grid attacks and FDIAs on state estimation processes [10]. The author’s survey illustrates how attackers exploit vulnerabilities in the grid, e.g., manipulation of sensor measurements, to initiate cascading failures and highlights the need to develop scalable detection methods to ensure system reliability. Similarly, Ghiasi et al. provide a comprehensive survey of cyberattacks in smart grids, including adversarial attacks, and recommend intelligent methods like deep learning to detect and mitigate threats [11]. Their work verifies the effectiveness of unsupervised learning to detect stealthy FDIAs, demonstrating its potential for field implementation in power systems.

On the mitigation front, researchers discuss cybersecurity within cyber-physical power systems, advocating defense-in-depth strategies that combine device, network, and physical security to prevent adversary attacks [12]. They cite the historical contribution of such attacks as the 2015 Ukraine blackout and advise multi-layer protection to help resilience in the grid. Besides, probabilistic methods are proposed for assessing the vulnerability of power systems to cyberattacks, offering a framework for threat modeling and vulnerability modeling. The approach offers an anticipatory mechanism for operators to make their defenses stronger [13].

In parallel, researchers such as Sayghe et al. contribute a novel perspective by proposing a deep learning-based method for detecting adversarial attacks on power system state estimation [14]. Their approach stands out for its capability to identify adversarial examples generated by various attack methods. Experimental validation of the method on the IEEE 14-bus system showcases its practical effectiveness, bolstering the arsenal of tools available to safeguard power systems from adversarial threats.

These studies collectively deepen our knowledge of adversarial attacks on machine learning models in power systems. They offer insights into detection and mitigation strategies, vulnerability assessment, and proactive measures to enhance the security and resilience of critical power infrastructures. The IEEE 14-bus and 118-bus systems were crucial testing grounds for these studies, allowing researchers to assess the real-world applicability and effectiveness of their proposed methods. These power system test cases serve as representative models for a wide range of practical scenarios, making the research findings applicable to diverse power grids and configurations. By using these standardized test systems, the studies ensure that their contributions are widely relevant to the power industry. They have provided valuable tools and insights for addressing the ever-evolving challenges of adversarial attacks on machine learning models in power systems. By introducing innovative methods for detection, mitigation, and vulnerability assessment, they contribute to the ongoing efforts to secure and enhance the resilience of critical power infrastructures. The research conducted by various researchers collectively represents significant progress in understanding and addressing the complex issues of power system security in the face of adversarial threats.

Another study, undertaken by Manandhar, et al., investigated the vulnerability of AI-based demand response systems in power grids [15]. They created adversarial assaults on demand response models using the Fast Gradient Sign Method (FGSM). The findings indicated that these attacks may be used to alter demand projections, potentially resulting in inefficient energy distribution and financial losses for energy suppliers. This study underlines the significance of protecting AI models in power systems from FGSM-based attacks.

While these studies provide useful information about the risks of DL in power systems, there are significant flaws and places for improvement. One prevalent issue is a focus on single attack methods while ignoring the potential for numerous attack strategies to be combined [16]. Future research should investigate the robustness of power systems to various attack techniques.

Several research have highlighted the seriousness of adversarial attacks on AI-powered power systems. Pandey et al. mentioned that adversarial attacks can have severe consequences, ranging from power supply disruptions to financial losses and compromised safety [17]. Such effects highlight the importance of solving this issue. Another study proposes that while designing AI-based solutions, we should consider AI and adversarial resilience as a critical design requirement. This method can assist in reducing the impact of hostile attacks [18].

In summary, past research has shed light on adversarial attacks in various applications, with an emphasis on specific attack methods such as JSMA, the FGSM, and DeepFool. However, more comprehensive research to solve this significant issue in critical infrastructure, such as power systems, is needed. Below in

Table 1. Summarization of related work.

Research Initiatives	Methods	Techniques	Objectives	Advantages	Limitations
Hug. et al. [7]	Demonstrated JSMA attacks on state estimation process	Jacobian-based Saliency Map Attack (JSMA)	Expose vulnerabilities in state estimation to adversarial attacks	Significantly degrades classification accuracy disrupting state estimation	Need for strong defenses against adversarial attacks
Manandhar et al. [15]	FGSM attacks on demand response models can alter demand projections	Fast Gradient Sign Method (FGSM)	Highlight risks of FGSM attacks on demand response models	Reduces accuracy, skewing demand forecasts and risking grid instability	Significance of protecting AI-models from FGSM-based attacks
Mirzaee et al. [19]	Overemphasis on single attack methods	Single attack method analysis	Identify gaps in focusing on single attack methods	Reveals how single attacks like DeepFool can drop multiclass accuracy exposing broader vulnerabilities	Focus on multifaceted attack strategies
Pandey and Misra [17]	Adversarial attacks can disrupt power supply, cause financial losses, and compromise safety	Adversarial attack simulation	Assess impact of adversarial attacks on power system operations	Causes significant performance drops, leading to operational disruptions	Inclusion of adversarial resilience in AI-model design
Kurakin et al. [20]	Emphasized the importance of adversarial resilience from the design phase	Adversarial resilience design principles	Advocate for resilience in AI design for critical infrastructure	Highlights need for resilience against attacks like DeepFool, which reduces accuracy, preventing major disruptions	Comprehensive defense strategies and holistic approach needed in critical infrastructure security
Sahay et al. [21]	Defended adversarial attacks on deep learning-based power allocation using denoising autoencoders	Denoising autoencoders	Mitigate adversarial attacks on power allocation systems	Mitigates specific attacks, maintaining power allocation stability despite attacks like FGSM	Limited to specific types of attacks
Heinrich et al. [22]	Targeted adversarial attacks on wind power forecasts	Targeted adversarial attack simulation	Evaluate impact of adversarial attacks on wind power forecasting	Disrupts wind power forecasting, dropping accuracy with the FGSM, affecting renewable energy integration	Lack of generalizability across different deep learning architectures
Wang et al. [23]	Surveyed adversarial attacks and defenses in machine learning models	Literature survey	Provide an overview of adversarial attacks and defenses in ML models	Identifies attacks like JSMA that lower accuracy aiding in understanding network performance degradation	High computational cost and complexity
Ding et al. [9]	Highlighted vulnerabilities in power grids due to communication reliance	AI-based detection methods	Identify cybersecurity challenges in power grids and propose AI-based solutions	Detects vulnerabilities exploited by attacks like DeepFool, improving grid security awareness	Limited focus on smaller grid operators’ vulnerabilities
Ghiasi et al. [11]	Reviewed cyberattacks and proposed intelligent methods for detection	Unsupervised learning for FDIA detection	Enhance smart grid security through intelligent detection and mitigation	Enhances detection of stealthy attacks like JSMA reducing false negatives in smart grids	Assumes static attack models, may not address adaptive threats

summarizes the research on adversarial attacks, their methods, and the limitations.

3. Background: Adversarial Attacks

Adversarial attacks or assaults are intentional modifications of neural network model input data with the objective of creating misclassifications or inaccurate predictions. These assaults target neural network vulnerabilities and potentially have serious consequences for AI-driven systems, including power systems. DeepFool, Jacobian-based Saliency Map Attack (JSMA), and the Fast Gradient Sign Method (FGSM) are three well-known adversarial attacks. This section aims to introduce the three different adversarial attacks that were used during the experiment to highlight the impacts they have on power systems.

3.1. DeepFool

DeepFool, developed by Moosavi-Dezfooli, is an iterative approach that seeks the smallest perturbation required to move an input data point across a neural network’s decision boundary, making it a highly efficient and stealthy adversarial attack [24]. The attack iteratively calculates the linear approximation of the decision boundary and advances the input in the direction of misclassification towards the boundary, ensuring minimal changes to the input while achieving misclassification. DeepFool is particularly effec-tive because it exploits the linearity of neural network decision boundaries, allowing it to be effective even for deep models with complex architectures. In the context of power systems, the impact of DeepFool can be severe, as it can subtly manipulate critical measurements like voltage, current, or frequency data, leading to incorrect state estimations or fault detections that may go unnoticed by operators. For instance, in a smart grid, a DeepFool attack could alter sensor data to mimic a legitimate fault, triggering unnecessary breaker trips and causing blackouts, or mask a real fault, leading to cascading failures and potential equipment damage. This stealthy nature of DeepFool poses a significant threat to the reliability and safety of cyber-physical systems, where even small perturbations can have catastrophic real-world consequences, such as widespread power outages or financial losses due to operational disruptions. It can produce minor perturbations that are unnoticeable to the naked eye but cause the model to misclassify the input [17]. Below is the equation used in producing a DeepFool adversarial example.

$$r_{\ast }\left(X_0\right) = arg min \left|\right|r|{|}_2$$

(1)

-

$r_{\ast }$: The pertubations that will fool the classifier

-

$X_0$: Our original input

-

arg min $|\left|r\right|{|}_2$: The smallest possible perturbation that causes misclassification

3.2. JSMA

Papernot presented the Jacobian-based Saliency Map Attack (JSMA) as an attack that targets manipulating input data so that it forces a neural network to classify it into a particular target class and thus becomes a threat factor in adversarial contexts [25]. JSMA does this by computing the Jacobian matrix, or the model’s sensitivity to small input feature changes, and then finding the most important features that, when modified, will most significantly nudge the input in the direction of the target class. By using this targeted approach, JSMA is able to generate effective perturbations with extremely small modifications to the input, making the attack stealthy. In power systems, the impact of JSMA can be particularly crippling as it could manipulate critical input features such as voltage or current measurements to mislead deep learning algorithms used for fault detection or state estimation to deceive the system into labeling a nominal operating condition as a fault situation or conversely. For example, in a smart grid, JSMA can attack the input data of a relay to falsely report a fault on a specific line, triggering an unwarranted breaker trip that disrupts power supply and can lead to blackouts, or it can conceal an actual fault, leading to delay and equipment damage or system-wide failures. The precision and targeted scope of JSMA make it a significant threat to the dependability and safety of cyber-physical systems, where such misclassifications can result in losses of efficiency in operations, economic loss, and safety hazards due to unexpected power loss.

$${adv}_x = x + arg max$$

(2)

-

${adv}_x$: This represents our modified input that fools the classifier

-

$x$: This is our original input sample

-

$arg max$: This operator finds the argument that maximizes the saliency map

3.3. FGSM

Goodfellow proposed the Fast Gradient Sign Method (FGSM), a simple yet effective approach that perturbs input data by utilizing gradient information of the loss function with respect to the input features, which has been a widely used adversarial attack as it is so effective [26]. The attack computes the gradient of the loss function with respect to the input and perturbs the input by a small amount in the direction that maximizes the loss (for targeted attacks) or minimizes the loss (for non-targeted attacks), resulting in large misclassifications with minimal changes to the input. The FGSM is computationally efficient because of its one-step perturbation approach, making it effectively generate adversarial examples, though with reduced success on strongly trained models or those with adversarial defenses, as indicated in earlier studies [27]. In power systems, the impact of the FGSM can be devastating, as it has the potential to subtly alter critical measurements like voltage, current, or frequency data fed to deep learning models for use cases like state estimation or load forecasting, such that it generates erroneous predictions that disrupt system operations. For instance, in a smart grid, an FGSM attack can perturb sensor data to deceive a model into predicting an incorrect load demand, resulting in over or under generation of power, which can lead to grid instability, blackout, or increased operational costs. In addition, the FGSM can deceive fault detection mechanisms by introducing slight perturbations that look like normal operating conditions when a fault actually exists, delaying response and causing cascading failures or equipment damage, thus posing a significant risk to the safety and reliability of cyber-physical systems in which even minimal misclassifications can have catastrophic real-world consequences. Below is the equation used in producing an FGSM adversarial example.

$$ad{v}_x = x + ϵ \times sign\left({\mathsf{\nabla }}_{x}J\left(\theta ,x,y\right)\right)$$

(3)

-

$ad{v}_x$: the perturbed input that will fool the classifier;

-

$x$: this is our original input sample;

-

$ϵ$: this is a small constant that controls the magnitude of the perturbation;

-

$\left({\mathsf{\nabla }}_{x}J\left(\theta ,x,y\right)\right)$: the gradient of the loss function with respect to the input.

4. Materials and Methods

There are five main procedures that make up the methodology for this research work. As a first step in our experiment, we start by developing deep learning models that are specifically designed for the adversarial dataset. Then, using the adversarial-specific dataset and training data, we design and train deep neural networks. The third phase includes creating and incorporating adversarial samples that imitate actual adversarial attacks into the dataset. Then, using these tainted, adversarial-specific datasets that reflect the changing data landscape, we create and train neural networks.

Finally, we conduct a comparative analysis, carefully comparing the model’s performance on the original, unspoiled dataset versus its performance on the poisoned, adversarial dataset, to determine the true impact of adversarial attacks on the model performance.

4.1. Proposed DL Framework Under Adversarial Attacks

Figure 1 illustrates the workflow of a deep learning (DL) model, starting with training and testing datasets fed into a Deep Neural Network (DNN). The DNN model becomes the victim model after using methods like DeepFool, JSMA, and the FGSM to create adversarial examples, which intentionally modified inputs designed to deceive the victim model into making incorrect predictions.

In this work, we utilized three common gradient-based adversarial attacks on our DL-based botnet detector (victim model), including the Fast Gradient Sign Method (FGSM), DeepFool, and JSMA.

The diagram depicts a deep learning-based botnet detection. An adversarial example generator is integrated to test the resilience of the deep learning model under adversary attacks. The process begins with the “Input” stage where data is divided into two sections, namely “Train” and “Test”. These input data, denoted as X, constitute the input to the DNN constituting for binary, triple, and multiclass classification.

The input provided is taken, and the output is obtained by processing it in fully connected layers to learn and extract relevant features for data segmentation, which are then forwarded to the “Victim Model” where the output predictions Y are generated. Then, the output from the DNN is fed to the “Victim Model” wherein if the predictions Y match the true labels, then the model is correct.

However, to test the detector of this botnet for its robustness, the “Generic Example Generator” component is used in the system generating adversarial examples. It relies on known techniques of adversarial attacks such as the FGSM, DeepFool, or JSMA methods to generate perturbed inputs X ′ from the input data X. Such adversarial examples are intentionally generated to mislead the victim model to output incorrect predictions Y ′, different from the original Y. The overall system aims to identify the vulnerabilities of the deep learning models on the ground by systematically poisoning this deep learning model with adversarial inputs. The DL model employs the deep learning neural network as shown in Figure 2.

4.2. Algorithm

This section presents Algorithm 1 to evaluate the effects of adversarial attacks under power systems.

Algorithm 1: Effects of Adversarial Attacks on Deep Learning Algorithm

Import Modules: a. Numpy, Pandas, os, time; File path exploration identifies and locates the input data through the directory structure; Import input data: a. Binary Class dataset; b. Triple Class Dataset; c. Multiclass Dataset; Data cleaning: a. Dropping duplicate rows; b. Removing rows with missing values; Exploratory data analysis: a. Generating summary statistics; b. Plotting class distributions; c. Analyzing the dataset’s features; Data encoding includes numeric columns to be z-score normalized and text values are encoded as dummy; Train–Test split: a. 80% training data, 20% testing data; b. Standard scaling applied to feature vectors; Deep learning model: a. Created for different subsets of data, such as DeepFool-specific, JSMA-specific, and FGSM-specific datasets; b. ReLu activation: i. $f\left(x\right) = max\left(0,x\right) = x \times \vee \left(x > 0\right)$; ii. The output of the ReLU activation function is f(x) and x is the activation function’s input; c. Sparse Categorical Cross Entropy Loss Function: $$L(y,y^) = -{\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\mathrm{l}\mathrm{o}\mathrm{g}\left(p_i\right)$$ $L\left(y,y^\right)$ is the loss, which measures how well the predicted class probabilities match the true class labels (y). N is the number of classes. pi is the predicted probability of the true class label for the i-th sample. d. Adam Optimizer: i. $m_t = {\beta }_1 \times m_{t-1} + \left(1 - {\beta }_1\right) \times \mathsf{\nabla }J\left({\theta }_t\right)$; ii. $m_t$$\mathrm{is} \mathrm{the} \mathrm{moment} \mathrm{estimate} \mathrm{for} \mathrm{a} \mathrm{specific} \mathrm{time} t. \mathrm{The} {\beta }_1$$\mathrm{parameter} \mathrm{control} \mathrm{the} \mathrm{exponential} \mathrm{decay} \mathrm{rates} \mathrm{for} \mathrm{the} \mathrm{moment} \mathrm{estimates}. J\left({\theta }_t\right)$$\mathrm{is} \mathrm{the} \cos \mathrm{t} \mathrm{functionat} \mathrm{time} \mathrm{step} t, \mathrm{and} {\theta }_t$ are the model parameters at the step; e. Constructed using keras with an input layer, 2 hidden layers, and an output layer; f. The model takes in the training data with an input layer of 256 neurons, then forward propagates to 128 neurons and 64 neurons in the hidden layers and outputs the total number of classes; Model training is monitored and early stopping techniques are employed for optimization and to prevent overfitting; Adversarial sample generation are crafted using different attack methods including DeepFool, JSMA, and the FGSM using the Equations (1)–(3) respectively Poisoned model training: a. Neural network models that are trained on adversarial samples; b. Known as “target models”; Performance evaluation on the testing data of the original and adversarial dataset: Calculating the Precision, F1, Recall, and Accuracy metrics on the DeepFool-, JSMA-, and FGSM-specific dataset; Model training is monitored and early stopping techniques are employed for optimization and to prevent overfitting.

5. Experiments and Results

5.1. Experimental Setup

Datasets

There are two data sets used in the experiments. One is generated by Mississippi State University and Oak Ridge National Laboratory on 15 April 2014, consists of power system event data designed for the purpose of assessing natural events, normal operations, and attack events [28,29]. We name it as ORNL dataset in this paper. It originates from a unified original dataset having 15 groups, each group containing 37 scenarios, leading to a sum of 555 scenarios, which are categorized as Natural Events (8 scenarios such as short-circuit faults and line maintenance), No Events (1 scenario replicating normal operations), and Attack Events (28 scenarios such as remote tripping command injection, relay setting manipulation, and data injection attacks). The 1% randomly sampled dataset is broken down into three classification formats—Binary (Attack and Normal Operation), Three-class (Attack, Natural, No Events), and Multiclass (all 37 scenarios as distinct classes)—and published in ARFF and CSV file types to support tool compatibility with Weka. The power system model includes two generators (G1, G2), four Intelligent Electronic Devices (IEDs, R1–R4) that act on breakers (BR1–BR4), and two transmission lines (Line 1: BR1 to BR2; Line 2: BR3 to BR4), with IEDs implementing a distance protection scheme. The input data is comprised of 128 features, primarily of four Phasor Measurement Units (PMUs), which each record 29 electrical features (e.g., voltage and current phase angles/magnitudes, impedance, frequency), equivalent to 116 PMU measurement columns. There are other features in the form of 12 columns to utilize for relay logs, Snort alarms, and control panel logs, and the 12th column as a marker encoding scenario information, load levels (in Megawatts), and fault locations (e.g., “085” for a fault at 85% on a line, or “000” for non-faulting conditions such as line maintenance). This organized input data allows the experiment to examine the effect of adversarial attacks on power system state estimation and assess detection or mitigation methods.

Further testing was executed to provide additional value to our initial experimental results by demonstrating our analysis of adversarial impacts against a second benchmark—the UCI Electrical Grid Stability Dataset [30]

$$\mathrm{Precision}={\displaystyle \frac{TruePositive}{TruePositive+FalsePositive}}$$

(4)

$$\mathrm{Recall}={\displaystyle \frac{TruePositive}{TruePositive+FalseNegative}}$$

(5)

$$\mathrm{F}1={\displaystyle \frac{2\left(Precision\ast Recall\right)}{Precision+Recall}}$$

(6)

$$\mathrm{Accuracy}={\displaystyle \frac{TruePositive+TrueNegative}{TruePositive+TrueNegative+FalsePositive+FalseNegative}}$$

(7)

The baseline performance of this model is such that it can be considered a good choice to compute adversarial attacks for several reasons. Based on the confusion matrix shown in Figure 3, the model demonstrates a77.08% accuracy and well-balanced metrics (Precision 90.82%, Recall 79.76%, F1 84.95%),showing that the model is competent enough to differentiate classes (such as Attack and Normal Operation in the power system dataset), providing a good starting point to gauge the impact of adversarial attacks. A good-baseline model ensures that attack impacts are not confused with inherent model weaknesses, thus attacks such as JSMA are more clearly observed to be exploiting some characteristics (e.g., reducing accuracy to 47.186% by attacking salient features) or how gradient-based perturbations of the FGSM disrupt decision boundaries (reducing accuracy to 54.742%). Second, the complexity of the model structure, as specified in the code—a neural network with 512 input neurons, 256 and 128 neuron hidden layers, and softmax output—is sufficiently high to capture deep learning models in power systems, which are generally vulnerable to adversarial attacks due to their reliance on high-dimensional data like voltage and current measurements. This density ensures that the model is able to capture non-linear relationships, and this makes it a likely candidate to be vulnerable to attacks like DeepFool, which incrementally pushes data along decision boundaries (accruing a diminished but nonetheless notable accuracy drop to 68.856%). Third, the preprocessing steps of the dataset (e.g., z-score normalization, duplicates removal, one-hot encoding of labels) ensure standardized inputs, which are critical to the unbiased evaluation of adversarial attacks because attacks like the FGSM rely on gradient computations, which are sensitive to input scaling. However, the model’s sufficiency is offset by its vulnerability to attacks, as shown by the sudden accuracy drop following attacks, meaning that although it is adequate for testing attack effects, it is not inherently robust against adversarial perturbations and needs such defenses as Adversarial Training or Feature Squeezing. To make it more useful for adversarial attack research, future versions might include Adversarial Training at the time of model building to resist attacks better and enable more stable benchmarking of the effects of attacks in power system applications.

Following the calculation of the initial performance measurements, adversarial samples are generated, which substitute values that were previously in the original dataset while maintaining the number of values in the dataset consistent. Following that, each adversarial attack’s training data is trained using the adversarial samples inserted into its original dataset. After training, the testing data are utilized to calculate the performance in terms of the Recall, Precision, F1, and Accuracy rates. This entire procedure is repeated ten times. Because the deep neural network predicts on each dataset, we compute results as shown in the tables and figures below.

Table 2. Performance measurements for binary, triple, & multiclass ORNL dataset.

Binary Class	Original					Poisoned
		Precision (%)	F1 (%)	Recall (%)	Accuracy (%)	Precision (%)	F1 (%)	Recall (%)	Accuracy (%)
	DeepFool	77.028	75.602	74.859	77.022	68.966	58.343	60.055	68.856
	JSMA	76.994	75.635	74.966	76.999	47.181	58.242	49.298	47.186
	FGSM	77.043	75.699	75.177	77.049	54.732	61.534	56.276	54.742
Triple Class	DeepFool	78.015	76.244	75.596	78.01	59.31	56.571	53.898	59.778
	JSMA	78.215	76.487	79.064	78.222	59.768	64.333	59.583	59.778
	FGSM	78.539	76.576	76.482	78.551	46.551	59.811	50.451	46.561
Multi- Class	DeepFool	62.974	63.534	62.805	60.983	4.638	6.944	3.551	4.6
	JSMA	61.482	62.179	61.372	61.493	15.889	18.317	15.306	15.91
	FGSM	60.974	61.579	60.771	63.983	8.914	12.097	8.883	8.926

depicts the average percentages for each performance metric that are used to distinguish the effect of each adversarial attack on its respective original dataset over the course of 10 trials.

5.2. Results and Discussions

As can be seen in Table 2 above, we are able to indicate the performance metrics of the model when simulating against the testing data as well as the effects of the adversarial attacks after they have been deployed. The results shown were tested over a series of 10 trials. They achieved an averaging above 75% accuracy for both the binary and triple class and just above 60% accuracy in the multiclass. However, in this experiment we have used the DeepFool, JSMA, and FGSM adversarial attacks to see the effects it has against the model’s overall performance.

There have been results of similar research done studying the model degradation of adversarial attacks especially for multiclass classification. For instance, Xu conducted an in-depth investigation into the vulnerability of multiclass image classification models to adversarial assaults [31]. Their study centered around a cutting-edge image classification model, and their goal was to see how the algorithm’s accuracy would change when exposed to hostile cases. To do this, they painstakingly built adversarial perturbations for a wide range of photos from various classes. These perturbations were intended to be unnoticeable to the naked eye while having a major impact on the model’s predictions. Prior to the adversarial attack, the model performed admirably on a conventional test dataset, with an accuracy of 80%. When subjected to the contrived adversarial samples, however, its accuracy dropped to a measly 10%. This significant decline in accuracy highlighted the model’s vulnerability to adversarial perturbations. The study also highlighted the need for increased adversarial robustness in multiclass classification models. It also highlighted concerns about the real-world ramifications of such flaws, especially in applications requiring great accuracy and reliability, such as driverless vehicles, medical diagnostics, and security systems.

In a similar vein, Sahay et al. proposed the utilization of denoising autoencoders (DAEs) to defend against adversarial attacks on deep learning models for power allocation in massive MIMO networks [21]. They discovered that adversarial examples generated by attacks like the FGSM and PGD could significantly degrade the performance of deep neural networks used for power allocation, causing them to produce infeasible solutions.

Their DAE defense was designed to train autoencoders to map perturbed inputs to clean reconstructions. The results demonstrated over a 90% detection rate for attacks and an 85% recovery of degraded capacity on attacked models when employing DAEs. The DAE defense proved effective against semi-white box and black box attacks, encompassing MR and MMMSE precoding schemes, and various perturbation magnitudes. These findings underscored the vulnerability of deep learning models for power allocation to adversarial attacks and highlighted the effectiveness of DAEs as a defense strategy.

Manoj et al. delved into the investigation of adversarial attacks on deep learning models for power allocation in massive MIMO networks [32]. They extended the FGSM, momentum iterative FGSM, and PGD attacks to the context of power allocation, revealing that these gradient-based attacks could substantially reduce network capacity by causing deep neural networks to output infeasible power allocations. White box attacks were demonstrated to result in up to 86% infeasible solutions with small input perturbations. These findings exposed the susceptibility of deep learning models for power allocation to adversarial examples and emphasized that attacks could transfer across various models and conditions, emphasizing the critical need for defending against such attacks to ensure reliable deep learning-based power allocation.

Furthermore, Sayghe et al. examined evasion attacks using adversarial deep learning against power system state estimation [14]. They generated adversarial examples using Carlini-Wagner and projected gradient descent attacks, demonstrating that these attacks could successfully evade detection by state estimation neural networks, even with small perturbations. This evasion resulted in substantial errors in state estimates, revealing the potential threat of adversarial examples to power system state estimation based on deep learning models. Their research emphasized the urgent need for effective defenses to prevent adversarial manipulation of state estimates, which are vital for power system monitoring and control.

Figure 4 presents the overall framework and components of the research on defensive countermeasures against adversarial attacks in power systems. The core of the framework consists of machine learning models, including deep neural networks and recurrent neural networks, which are applied in various applications in power systems.

These ML models take raw data as input, {x, y}, and out-put predictions, y_pred}. However, the authors have also considered the possibility of adversarial attacks-some tech-niques for generating adversarial examples to mislead ML models include the Fast Gradient Sign Method (FGSM), Jacobian Saliency Map (FGSM), and DeepFool.

In order to mitigate such adversarial threats, the researchers have integrated state-of-the-art countermeasures in the domain, namely, PCA and autoencoder-based defenses and others. These defenses materialized via a neural network architecture processing raw data and adversarial examples for more secure predictions.

The overall defense algorithm then integrates all these ML models, adversarial attacks, and defensive countermeasures to make the power system applications resilient against malicious threats. The picture shows that there exists a tug of war between the adversarial attack and the respective defensive strategies, indicating the need toward an integrated multilayer approach to secure and perform reliable deep learning-based applications in the context of smart power systems.

Figure 5, Figure 6, Figure 7 and Figure 8 are bar charts of the accuracy measurement derived from Table 2 results, demonstrating the various im-pacts of DeepFool, JSMA, and FGSM attacks on various classification tasks. Figure 4 shows that the JSMA attack causes a much more severe impact on the Binary Class Da-taset, reducing accuracy to around 47% (~29.8% drop), compared to DeepFool’s 68.856% (~8.2% drop) and the FGSM’s 54.742% (~22.3% drop), as expected for JSMA’s targeted feature tampering being extremely effective in exploiting binary decision boundaries despite being computationally more costly. In Figure 5, the FGSM attack is strongest on the Triple Class Dataset, lowering accuracy to 46.561% (a ~31.9% decrease), while DeepFool and JSMA have an accuracy of 59.778% (a ~18.2% decrease) after 10 attempts, demonstrating the FGSM’s surprising effectiveness as the least computationally intensive attack, likely due to its broad gradient-based perturbations affecting many classes. Figure 7 shows a sharp drop in model performance for all three at-tacks in the MultiClass Dataset, with DeepFool causing the most destruction, dropping accuracy to 4.6% (a ~56.4% decrease), followed by the FGSM at 8.926% (a ~53.1% decrease) and JSMA at 15.91% (a ~45.6% decrease), confirming the hypothesis that DeepFool’s iterative minimal perturbations are particularly disastrous in complex multiclass settings with 41 classes where small changes can easily cross numerous decision boundaries. As expected, all of the attacks result in a reduction in performance metrics relative to the original data, illustrating the vulnerability of neural networks to adversarial machine learning and the inherent loss in model performance when such attacks are applied.

Figure 9 gives a well-rounded comparison of three adversarial attack methods, DeepFool, JSMA, and the FGSM, on the multiclass dataset after running for 10 trials. The y-axis performance metric here is in seconds, meaning the time taken by each method to run one trial. This therefore gives the efficiency and behavior under the change of conditions of the methods.

The FGSM, in the red line, has the most dynamic performance profile. Starting off as the fastest method, possibly suggesting initial efficiency, its performance is wildly fluctuating across trials. Important peaks at trials 6 and 9 indicate instances of remarkably fast computation times. The highs are indeed punctuated by some very noticeable drops, most pronounced at trials 4 and 10. This variability might suggest that the success of the FGSM can vary greatly with other specifics in individual trials, such as input data complexity or tar-get model structure. The blue line, representing JSMA, contains an interesting performance trajectory. JSMA starts at a moderately efficient level, standing between the FGSM and DeepFool. A major loss in performance happens around the fourth trial; probably, this is to show a specific instance of high hardness for the algorithm of JSMA. What is most notable, however, is that JSMA generally experienced an upward trend throughout the series. This progressive performance increase indeed shows that JSMA might be getting more and more effective as it processes further input or becomes better attuned to the subtleties of the dataset. In the last trial, JSMA is the best-performing method, indicating its potential in optimization for a long-running application or with big datasets.

DeepFool, shown by the green line, stands out because of its extraordinary consistency through all the trials. Although it generally has the worst performance regarding speed, its stability is far better compared to the other two methods. This consistent behavior could hint at DeepFool being less sensitive to variations in either the input data or model characteristics. This could be highly valued in scenarios where resource allocation predictability is key, even at the cost of execution time being a bit slower.

The richly varying patterns that emerge underscore the complexity of adversarial attacks and their trade-offs in methodology. Such issues as the intricacy in datasets, the target neural network architecture, and the algorithmic nuances of each attack method may be responsible for these effects. FGSM’s speed and volatility may come from its straightforward, gradient-based approach, while JSMA’s improving trend could reflect its capability to refine its saliency mapping over time. This may be due to the consistency of DeepFool, which iteratively searches for minimal perturbation.

This comparison of performance provides a rich landscape of trade-offs in methodologies of adversarial attack: the FGSM allows for fast computation with high variability, JSMA offers interesting adaptivity, and DeepFool attains unmatched consistency. Knowing such features will turn out to be important in the elaboration of more robust defense mechanisms and enhance the general understanding of neural network vulnerabilities. Conclusively, as the adversarial machine learning domain keeps growing, insights from such comparative analyses will be especially important in the formulation of more effective and trustworthy attack and defense strategies.

The experiment on the UCI Electrical Grid Stability dataset is reported in

Table 3. Performance measurements for UCI datasets.

UCI Class Dataset—DeepFool.
	Recall (%)	Precision (%)	F1 (%)	Accuracy (%)
Original	78.58	76.59	77.57	77.85
DeepFool	34.99	30.87	24.71	35.04
UCI Class Dataset—JSMA
	Recall (%)	Precision (%)	F1 (%)	Accuracy (%)
Original	78.76	76.62	77.76	77.46
JSMA	36.17	48.75	24.00	36.17
UCI Class Dataset—FGSM
	Recall (%)	Precision (%)	F1 (%)	Accuracy (%)
Original	78.16	76.52	77.16	77.16
FGSM	65.38	50.41	51.85	65.42

. The aim of the follow-up, for purposes of this evaluation, was to confirm whether the overall efficacy of adversarial attacks is generalizable across similar, yet distinct datasets used for monitoring and classifying power systems. Therefore, we evaluated this additional dataset under the same adversarial scenarios to determine if the vulnerabilities previously noted are isolated incidents or systemic issues related to the DL-based power classifiers.

In addition to the quantitative results from the experiment, Figure 10, Figure 11 and Figure 12 illustrate the degradation of classifier performance in different scenarios, thus pairing the numbers with visual depictions to show how each of the metrics of recall, precision, F1 score, and accuracy (for clean and perturbated data) were impacted differently.

Figure 10 summarizes the results of the DeepFool attack. Based on the graphical trend, there was a steady and significant decline across performance measures with all performance measures cascading down from the clean data. The most significant decline was for F1 score and accuracy. This behavior highlights the vulnerability of the model to DeepFool attacks when perturbations are small perturbations with an L2 norm, given how confidently misclassified labels can strip credibility from the model.

Figure 11 summarises the JSMA results. In contrast to the previous performance trends, there was an overall performance decline, but the contrast of each metric showed that precision was still relatively high compared to recall and F1 score on the Y axis. This pattern supports the idea of JSMA as attacks are targeted perturbations characterized by high saliency values on input variables to generate confidently incorrect outputs. The imbalanced proportion of precision and recall also suggests the classifier’s inner workings were distorted or unbalanced with regard to probabilities of the labels through targeted perturbations.

Figure 12 shows the impact of the FGSM on performance. Although it is less severe than the past attacks, it did result in a significant drop in precision, as well as a drop in accuracy and F1 score metrics. The FGSM is a one step gradient attack that is quick and therefore has a low computational burden on the model, but it can still degrade model reliability significantly, further illustrating the applicability of fast adversarial attacks for real-time monitoring of power system operation.

6. Conclusions

The results of 10 experiments averaged together show the performance of the machine learning models on the Mississippi State University and Oak Ridge National Laboratory power system dataset with modest accuracies at 77.022% for binary classification, 78.015% for triple-class classification, and 60.983% for multiclass classification under normal conditions. However, the application of adversarial attacks—DeepFool, JSMA, and the FGSM—catastrophically ruined model performance in all the classification settings. Accuracy in binary classification dropped to 40–70%, with JSMA experiencing the biggest drop to 47.186% (~29.8% down), followed by the FGSM to 54.742% (~22.3% down) and DeepFool to 68.856% (~8.2% down), reflecting the strength of JSMA in targeting binary decision boundaries by selectively manipulating features. For triple-class classification, with a baseline accuracy of 78.015%, FGSM attack registered the highest drop, reducing accuracy to 46.561% (a ~31.9% drop), while DeepFool and JSMA both reduced accuracy to 59.778% (a ~18.2% drop), reflecting the power of the FGSM in multi-class settings despite its lower computational cost. In multiclass classification with baseline accuracy 60.983% in 41 classes, all three attacks caused severe degradation, with DeepFool producing the largest reduction to 4.6% (a ~56.4% reduction), then the FGSM to 8.926% (a ~53.1% reduction) and JSMA to 15.91% (a ~45.6% reduction), which can be explained by the huge class size amplifying the effect of the attacks, particularly DeepFool’s tiny perturbations in a large-dimensional space. These findings emphasize the devastating potential of adversarial attacks on machine learning models in power systems to hijack state estimation, hijack demand response systems, and induce faults in core decision-making frameworks, potentially triggering power outages, economic losses, environmental disasters, and violations of public security. This study, drawing on earlier work, emphasizes the utmost significance of protecting AI models from such attacks, as attacks such as DeepFool, JSMA, and the FGSM can have a profound impact on the security and reliability of AI-powered power systems and need robust defenses to ensure the resilience of critical infrastructure that contemporary society relies on. For future research, exploration of adaptive defense strategies, such as hybrid approaches combining Adversarial Training and Feature Squeezing, could further enhance model resilience to evolving attack strategies. Moreover, implement the model in a real-time power system environment to evaluate its impact on system responsiveness and network performance. We will also consider increasing the model performances using more expressive model architectures. In addition, investigating the scalability of such defenses in real-time operation of actual power systems, particularly for small grid operators with limited resources, and developing standardized cybersecurity protocols for smart grid equipment, such as inverters and converters, can potentially further improve the security of AI-based power systems against adversarial attacks.