Mitigation of Black-Box Attacks on Intrusion Detection Systems-Based ML

: Intrusion detection systems (IDS) are a very vital part of network security, as they can be used to protect the network from illegal intrusions and communications. To detect malicious network trafﬁc, several IDS based on machine learning (ML) methods have been developed in the literature. Machine learning models, on the other hand, have recently been proved to be effective, since they are vulnerable to adversarial perturbations, which allows the opponent to crash the system while performing network queries. This motivated us to present a defensive model that uses adversarial training based on generative adversarial networks (GANs) as a defense strategy to offer better protection for the system against adversarial perturbations. The experiment was carried out using random forest as a classiﬁer. In addition, both principal component analysis (PCA) and recursive features elimination (Rfe) techniques were leveraged as a feature selection to diminish the dimensionality of the dataset, and this led to enhancing the performance of the model signiﬁcantly. The proposal was tested on a realistic and recent public network dataset: CSE-CICIDS2018. The simulation results showed that GAN-based adversarial training enhanced the resilience of the IDS model and mitigated the severity of the black-box attack.


Introduction
Nowadays, the use of the Internet, in general, and reliance on cloud-based resources is growing at an exponential rate. Operations are concentrating on their core businesses while transferring their information technology (IT) services to the cloud. Many more factors encourage businesses to use internet-based offerings. Likewise, malicious traffic has increased at a rapid rate [1]. Today's cyberattacks are becoming more diversified and broad. The purpose of these assaults is to obtain unauthorized access to remote data or to create service interruptions for consumers. These attacks have a tremendous influence not only on the economy and finances of a country but also at the national level, in addition to cultural security [2,3]. Therefore, such assaults should be prevented from both inside and outside as well as from governmental and private institutions [4][5][6]. As a result, it is critical to rely on automated powerful systems for quickly and reliably identifying threats. Interestingly, intrusion detection systems (IDS) have been considered an excellent solution to further boosting the security level of a system [7].
IDS is a type of security software that observes network traffic and gives warnings once an unusual behavior is discovered [8]. Generally, there are two kinds of IDS: host-systems are continually probed by adversaries with inputs that are specifically meant to evade the system and generate a false prediction. Furthermore, malicious attacks have become more common, and ML models' defense and resistance against them must be addressed. Several studies in text and image recognition fields have looked at the danger and provided viable countermeasures. Unfortunately, not much research on the NIDS sector that addresses the problems of adversarial attacks has been undertaken [12]. In addition, the learning model and dataset quality are both closely connected to the efficiency of the IDS. Many researches have been dependent on datasets that have significant shortcomings such as simulated traffic (i.e., not from an actual production network), anonymity, redundancy, and outdated attack traffic, e.g., denial of service (DDoS) [20,23]. Other studies have concentrated on the adversary knowledge factor, such as white-box attacks, and shown that such attacks are strong in targeting a system under the assumption that opponents have full access and knowledge of the classifier [24,25]. In practice, having such an ability by an attacker seems to be elusive. It has been proven that a GAN is a very serious and powerful attack compared with other existing attacks [26]. Contrary to white-box attacks, GAN-based black-box attacks are considered weak, as attackers have no knowledge or only have superficial information about the victim classifier. A GAN-based adversarial ML attack has been proposed and validated on a black-box IDS, and it turned out that GAN is a powerful technology for bypassing an IDS due to the fact of its potential to generate data that have a similar distribution to the original dataset [20]. In general, there is a lack of research studies investigating and evaluating the effectiveness of existent adversarial defensive mechanisms. Accordingly, it is necessary to ensure the resilience of the proposed methods against adversarial attacks and to pay more attention to proposing attack-agnostic defense mechanisms that address the increasing variety of adversarial attacks, rather than focusing only on a narrow range of attacks [27]. Therefore, the above reasons served as motivation to propose the main contribution in this paper.

The Contribution of the Paper
The major contribution of this paper is as follows: We designed a defensive model for NIDS-based random forest classifier and enhanced the proposed model using GAN-based adversarial training, where the generated adversarial examples are used for training the model and measuring the model resistance in two phases. The first phase was utilized to train the proposed technique on a noncrafted dataset, and the second phase was related to improving the robustness and accuracy of the first phase by retraining the proposed model on a combined dataset that included a non-crafted dataset, generated dataset from a GAN; • Our proposal was further improved by carefully training our model with valuable features selected by PCA with the generated adversarial examples; • We implemented a black-box-based ZOO attack to evaluate the resistance of the proposed random forest model in which this attack was capable of generating adversarial examples that the model had never seen before.
To the best of our knowledge, there is no recent work concentrated mainly on the improvement of ML-based IDS as a defense model and evaluating the resilience of the model by thwarting new unseen attacks.
The remainder of this paper is structured as follows: Section 2 tackles, in brief, the random forest model, GAN-based defense technique, black-box-based ZOO attack, and feature reduction methods. Section 3 tackles the curriculums of related work including adversarial attack and defense techniques. Our proposed methodology is described in detail in Section 4. The experimental setup and results of the proposal are illustrated in Section 5. A comparison with other prior work is given in Section 6. Finally, we provide the conclusion in Section 7.

Background
In this section, we present the fundamentals of the random forest classifier and explain the GAN architecture in detail. Afterward, we describe the realistic threat model scenario, ZOO attack, which was considered in our work, and then we explain the current feature selection and reduction methods.

Random Forest (RF) Classifier
RF is one of the most powerful methods employed to solve classification and regression issues in machine learning. It is a class of supervised classification algorithms. The random forest requires two steps: one is to tune the random forest configuration, and the other is to predict the incoming results obtained from step one [28]. The random forest algorithm is implemented based on building multiple decision trees; each one represents a classifier. Every tree in the forest is sampled from the original dataset to create a sub-dataset. Then, subsets of data are placed in each decision tree, and each decision tree produces results. The result of the final decision is determined via a vote by all decision trees. A tree does not select all the features, instead only some features are randomly chosen; then, from the chosen features, only the optimal features are selected. Because of this randomness, its variance decreases, and a better overall classification model is also produced [29].

GAN-Based Defense Methodology
A GAN is a deep learning approach that is composed of two NNs, each one against the other in a game setting as shown in Figure 1 [30]. It has been studied in depth in the field of security, as a GAN is capable of generating new unseen threats. The usage of a GAN as a defense mechanism renders the model more robust against future attacks. The main objective of a GAN is to detect unknown or unseen attacks and protect systems from various vulnerabilities [18]. In a zero sum game context, a GAN has two NNs competing against each other. One is leveraged for producing regression and labeled as a generator (G), while the second is labeled as a discriminator (D). Usually, the purpose of the generator is to take random noise (V) as input, transform it using the NNs, and create false instances, whereas the aim of the discriminator is to use a NN to separate the infected data generated via the generator from the actual one [31,32]. When the process reaches equilibrium, the discriminator is unable to recognize between real and bogus data. The generator, therefore, accepts random noise (V) as input and produces actual instances as output. That is to say, the generator has found how the data is distributed [26,33]. The adversarial loss for both G and D is given in Equations (1) and (2), respectively [34].
In the above equations, S refers to the data collected from the generator and leveraged to train the discriminator, while the variable E is the expected volume of the produced data that is indicated to be an attack or benign. B benign is a variable for the benign data, and B attack is the attack data.

Black-Box-Based ZOO Attack
The ZOO-attack-based method was first introduced in [35] to generate adversarial examples (AEs). Note that white-box-attack-based methods differ from black-box attacks, as black-box methods do not rely on the gradient information of the target model. The black-box attack process represents a targeted misclassification by which the data are crafted to generate AEs. The generation of such examples relies on the modification of optimization parameters and on a conjecture of confidence, rather than the gradation.
When an attacker generates these examples, he or she utilizes them to violate IDS [36]. In this paper, the threat model settings assume an attacker only queries the model for relevant labels and has no access to the IDS model, including its hyperparameters. The goal of such an attacker is to generate AEs that are hard to detect via the IDS model, and this makes the model vulnerable to many threats.
Computers 2022, 11, x FOR PEER REVIEW 5 of 18 In the above equations, S refers to the data collected from the generator and leveraged to train the discriminator, while the variable E is the expected volume of the produced data that is indicated to be an attack or benign. Bbenign is a variable for the benign data, and Battack is the attack data.

Black-Box-Based ZOO Attack
The ZOO-attack-based method was first introduced in [35] to generate adversarial examples (AEs). Note that white-box-attack-based methods differ from black-box attacks, as black-box methods do not rely on the gradient information of the target model. The black-box attack process represents a targeted misclassification by which the data are crafted to generate AEs. The generation of such examples relies on the modification of optimization parameters and on a conjecture of confidence, rather than the gradation. When an attacker generates these examples, he or she utilizes them to violate IDS [36]. In this paper, the threat model settings assume an attacker only queries the model for relevant labels and has no access to the IDS model, including its hyperparameters. The goal of such an attacker is to generate AEs that are hard to detect via the IDS model, and this makes the model vulnerable to many threats.

Reduction Techniques
To further increase the IDS model's resilience, the most valuable features should be extracted from the collected dataset. This will also help to decrease the data's dimensions and the model's complexity. Such a method is known as a reduction method in which only valuable features are chosen during the classification. In this work, two reduction techniques, known as "PCA" and "RFE", are tackled in the following.

•
Principle component analysis (PCA) is widely employed to extract preferable features and compress them, in which the dimensions of the feature are reduced. Note that this also leads to the diminishment of the computational time and the model's complexity. The subsets of the feature set are extracted via PCA, and this helps diminish the search range [37]. In fact, the general usage of PCA is to extract important features for traffic analysis [38]; • Recursive feature elimination (Rfe) is utilized to select some valuable features out of all of the features in the dataset. Only features with high ranking are selected, and reset features (e.g., those with low ranks) are eliminated one by one. Rfe technique removes duplicated features and extracts only preferable and valuable features from all dataset features. The goal of Rfe is to choose the best subsets of valuable features [10].

Reduction Techniques
To further increase the IDS model's resilience, the most valuable features should be extracted from the collected dataset. This will also help to decrease the data's dimensions and the model's complexity. Such a method is known as a reduction method in which only valuable features are chosen during the classification. In this work, two reduction techniques, known as "PCA" and "RFE", are tackled in the following.

•
Principle component analysis (PCA) is widely employed to extract preferable features and compress them, in which the dimensions of the feature are reduced. Note that this also leads to the diminishment of the computational time and the model's complexity. The subsets of the feature set are extracted via PCA, and this helps diminish the search range [37]. In fact, the general usage of PCA is to extract important features for traffic analysis [38]; • Recursive feature elimination (Rfe) is utilized to select some valuable features out of all of the features in the dataset. Only features with high ranking are selected, and reset features (e.g., those with low ranks) are eliminated one by one. Rfe technique removes duplicated features and extracts only preferable and valuable features from all dataset features. The goal of Rfe is to choose the best subsets of valuable features [10].

Literature Review
Given the most recent resurgence of DL effectiveness models and ML approaches, studies in different domains have been accomplished to resolve prominent challenges in the various realms across the world [39]. ML and DL models have been widely leveraged in data generation, network security classification, network attack modification, and forecasting. This section tackles prior work on AML attacks and possible defense techniques for NIDS.

Adversarial Attack Approaches
Researchers have investigated adversarial attacks and shown how easily they may fool ML models [40]. White-box and black-box threats are two types of adversarial attacks. The former necessitates that the white-box attack has access to the variables of the detector, whereas the black-box setting does not [25]. Lin et al. proposed a framework, namely, IDSGAN, to produce AEs that can fool the IDS model by making prediction mistakes. IDSGAN employs a Wasserstein generative adversarial network (WGAN) to produce malicious traffic records that are hard to detect by IDS. A WGAN is composed of three major parts: a generator (G), discriminator (D), and a black-box IDS. The G generates hostile unlawful data from the incoming mixed malicious records with noise. The black-box IDS is used to foretell the malicious records from the normal ones by producing predicted labels as targets, in which the D uses these targets to impart the black-box IDS [41]. In 2019, researchers proposed an AML attack via the utilization of GANs to create a large adversarial variation in the original network dataset. This attack aimed to evade a blackbox IDS. Then, GANs were used as a defense mechanism during the training phase to render the system more robust against adversarial threats. The KDD99 dataset, which is extensively used to measure IDS performance, was used to examine the proposed GAN. The experiments showed that the highest accuracy was 65.38 percent for gradient boosting (GB), and the lowest accuracy was 43.44 percent for support vector machine (SVM). After training with the GAN, the classifiers' performance improved, where the accuracy rate reached 86.64 and 79.31 percent for LR and KNN, respectively [20]. Later in 2021, a new aggressive framework, called anti-intrusion detection auto encoder (AIDAE), was proposed, where GAN was used to create features for deactivating IDS. This framework has an encoder that converts some characteristics to embedding space, and many decoders to gather discrete and continuous characteristics. Then, a GAN is employed to impart the previous distribution of the embedded space. The framework learns the typical feature spread to produce irregular features, and this does not require IDS feedback during the training operation. In addition, the proposal maintained the correlation between the created discrete and continuous characteristics. The test was carried out on the NSL-KDD, UNSW-NB15, and CICIDS2017 datasets, with six classifiers (i.e., LR, K-NN, DT, AdaBoost, RF, and CNN+LSTM). The experimental results demonstrated that the generated characteristics were capable of weakening the baseline IDS, implying that researchers need to take into consideration defending against such attacks in the future [42].

Recent Work on Defense
Many researchers have introduced novel defense techniques to prohibit various existing threats via leveraging GANs to render stronger IDS. In 2018, the author, Mirza, introduced an ensemble learning method to enhance system resilience. The results in all classifiers were merged by collecting the most valuable information from all classifiers (e.g., LR, NN, and DT) during both the training and testing phases. Afterward, a weighted majority voting mechanism was applied to each individual classifier, and the results were released to determine whether each sample was abnormal. The general accuracy for the training and testing was 96.66% and 96.13% for LR, 90.67% and 89.83% for NN, and 92.08% and 91.66% for DT [43]. In the same year, Zenati et al. proposed an anomaly detection method-based bidirectional GAN, called adversarial learned anomaly detection (ALAD). The GAN learned the distribution of the features to perform the anomaly detection goal. Afterward, recreation errors based on the adversarial features were leveraged to specify whether a given sample was malicious or benign. ALAD is constructed on the last level to guarantee "data-space", "latent-space", and "cycle-consistencies" and to stabilize a GAN during training. It was proven that the proposal elevates anomaly detection performance significantly compared to state-of-the-art studies, where the KDD99 and Arrhythmia tabular datasets and the SVHN and CIFAR-10 picture datasets were employed during the evaluation [44]. A novel intelligent IDS was introduced in which a lower number of features was used to detect intrusions. The genetic algorithm (GA) is leveraged to extract preferable features in order to minimize resource usage and time complexity. After employing the GA to remove the redundant and unnecessary data from the dataset, the GA output predicts the best features via using a specific number of comparisons. The true positive rate was enhanced when feature ranking was accomplished according to the results obtained from averaging the values in the dataset [45]. In 2021, McCarthy et al. proposed a defensive strategy for measuring feature susceptibility to AEs generated by the fast gradient signed method (FGSM) attack. The FGSM is a combination of white-box and misclassification models, which is used to trick a NN model via rendering incorrect predictions. The presented strategy aims to strike a balance between classification results and diminishing potential attacks on the feature space. The proposal was evaluated on the CICIDS2017 dataset. The authors found that there are data features vulnerable to attack. Defense solutions are given for algorithmically generated AEs. In addition, Rfe was employed to eliminate the vulnerability features that had the largest absolute difference during the FGSM attack. Furthermore, regular feature selection for training enhanced the model's durability against AEs. With limited features, the method achieved high accuracy. When all features were taken into account, the model had the highest accuracy; however, the accuracy under assault seldom reached 60%. The results indicate that incorporating feature selection increases the accuracy rate of the model when an FGSM attack exists [25].

Proposed Research Methodology
In this section, we present in detail our proposed technique's structure, the dataset's preparation and preprocessing, and the evaluation metrics.

Model Structure
A framework diagram of the proposed model is demonstrated in Figure 2. It consists of three main parts. The first part is data preprocessing, which is used to prepare the original data for the ML models and apply feature reduction methods to improve the accuracy and reduce the complexity. Specifically, we used PCA and RFE to reduce the data's dimensions by selecting only the relevant features that are needed for the classification task. The second part is the defender model, which consists of the classifier model and the GAN model. The classifier is an ML model used for binary classification. The GAN model aims to generate adversarial examples (AEs) from the original dataset using arbitrary latent vector (noise vector) and retrain our classifier on the new dataset (original and synthetic dataset) to make it more resistant and powerful against known and unknown attacks in the future. The last part, i.e., the attacker model, is a black-box attack method. This model generates new AEs that aim to evade the detection system, the defender model, and influence on the predictions of the classifier to determine its robustness.

Dataset
The CSE-CIC-IDS2018 is an intrusion detection dataset created by the Communications Security Establishment and Canadian Institute for Cybersecurity on AWS (Amazon Web Services), located at Fredericton, Canada, in 2018 [46]. The IDS2018 is the updated version of the IDS2017 dataset and the latest and most comprehensive intrusion dataset, collected for launching real attacks, which is publicly available. The dataset includes the necessary standards for the attack dataset and contains many different kinds of attacks. This dataset also comprises network traffic, system logs, and 80 features [47]. To better model the attacks, a topology with a machine diversity similar to real-world networks was created [48]. The infrastructure of the network included 50 attacker machines, 420 victim machines, and 30 servers. The details of the dataset's features are provided in Table 1.
The intrusions in the CSE-CICIDS2018 dataset were normalized into two kinds, namely, benign and malicious. The number of benign and malicious network traffics is given in Table 2.

Data Preprocessing
The CSE-CICIDS2018 dataset consists of over 1,000,000 records. The dataset consists of the original traffic in the packet capture (pcap) files, the logs, the preprocessed labels, and the feature-selected comma-separated values (CSV) files. The CSV files are categorized into two classes benign (class-0) and malicious (class-1). The dataset does not contain blanks or errors. Therefore, we applied some preliminary data processing procedures which are presented as follows:

1.
Numerical standardization: To provide data consistency, the data were standardized using the technique of obtaining the Z-Score in which the standard deviation was set to 1, and the average value of each feature was set to 0.

2.
Outliers: We deleted two features (i.e., the timestamp (date and time) and Fwd packets features) from the CSE-CIC-IDS2018 dataset, because they have a neglected influence on the model training. Therefore, the total number of features was 78. 3.

Replacement of default values:
In the leveraged dataset, the packet length Std feature has a value of infinity. We fixed this by changing its value to 0 in the database.
For all the experiments in Section 5, 75% of the dataset was employed to train the ML model, and the remaining 25% was employed to test the model. However, 70% of the dataset was used to train the GAN model, and the remaining 30% was used to test the model. collected for launching real attacks, which is publicly available. The dataset includes the necessary standards for the attack dataset and contains many different kinds of attacks. This dataset also comprises network traffic, system logs, and 80 features [47]. To better model the attacks, a topology with a machine diversity similar to real-world networks was created [48]. The infrastructure of the network included 50 attacker machines, 420 victim machines, and 30 servers. The details of the dataset's features are provided in Table 1.

Count Description 4
Basic features of network connections 11 Features of network packets 5 Features of network flow 22 Statistic of network flows 17 Content-related traffic features 3 Features of network sub-flows 18 General purpose traffic features

Evaluation Metrics
There are several classification metrics for IDS. The confusion matrix (CM) of a twoclass classifier was used to compute the performance metrics because, in our work, the experiments were conducted broadly to distinguish between malicious and normal records. The abbreviations of the CM are as follows:

Results
This section presents three experimental setups with the results. In the first setup, we used all 78 features of the dataset as the model's input. However, we used dimensionality reduction methods (i.e., RFE and PCA) to reduce the number of features in experiments II and III, respectively. Each experiment included three parts: training the ML model on the original dataset; using GAN to generate adversarial examples and retrain the ML model on the original and generated dataset; evaluating the performance of the ML model when the black-box ZOO attack was applied.

Experiment I
In the first experiment, we used all 78 features to train our IDS classifier, the random forest. The classifier was used to classify the dataset into two classes (i.e., benign and malicious). The hyperparameters of the random forest classifier are illustrated in Table 3.
The second part of this experiment was to increase the ability of this model to handle more than a real dataset. Therefore, the proposed GANs with all 78 features was built to generate adversarial examples to increase the defense mechanism. The architecture of the proposed GAN consisted of two neural networks (i.e., generator (G) and discriminator (D)). The G neural network model had three layers and a Relu activation function, including an input layer with 79 units to meet the formula of the input vector after preprocessing. The hidden layer consisted of 100 units, and the output layer of the proposed generator had 79 (78 features, 1 label), which are referred to as the vectors of the fake record generated from noise V. On the other hand, the proposed D network was designed to classify (fake or real) data generated by the G network. It was also used to update the noise vectors depending on the feedback loss function from the network. This D network consisted of three layers: an input layer with 78 units followed by the activation function Relu; a hidden layer with 100 units followed by a dropout layer with a dropping rate of 0.4 used to avoid the overfitting problem; finally, the sigmoid output layer was used for binary classification: 0 for real, 1 for fake. In this experiment, the Adam optimizer was used to update the trainable parameter at a learning rate of 0.001 as shown in Table 4.  After 2000 epochs, the D completely failed to classify the output from the G network, which is known as fooling the D model to distinguish between the real and generate fake samples. The loss rate of G reached the lowest value at 0.002, while the D loss rate reached 17.12 as shown in Figure 3. The process of training the GAN led to the generation of 230,000 samples. After this process, the generated data were merged with the real data and used to retrain the proposed RF model. Table 5 shows the classification results of the proposed RF model with all 78 features. The third part of this experiment was to evaluate the proposed IDS model using a black-box attack. We used the ZOO method as an attacker model to generate adversarial examples that were used for launching against the proposed RF classifier. Our goal was to assess the ability and susceptibility of the system after the adversarial training process. We modified this method and used it on one vector. The adversarial setting of the proposed ZOO model is explained as follows: the Adam optimizer with β1 = 0:8; β2 = 0:899 was applied to minimize the loss with a learning rate of 0.001 and ∈ = 0.0000001. The maximum epoch was set to 2000. The classification results of the proposed RF classifier after applying the ZOO attack are shown in the third row of Table 5. Figures 4a-c and 5a-c summarize the confusion matrices (CMs) and AUC-ROC of all three parts of experiment I.    Table 5 summarizes the classification results of the proposed random forest classifier (IDS) when 78 features were used. Even though RF before the GAN provided good results compared to RF after the GAN and ZOO, the performance could be improved by removing unrelated or redundant features from the dataset. Therefore, it was necessary to choose the best and most effective features from the dataset to improve the performance of our IDS classifier. As can be seen in Table 5, the generated data by the GAN resulted in a decline in accuracy. This may indicate that these generated samples were somehow symmetrical to the real data and, therefore, identification by the trained model is hard. Moreover, the selected values of the parameters (e.g., epochs) help the GAN model produce strong adversarial examples. It is worth mentioning that even though the ROC was close to 1, as seen in Figure 5b, it did not help much in detecting infected samples generated by the GAN. Moreover, the accuracy of the RF was significantly decreased under the influence of the black-box attack (ZOO), because the samples generated by the ZOO attack could not be easily detected by the IDS. This confirms two things: (1) the process of the ZOO adversarial black-box attack generated new and strong infected instances that were hard for our model to detect, and this reduced the model's efficiency; (2) RF had the advantage of good accuracy on the original dataset with imbalanced high diminutions. Based on the aforementioned, the RF was successful because it did not have the problem of nominal data and did not overfit the data. Note that our proposal was still efficient at detecting unknown attacks as shown in Figure 5c.

Experiment II
The objective of this experiment was to train and test the proposed IDS model on fewer features using feature reduction methods, because we concluded from previous experiments that taking all of the features did not provide an optimal performance probably, as some of the features were unrelated and redundant. Accordingly, we turned to leveraging methods to minimize the features to obtain a better result. Specifically, we applied the RFE feature selection method to the original dataset to select only eight features. The RFE method was efficient in choosing the robust features and neglecting the weaker ones. Furthermore, it reduced the dependencies and the interlinear relationships that may exist in the dataset. The most important eight features described in Table 6 were utilized to retrain the proposed RF classifier. The first row of Table 7 illustrates the classification results of the RF classifier with the eight input features.
Similar to experiment I, we applied the strategy of adversarial training based on the GAN to generate new samples based on the same eight selected features. We also tested the model when the ZOO model was used to attack the classifier. The proposed GAN was modified to use eight features instead of 78. After 2000 epochs, the G's success in fooling the D can be seen in Figure 6. The proposed RF model was retrained with 240,000 adversarial samples that were generated by the GAN. The classification results of the proposed RF after applying the GAN are shown in the second row of Table 7. While the results after applying the ZOO attacker are shown in the third row of Table 7. Figures 7a-c and 8a-c summarize the confusion matrices and AUC-ROC of all three parts of experiment II.

Experiment II
The objective of this experiment was to train and test the proposed IDS model on fewer features using feature reduction methods, because we concluded from previous experiments that taking all of the features did not provide an optimal performance probably, as some of the features were unrelated and redundant. Accordingly, we turned to leveraging methods to minimize the features to obtain a better result. Specifically, we applied the RFE feature selection method to the original dataset to select only eight features. The RFE method was efficient in choosing the robust features and neglecting the weaker ones. Furthermore, it reduced the dependencies and the interlinear relationships that may exist in the dataset. The most important eight features described in Table 6 were utilized to retrain the proposed RF classifier. The first row of Table 7 illustrates the classification results of the RF classifier with the eight input features. Similar to experiment I, we applied the strategy of adversarial training based on the GAN to generate new samples based on the same eight selected features. We also tested the model when the ZOO model was used to attack the classifier. The proposed GAN was modified to use eight features instead of 78. After 2000 epochs, the G's success in fooling the D can be seen in Figure 6. The proposed RF model was retrained with 240,000 adversarial samples that were generated by the GAN. The classification results of the proposed RF after applying the GAN are shown in the second row of Table 7. While the results after applying the ZOO attacker are shown in the third row of Table 7. Figures 7a-c and 8a-c summarize the confusion matrices and AUC-ROC of all three parts of experiment II.         According to the results in Table 7, the accuracy did not differ much from the accuracy of the random forest in experiment I. However, after applying the GAN and retraining the RF model, the accuracy improved compared to the GAN in experiment I. The MSE also improved from 0.14 to 0.09, as well as the FN in the confusion matrix. In addition, an improvement in the ability of the classifier ROC curve from 0.93 to 0.97 is shown in Figure 8b. The performance of the RF model against the ZOO attacker in experiment II was lower than in experiment I. Where the accuracy dropped from 0.69 to 0.48. This suggests selecting more robust features from the original dataset by using a different feature selection method. Therefore, we performed the third experiment.

Experiment III
In this experiment, we used a different a feature selection method to improve the classifier's performance and handle the issue in experiment II when the ZOO attack was applied. Specifically, we used the PCA feature selection method to select the most efficient features from the original 78 features. The experimental results, illustrated in Table 8 Table 8. Similar to the previous two experiments, we evaluated the RF classifier using the ZOO attack. The results are shown in the third row of Table 8. Figures 9a-c and 10a-c summarize the CMs and AUC-ROCs of all three parts of experiment II. According to the results in Table 7, the accuracy did not differ much from the accuracy of the random forest in experiment I. However, after applying the GAN and retraining the RF model, the accuracy improved compared to the GAN in experiment I. The MSE also improved from 0.14 to 0.09, as well as the FN in the confusion matrix. In addition, an improvement in the ability of the classifier ROC curve from 0.93 to 0.97 is shown in Figure  8b. The performance of the RF model against the ZOO attacker in experiment II was lower than in experiment I. Where the accuracy dropped from 0.69 to 0.48. This suggests selecting more robust features from the original dataset by using a different feature selection method. Therefore, we performed the third experiment.

Experiment III
In this experiment, we used a different a feature selection method to improve the classifier's performance and handle the issue in experiment II when the ZOO attack was applied. Specifically, we used the PCA feature selection method to select the most efficient features from the original 78 features. The experimental results, illustrated in Table 8 Table 8. Similar to the previous two experiments, we evaluated the RF classifier using the ZOO attack. The results are shown in the third row of Table 8. Figures 9a-c and 10a-c summarize the CMs and AUC-ROCs of all three parts of experiment II.
The improvement in the RF classifier results shown in Table 8 indicate that the PCA method was more robust than the Rfe method. The RF with PCA accuracy was 0.863, while the RF with RFE accuracy was 0.85. The results also show that using GAN with the features selected by PCA achieved the highest accuracy of 99.9 and a significant decrease in the MSE from 0.014 to 0.0001. The excellent results of this experiment were not limited to the adversarial training but also extended to the black-box attack. Specifically, the system repelled the ZOO attack and obtained the highest accuracy of 0.759; the accuracies of experiment I and II when ZOO attack is applied were 0.69 and 0.487, respectively.

Comparison with Previous Studies
In this section, we compare our proposed ML-based IDS with state-of-the-art ML/DLbased IDS that only evaluate the model' performance without measuring the model resistance. Each of the prior work was implemented with different methods (e.g., some of them used a single model and others used multiple models) on same dataset. The results show that our proposal offers better accuracy compared to other existing works as shown in Table 9. It is worth mentioning that we did not compare our work with work that handled class imbalance issues by modifying the original dataset (e.g., the work in [47,49]), because it was beyond the scope of this paper, and it could be a complement to our proposed model. Although DL has been proven to be effective in the field of NIDS, our proposed ML model achieved better results. This might be because DL algorithms deal very well with complex tasks that require discovering relationships among a large number of different features. However, our experimental results show that reducing the number of features of our targeted task led to improving the overall accuracy. In addition, a recent study showed that using such techniques (i.e., using PCA to minimize the dimensionality of the dataset) with IDS reduced the performance of the model. Table 9. Comparison with some related works on the CSE-CICIDS2018 dataset.

Authors
Year Models Acc Usama, Asim et al. [20] 2019 LR 0.866 Amaizu, Nwakanma et al. [50] 2020 DNN 0.764 Fitni and Ramli [11] 2020 Ensemble model 0.988 Sawadogo, Bassolé et al. [51] 2021 CNN 0.975 Our proposed method 2022 RF 0.999 We compared our work with a related work [20] to measure the proposed model resistance against different attacks, where they used GAN as a defense method based on adversarial training. The test was conducted before and after applying GAN and ZOO attacks with and without applying adversarial training. Our model outperformed the work in [20] in all testing stages when these two attacks were applied as shown in Table  10. We obtained better results due to the technique used for feature selection and number of epochs. Specifically, we used PCA for feature selection instead of dividing the features into functional and nonfunctional as done in [20]. Moreover, since the epochs used in [20] were only 100, this would not be enough to generate sufficiently strong fake samples for training the model effectively. To address this dilemma and generate strong samples, we increased the number of epochs to 2000. The improvement in the RF classifier results shown in Table 8 indicate that the PCA method was more robust than the Rfe method. The RF with PCA accuracy was 0.863, while the RF with RFE accuracy was 0.85. The results also show that using GAN with the features selected by PCA achieved the highest accuracy of 99.9 and a significant decrease in the MSE from 0.014 to 0.0001. The excellent results of this experiment were not limited to the adversarial training but also extended to the black-box attack. Specifically, the system repelled the ZOO attack and obtained the highest accuracy of 0.759; the accuracies of experiment I and II when ZOO attack is applied were 0.69 and 0.487, respectively.

Comparison with Previous Studies
In this section, we compare our proposed ML-based IDS with state-of-the-art ML/DLbased IDS that only evaluate the model' performance without measuring the model resistance. Each of the prior work was implemented with different methods (e.g., some of them used a single model and others used multiple models) on same dataset. The results show that our proposal offers better accuracy compared to other existing works as shown in Table 9. It is worth mentioning that we did not compare our work with work that handled class imbalance issues by modifying the original dataset (e.g., the work in [47,49]), because it was beyond the scope of this paper, and it could be a complement to our proposed model. Although DL has been proven to be effective in the field of NIDS, our proposed ML model achieved better results. This might be because DL algorithms deal very well with complex tasks that require discovering relationships among a large number of different features. However, our experimental results show that reducing the number of features of our targeted task led to improving the overall accuracy. In addition, a recent study showed that using such techniques (i.e., using PCA to minimize the dimensionality of the dataset) with IDS reduced the performance of the model. Table 9. Comparison with some related works on the CSE-CICIDS2018 dataset.

Authors
Year Models Acc We compared our work with a related work [20] to measure the proposed model resistance against different attacks, where they used GAN as a defense method based on adversarial training. The test was conducted before and after applying GAN and ZOO attacks with and without applying adversarial training. Our model outperformed the work in [20] in all testing stages when these two attacks were applied as shown in Table 10. We obtained better results due to the technique used for feature selection and number of epochs. Specifically, we used PCA for feature selection instead of dividing the features into functional and nonfunctional as done in [20]. Moreover, since the epochs used in [20] were only 100, this would not be enough to generate sufficiently strong fake samples for training the model effectively. To address this dilemma and generate strong samples, we increased the number of epochs to 2000.

Our Findings and Future Work
To identify abnormal and malicious behavior in networks, IDS have been used. Many ML techniques have been utilized to adopt different types of such systems to protect the network. Improving the system's performance and analyzing large amounts of network traffic requires providing robust and efficient systems to counter possible unknown attacks. To cover this issue, this paper proposed a theoretical-game-based approach to create a defensive system and train adversarially based on a GAN to ensure the system's reliability against black-box attacks. This system was then attacked to evaluate its strength and resilience in capturing the samples that were distorted by the adversary. This process used a three-stage framework for each experiment. All of the features were used, and then feature selection methods were performed to determine the right features for good results with less complexity and execution time. Evaluation of these experiences was conducted on the recent CSE-CICIDS2018 dataset. The outcomes showed that the quality of the data that our IDS trained on and the features that were selected as well as the rate of perturbed samples by the attacker were factors that influenced the results of the system. The PCA method was the best with lower implementation times compared to other trials. The use of a GAN as a defense technique is a good decision to protect networks from modern attacks. For future actions, we recommend that a GAN can be used in security domains other than image and encryption areas to train the system to defend itself against adverse attack scenarios. It can also be applied to deep learning techniques to determine their effectiveness with high-dimensional data. Proposing new defense methodologies against such attacks is necessary. While we have focused on the problem of binary classification in this work, it is important to extend this research to the problem of multiclass classification to classify separate types of attacks, and this will be one of our key future works. This could be important in reducing the complexity and execution time of the ML model.

Conflicts of Interest:
The authors declare no conflict of interest.