GAN-AHR: A GAN-Based Adaptive Hybrid Resampling Algorithm for Imbalanced Intrusion Detection

Abstract

With the recent proliferation of the Internet and the ever-evolving threat landscape, developing a reliable and effective intrusion detection system (IDS) has become an urgent need. However, one of the key challenges hindering the success of IDS development is class imbalance, which often leads to biased models and poor detection rates. To address this challenge, this paper proposes a GAN-AHR algorithm which adaptively balances the dataset by augmenting minority classes using CGAN or BSMOTE, based on class-specific characteristics such as compactness and density. By leveraging BSMOTE to oversample classes with high compactness and high density, we can exploit its simplicity and effectiveness. However, the quality of BSMOTE-generated data is significantly lower when the classes are sparse and lacking clear boundaries. In such cases, CGAN is better suited in this scenario given its ability to capture complex data distributions. We present empirical results on the NF-UNSW-NB15 dataset using a Random Forest (RF) classifier, reporting a significant improvement in the precision, recall, and F1-score of several minority classes. Specifically, a remarkable increase in the F1-score for the Shellcode and DoS classes was reported, reaching 0.90 and 0.51, respectively.

1. Introduction

The rapid proliferation of digital communication and its central role in daily life have increased the need for network protection. Unfortunately, the frequency of networks attacks continues to rise, necessitating the development of accurate and timely protection mechanisms [1,2]. One such mechanism is an Intrusion detection system (IDS) which is a software or a hardware system that maintains the security of computer systems by monitoring and identifying malicious actions [3]. IDSs are classified into three categories: signature-based, anomaly-based, and hybrid IDSs. In signature-based IDSs, a labeled dataset is available, and a machine learning classifier is trained on this dataset. In contrast, anomaly-based IDSs do not require a labeled dataset and instead use clustering algorithms to differentiate normal traffic from malicious attacks. Hybrid approaches combine the two approaches, utilizing their respective strengths.

Signature-based IDSs have shown great success in protecting computer networks. However, their reliance on labeled data presents several challenges. One of the major challenges is class imbalance, as generally attack records are far fewer than benign records, reflecting real-life scenarios [4,5]. To bridge this gap, several oversampling techniques, such as Synthetic Minority Over-sampling (SMOTE) and Borderline Synthetic Minority Over-sampling (BSMOTE), have been introduced to increase the number of minority class samples. Despite their simplicity and effectiveness, traditional oversampling methods suffer from major drawbacks. One critical drawback is that they lack diversity since they aim to produce similar samples. In contrast, Generative Adversarial Networks (GANs) have emerged as a powerful synthetic data generation mechanism capable of capturing the true distribution of data, and thus, producing a diverse set of samples. One critical shortcoming of GANs is their computational complexity and cost.

In this paper, we aim to shed light on class imbalance in IDSs by proposing a novel adaptive data resampling technique that dynamically selects between the Conditional GAN (CGAN) and BSMOTE depending on data characteristics. The choice is guided by two proposed metrics, class-specific density and compactness. Compactness indicates how close the class samples are to each other, with a higher compactness indicating that the class is highly clustered and thus that traditional BSMOTE would work well since the boundaries are well defined. In such cases, the complexity of the CGAN is avoided. However, the CGAN should be utilized in cases with low compactness due to the potential high data distribution and less clear class boundaries, a characteristic commonly associated with class overlap. Class density is also considered in the decision process since dense classes potentially reduce the chance of overlapping, thereby enhancing the effectiveness of BSMOTE. The proposed approach has been tested using the NF-UNSW-NB15 dataset [6], and the class-wise classification results suggest that it significantly increases the detection rate of several minority class attacks, such as Worms and DoS. A comprehensive evaluation against several existing IDS methods using the same dataset highlights that the proposed approach outperforms existing IDS methods in terms of all metrics’ precision, recall and F1-score.

The key contributions of this study are summarized as follows.

• Developing a novel adaptive, hybrid approach for data balancing that dynamically chooses between the CGAN and BSMOTE depending on data characteristics.
• Introducing two measures guiding the selection of the CGAN or BSMOTE based on class-specific density and compactness.
• Introducing a new metric called class imbalance which serves as a gate to the resampling stage, avoiding resampling classes with a sufficient number of samples.
• Improving classification performance, especially regarding macro F1-score on the NF-UNSW-NB15 dataset.

This paper is structured as follows. Section 2 briefly describes recent research on three topics: oversampling IDS methods, GAN-based IDS methods, and hybrid IDS methods. Section 3 summarizes the proposed method, describing its general framework and detailing the steps within the framework. Section 4 details the results and discussion, comparing the proposed method against other approaches. Lastly, Section 5 concludes this paper and summarizes key future research directions.

2. Related Work

Given the increasing number of attacks on computer networks, a considerable amount of research has been published on IDSs. These studies tackled the problem from diverse perspectives. One of the most promising areas of research within this field is data balancing using oversampling techniques [7,8]. In addition, recent developments in generative networks have led to a renewed interest in adapting them for intrusion detection, specifically for synthetic data generation, which addresses the class imbalance issue. These two research avenues will be investigated in the following subsections.

2.1. Oversampling IDS Methods

Despite the progress in oversampling IDS methods, there is increasing concern over the quality of the sampled data. Recently, researchers have examined the effectiveness of hybridization oversampling methods with under-sampling methods in increasing the performance of IDSs. For instance, Abedzadeh et al. [9] introduced an algorithm called the Reinforcement Learning Framework with Oversampling and Under-sampling Algorithm (RLFOUA). Interestingly, their approach is adaptive, whereby the decision to use oversampling or under-sampling is based on the classification performance. Specifically, they employed oversampling for falsely detected examples whilst using under-sampling for correctly classified examples. Their approach was tested on a well-known IDS dataset: CSE-CICIDS2018. They reported a precision of 90.55% and a recall of 91.18%, outperforming existing algorithms. Similarly, Abdelkhalek et al. [10] introduced a framework that combines oversampling with under-sampling in IDSs, focusing on increasing the detection rate of minority classes in the NSLKDD dataset. For data oversampling, they employed the Adaptive Synthetic Sampling Approach (ADASYN) method, while they used TomekLinks for data under-sampling. Later, they classified the resampled data using four deep learning algorithms, such as the deep neural network (DNN) and convolutional neural network (CNN). For comparison, they compared the proposed model against the baseline model (with no sampling) and showed that the proposed approach outperforms the baseline model, with an accuracy of 99.8%.

The hybridization of oversampling and under-sampling has also been explored in the Internet of Things (IoT) field. Researchers in [11] addressed the class imbalance issue in the smart home dataset, as there are far more benign samples than malicious ones. The central question in that study asks how different oversampling and under-sampling methods perform with five machine learning models, namely, the decision tree, multi-layer perception, Random Forest (RF), XGBoost, and CatBoost. Their experimental results revealed that the combination of the SMOTE oversampling method and the XGBoost classifier yielded a high F1 score of 78%.

Oversampling methods have also been used with dimensionality reduction methods, such as in Talukder et al. [12], where they proposed an IDS that tackles both the imbalance issue using random oversampling (RO) methods and dimensionality reduction using Stacking Feature Embedding, with Principal Component Analysis (PCA). For evaluation, they tested the proposed approach using three publicly available datasets: UNSW-NB15, CIC-IDS-2017, and CIC-IDS-2018. Specifically, using the UNSW-NB15 dataset and in both single class and multi-class classification, they reported an accuracy of 99%.

Class imbalance has also been addressed alongside other issues such as model bias. Malik et al. [13] addressed class imbalance and model bias that resulting from irrelevant features. In their proposed framework, they employed three feature selection methods namely, Recursive Feature Elimination (RFE), Sequential Feature Selection (SFS), and statistical feature selection, to filter features into a smaller subset that only consists of predictive attributes. Then, they adopted several oversampling methods to balance the dataset, such as SMOTE Edited Nearest Neighbor (SMOTE-ENN), ADASYN, and SMOTE Tomek Links (SMOTE_Tomek). Interestingly, their proposed pipeline reported promising results, using the ADASYN resampling method, coupled with an RF model, achieving an accuracy of 99.9% for the minority class (botnet).

2.2. Generative Adversarial Networks in IDSs

GANs have revolutionized the machine learning field and have rapidly garnered research interest both in industry and academia [14]. Specifically, in the IDS field, there has been increasing interest in utilizing GANs for synthetic data generation, which significantly mitigates the class imbalance issue. Alobrah [15] proposed an IDS pipeline that consists of feature selection, data balancing, machine learning, and evaluation. For feature selection, Chi-Square testing was used, whilst a GAN was utilized to generate minority class (attack) samples. Classification, both single-class and multi-class, was conducted using six classification algorithms. Evaluation was conducted by comparing the newly generated dataset with the original UNSW-NB15 dataset, and the results revealed that the latter yielded a higher accuracy of 98.14% and a precision of 98.14%. A similar pipeline was introduced by Park et al. [16], who proposed a four-step pipeline that consists of preprocessing, GAN training, autoencoder training, and classification. They employed a form of generative networks, called Boundary Equilibrium Generative Adversarial Networks (BEGANs), to generate minority class samples. The resulting data were then fed to autoencoder-driven DNNs and CNNs for training and feature reduction. Lastly, classification algorithms such as DNNs, CNNs, and long short-term memory (LSTM) were implemented. This proposed framework was validated using several datasets such as NSL-KDD, UNSW-NB15, and IoT. The experimental results suggested that this framework delivers promising results, with an accuracy of 93.2%

A large volume of published studies describes the applicability of more advanced GAN models, such as the CGAN, in the IDS field. Babu et al. [17] introduced a novel algorithm called the Modified Conditional Generative Adversarial Network (MCGAN), which is based on game theory, to solve class imbalance. Unlike the vanilla GAN, the proposed MCGAN generates new attack samples based on a specific attack class. The data are then fed to a Bi-LSTM classifier for classification. Their experiments aimed to apply the classification technique on two datasets, the original NSL-KDD+ dataset, and the modified one called NSL-KDD+20. The results revealed that the latter dataset yielded better accuracy at 95.16%, compared with 91.76% on the original dataset. Similarly, Alabsi et al. [18] proposed a framework that utilizes the Conditional Tabular Generative Adversarial Network (CTGAN) for specifically detecting DDoS and DoS attacks. Their GAN generator model is conditioned to generate DDoS and DoS attacks, whilst the discriminators’ task is to differentiate between normal and malicious attacks. The resulting data were classified using several machine learning and deep learning algorithms, on both the original Bot-IoT dataset and the one generated by their framework. Their evaluation results revealed that the latter dataset yielded an increase in accuracy, especially with the RNN model, which reported the highest accuracy increase of 0.293.

Several research studies have adopted a more complex GAN architecture, utilizing several generators instead of one, such as the study by Ding et al. [19]. They proposed a novel model called TMG-IDS, which is composed of several generators, each focusing on generating a specific attack type. Furthermore, to improve the quality of the generated data, they calculated the classification loss and cosine similarity between the synthetic examples and the original samples. The loss is used by the generator to guide data generation based on realism and performance. However, the cosine similarity is utilized such that the distribution of the synthetic data matches that of the original dataset. Their experimental results on two public datasets, CICIDS2017 and UNSW-NB15, proved the usefulness of their approach.

2.3. Combining GAN and Oversampling in IDSs

Several studies have employed both GANs and oversampling to advance intrusion detection. Yang et al. [20] proposed an algorithm called Self-Paced Ensemble and Auxiliary Classifier Generative Adversarial Network (SPE-ACGAN). Self-Paced Ensemble is utilized to under-sample the majority class, whilst the Auxiliary Classifier Generative Adversarial Network is used to generate new synthetic samples that belong to the minority classes. For evaluation and to increase the degree of imbalance, they proposed a new dataset that resulted from merging two well-known datasets: CICIDS-2017 and CICIDS-2018. Their results suggested that SPE-ACGAN increased the F1-score by 5.5% compared with traditional oversampling methods.

Several attempts have been made to compare the performance of GANs and oversampling methods. For instance, researchers in [21] have implemented two IDS approaches: a GAN-based IDS, and an IDS that utilizes SMOTE. They proposed a novel Auxiliary Classifier Generative Adversarial Networks (ACGAN) model for generating minority attacks. The remarkable feature of this model is that its discriminator not only differentiates the normal and malicious samples but also predicts the class label. Their experimental results suggested that the proposed approach outperforms the SMOTE approach in all reported metrics: recall, precision, and F1. Similarly, Kumar et al. [22] introduced a hybrid autoencoder called the Wasserstein Conditional Generative Adversarial Network (WCGAN). First, autoencoders are employed to learn the complex patterns of the data. Then, the learned features are fed to the WCGAN, which generates samples based on a specific class. This is enhanced using Wasserstein loss to maintain data stability and quality. Classification was accomplished by using an XGBoost classifier. They ran several experiments to validate the proposed model. First, they conducted comparisons with other GAN models, such as the vanilla GAN and CGAN, and reported that the proposed WCGAN yielded better classification performance. Then, they compared the proposed model against traditional oversampling methods, such as SMOTE, using three datasets, namely, NSL-KDD, UNSW-NB15, and BoT-IoT, and reported that the proposed model achieved a higher precision, recall, and F1-score.

A broader perspective was adopted by Chui et al. [23], who incorporated two generative networks GAN and a Variational Autoencoder (VAE). They proposed a three-step synthetic data generation pipeline consisting of a GAN, a VAE, and SMOTE. First, they employed SMOTE to generate minority attack classes. Despite its simplicity, SMOTE struggles to generate realistic samples. To bridge this gap, they applied GAN, which excels in learning data distribution, to generate more realistic samples. Then, to improve the data generated by the GAN in the previous stage, they adopted a VAE to filter noise and smooth data. The data generation pipeline then provides data to a CNN model for feature extraction. Classification is accomplished using a Support Vector Machine (SVM), resulting in an accuracy of 91.9–96.2% on four public datasets.

3. Materials and Methods

3.1. Methodology

The proposed GAN-AHR methodology, which addresses class imbalance issues in IDSs, consists of multiple stages as shown in Figure 1, each of which contributes to generating high-quality synthetic samples. In the first stage, a three-step data preprocessing procedure is applied to the original data to ensure data quality and consistency. Subsequently, the processed dataset is then divided into folds using K-fold cross-validation to be later used in evaluation. The key component of the methodology is the Adaptive Resampling Decision Module, which first decides whether synthetic data are needed and then dynamically chooses between BSMOTE and the CGAN according to class compactness and data density criteria. At this stage, the data are balanced, and the classification model can be trained and tested. Finally, the predictions are evaluated using key criteria such as F1-score. The logic of the proposed algorithm is illustrated in Algorithm 1, whilst the details of every stage are described in the following subsections.

Algorithm 1: Adaptive Hybrid Resampling Algorithm for Imbalanced Intrusion Detection

Input:  Raw dataset D;     imbalance threshold T;     number of folds K Output: Average macro F1,     weighted F1 scores Begin   Load raw dataset D. Preprocess: remove missing and duplicate records; drop unnecessary features. Encode categorical labels; normalize features to the range [−1, 1]. Initialize stratified K-fold cross-validation with K folds. For each fold k = 1, 2, …, K: Split D into training set D_train and test set D_test. Determine the maximum class size N_max in D_train. For each class c in D_train: Compute required samples Assess class compactness (based on variance) and density (average nearest-neighbor distance). If N_c > 0 and compactness/density criteria are met: ● Train a Conditional GAN with adversarial loss, feature matching, and diversity loss. ● Generate synthetic samples for class c. ● Filter samples: keep those with high discriminator confidence and sufficient diversity; apply fallback filter if needed. Else: apply BorderlineSMOTE for class c. Combine synthetic or oversampled data with the original training set. Train a Random Forest classifier. Evaluate on D_test and record macro and weighted F1 scores. End For. End

3.2. Data Preprocessing

Prior to model training and classification, the dataset undergoes rigorous preprocessing to ensure data quality. First, missing values and duplicate records are removed. Then, unnecessary features that do not add analytical value, such as the source and destination IP addresses, are removed, which contributes to faster and more efficient processing. A crucial step in preprocessing is encoding, whereby categorical attack labels are encoded into numerical values using a label encoder. Lastly, Min-Max normalization is conducted to transform features into a limited scope [−1, 1]. The described preprocessing steps not only ensure data quality but also reduce the model’s inaccuracies [24].

3.3. Adaptive Resampling Decision Module

Data imbalance has always been a major concern in cybersecurity as datasets are often imbalanced (more benign samples than suspicious ones), which leads to models with low generalizability [25] and biased performance [26]. Traditionally, many researchers have subscribed to the belief that oversampling techniques such as BSMOTE, which work by increasing the number of minority class instances, are sufficient for balancing data. A primary concern, however, with oversampling techniques is that they often fail to generate a diverse set of data because they focus primarily on local information [27]. In contrast to oversampling techniques, generative networks have shown great capability in generating synthetic data that maintains the original distribution [28,29], rather than merely producing data similar to the existing data. They also excel with complex datasets which are hard to balance with oversampling techniques [30]. However, a persistent challenge with generative networks is that they require considerable computational resources [31]. Therefore, to achieve the strengths of both approaches, we designed an adaptive decision model, which takes advantage of the simplicity of BSMOTE while preserving data distribution with generative networks.

3.3.1. General Framework

This module represents the core of GAN-AHR, where data augmentation is performed based on the characteristics of the data. The flowchart of this module is shown in Figure 2 and the following subsections will describe the steps in detail.

3.3.2. Resampling Criteria Analysis

With a sufficient number of samples in one class, caution must be applied, as augmenting this class does not add value, but could potentially contribute to overfitting [32]. Therefore, we incorporated a metric called class imbalance, as shown in Equation (1), which serves as a gate to the resampling stage. It calculates the difference between the number of instances in any given class and the number of instances of the majority class to determine how many samples, if any, are needed. It also limits the number of generated samples to three times the original number of samples, ensuring that synthetic samples do not overpower the original data. While traditional oversampling methods such as BSMOTE inherently solve class imbalance by using an imbalance ratio and generating synthetic samples to match the majority class size [33], generative networks do not address this explicitly.

$$N_{\mathrm{needed}}^{\left(c\right)}=\min \left(\max \left(N_{\max }-N_c,0\right),C\times N_c\right)$$

(1)

The number of required synthetic samples for a class c (N_needed ^c), is computed as the difference between the majority class size N_max and the current class size N_c, the constant C is a hyperparameter to be tuned controlling the upper bound of generation per class. Based on our experiments, we set this number to 3 based on our evaluation but it can be tuned in the future.

3.3.3. Resampling Strategy Selection

This part describes the adaptive nature of the proposed algorithm in which data augmentation is carried out to balance the dataset, based on data characteristics such as class compactness and data density. Once the class is marked for resampling, further data measures are examined, namely class compactness and data density.

Class compactness is a measure of how distinct the samples of different classes are from one another, as overlap between classes is a persistent challenge in classification [34]. If data is not compact, BSMOTE will struggle to find a boundary that is the basis of data generation, leading to lower quality data generation. In contrast, generative networks do not consider class boundaries and rather learn from the general distribution of the data [28,29]. Class compactness, inspired by Fisher Linear Discriminant Analysis [35], is calculated according to Equation (2). Here, c refers to a specific class, d is the number of features in the class, and $X_j^{\left(c\right)}$ is the value of feature j in class c.

$$S^{\left(c\right)}={\displaystyle \frac{1}{d}}{\sum }_{j=1}^d\mathrm{Var}\left(X_j^{\left(c\right)}\right)$$

(2)

The second measure used by GAN-AHR is class density, which determines how close the samples of the same class are to one another, a measure that plays a crucial role in oversampling techniques [36]. It is calculated according to Equation (3), which averages the Euclidean distance between a data point and its nearest five neighbors.

$${\mathrm{Density}}^{\left(c\right)}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N\left({\displaystyle \frac{1}{k}}{\sum }_{j=1}^k|x_i-x_j|\right)$$

(3)

GAN-AHR calculates both measures, and based on empirical experiments, on the dataset, the density threshold was set to 0.3, while the compactness threshold was set to 0.4.

3.3.4. CGAN Generation and Sample Filtering

The CGAN architecture consists of two components the generator, which produces synthetic samples, and the discriminator, which verifies them. When data are qualified for CGAN generation, the generator takes two inputs a noise vector and a desired class label, where the latent dimension is set to 64. The generator’s architecture consists of several layers, namely, the fully connected, dropout, and dense layers. Additionally, the tanh activation function is adopted to ensure data normalization, [−1, 1].

The discriminator, which controls the quality of synthetic data, incorporates the class label as a condition for classifying generated samples as real or fake. It takes two inputs a generated sample and an associated class label. These inputs are then fed to a series of layers, namely, the dense and dropout layers. Then, the LeakyReLU activation function is employed, followed by a sigmoid function that produces a single number indicating the probability score. The hyperparameters settings of GAN-AHR are shown in

Table 1. CGAN’s hyperparameters.

Component	Setting
Label embedding size	Equal to latent dimension (64)
Activation functions	LeakyReLU (α = 0.2), tanh (output layer)
Batch size	32 samples
Epochs	150
Optimizer	Adam (learning rate = 0.0002)
Loss functions	Adversarial loss (binary cross-entropy) + Feature Matching Loss + Diversity Loss (trace of covariance)
Label smoothing	Real label = 0.9
Dropout rate	0.3
stopping criteria	Epoch size (150) per class
No. of generator’s layers	2 dense layers (256 and 128).
No. of discriminator’s layers	2 dense layers (256 and 128).
Dropout Rate	0.3

, whereas the architecture diagram is shown in Figure 3.

3.3.5. BSMOTE

BSMOTE is employed in GAN-AHR as a simple approach to data balancing when the class exhibits high compactness and high density. We implemented BSMOTE using the BorderlineSMOTE class from the imblearn library [37]. The number of required samples is determined by the class imbalance ratio in relation to the majority class.

3.4. Classification Model

RF [38] is employed as the classification model, as our empirical evaluation showed its superiority over other classification algorithms. The RF depends on several hyperparameters, such as the number of estimators, which was set to 300, and the minimum split size which was set to 5. The classifier was trained and tested using 10-fold cross-validation, which divides the data into ten subsets, nine for training and one for testing ensuring reliable and consistent prediction.

3.5. Experimental Setup

GAN-AHR was developed in Python version 3.1, using the Google Colab Pro+ platform. Several libraries were incorporated, such as scikit-learn, imbalanced-learn, TensorFlow, and Keras. For all experiments, L4 GPU was utilized with high RAM settings to facilitate faster processing.

To assess the feasibility of our proposed algorithm, we measured the training time, and GPUusage. The training time which is the time taken by the model to train per fold, was 678 s on average, whilst GPU memory usage was around 447 MB. Moreover, the theoretical complexity of the algorithm is polynomial in nature, indicating that there are no steps that require exponential computations. Steps such as GAN training, BSMOTE, and filtering scale in polynomial time in relation to the dataset size are conducted. These results suggest that our proposed algorithm is computationally efficient and feasible.

4. Results and Discussion

4.1. Dataset

The dataset used for evaluation is the NF-UNSW-NB15 dataset [39], which is derived from the original Netflow dataset. It includes ten classes: Benign, Exploits, Fuzzers, Reconnaissance, Generic, DoS, Analysis, Backdoor, Shellcode, and Worms. The class distribution is shown in Figure 4. This dataset is highly imbalanced, with the benign class constituting approximately 93% of the data. There are 12 features, as shown in

Table 2. NF-UNSW-NB15 dataset features.

Feature	Description	Significance
IPV4 SRC ADDR	IPv4 source address	Uniquely determines source.
IPV4 DST ADDR	IPv4 destination address	Uniquely determines destination.
L4 SRC PORT	IPv4 source port number	Uniquely determines source application.
L4 DST PORT	IPv4 destination port number	Uniquely determines destination application.
PROTOCOL	IP protocol identifier byte	Determines the protocol for network diagnosis.
TCP FLAGS	Cumulative of all TCP flags	Determines the state of the TCP connection.
L7 PROTO	Layer 7 protocol (numeric)	States the application layer protocol.
IN BYTES	Incoming number of bytes	For identifying the network traffic pattern.
OUT BYTES	Outgoing number of bytes	Determines the volume of data leaving the network, which is essential for finding anomalies.
IN PKTS	Incoming number of packets	Key in identifying anomalies, and to diagnose packets’ loss.
OUT PKTS	Outgoing number of packets	Useful in determining the packets’ behavior and highlighting heavy applications.
FLOW DURATION MILLISECONDS	Flow duration in milliseconds	Useful in finding abnormally long sessions (suspicious)

, and the correlation matrix of the features is shown in Figure 5.

4.2. Evaluation Metrics

Given that intrusion detection is mainly a classification problem, the most suitable metric would naturally be classification accuracy. Nonetheless, attack incidents are far fewer than benign ones, making it a highly imbalanced classification problem. In such cases, accuracy alone does not measure the true performance [33]. Thus, other more useful measures that precisely measure the detection capabilities were studied, namely, the F1-score, macro F1-score, precision, and recall. All four metrics are obtained from a matrix known as a ‘confusion matrix’, which includes four values: false positive (FP), false negative (FN), true positive (TP), and true negative (TN). Recall refers to the percentage of correctly classified positive examples compared with the examples in the actual class. It is calculated using the following equation.

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(4)

Precision refers to the percentage of correctly classified positive examples compared with the total predicted positive examples. It is calculated according to the following equation:

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(5)

F1-score covers both precision and recall; it is key in imbalanced classification as it assesses the correctly classified examples as well as the number of wrongly classified examples [39]. It is determined using the following equation:

$$F1=2\times {\displaystyle \frac{\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(6)

In multi-class classification problems, an additional measure called macro F1-score is essential. It averages the F1 score of all classes, regardless of the class size [40]. It is calculated according to the following equation, where N represents the total number of classes:

$$\mathrm{Macro}\text{-}F1={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{F}_{1_i}$$

(7)

4.3. Baseline Performance

For evaluation and to demonstrate that the proposed algorithm effectively enhances intrusion detection performance, we implemented three baseline approaches: Random Forest without any oversampling or use of the GAN (RF), Random Forest with BSMOTE oversampling (BSMOTE), and Random Forest with CGAN-based synthetic data generation (CGAN). Specifically, we compared the baseline algorithms with the proposed one in terms of all measures, focusing primarily on the F1-score.

As evident from the results in

Table 3. Performance comparison between the proposed algorithm vs. CGAN and BSMOTE approaches.

Method	Precision (Macro)	Recall (Macro)	F1 Score (Macro)	F1 Score (Weighted)
RF	0.66	0.62	0.61	0.97
BSMOTE	0.56	0.56	0.56	0.96
CGAN	0.57	0.59	0.61	0.97
Proposed (GAN-AHR)	0.67	0.66	0.65	0.97

, the proposed GAN-AHR algorithm outperforms the baseline approaches across all evaluation metrics. Notably, the macro F1-score which evaluates performance regardless of class size, increased by over four percentage points. This indicates that our algorithm improved the F1-score of minority classes, which are often challenging to detect. Similarly, in terms of both precision and recall, the proposed algorithm also outperforms the baselines, achieving scores of 0.67 and 0.66, respectively.

On the other hand, the F1 scores of all four approaches are very close, and this was highly predicted since the F1-score is highly influenced by the detection of the majority class. Thus, this measure alone is not sufficient in imbalanced classification problems. Collectively, the results presented here confirm that the hybridization of the CGAN and BSMOTE enhances prediction performance and yields higher evaluation metrics compared with each approach individually.

4.4. Threshold Sensitivity Analysis

The adaptive mechanism of the proposed approach (GAN-AHR) depends on the values of the thresholds, which are density and compactness, thus, in this section we empirically justify the thresholds used in the algorithm. A grid search over multiple values of density and compactness was conducted and macro F1-score was recorded for each combination of values, as shown in

Table 4. Macro F1-scores across different compactness and density thresholds.

Compactness/Density	Den. 0.3	Den. 0.4	Den. 0.5
Comp. 0.2	0.622	0.623	0.621
Comp. 0.3	0.623	0.622	0.621
Comp. 0.4	0.624	0.621	0.623
Comp. 0.5	0.623	0.624	0.623

.

As shown in Table 4, the macro F1-score is generally stable across many threshold combinations, however with a density of 0.3 and compactness of 0.4, the algorithm achieved the highest macro F1-score (0.624). The results illustrate that the algorithm’s performance is robust, remaining consistently high across a variety of threshold settings.

4.5. Class-Wise Performance Analysis

The performance of the proposed algorithm, as discussed in the previous section, demonstrates its effectiveness compared with the baseline approaches. However, a detailed investigation of the performance metrics at the class level is essential.

Table 5. Class-wise macro F1-score comparison of the proposed algorithm vs. the CGAN and BSMOTE approaches.

Class	CGAN	BSMOTE	Proposed
Analysis	0.13	0.13	0.08
Backdoor	0.16	0.15	0.13
Benign	0.99	0.99	0.99
DoS	0.41	0.32	0.51
Exploits	0.83	0.79	0.84
Fuzzers	0.55	0.56	0.60
Generic	0.72	0.62	0.71
Reconnaissance	0.81	0.81	0.81
Shellcode	0.88	0.85	0.90
Worms	0.67	0.48	0.90

summarizes the results in terms of F1 score for each class. The benign class, which is the majority in the dataset, consistently shows a stable and high performance regardless of the approach used. In contrast, minority classes show varying performance depending on the approach used. For example, the worm class, the smallest class of the dataset, shows a significant increase in F1 score of 0.90 compared with 0.67 with the CGAN and 0.48 with BSMOTE. Similarly, two other minority classes, namely Shellcode and DoS, reveal a considerable increase in F1-score (0.90 and 0.51, respectively). Likewise, the Exploits and Fuzzers classes were better detected by the proposed algorithm compared with the baseline approaches. The gain in F1 score across classes is illustrated in Figure 6, demonstrating an improved detection rate for many of the minority classes. Overall, class-wise analysis confirms that the proposed algorithm considerably increases the F1 score of most of the minority classes, whilst maintaining the high F1-score of the majority class.

4.6. Statistical Evaluation

Statistical evaluation plays a vital role in validating the performance of machine learning models and in conducting fair comparisons among different approaches [41]. We employed two important statistical measures: paired t-tests and Wilcoxon signed-rank tests to compare the proposed GAN-AHR against both BSMOTE and the CGAN using 5-fold cross-validation as shown in

Table 6. Statistical evaluation of the proposed algorithm vs. BSMOTE and the CGAN.

Method	Mean Macro F1	95% Confidence Interval	t-Test p-Value vs. GAN-AHR	Wilcoxon p-Value vs. GAN-AHR
BSMOTE	0.564	[0.557, 0.571]	0.00021	0.0625
CGAN	0.618	[0.608, 0.628]	0.016	0.0625
Proposed (GAN-AHR)	0.630	[0.621, 0.639]	–	–

. The Wilcoxon signed-rank test is a particularly important measure since it is a safe measure, one that does not assume normal distribution of data [42]. As shown in the table, the proposed GAN-AHR achieves the highest average macro F1-score (0.630) with a 95% confidence interval of [0.621, 0.639], demonstrating its consistent and high performance. The reported t-test p-value confirms a statistically significant difference between GAN-AHR and both BSMOTE (p = 0.00021) and the CGAN (p = 0.016), confirming that the proposed algorithm consistently outperforms baseline algorithms across folds. In terms of the Wilcoxon signed-rank test, both baseline approaches resulted in a value of 0.0625, which does not show a statistically significant difference at α = 0.05.

4.7. Comparison with Existing Approaches

This section describes the experiments conducted to validate the proposed algorithm against several approaches that utilize the same dataset [6]. Four relevant approaches were identified from the available literature, and since the macro F1-score is not always reported, the comparison solely focuses on the weighted values of precision, recall, and F1-score.

Table 7. Comparisons with previous studies.

Method	Precision	Recall	F1 Score
Rahman et al. [43]	91%	90%	89%
Shaker et al. [44]	-	-	95%
Li et al. [45]	-	-	89%
Ajagbe et al. [46]	90.5%	90.3%	90.4%
Gao et al. [47]	-	94.7%	94.5%
Proposed	98%	97%	97%

summarizes the comparisons, and as it is clearly seen, the proposed GAN-AHR outperforms other approaches in all three metrics: precision, recall, and F1-score.

5. Limitations and Future Work

While the proposed algorithm was validated using the NF-UNSW-NB15 dataset which is known for its reliability, and diverse and multi-class characteristics, further validation on other datasets such as CICIDS2017 would prove the generalizability of our approach.

Moving forward, there are several avenues for future research. First, it is necessary to employ automatic hyperparameter tuning methods, such as optimization algorithms, since hyperparameters significantly impact detection performance. Second, exploring other GAN variants and measuring their performance would indeed advance our understanding of GAN applicability within the IDS field. Third, federated learning (FL) approaches should be explored as they add an extra security layer, and the applicability of the proposed approach should be investigated in the federated setting. Fourth, explainable AI should be incorporated such that network managers can understand the rationale behind the proposed approach. Moreover, overlap measures such as interclass distance or Disagreeing Neighbours could enhance the adaptive mechanism of the proposed algorithm.

6. Conclusions

The research area of intrusion detection is fast-growing and involves diverse mechanisms. However, several challenges hinder the effectiveness of these mechanisms, such as data imbalance, where attack samples are far fewer than benign samples, leading to biased models. By harnessing the potential of a generative network coupled with BSMOTE, we proposed GAN-AHR, a novel, adaptive, hybrid approach for data balancing that dynamically chooses between the CGAN and BSMOTE depending on data characteristics. Two characteristics were considered: data compactness and density. Then, an RF was employed for classification on the augmented data. The experimental results on the NF-UNSW-NB15 dataset suggest that the proposed approach improves classification performance, resulting in a higher precision, recall, and F1-score compared with other existing approaches.