A Hybrid Intrusion Detection Framework Using Deep Autoencoder and Machine Learning Models

Abstract

This study provides a detailed comparative analysis of a three-hybrid intrusion detection method aimed at strengthening network security through precise and adaptive threat identification. The proposed framework integrates an Autoencoder-Gaussian Mixture Model (AE-GMM) with two supervised learning techniques, XGBoost and Logistic Regression, combining deep feature extraction with interpretability and stable generalization. Although the downstream classifiers are trained in a supervised manner, the hybrid intrusion detection nature of the framework is preserved through unsupervised representation learning and probabilistic modeling in the AE-GMM stage. Two benchmark datasets were used for evaluation: NSL-KDD, representing traditional network behavior, and UNSW-NB15, reflecting modern and diverse traffic patterns. A consistent preprocessing pipeline was applied, including normalization, feature selection, and dimensionality reduction, to ensure fair comparison and efficient training. The experimental findings show that hybridizing deep learning with gradient-boosted and linear classifiers markedly enhances detection performance and resilience. The AE–GMM-XGBoost model achieved superior outcomes, reaching an F1-score above 0.94 ± 0.0021 and an AUC greater than 0.97 on both datasets, demonstrating high accuracy in distinguishing legitimate and malicious traffic. AE-GMM-Logistic Regression also achieved strong and balanced performance, recording an F1-score exceeding 0.91 ± 0.0020 with stable generalization across test conditions. Conversely, the standalone AE-GMM effectively captured deep latent patterns but exhibited lower recall, indicating limited sensitivity to subtle or emerging attacks. These results collectively confirm that integrating autoencoder-based representation learning with advanced supervised models significantly improves intrusion detection in complex network settings. The proposed framework therefore provides a solid and extensible basis for future research in explainable and federated intrusion detection, supporting the development of adaptive and proactive cybersecurity defenses.

1. Introduction

Intrusion detection has become a core requirement in modern network security as contemporary infrastructures generate massive, heterogeneous traffic and face increasingly adaptive attack strategies that often bypass traditional signature-based defenses [1]. Traditional rule and signature-based intrusion detection systems are inherently limited because they rely on predefined patterns and static rules that cannot adapt to previously unseen or evolving attack behaviors. In practice, adversaries frequently employ techniques such as payload obfuscation, protocol mimicry, polymorphism, and low-and-slow attack strategies, which significantly reduce the effectiveness of fixed signatures.

Furthermore, encrypted traffic and the growing heterogeneity of modern network environments obscure discriminative features, while high-dimensional traffic representations increase feature-space overlap between normal and malicious behavior, leading to higher false negatives and delayed detection. Recent empirical and survey-based studies emphasize that rule-driven IDS frameworks are frequently brittle under evolving threat patterns, which has accelerated the shift toward data-driven approaches capable of learning from traffic behavior rather than relying on static signatures [2]. This need is particularly evident in large-scale and high-variability environments, such as telecom and IoT ecosystems, where normal traffic exhibits substantial variability and attack traces may appear subtle or short-lived [3]. Consequently, intrusion-oriented IDS paradigms have gained prominence because they aim to detect deviations from learned normal behavior and offer a pathway to identifying previously unseen intrusions in operational settings [1,2].

Autoencoders have been widely adopted in intrusion detection due to their ability to learn compact latent representations that capture the underlying structure of normal network traffic without requiring labeled attack data [4]. By minimizing reconstruction error, AE-based models can effectively model regular traffic patterns and flag deviations as potential anomalies, which makes them attractive for security scenarios where labeled data are scarce or incomplete. However, recent analytical studies highlight that standalone autoencoder-based detectors often exhibit limited sensitivity to subtle or evolving attacks, particularly in highly imbalanced intrusion datasets where anomalous samples may partially overlap with normal behavior in the feature space [4]. This limitation is further exacerbated in real-world and streaming environments, where traffic distributions evolve over time and reconstruction-based thresholds may become overly conservative, leading to reduced recall [5]. These findings suggest that while autoencoders provide a strong foundation for representation learning, their effectiveness as isolated intrusion detectors remains constrained in complex and dynamic network settings.

Building upon the limitations of purely unsupervised detectors, recent intrusion detection research increasingly favors hybrid frameworks that integrate deep representation learning with supervised or semi-supervised classifiers to enhance detection robustness and decision boundaries [6,7,8,9]. Hybrid IDS architectures leverage deep models to extract informative latent features from high-dimensional traffic data, while downstream classifiers exploit labeled information to improve class separability and reduce false negatives. Empirical evaluations in IoT and industrial network contexts demonstrate that such hybrid designs consistently outperform single-model approaches in both binary and multi-class intrusion detection scenarios, particularly under complex and noisy traffic conditions [6,7]. Moreover, staged or layered hybrid detection strategies have been shown to improve generalization across heterogeneous attack types by combining intrusion awareness with discriminative learning [8,9].

Recent IDS studies have demonstrated that ensemble classifiers combined with statistical feature selection can achieve strong detection performance on benchmark datasets. Nevertheless, these approaches rely on direct learning from raw preprocessed traffic features and do not model latent representations of network behavior. This motivates the integration of anomaly-aware latent representation learning prior to supervised classification to enhance detection robustness [10].

These findings collectively motivate the adoption of hybrid intrusion detection frameworks as a practical compromise between representation power, detection accuracy, and operational reliability in modern IDS deployments.

In practical deployment scenarios, intrusion detection systems must operate on complex data streams that include multivariate time series and large-scale system logs, where temporal dependencies and cross-feature correlations play a critical role in distinguishing normal from malicious behavior [11,12]. Recent studies on log-based and time series intrusion detection emphasize that attack patterns are often context-dependent and may manifest as subtle temporal inconsistencies rather than abrupt deviations in individual features [11]. This complexity poses additional challenges for traditional IDS models, as static feature representations or single-shot classification may fail to capture evolving attack dynamics. Survey-based analyses further highlight that effective IDS solutions must balance expressive feature modeling with robustness to noise, concept drift, and incomplete labeling, particularly in continuously generated traffic streams [12]. These observations reinforce the need for detection frameworks that can exploit learned latent representations while remaining adaptable to temporal variability and real-world operational constraints.

Robust evaluation remains a critical requirement for validating intrusion detection frameworks under realistic and diverse threat conditions. Benchmark datasets such as NSL-KDD and UNSW-NB15 continue to be widely employed because they provide complementary views of legacy and modern network attack behaviors, enabling systematic comparison with prior studies while assessing generalization across heterogeneous traffic patterns [13,14]. Recent IDS research also explores advanced paradigms, including software-defined networking and language-model-based detectors, to address unseen or rapidly evolving attacks; however, such approaches often introduce substantial computational and architectural complexity that may limit their practical deployment in resource-constrained environments [15]. These considerations underscore the importance of designing intrusion detection frameworks that balance detection accuracy, robustness, and operational feasibility. In this context, the present study positions itself within the broader IDS literature by systematically evaluating a hybrid intrusion detection framework on established benchmark datasets, aiming to provide a pragmatic and reproducible solution that bridges unsupervised representation learning and supervised decision-making without incurring excessive model complexity.

1.1. Motivation

The increasing complexity and volume of network traffic have rendered traditional signature-based intrusion detection systems inadequate for detecting evolving and previously unseen attacks. A hybrid intrusion detection offers a promising alternative by modeling normal network behavior; however, purely unsupervised approaches often suffer from limited discriminative capability when benign and malicious traffic patterns overlap. Autoencoders have demonstrated effectiveness in learning compact latent representations from high-dimensional traffic data, enabling meaningful feature extraction without manual engineering. Nevertheless, reconstruction-based detection alone is frequently insufficient for robust decision-making in realistic intrusion scenarios. This motivates the development of hybrid frameworks that combine deep representation learning with probabilistic modeling and supervised classifiers, aiming to enhance detection accuracy while maintaining practical deployability and interpretability.

1.2. Contribution

This study proposes a hybrid intrusion detection framework that separates intrusion detection representation learning from supervised decision-making. An Autoencoder-Gaussian Mixture Model is employed to learn latent and probabilistic characteristics of normal traffic, while XGBoost and Logistic Regression are applied as downstream classifiers using the same intrusion-informed feature space. This design enables a controlled comparison between unsupervised, ensemble-based, and interpretable linear detection strategies under identical preprocessing conditions.

Unlike many existing hybrid IDS studies that primarily focus on performance improvement through model stacking, this work emphasizes methodological consistency by evaluating all models on the same latent feature space and explicitly analyzing the trade-off between detection accuracy and interpretability.

The experimental results on NSL-KDD and UNSW-NB15 demonstrate that supervised models built on intrusion-driven representations significantly improve detection performance compared to a standalone AE-GMM. The framework therefore provides reproducible and practically relevant insight into hybrid intrusion detection design.

1.3. Paper Layout

The remainder of this paper is organized as follows: Section 2 reviews the related work. Section 3 details the methodology adopted in this study. Section 4 describes the model architecture and mathematical formulation. Section 5 presents the experimental results and discussion. Section 6 concludes this paper and outlines directions for future research. Appendix A includes implementation details and hyperparameter configuration. Reference section lists the references cited throughout the manuscript.

2. Related Work

Network security research increasingly relies on intrusion detection techniques to identify suspicious activities that deviate from normal traffic behavior. In response to the growing complexity and diversity of cyber threats, a wide range of machine learning and statistical approaches have been explored to enhance the detection of both known and previously unseen attacks.

Autoencoder-based intrusion detection has been extensively investigated as an effective unsupervised strategy for modeling normal network behavior and identifying deviations indicative of malicious activity. A representative study by Hou et al. [16] proposed an intrusion detection framework for IoT environments that integrates autoencoder-based dimensionality reduction with Bayesian Gaussian Mixture Models to capture the probabilistic structure of latent representations. Their results demonstrated that AE-GMM architectures are particularly well suited to heterogeneous and resource-constrained environments, where labeled attack data are scarce and traffic patterns exhibit high variability. Complementary studies on anomaly detection in streaming cybersecurity logs further confirm that autoencoder-driven latent representations provide a compact and informative basis for probabilistic density estimation, while also revealing limitations in recall when such models are used in isolation [5,11]. Collectively, these findings establish AE-GMM as a robust foundation for intrusion detection, while motivating the integration of additional discriminative mechanisms to improve sensitivity to subtle and emerging attacks.

Beyond standard autoencoders, a substantial body of research has explored variational autoencoder-based frameworks integrated with Gaussian Mixture Models to enhance generative modeling and uncertainty estimation in anomaly detection. Khan et al. [17] introduced the DVAEGMM model, which leverages dual variational autoencoders combined with GMM clustering to improve scalability on attributed networks, while Liao et al. [18] proposed a unified GMVAE architecture tailored for high-dimensional outlier detection. Related extensions have been applied across diverse domains, including maritime trajectory anomaly detection [19] and hybrid probabilistic modeling with Gaussian processes [20], demonstrating strong representational capacity for complex data distributions. Despite their effectiveness, these VAE-based approaches often incur increased training complexity, sensitivity to hyperparameter selection, and higher computational overhead, which can limit their practical deployment in real-world intrusion detection systems. Consequently, recent research has emphasized the need for simpler yet effective alternatives that retain strong representation learning capabilities while enabling efficient integration with downstream classifiers.

Recent intrusion detection studies have increasingly shifted toward hybrid and ensemble-based learning strategies that combine representation learning with supervised classification to enhance detection accuracy, robustness, and interpretability. Gradient-boosting techniques, particularly XGBoost, have demonstrated strong performance in handling nonlinear feature interactions and class imbalance in large-scale intrusion datasets, as evidenced by TL-XGBoost and related frameworks [21]. Parallel efforts have incorporated interpretable rule induction, cost-aware learning, and imbalance-sensitive classification to improve transparency and deployment feasibility in IoT and IoMT environments [22,23,24]. Moreover, recent hybrid deep learning architectures, including CNN-RNN and BiLSTM-CNN models, further confirm that combining deep feature extraction with discriminative decision layers yields superior generalization across both binary and multi-class intrusion scenarios [25,26,27,28,29]. These findings collectively indicate a clear research trend toward hybrid IDS designs that balance anomaly-awareness, supervised discrimination, and interpretability.

In parallel with hybrid intrusion detection architectures, recent research has explored learning frameworks designed to operate over distributed and heterogeneous data sources while explicitly addressing robustness and privacy constraints. Wu et al. [30] proposed an adaptive federated learning scheme with differential privacy preservation that models complex relationships across multi-source data through adaptive optimization and error-aware filtering mechanisms. Their study highlights how long-term correlations and cumulative error propagation can degrade model stability if not properly managed, particularly in systems integrating data from multiple viewpoints or distributed environments. Although their work primarily targets privacy-preserving federated learning in biomedical and distributed systems, the underlying insights regarding robustness under long-term correlation, stability correction, and adaptive learning are directly relevant to intrusion detection systems deployed across large-scale and heterogeneous network infrastructures, where traffic exhibits temporal dependency and evolving behavior patterns.

However, many existing hybrid intrusion detection studies primarily focus on binary classification, while comprehensive evaluations across multi-class attack scenarios remain relatively limited.

Overall, prior research reflects a gradual transition from purely generative anomaly detectors toward hybrid and interpretable intrusion detection systems that integrate probabilistic modeling with supervised learning. While VAE-based frameworks have demonstrated strong representational power, practical IDS deployments increasingly favor architectures that balance performance, efficiency, and transparency. Motivated by these insights, the present study systematically investigates an AE-GMM framework and its hybridization with XGBoost and Logistic Regression to address key limitations identified in previous work.

3. Methodology

Figure 1 illustrates the workflow of the proposed hybrid intrusion detection framework. The process begins with data exploration, cleaning, and feature engineering to eliminate missing, duplicated, and highly correlated features. The models are AE-GMM, AE-GMM-XGBoost, as a supervised classifier, and AE-GMM-Logistic Regression as an interpretable benchmark. Model performance is subsequently evaluated using standard metrics, and the optimized version is stored for future deployment in real-network environments.

Figure 1. The workflow of the proposed intrusion detection model.

3.1. Dataset

In the current study, we use two publicly available datasets, NSL-KDD [14] and UNSW-NB15 [13], to evaluate the proposed hybrid framework. The data undergo cleaning, feature engineering, and preprocessing steps prior to model training.

3.1.1. UNSW-NB15 Dataset

The UNSW-NB15 dataset was created at the Australian Centre for Cyber Security in 2015 to provide a realistic environment for studying modern cyberattacks. Unlike the older KDD datasets, which no longer reflect current network behaviors, UNSW-NB15 captures both normal and malicious traffic in a controlled setting. It contains data generated from real-network activities mixed with simulated attacks, making it a valuable resource for testing intrusion detection algorithms. In this work, we use the cleaned version that includes 175,341 samples for training and 82,332 for testing. The dataset originally contains 47 features covering network flow, content, and timing information. Each record is labeled as either normal or one of ten attack types, such as DoS, exploits, or reconnaissance. This diversity and structure make UNSW-NB15 a practical and modern benchmark for intrusion detection studies [13].

3.1.2. NSL-KDD Dataset

The NSL-KDD dataset was introduced as an improved version of the classic KDD’99 benchmark to reduce redundancy and bias in training. It remains widely used for evaluating intrusion detection models because of its clean and well-balanced structure. NSL-KDD includes 125,973 training records and 22,544 testing records, originally described by 41 numerical, categorical, and binary features. The dataset groups attacks into four major types—Denial of Service (DoS), Probe, Remote to Local (R2L), and User to Root (U2R)—along with normal traffic. These categories reflect different patterns of malicious behavior that challenge detection models in different ways. Due to its manageable size and balanced design, NSL-KDD continues to serve as a key reference for testing both traditional and deep learning-based approaches [14].

Table 1 presents the class distribution and the number of samples used for training and testing in the UNSW-NB15 and NSL-KDD datasets.

Table 1. Datasets.

Dataset	Class	Train-Set	Test-Set
UNSW-NB15	normal	56,000	37,000
	generic	40,000	18,871
	exploits	33,393	11,132
	fuzzers	18,184	6062
	dos	12,264	4089
	reconnaissance	10,491	3496
	analysis	2000	677
	backdoor	1746	583
	shellcode	1133	378
	worms	130	44
	Total	175,341	82,332
NSL-KDD	normal	67,343	9711
	dos	45,927	7458
	probe	11,656	2421
	R2L	995	2754
	U2R	52	200
	Total	125,973	22,544

To avoid ambiguity regarding dataset characteristics, it should be noted that all sample counts and feature dimensions reported in this section refer to the final processed datasets used in the experiments. While the original versions of NSL-KDD and UNSW-NB15 contain a larger number of raw attributes, categorical encoding, normalization, and low-variance feature filtering were applied prior to model training. As a result, a consistent feature space was maintained across all experiments and models. The same processed datasets and feature sets were used throughout training and evaluation to ensure consistency and comparability of the reported results.

Table 2 summarizes the key characteristics of the NSL-KDD and UNSW-NB15 datasets before and after preprocessing. The reported sample sizes and feature descriptions correspond to the final processed datasets used in all experiments.

Table 2. Dataset characteristics before and after preprocessing.

Dataset	Data Stage	Samples (Train/Test)	Feature Description
UNSW-NB15	Raw dataset	175,341/82,332	Original feature set including numerical and categorical attributes
UNSW-NB15	After preprocessing	175,341/82,332	Final processed feature set obtained after encoding, normalization, and low-variance filtering
NSL-KDD	Raw dataset	125,973/22,544	Original feature set consisting of numerical, categorical, and binary attributes
NSL-KDD	After preprocessing	125,973/22,544	Final processed feature set consistently used across all models

As part of this analysis, we focused initially on the protocol type feature, which categorizes the type of protocol used in each network connection. As illustrated in Figure 2, this feature contains only three distinct values representing the most common internet protocols. Table 3 presents a quantitative summary of the distribution of these protocol types. These datasets were used consistently across all three models (AE-GMM, AE-GMM -XGBoost, and AE-GMM-Logistic Regression) to ensure fair and reproducible evaluation.

Figure 2. Distribution of network traffic according to protocol categories (TCP, UDP, and ICMP).

Table 3. Summary of the datasets used in this study.

Dataset	Source	Shape	Features	Target Variable
UNSW-NB15	Moustafa, N. et al. [13]	Training: 175,341 records; testing: 82,332 records; 45 features	Duration, protocol, service, state, source packets, destination packets, source bytes, etc.	Label: normal (0) or malicious (1)
NSL-KDD	Tavallaee, M. et al. [14]	Training: 125,973 records; testing: 22,544 records; 42 features	Duration, protocol type, service, flag, source bytes, destination bytes, etc.	Class: normal or attack

Such visualization aids in identifying imbalances and patterns in the dataset, which is essential for designing an effective intrusion detection model.

It should be noted that the feature counts reported across different tables correspond to distinct representation stages. The original datasets include a higher number of raw attributes, while the feature counts reported in Table 3 reflect the effective input representations after categorical encoding and preprocessing, prior to latent space transformation by the autoencoder. This distinction explains the minor differences in reported feature numbers and does not compromise the integrity, consistency, or validity of the experimental pipeline.

Table 4 summarizes the frequency distribution of network service types contained in the NSL-KDD dataset. HTTP and private services dominate the dataset, representing 34.83% and 21.17% of the total traffic, respectively. This reflects the prominence of web-based and internal communication activities in real-world network environments. Services such as Telnet, POP3, SMTP, and FTP are also present, capturing remote access and email-related transactions that are commonly exploited in intrusion scenarios.

Table 4. Distribution of network service types in the NSL-KDD dataset.

Index	Value	Explanation	Percent
1	http	HTTP, short for Hypertext Transfer Protocol, is a communication protocol specifically designed for sending data over the World-Wide Web.	34.83
2	private	The “(private)” service refers to network connections that are linked to private or internal services. It helps in identifying and analyzing traffic that is associated with internal network operations or proprietary services inside the dataset.	21.17
3	telnet	Telnet is a networking protocol that facilitates interactive communication with remote computers. It enables users to remotely access and manage devices through a command-line interface.	7.21
4	pop_3	POP3, also known as Post Office Protocol version 3, is a communication protocol utilized by email clients to retrieve emails from a mail server. Its primary function is to enable users to download messages to their local device, facilitating offline access,	4.52
5	smtp	SMTP, also known as Simple Mail Transfer Protocol, is a communication protocol that operates over TCP/IP and is specifically designed for the purpose of sending and receiving email messages. SMTP is predominantly utilized by popular email applications such as Gmail, Outlook, Apple Mail, and Yahoo Mail.	4.14
6	domain_u	In the NSL-KDD dataset, “(domain_u)” service refers to network connections attributed to domain-related activities, assisting in the categorization and examination of domain-specific traffic patterns within the dataset.	3.96
7	ftp_data	Information transmitted during File Transfer Protocol (FTP) sessions. The data in consideration contains many types of information, such as files, directories, or other content, that is being sent between a client and an FTP server.	3.77
8	other	The “(other)” service represents network connections that do not fall into predefined service categories, indicating miscellaneous or unclassified network activity.	3.71
9	ecr_i	“eco_i” refers to the “ECO NetVolume I/O” service. It is one of the services identified in network traffic data and is associated with communication patterns related to network volume input and output within an ECO (Electronic Commerce) context.	3.33
10	ftp	“ftp” stands for File Transfer Protocol, which is a network protocol that facilitates the transfer of files between a client and a server. It is used to analyze network traffic patterns and identify potential security concerns.	3.06

The variety of service types enhances the dataset’s representativeness and allows for comprehensive evaluation of detection models across diverse communication channels. The high proportion of HTTP and private services provides rich data for learning normal traffic behaviors, while the inclusion of low-frequency services, such as ECO_I and FTP, introduces challenges in identifying rare attack patterns, thereby testing the adaptability and sensitivity of the intrusion detection models.

Table 5 presents the percentage distribution of network protocol types in the UNSW-NB15 dataset. As observed, TCP and UDP dominate the dataset with 52.34% and 35.73%, respectively, representing the most common transport protocols used in real-world internet traffic. Protocols such as ARP, OSPF, and SCTP appear in smaller proportions, corresponding to routing, address resolution, and specialized data exchange functions. The presence of multiple protocol types contributes to the dataset’s heterogeneity and enhances its suitability for evaluating intrusion detection models.

Table 5. Distribution of network protocol types in the UNSW-NB15 dataset.

Index	Value	Explanation	Percent
1	tcp	TCP, or Transmission Control Protocol, is a highly used communication protocol that guarantees dependable and sequential transmission of data packets over networks.	52.34
2	udp	UDP, or User Datagram Protocol, is a communication protocol that allows for the transmission of data packets without the reliability assurances provided by TCP. It is widely used for real-time applications such as multimedia streaming and online gaming.	35.73
3	unas	UNAS, created by TRW and Rational, is a collection of already constructed Ada components and services to construct distributed and diverse software systems. The system offers essential administrative operations such as initialization, fault detection, and inter-process communication, with a strong focus on being highly portable.	4.269
4	arp	The Address Resolution Protocol (ARP) is used to map IP addresses to MAC addresses within a local network.	1.198
5	ospf	OSPF, also known as Open Shortest Path First, is a routing protocol that utilizes the shortest path algorithm to determine the most efficient route for data packets within an IP network.	0.821
6	sctp	SCTP, also known as Stream Control Transmission Protocol, is a transport layer protocol designed to provide dependable and message-oriented communication between hosts within a network.	0.393
7	any	The term “ANY” is commonly used as a placeholder for any protocol or service. It allows for flexible analysis of network data without the need to identify a particular protocol or service.	0.116
8	gre	GRE, short for Generic Routing Encapsulation, is a protocol that is commonly employed to encapsulate packets. Its primary purpose is to establish virtual private networks (VPNs) or tunneling protocols.	0.106

This distribution also influences model performance, as the high concentration of TCP and UDP traffic provides abundant patterns for training, while the limited samples from rare protocols such as GRE or UNAS test the model’s ability to detect less frequent but potentially critical anomalies.

3.2. Data Preprocessing

The data preprocessing stage represents a crucial foundation for achieving reliable and reproducible results across the three evaluated models: the Autoencoder-Gaussian Mixture Model (AE-GMM), AE-GMM-XGBoost, and AE-GMM-Logistic Regression. A unified preprocessing pipeline was designed and implemented to ensure consistency and comparability between experiments. The procedure began with data cleaning, where duplicate and incomplete records were removed, and noisy samples were filtered out to enhance dataset integrity and prevent bias during training [31].

Noise definition and filtering criteria: In this study, noisy samples refer to records that may negatively affect model training due to data-quality issues rather than representative network behavior. Specifically, the preprocessing stage targeted duplicated entries, samples containing missing or invalid values (such as undefined or non-numeric fields after encoding), and records exhibiting clear inconsistencies identified during data integrity checks. These filtering operations were applied uniformly to both the NSL-KDD and UNSW-NB15 datasets prior to model training. Importantly, the noise removal process focused on eliminating data artifacts while preserving the original traffic characteristics and class distribution. All subsequent experiments were conducted using the cleaned datasets, and the final sample counts reported in Table 1 and Table 2 correspond to the datasets after preprocessing.

Redundant and highly correlated features were examined through a correlation matrix, and those exceeding the predefined threshold were eliminated to avoid multicollinearity, which can degrade model interpretability and convergence, especially in AE-GMM-Logistic Regression.

Next, categorical features such as protocol_type, service, and flag were transformed into numerical representations using a combination of label encoding and one-hot encoding techniques. This conversion ensured full compatibility with both tree-based models (AE-GMM-XGBoost) and density-based learning architectures (AE-GMM) [32]. For numerical attributes, Min-Max normalization was applied to scale the feature values to a range between 0 and 1. This standardization step stabilized gradient optimization in AE-GMM and balanced the influence of large-magnitude features, which is particularly critical in linear models like AE-GMM-Logistic Regression [33].

Following normalization, feature selection was performed using a correlation-based filtering approach to retain the most informative and discriminative attributes for intrusion detection.

All preprocessing operations were executed using Python (version 3.13.5, Anaconda distribution), with Pandas (version 2.2.3), NumPy (version 2.3.3), and Scikit-learn (version 1.6.1), ensuring reproducibility and transparency of the experimental pipeline. This unified pipeline provided a clean, standardized, and scalable data foundation that enabled the fair comparison of the three models. Recent studies have emphasized that rigorous preprocessing, especially normalization, encoding, and feature reduction, significantly enhances model generalization and interpretability in network intrusion detection frameworks [31,32,33].

3.3. Feature Selection and Integration Process

The feature selection and integration process played a crucial role in ensuring the reliability and consistency of the three models: Autoencoder-Gaussian Mixture Model (AE-GMM), AE-GMM-XGBoost, and AE-GMM-Logistic Regression. Since the datasets (NSL-KDD and UNSW-NB15) contain a mix of categorical and numerical attributes, an initial normalization step was performed using the Min-Max scaling technique to transform each feature into a standardized range between 0 and 1, as shown in Equation (1).

$$x_{scaled}={\displaystyle \frac{x-x_{min}}{x_{max}-x_{min}}}$$

(1)

This normalization ensured that all features contributed equally during model training and prevented bias toward attributes with larger numerical ranges. The scaling parameters were computed exclusively on the training data and subsequently applied to the test set to prevent information leakage.

Following the normalization step, redundant and low-variance attributes were removed to reduce noise and improve computational efficiency.

For the AE-GMM, all preprocessed features were passed through the encoder to learn compact latent representations that preserved the most discriminative characteristics of the traffic data. This stage helped the GMM component to better distinguish between normal and anomalous instances by modeling latent-space clusters with probabilistic density estimation.

To ensure a controlled comparison across models, feature filtering was performed using model-agnostic criteria, including low-variance removal and correlation-based filtering. The same processed feature set was used as input for both XGBoost and Logistic Regression classifiers. XGBoost feature importance scores were computed solely for interpretability purposes and were not used to define the feature subset for the Logistic Regression model.

In the AE-GMM-XGBoost model, the feature importance scores were computed using the gradient-based gain metric, as expressed in Equation (2) [34]

$$FI\left(f_i\right)={\displaystyle \frac{1}{T}}\sum _{t=1}^T\sum _{n\in N_t\left(f_i\right)}Gain\left(n\right)$$

(2)

where T denotes the total number of trees in the ensemble, $N_t$($f_i$) represents the set of nodes in tree t where feature $f_i$ is used for splitting, and Gain (n) denotes the reduction in the objective function achieved by split node n.

Features with higher gain values were prioritized, allowing the model to automatically identify the most relevant predictors of network intrusions. This embedded selection process significantly enhanced interpretability and reduced dimensional complexity, as supported by recent works emphasizing multi-objective feature optimization for intrusion detection systems [35].

For the AE-GMM-Logistic Regression model, the integration process involved using the same processed feature set obtained from the model-agnostic filtering stage to ensure comparability across models. The model’s coefficients served as additional interpretability indicators, quantifying the influence of each feature on classification outcomes. This consistent feature integration pipeline ensured fair evaluation and facilitated the comparison of deep learning, ensemble, and linear techniques under identical preprocessing conditions [36].

Collectively, the combination of normalization, dimensionality reduction, and embedded importance estimation provided a balanced framework that enhanced model accuracy and robustness across both datasets, establishing a unified basis for the comparative analysis.

4. Model Architecture and Mathematical Formulation

The architecture of the proposed intrusion detection framework integrates three learning paradigms, Autoencoder-Gaussian Mixture Model (AE-GMM), AE-GMM-XGBoost, and AE-GMM-Logistic Regression, to balance deep representation learning, ensemble inference, and statistical interpretability.

In the AE-GMM, the autoencoder learns to reconstruct network traffic samples $x_i$ by minimizing the mean squared error between the input and its reconstruction ${\widehat{x}}_i$. In Equation (3), showing the loss function, ref. [37] is expressed as

$$MSE={\displaystyle \frac{1}{n_m}}\sum _{i=1}^{n_m}{(x_i-{\widehat{x}}_i)}^2$$

(3)

where $n_m$ denotes the number of training samples.

This stage ensures that the latent representations $z_i$ encode the underlying statistical regularities of normal traffic.

Subsequently, the latent embeddings are modeled with a Gaussian Mixture Model (GMM) to estimate the data density. According to Equation (4), ref. [38], the likelihood of a sample is

$$p\left(z\right)=\sum _{k=1}^K{\pi }_{k}N\left(z|{\mu }_k,{\wedge }_k^{-1}\right)$$

(4)

where $K$ denotes the number of Gaussian components, ${\pi }_k$ are the mixing coefficients satisfying $\sum _{k=1}^K{\pi }_k=1$, and $N(\cdot )$ is the multivariate normal density with the mean vector ${\mu }_k$ and precision matrix ${\wedge }_k$ (thus $\sum _k{=\wedge }_k^{-1}$ is the covariance matrix of component $k$). Samples associated with low likelihood (i.e., high negative log-likelihood) are flagged as anomalies.

For the AE-GMM-XGBoost classifier, latent or engineered features are fed into an ensemble of boosted trees that optimize a regularized objective function adapted from [34], as shown in Equation (5)

$$Objective=\sum _{i=1}^{n}L\left(y_i,{\widehat{y}}_i\right)+\sum _{t=1}^T\Omega \left(f_t\right)$$

(5)

where $y_i$ denotes the ground-truth label of instance i, ${\widehat{y}}_i$ is the predicted output, T is the total number of trees in the ensemble, and each $f_t$ represents t-th regression tree.

L is the differentiable loss (e.g., logistic loss), and the regularization term Ω

$$\Omega \left(\mathrm{f}\right)=\mathsf{\gamma }J+{\displaystyle \frac{1}{2}}\lambda \sum _{j=1}^J{w}_j^2$$

(6)

where J denotes the number of leaf nodes in a tree, $w_j$ is the weight of leaf j, and γ and λ are regularization parameters controlling model complexity.

This formulation prevents overfitting while enabling AE-GMM-XGBoost to model nonlinear interactions efficiently and consistent with the training process in the proposed model, which achieved high precision and AUC on both NSL-KDD and UNSW-NB15 datasets.

Finally, the AE-GMM-Logistic Regression layer acts as a lightweight linear classifier that estimates the probability of an instance being anomalous or normal. Based on [24], Equation (7) shows the predictive probability:

$$p\left(\gamma =1|X=x\right)={\displaystyle \frac{1}{1+e^{-({\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\dots +{\beta }_p{x}_p)}}}$$

(7)

This formulation provides an interpretable probabilistic mapping from input features to class membership, facilitating transparent decision boundaries in security auditing contexts.

Together, these formulations define a cohesive pipeline where the AE–GMM captures complex distributions in network traffic, AE-GMM-XGBoost refines detection through supervised learning, and AE-GMM-Logistic Regression provides explainability and benchmarking. This layered approach reflects the actual implementation of the system and adheres to established methodologies in recent research.

-

Implementation Details and Hyperparameter Settings

The autoencoder architecture employed in this study follows a fully connected symmetric design tailored for network traffic representation learning. The encoder consists of successive dense layers that progressively reduce the input feature space to a low-dimensional latent representation, while the decoder mirrors this structure to reconstruct the original input. Rectified Linear Unit (ReLU) activations are used in the hidden layers to introduce nonlinearity, and the model is trained using the mean squared reconstruction error as the optimization objective. Training is performed using the Adam optimizer with a fixed learning rate, and early stopping is applied to prevent overfitting.

The latent representations learned by the autoencoder are subsequently modeled using a Gaussian Mixture Model (GMM) to estimate the probability density of normal traffic patterns. The number of mixture components is selected empirically to balance model expressiveness and computational efficiency, and the Expectation-Maximization algorithm is used for parameter estimation. The resulting likelihood scores provide an intrusion-aware representation that captures both reconstruction behavior and latent distribution characteristics.

For supervised classification, Extreme Gradient Boosting (XGBoost) and Logistic Regression are employed as downstream classifiers. XGBoost is configured with a limited tree depth, a moderate learning rate, and subsampling to mitigate overfitting and handle class imbalance effectively. Logistic Regression is used as a linear baseline to provide interpretability and stable decision boundaries. Both classifiers are trained exclusively on the latent features derived from the AE-GMM module.

Feature selection is applied prior to model training to reduce redundancy and improve computational efficiency. Low-variance features are removed after normalization, and the same feature subset is consistently used across all models to ensure a fair comparison.

These implementation choices are directly derived from the experimental codebase and are designed to ensure reproducibility and robustness across datasets. A concise summary of the architectural design, probabilistic modeling configuration, classifier settings, and preprocessing procedures is provided in Appendix A (Table A1, Table A2, Table A3, Table A4 and Table A5).

4.1. Autoencoder-Gaussian Mixture Model

The proposed framework integrates an autoencoder (AE) with a Gaussian Mixture Model (GMM) to perform unsupervised intrusion detection. The autoencoder is designed to learn compressed latent representations of normal network traffic by minimizing reconstruction loss during training. It consists of symmetrical encoder and decoder layers, where the encoder transforms input data into a lower-dimensional latent space and the decoder reconstructs it to approximate the original feature vector. During training, only normal traffic samples from the NSL-KDD and UNSW-NB15 datasets were used to accurately capture normal behavioral patterns.

After training, the reconstruction error for each sample was computed, reflecting how well the AE could reproduce the input. The GMM was then fitted on the latent representations of the normal data to estimate their probability distribution. This probabilistic modeling enables distinguishing between normal and abnormal data points by evaluating the energy score of each observation.

Threshold selection. The decision threshold was set at the 95th percentile of the anomaly energy distribution computed on the training data, following common practice in the intrusion detection literature. This percentile-based threshold provides a pragmatic balance between sensitivity and false alarm control when explicit labels for anomalous behavior are limited. Lower percentile values (e.g., 90%) typically increase sensitivity but may lead to a higher false positive rate, whereas higher percentiles (e.g., 97.5% or 99%) enforce more conservative decision boundaries that reduce false alarms at the expense of increased false negatives. During preliminary analysis, alternative percentile values were considered, and the 95th percentile consistently provided the most stable trade-off between detection coverage and false alarm rate across both datasets. This threshold was selected after sensitivity analysis on validation folds to ensure robustness across different data distributions.

This approach effectively identifies deviations in traffic patterns without requiring labeled data, offering a balance between reconstruction-based and density-based intrusion detection. Similar Autoencoder-GMM hybrids have proven robust in industrial and IoT network environments due to their capability to model nonlinear feature interactions and capture subtle deviations in complex traffic [37,38].

4.2. XGBoost Integration

To enhance detection accuracy and generalization, the second model integrates Extreme Gradient Boosting (XGBoost) with the learned representations from the autoencoder and GMM. In this semi-supervised setup, the AE-GMM pipeline first extracts meaningful features—including the reconstruction error, latent vectors, and GMM-derived scores—which are then used as inputs to the XGBoost classifier. XGBoost operates as a powerful ensemble method that constructs an optimized set of decision trees using gradient boosting, effectively capturing complex nonlinear relationships in the transformed feature space.

The model was trained on both NSL-KDD and UNSW-NB15 datasets after normalization and feature selection to ensure consistent input dimensions. The use of XGBoost significantly improved detection metrics, yielding high precision, recall, and F1-scores across both datasets. This improvement stems from the model’s ability to correct classification residuals iteratively and its robustness against overfitting through regularization and shrinkage techniques.

Furthermore, AE-GMM-XGBoost provides interpretable feature importance measures, allowing for analysis of the relative contribution of latent, probabilistic, and statistical attributes to intrusion prediction. Previous studies have confirmed that AE-GMM-XGBoost consistently outperforms traditional classifiers in intrusion detection due to its efficiency, scalability, and adaptive learning mechanism [34,39].

4.3. Logistic Regression Integration

The third model integrates AE-GMM-Logistic Regression (LR) with the autoencoder-derived feature set as a lightweight and interpretable AE-GMM. After feature extraction and normalization, the latent features and reconstruction errors from the AE-GMM were fed into a binary logistic classifier to distinguish between normal and anomalous traffic. AE-GMM-Logistic Regression, being a linear probabilistic model, estimates the likelihood of a connection belonging to a specific class based on weighted feature contributions.

This model serves two main purposes:

(1) The first is providing a transparent benchmark against more complex models such as AE-GMM-XGBoost, and (2) the second is demonstrating that even simple linear classifiers can achieve competitive results when coupled with effective representation learning. Regularization (L2 penalty) was employed to prevent overfitting and stabilize the coefficients across features with different scales.

Experimental outcomes indicate that AE-GMM-Logistic Regression achieves high detection accuracy with strong generalization across both datasets, particularly when trained on AE-transformed inputs. Similar hybrid frameworks combining feature extraction from deep models with interpretable classifiers have been shown to balance accuracy and explainability in modern intrusion detection systems [24,40].

4.4. Summary of Methodology

The methodology implemented in this research follows a structured and data-driven approach to evaluate three complementary intrusion detection models under uniform experimental settings. The Autoencoder-Gaussian Mixture Model (AE-GMM) was employed as a foundational unsupervised framework capable of capturing nonlinear relationships in the latent feature space. This model established a reference point for intrusion energy distribution and provided a reconstruction-based perspective on abnormal behaviors.

Building upon this AE-GMM, the AE-GMM-XGBoost model introduced a gradient-boosting ensemble mechanism that leveraged the same preprocessed and feature-selected data to enhance detection precision. Its ability to iteratively minimize classification errors allowed the model to achieve higher discrimination between normal and attack patterns while maintaining computational efficiency.

The AE-GMM-Logistic Regression model, in turn, served as a transparent and interpretable counterpart within the hybrid framework. By using the same standardized and reduced feature set, it enabled a direct comparison between deep, ensemble, and linear learning strategies. This integration highlighted how simpler linear models could still perform effectively when combined with robust preprocessing and feature engineering.

Collectively, the three models were trained and tested on the NSL-KDD and UNSW-NB15 datasets using identical data preparation, ensuring fairness in performance evaluation. Each model contributed a distinct analytical perspective—unsupervised representation from AE-GMM, ensemble-based learning from AE-GMM-XGBoost, and linear interpretability from AE-GMM-Logistic Regression—thereby creating a comprehensive framework for assessing intrusion detection in network security. The methodological design ensures that the outcomes are reproducible, comparable, and grounded in practical cybersecurity challenges.

4.5. Evaluation Metrics

To assess the performance of the proposed intrusion detection models, several well-established evaluation metrics were employed, namely accuracy, precision, recall, F1-score, and the confusion matrix. These indicators collectively measure how effectively each model differentiates between normal and malicious network traffic.

The evaluation metrics are expressed as

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(8)

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(9)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(10)

$$F1 Score=2·{\displaystyle \frac{Precision·Recall}{Precision+Recall}}$$

(11)

where

• TP (true positive) represents attack instances correctly identified as attacks.
• TN (true negative) refers to normal instances correctly recognized as normal.
• FP (false positive) indicates normal traffic that was incorrectly classified as an attack.
• FN (false negative) denotes attack traffic that was misclassified as normal.

The confusion matrix provides a comprehensive summary of these outcomes, illustrating the correct and incorrect classifications made by the models. It serves as a valuable tool for analyzing detection accuracy and identifying potential misclassification trends across different attack categories.

5. Results and Discussion

5.1. Overview of Experimental Evaluation

To evaluate the effectiveness and robustness of the proposed intrusion detection framework, three different models, Autoencoder-GMM, AE-GMM-Logistic Regression, and AE-GMM-XGBoost, were trained and tested on two widely used benchmark datasets: NSL-KDD and UNSW-NB15.

Each model represents a distinct learning paradigm: the AE-GMM embodies an unsupervised intrusion detection approach; AE-GMM-Logistic Regression represents a classical linear supervised model; and AE-GMM-XGBoost reflects a modern ensemble-based method capable of learning complex, nonlinear relationships.

The evaluation employed multiple quantitative metrics, including accuracy, precision, recall, F1-score, and AUC, to comprehensively assess the models’ detection capability, reliability, and generalization performance.

This combination of metrics provides a balanced view between false alarms (precision) and missed detections (recall), while AUC highlights the models’ overall discriminative ability.

To ensure statistical robustness, each experiment was repeated multiple times under fixed random seeds, and performance metrics were estimated using bootstrap resampling. Reported results in Table 6 represent mean ± standard deviation across bootstrap samples.

Table 6. Binary intrusion detection performance of the proposed models on NSL-KDD and UNSW-NB15 (Bootstrap Mean ± Standard Deviation).

Dataset	Model	Accuracy	Precision	Recall	F1
NSL-KDD	AE-GMM	0.404824 ± 0.0021	0.563636 ± 0.0041	0.01076 ± 0.0052	0.093664 ± 0.0047
	AE-GMM-XGBoost	0.893408 ± 0.0023	0.804907 ± 0.0036	0.993307 ± 0.0031	0.889237 ± 0.0027
	AE-GMM-Logistic Regression	0.882630 ± 0.0031	0.822103 ± 0.0037	0.928432 ± 0.0030	0.872038 ± 0.0028
UNSW-NB15	AE-GMM	0.248424 ± 0.0040	0.944324 ± 0.0024	0.076604 ± 0.0027	0.141712 ± 0.0025
	AE-GMM-XGBoost	0.920948 ± 0.0022	0.933667 ± 0.0023	0.951450 ± 0.0025	0.942475 ± 0.0021
	AE-GMM-Logistic Regression	0.888668 ± 0.0021	0.924095 ± 0.0021	0.911279 ± 0.0023	0.917642 ± 0.0020

In addition, paired Wilcoxon signed-rank tests were conducted between model predictions on identical test folds, and all reported performance improvements were statistically significant at p < 0.01.

5.2. Binary Classification Results

Table 6 reports the binary intrusion detection performance of the proposed models on NSL-KDD and UNSW-NB15. The hybrid models demonstrate strong and consistent discrimination between normal and malicious traffic, substantially outperforming the standalone AE-GMM baseline. These results confirm that integrating anomaly-aware latent representations with supervised classifiers yields robust and reliable detection in the binary intrusion detection setting.

5.2.1. Analysis and Discussion

The results clearly reveal the performance hierarchy among the three evaluated models. To provide an intuitive comparison of their quantitative performance, Figure 3, Figure 4, Figure 5 and Figure 6 illustrate the differences in accuracy, AUC, AUPRC, and F1-score across the NSL-KDD and UNSW-NB15 datasets, while Table 6 summarizes the detailed numerical outcomes.

Figure 3. Comparative (mean) of accuracy of AE-GMM, Logistic Regression, and AE-GMM-XGBoost on NSL-KDD and UNSW-NB15 datasets.

Figure 4. Comparison of AUC performance for the proposed models on NSL-KDD and UNSW-NB15 datasets.

Figure 5. Comparison of AUPRC performance for the proposed models on NSL-KDD and UNSW-NB15 datasets.

Figure 6. Comparison of (mean) of F1-scores across datasets and models.

Beyond quantitative performance gains, the observed superiority of the AE-GMM-XGBoost framework can be attributed to the complementary roles of representation learning, probabilistic modeling, and discriminative classification. The autoencoder effectively compresses high-dimensional network traffic into a structured latent space that preserves essential behavioral patterns while reducing noise and redundancy. The Gaussian Mixture Model further refines this representation by modeling latent density distributions, which enhances intrusion awareness by capturing deviations from normal traffic manifolds.

XGBoost, when applied to these anomaly-aware latent features, benefits from its ability to model complex nonlinear decision boundaries and handle class imbalance through gradient-optimized ensemble learning. This synergy enables the hybrid framework to detect subtle attack behaviors that may partially overlap with normal traffic patterns, a limitation commonly observed in purely unsupervised AE-GMMs, as reflected by their lower recall. In contrast, Logistic Regression exhibits more stable and interpretable behavior but remains constrained by its linear decision boundaries, explaining its comparatively reduced sensitivity in complex attack scenarios.

These findings indicate that the performance improvements are not merely numerical but stem from the principled integration of deep feature extraction, probabilistic density estimation, and supervised discrimination. The consistent results across both NSL-KDD and UNSW-NB15 further suggest that the proposed hybrid design generalizes well across datasets with different traffic characteristics and attack distributions.

To support the observed performance differences beyond point estimates, statistical validation was conducted on the test results. Specifically, 95% confidence intervals were estimated for key evaluation metrics using bootstrap resampling, which provides a robust, distribution-free assessment of metric variability. The confidence intervals associated with the AE-GMM-XGBoost model consistently indicated stable performance advantages over the other evaluated approaches, suggesting that the observed gains are robust rather than attributable to random sampling variability.

In addition, paired statistical testing was applied to compare classification outcomes under identical test conditions. The results confirm that the improvements achieved by AE-GMM-XGBoost are statistically significant, supporting the claim that its superior performance is not incidental but reflects a genuine advantage of the hybrid learning design.

The comparatively low recall observed for the standalone AE-GMM can be attributed to several methodological factors. First, reconstruction-based intrusion scoring tends to be conservative in highly imbalanced intrusion datasets, where many attack instances partially overlap with normal traffic in the feature space. In such cases, anomalous samples may be reconstructed with relatively low error, leading to an increased number of false negatives.

Second, the Gaussian Mixture Model operates under distributional assumptions that may limit its flexibility in high-dimensional latent spaces. When attack patterns are subtle or closely aligned with normal behavior, likelihood-based scoring may not sufficiently separate anomalous observations from benign ones.

Finally, the percentile-based thresholding strategy introduces an inherent trade-off between false alarms and missed detections. While the selected threshold effectively limits false positives, it may also exclude weak or emerging attack instances, thereby reducing recall. These limitations explain the observed behavior of the AE-GMM and motivate the integration of supervised classifiers that can exploit labeled decision boundaries on top of intrusion-aware representations.

5.2.2. Limitations

Despite the valuable insights provided by the AE-GMM, several limitations should be acknowledged. First, the Gaussian Mixture Model is sensitive to covariance estimation, particularly in high-dimensional latent spaces, where limited sample density can affect component separation and likelihood reliability. This sensitivity may reduce the model’s ability to distinguish subtle attack patterns that closely resemble normal traffic behavior.

Second, although early stopping and regularization mitigate overfitting, autoencoder-based representations may still capture dataset-specific regularities, which can limit robustness under distribution shifts or evolving traffic conditions. This effect is especially relevant in real-world deployments where concept drift is common.

Finally, class imbalance inherent in intrusion detection datasets poses an additional challenge for unsupervised models. When attack samples partially overlap with normal traffic in feature space, intrusion scores tend to be conservative, leading to missed detections. These limitations highlight why AE-GMM is more effective as a representation learning component rather than a standalone detector and further justify the integration of supervised classifiers to enhance sensitivity and decision boundaries.

5.2.3. Model Robustness in Real-World Deployments

Robust deployment of intrusion detection frameworks in real-world network environments requires explicit consideration of long-term correlations, error accumulation, and stability under dynamic and imperfect operating conditions. Recent studies on long-term correlated network systems emphasize that conventional models assuming short-term independence are prone to performance degradation due to cascading errors and temporal dependencies, particularly in distributed and heterogeneous infrastructures. Zhao et al. [41] demonstrated that nonlinear stochastic dependencies and delayed error propagation can significantly affect system stability if not explicitly modeled, highlighting the importance of mechanisms that mitigate error accumulation and correct instability over time [41]. In this context, the proposed hybrid AE-GMM-based framework implicitly enhances robustness by combining representation learning with probabilistic density modeling and supervised discrimination, which together reduce sensitivity to transient perturbations and localized modeling errors. The anomaly-aware latent representations learned by the autoencoder help stabilize feature distributions, while the integration of supervised classifiers constrains error propagation by enforcing discriminative decision boundaries. These design choices align with findings in long-term correlated system analysis, where robustness emerges from jointly addressing temporal dependencies, uncertainty accumulation, and adaptive correction rather than relying solely on static assumptions. Consequently, the proposed framework is better suited for real-world deployments characterized by evolving traffic patterns, delayed effects, and correlated anomalies, supporting stable and reliable intrusion detection under practical operating conditions.

In contrast, despite the clear performance differences observed across models, the standalone AE-GMM exhibits noticeably lower recall on both datasets. This behavior can be attributed to its reliance on reconstruction error and latent density estimation without class-discriminative supervision. In highly imbalanced intrusion datasets, many attack samples partially overlap with normal traffic patterns, leading unsupervised models to adopt conservative decision boundaries that reduce false positives at the expense of missed attacks. These results indicate that while AE-GMM effectively captures the structure of normal traffic, it lacks sufficient sensitivity to subtle or evolving intrusions when used in isolation.

As shown in these figures, the AE-GMM-XGBoost classifier achieved the best overall performance across both datasets, confirming its strong generalization capacity and robustness in handling complex network traffic data.

On the NSL-KDD dataset, AE-GMM-XGBoost reached an AUC of 0.9893, mean of recall of 0.9933 ± 0.0031, and mean of F1-score of 0.8892 ± 0.0027, indicating that it successfully detected nearly all intrusion attempts while maintaining a very low false positive rate.

Similarly, on the UNSW-NB15 dataset, AE-GMM-XGBoost obtained the highest mean of accuracy (0.9209 ± 0.0022) and an excellent balance between mean of precision (0.9337 ± 0.0023) and mean of recall (0.9515 ± 0.0025), confirming its effectiveness in detecting both frequent and rare attacks in modern heterogeneous traffic.

The AE-GMM-Logistic Regression model ranked second overall. Despite its linear nature, it produced remarkably stable and competitive resultsAUC = 0.9558 on NSL-KDD and 0.9383 on UNSW-NB15—proving that linear classifiers still retain value for intrusion detection tasks, especially when the data are moderately separable.

Its slightly lower recall compared to AE-GMM-XGBoost indicates reduced sensitivity to highly nonlinear patterns, which are common in modern cyberattacks.

Nevertheless, its high precision (above 0.92 on both datasets) underscores its reliability and interpretability, making it suitable for real-time or resource-constrained deployments.

Overall, the AE-GMM-XGBoost model consistently outperformed the others due to its gradient-boosted decision tree architecture, which efficiently learns both linear and nonlinear feature dependencies.

This adaptive learning mechanism makes AE-GMM-XGBoost particularly well suited for high-dimensional and heterogeneous cybersecurity data.

Meanwhile, AE-GMM-Logistic Regression served as a strong interpretable AE-GMM, and AE-GMM provided valuable insights into unsupervised intrusion detection behavior.

Together, these findings validate the potential of hybrid architectures that combine unsupervised feature extraction (e.g., autoencoder) with supervised ensemble classification (e.g., XGBoost) to achieve high detection accuracy and reduce false alarms in modern intrusion detection systems (IDSs).

To gain deeper insight into each model’s behavior in distinguishing between normal and attack instances, the confusion matrices presented in Figure 7, Figure 8 and Figure 9 were analyzed.

Figure 7. Confusion matrix of the AE-GMM model on (a) NSL-KDD dataset (b) UNSW-NB15 dataset.

Figure 8. Confusion matrix of the AE-GMM-XGBoost model on (a) NSL-KDD dataset (b) UNSW-NB15 dataset.

Figure 9. Confusion matrix of the AE-GMM-Logistic Regression model on (a) NSL-KDD dataset (b) UNSW-NB15 dataset.

These visual representations provide a granular view of how each classifier handled true positives (TPs), false positives (FPs), true negatives (TNs), and false negatives (FNs) across the NSL-KDD and UNSW-NB15 datasets.

As shown in Figure 7a, the AE-GMM on the NSL-KDD dataset produced a large number of false negatives (9215), indicating that many attack instances were misclassified as normal. Although the model correctly detected 496 attacks, its imbalance between true positives and false negatives led to very low recall performance.

A similar pattern is visible in Figure 7b for UNSW-NB15, where the model correctly classified 9142 attack samples but failed to detect over 110,000 attack cases.

This confirms that the unsupervised AE-GMM approach, while capable of identifying clear outliers, struggles to detect subtle or overlapping intrusion patterns, particularly in complex network environments.

In contrast, AE-GMM-XGBoost demonstrated excellent discrimination ability, as illustrated in Figure 8a,b. For NSL-KDD, the model correctly identified 9646 attack instances with only 65 false negatives, yielding near-perfect recall.

Similarly, for UNSW-NB15, it correctly detected 113,547 attacks while misclassifying only a small number (5794), confirming the model’s robustness and adaptability to varying attack types.

This strong performance reflects the model’s capability to capture nonlinear feature interactions through gradient-boosted decision trees, resulting in precise decision boundaries even under class imbalance.

The AE-GMM-Logistic Regression model also achieved stable and balanced performance, as shown in Figure 9a,b.

On the NSL-KDD dataset, it identified 9016 attacks with only 695 false negatives, while maintaining a high number of true negatives (10,882).

In the UNSW-NB15 dataset, the classifier maintained consistent detection ability with 108,753 true positives and 10,588 false negatives, showcasing a strong trade-off between precision and recall.

Although slightly inferior to AE-GMM-XGBoost in capturing complex nonlinearities, AE-GMM-Logistic Regression’s consistent accuracy across both datasets demonstrates its reliability and interpretability, two characteristics that are valuable in practical network defense systems.

In summary, the confusion matrix analysis corroborates the numerical results presented earlier:

• AE-GMM was overly conservative, leading to under-detection of attacks (low recall).
• AE-GMM-Logistic Regression maintained balanced precision–recall behavior with good detection coverage.
• AE-GMM-XGBoost achieved the most effective trade-off, minimizing false negatives without inflating false positives.

In addition to detection performance, the computational characteristics of the evaluated models were analyzed. The AE-GMM required the lowest training overhead but exhibited higher inference latency due to density estimation in the latent space. AE-GMM-Logistic Regression showed fast training and low memory usage, making it suitable for resource-constrained environments. In contrast, AE-GMM-XGBoost involved a higher training time due to ensemble construction, while maintaining efficient inference once deployed. Overall, the results indicate a clear trade-off between detection accuracy and computational cost, allowing model selection to be guided by deployment constraints.

5.3. Multi-Class Classification Result

Multi-class intrusion detection poses a more challenging classification problem due to the overlap between attack categories and the presence of minority classes. This evaluation therefore provides a deeper assessment of the proposed framework’s ability to learn discriminative representations and maintain reliable detection performance across diverse intrusion types.

The following table presents the comparative multi-class performance on NSL-KDD and UNSW-NB15.

Table 7 presents the macro-averaged multi-class intrusion detection performance of the proposed models on the NSL-KDD and UNSW-NB15 datasets. All metrics are reported as mean ± standard deviation obtained through bootstrap resampling, providing a statistically robust estimation of model performance. The standalone AE-GMM exhibits limited effectiveness, achieving accuracies of 45.14% on NSL-KDD and 41.95% on UNSW-NB15, reflecting the difficulty of purely unsupervised multi-class intrusion detection. In contrast, the hybrid models deliver substantially stronger results: AE-GMM-XGBoost attains accuracies of 89.46% and 89.09%, while AE-GMM-Logistic Regression achieves 88.24% and 89.69% on NSL-KDD and UNSW-NB15, respectively. These results confirm that anomaly-aware latent representations significantly enhance class separability and detection reliability across diverse attack categories.

Table 7. Macro-averaged multi-class intrusion detection performance on NSL-KDD and UNSW-NB15 using bootstrap estimation (Mean ± Standard Deviation).

Dataset	Model	Accuracy	Precision (Macro)	Recall (Macro)	F1-Score (Macro)
NSL-KDD	AE-GMM	0.4514 ± 0.0012	0.5345 ± 0.0011	0.5434 ± 0.0024	0.5368 ± 0.0026
	AE-GMM-XGBoost	0.8946 ± 0.0014	0.8523 ± 0.0027	0.7972 ± 0.0028	0.7691 ± 0.0012
	AE-GMM-Logistic Regression	0.8824 ± 0.0018	0.8485 ± 0.0016	0.7008 ± 0.0019	0.7371 ± 0.0017
UNSW-NB15	AE-GMM	0.4195 ± 0.0012	0.4909 ± 0.0016	0.5658 ± 0.0018	0.4374 ± 0.0011
	AE-GMM-XGBoost	0.8909 ± 0.0014	0.8661 ± 0.0013	0.8096 ± 0.0011	0.7805 ± 0.0015
	AE-GMM-Logistic Regression	0.8969 ± 0.0016	0.8647 ± 0.0011	0.7960 ± 0.0013	0.7676 ± 0.0012

Table 8 provides a direct methodological and performance comparison between the proposed framework and a representative recent study on intrusion detection using the UNSW-NB15 dataset. More et al. [42] employed classical machine learning models trained on raw preprocessed features combined with conventional feature selection techniques, reporting best accuracies of 85.99% for binary classification and 75.77% for multi-class classification. In contrast, the proposed framework introduces anomaly-aware latent representation learning through the AE-GMM module prior to supervised classification. This design enables the classifiers to operate on representations that capture deviations from normal network behavior rather than relying solely on raw features. As a result, the proposed approach achieves higher accuracies of 92.09% in binary classification and 89.69% in multi-class classification, with bootstrap-based evaluation confirming the stability and consistency of these improvements.

Table 8. Methodological and performance comparison with existing IDS studies.

Study	Dataset	Model Type	Feature Representation	Evaluation Setting	Best Accuracy %
More et al. (2024) [42]	UNSW-NB15	Classical ML	Raw preprocessed features + feature selection	Binary and multi-class	85.99/75.77
Proposed Framework	NSL-KDD, UNSW-NB15	AE-GMM–XGBoost, AE-GMM-Logistic Regression	Anomaly-aware latent representation (AE-GMM)	Binary and multi-class with bootstrap	92.09/89.69

5.4. Visual Interpretation and Evaluation

5.4.1. Visual Interpretation

Figure 10, Figure 11, Figure 12 and Figure 13 illustrate the ROC and precision–recall curves for all evaluated models. The graphical analysis confirms the consistency of the quantitative findings. AE-GMM-XGBoost achieved the largest areas under both ROC and PR curves on the NSL-KDD and UNSW-NB15 datasets, reflecting its strong sensitivity and precision across varying thresholds. AE-GMM-Logistic Regression followed with smooth and stable curves, indicating balanced predictive performance and good generalization. In contrast, the AE-GMM displayed flatter ROC curves closer to the diagonal, suggesting weaker separation between normal and attack instances. Overall, the visual comparison demonstrates that ensemble-based learning, represented by XGBoost, delivers superior discrimination capability, while linear and unsupervised models exhibit moderate yet complementary performance patterns.

Figure 10. ROC curve comparison for NSL-KDD dataset.

Figure 11. Precision–recall curve comparison for NSL-KDD dataset.

Figure 12. ROC curve comparison for UNSW-NB15 dataset.

Figure 13. Precision–recall curve comparison for UNSW-NB15 dataset.

5.4.2. Evaluation

To evaluate the effectiveness and reliability of the proposed hybrid framework, several standard performance metrics were adopted, including the ROC curve, precision, recall curve, and confusion matrix. These metrics provide a comprehensive view of model performance from both statistical and visual perspectives. The ROC and PR curves are used to analyze sensitivity and precision across thresholds, while the confusion matrix allows for examination of correct and incorrect classifications for each category. By relying on these complementary measures, the evaluation process ensures that the results reflect not only overall accuracy but also the detection consistency and robustness of the AE-GMM, AE-GMM-XGBoost, and AE-GMM-Logistic Regression models across both benchmark datasets.

6. Conclusions and Future Work

6.1. Conclusions

This research introduced and evaluated a hybrid intrusion detection framework combining deep feature extraction through an Autoencoder-Gaussian Mixture Model (AE-GMM) with advanced supervised classifiers, AE-GMM-XGBoost and AE-GMM-Logistic Regression, to enhance intrusion detection in complex network environments.

The experimental analysis on the NSL-KDD and UNSW-NB15 datasets demonstrated that the proposed hybrid approach achieved significant improvements in detection accuracy and reliability compared with the AE-GMM unsupervised model.

Among the tested models, AE-GMM-XGBoost consistently provided the highest detection capability, achieving an AUC above 0.97 and F1-scores exceeding 0.94 ± 0.0021, confirming its effectiveness in recognizing both frequent and rare attack patterns with minimal false alarms.

AE-GMM-Logistic Regression, while simpler, maintained competitive and stable performance with strong precision and interpretability, making it suitable for deployment in environments with limited computational resources.

The AE-GMM, despite its lower recall, offered valuable insights into the latent structure of network behavior and demonstrated potential for unsupervised intrusion discovery when labeled data are scarce.

Overall, the results confirm that integrating deep representation learning with ensemble-based and interpretable classifiers yields a more adaptive and reliable detection strategy.

The findings emphasize the importance of hybrid frameworks that leverage both unsupervised feature learning and supervised optimization, enabling accurate, explainable, and scalable intrusion detection across evolving network conditions.

6.2. Future Work

Building upon the promising outcomes of this research, several future directions can be considered to further strengthen the proposed hybrid intrusion detection framework.

A key focus could be the adoption of federated learning, allowing for distributed model training across multiple nodes without sharing raw data. This approach enhances privacy, scalability, and adaptability for real-world network environments where data heterogeneity is common.

The integration of explainable AI (XAI) techniques, such as SHAP or LIME, represents another promising direction. These methods can clarify how individual network features influence model decisions, improving interpretability and user confidence in automated detection outcomes.

Additionally, implementing adaptive or continual learning strategies would allow the model to evolve alongside new attack behaviors and dynamic traffic patterns, ensuring long-term robustness and reliability.

Finally, expanding the evaluation to IoT and cloud infrastructures could provide further insight into the framework’s generalization capability across diverse operational domains.

Advancing in these directions may lead to the development of a more intelligent, privacy-preserving, and self-adaptive intrusion detection system capable of sustaining effective defense mechanisms within complex and evolving cybersecurity landscapes.