AutoBoost-IoT: A Hybrid Model for Intrusion Detection in IoT Networks

Abstract

The rapid growth of IoT ecosystem has significantly increased the potential threats and attack vectors in the recent times, thereby requiring intrusion detection mechanisms that are highly accurate and scalable in nature. This paper presents a hybrid intrusion detection system that involves the usage of both supervised and unsupervised machine learning methods to detect different kinds of attacks present in the IoT network. In the first step, Random Forest-based feature extraction is adopted to determine the most important features from the highly dimensional network traffic data. After this, the extracted features are compressed using the Deep AutoEncoder model into latent features that are fed into multiple classifiers to classify the traffic into various IoT attack classes and normal traffic class. Specifically, the classifiers used in the process include XGBoost, SVM, Logistic Regression, Naive Bayes and Multilayer Perceptron models. Multiple IoT benchmark datasets, such as N-BaIoT and CICIoT2023, are used to evaluate the performance of the proposed hybrid intrusion detection system. It was found that the XGBoost classifier performed better than others, obtaining an accuracy rate of 99.63% and 98.94% on the N-BaIoT and CICIoT2023 datasets, respectively. The above-discussed results show the high potential of the proposed architecture for generalization in various IoT environments. From the results, one can see that it is highly effective to integrate deep learning for extracting features from data and using boosting techniques for classification to develop an efficient IDS system.

1. Introduction

As the deployment of IoT devices continues to grow across different industries, such as healthcare, smart homes, manufacturing, and transport; it is changing how we connect with each other and automate tasks. Due to complicating factors, there are now many more opportunities for cybersecurity vulnerabilities owing to the extensive utilization of IoT technologies [1]. For one thing, most IoT devices have limited computational power and do not generally follow standard security protocols, which leaves them open to cyber-attacks like botnet attacks, DDoS attacks, and unauthorized access attempts [2].

Another complication is that traditional IDS systems were designed to work only within traditional IT networks, making them unfit for IoT networks due to their highly dynamic and resource-limited environment [3]. In addition, using traditional ML modeling techniques in IoT networks requires a large amount of time spent on manually engineering features; however, this does not capture or provide an accurate depiction of the complex relationships found within the vast amounts of traffic passing through the entire IoT network [4].

There has been an increased focus on developing hybrid systems that combine unsupervised deep learning [5] and supervised classification methods [6] to solve the issues noted above. In contrast to supervised classification techniques, autoencoders perform exceedingly well at producing more compact and abstract representations of the input data. However, XGBoost (an example of an ensemble-based classifier) has consistently performed better than other classifiers when used on structured networks.

In this work, a hybrid intrusion detection framework is proposed, integrating autoencoder-based latent feature extraction with XGBoost for multi-class attack detection in IoT networks [7]. Rather than introducing new algorithmic components, this study investigates whether such a hybrid design can effectively balance detection performance [8], robustness to high-dimensional and potentially noisy data, and computational efficiency. The autoencoder is employed to acquire a concise latent representation of network traffic, which is then provided as input to the XGBoost classifier to differentiate between benign and malicious activities.

Furthermore, the aim of this study is not to develop a novel standalone algorithm, but assess the efficacy of combining unsupervised representation learning with ensemble-based classification in the specific context of IoT intrusion detection. This design is motivated by the need to achieve a trade-off between detection accuracy and computational efficiency, particularly in scenarios involving high-dimensional and imbalanced IoT traffic data.

In order to analyze the efficacy of the proposed hybrid model, an experiment has been carried out using various IoT benchmark datasets like the N-BaIoT and the CICIoT2023 datasets. The N-BaIoT and CICIoT2023 datasets have been used that comprise the actual network traffic captured under the normal operation of IoT networks and during the attacks. In the N-BaIoT dataset, there have been traffic logs of nine devices that represent a broad range of attacks, which include various forms of malware such as Mirai and Gafgyt. In contrast, in the CICIoT2023 dataset, there have been various attack types carried out using a large-scale and complex IoT architecture. The experimental results indicate that the proposed hybrid model achieves performance comparable to or exceeding that of existing approaches in terms of detection accuracy and generalization across multiple attack categories and normal traffic conditions. Furthermore, the combination of deep unsupervised representation learning and robust ensemble classification improves the model’s ability to handle high-dimensional and potentially imbalanced IoT data.

Our proposed method harnesses the synergies between deep and ensemble learning that have traditionally been treated separately. In contrast to prior models that rely solely on either deep learning or ensemble systems and thus suffer from the drawbacks of manual feature development, our integrated model uses both deep learning systems and regularizing mechanisms from tree-based classification systems (such as XGBoost). Consequently, our method enables us to design a highly scalable and performant system that will perform effectively in real-time in smart IoT environments where low-latency responses to intrusions are required.

The key contributions of this paper are as follows:

• A lightweight hybrid intrusion detection framework is proposed, integrating unsupervised autoencoder-based feature extraction with XGBoost for multi-class attack detection in IoT networks.
• The effectiveness of combining latent feature representation learning with ensemble-based classification is investigated to address high-dimensional and potentially noisy IoT traffic data.
• A comprehensive analysis of computational efficiency is provided, including inference latency and memory footprint, highlighting the suitability of the framework for real-time IoT monitoring.
• Extensive experiments are conducted on multiple IoT datasets, including N-BaIoT and CICIoT2023, demonstrating the robustness and generalization capability of the suggested framework.

The structure of this paper is as follows: Section 2 reviews related works on hybrid IDS approaches. Section 3 details the suggested methodology, encompassing preprocessing, feature extraction, and classification. Section 4 delineates and examines the experimental findings. Finally, Section 5 summarizes the main findings of this study and exploring prospective avenues for further research.

2. Related Works

As the number of security threats increases to an alarming level, there is a growing amount of research being conducted using machine learning (ML), deep learning (DL), and similar technologies to create intelligent intrusion detection systems (IDS) for IoT networks [9]. In the past, many of the original researchers focused primarily on developing traditional machine learning classifiers like Support Vector Machines (SVM), Decision Trees, and Random Forests to identify anomalies in IoT-based traffic [10,11]. Generally speaking, most of these classifiers were easy to understand and implement but did not generalize well in high-dimensional feature spaces or when faced with a dynamic attack environment. Therefore, as IoT technology advances and becomes more complex, researchers are moving towards developing faster and more automated approaches to developing and deploying intrusion detection systems.

Through employing autoencoders, a type of deep learning model [12], to detect device-specific anomalies in IoT security, researchers have identified significant opportunity for improvement over traditional machine learning methods [13]. Autoencoders can learn meaningful features from high-dimensional input data with minimal human intervention, making them particularly well suited to the IoT environment. Typically, an autoencoder is trained on benign traffic and models the normal behavioral characteristics of the devices being monitored. Any deviation from this pattern is indicative of an attack on those devices and can be detected by the autoencoder. For instance, the authors of [14] introduced the N-BaIoT dataset and used deep autoencoders to detect device-specific anomalies, with high precision in identifying cyberattacks against IoT devices.

The authors of [15] introduced a hybrid method for detecting intrusions that combines a stacked sparse autoencoder (SSAE) for extracting deep features with an XGBoost classifier. Their primary goal in creating this model was to develop a way to help overcome the difficulties posed by the detection of unknown attacks as well as the problem of the imbalance of classes found in network traffic. The results of their experiments showed that their proposed model performed better with higher accuracy and efficiency than other current forms of intrusion detection when tested against the same data sets.

On the classification side, XGBoost has received a lot of attention as a scalable ensemble learning technique that has additional features such as built-in features for regularity and for greater accuracy on structured data [16]. This is due to the robustness of XGBoost and its utility in implementations in security, such as intrusion detection, malware analysis, and attack prediction.

In addition to XGBoost, the authors of [17] evaluated a total of six algorithms designed for use with Wireless Sensor Networks (WSN) to attempt detection of cyber-attacks; these algorithms included GBM, LightGBM, CatBoost, Random Forest, KNN, and Gaussian Naïve Bayes. Data from the WSN-DS dataset was used to perform evaluations of detection accuracy, detection probability, false alarm rate, memory consumption and processing times in real world traffic generated by sensors. The results demonstrated that boosting algorithms, specifically LightGBM had the highest degree of accuracy and lowest rates of misdetection. Thanks to these factors, boosting algorithms are well suited for use in resource-constrained environments such as those found in WSN and IoT systems.

In [18], the authors presented an optimized hybrid IDS framework which utilizes the information gain methodology for feature selection and an autoencoder as a dimensionality reduction technique while integrating a classifier ensemble consisting of XGBoost, Light-GBM, and CatBoost classifier algorithms. Using the CICIDS2018 dataset with more than 1 million records for evaluation purposes; their model produced results that were greater than 99% accurate and at least 80% recall for frequent attack types (such as DDoS); however, performance was significantly lower for less frequent types of attacks (such as botnet attacks or brute force). The research further indicates hybrid architectures are capable of improving scalability and modularity, which increases their applicability as a real-time intrusion detection system. Additionally, the research identifies additional areas where there is need for research related to addressing class imbalance and improving detection of rare attack types.

Recent advancements in deep learning have resulted in the emergenceof HIDS which incorporate hybrid techniques for anomaly detection. These methods utilize deep learning for feature extraction and supervised machine learning for algorithmic training. For example, ref. [19] have developed a novel hybrid model that identifies IoT devices based on the use of autoencoders to generate latent features and uses XGBoost for algorithmically identifying anomalies through supervised learning with an accuracy of 97.6% for identifying the type of device and an F1 score of 98.36% for identifying anomalies in Modbus TCP data. This indicates that the hybrid model may generalize across many types of connected devices. In addition, the authors of [20] developed a HIDS framework where low-dimensional, benign traffic feature representations, learned exclusively from benign data, were used as input for multiple ML and DL classifiers in order to identify anomalous behavior patterns. Their findings further confirmed the efficacy of employing autoencoder-based feature learning for improving detection accuracy in IoT environments.

The authors of [21] suggested a hybrid method that uses a stacked sparse autoencoder to reduce deep features and LightGBM to classify them. Their technique was more than 80% effective in reducing the feature set and got more than 99% accuracy on both the NSL-KDD and Bot-IoT datasets. This shows that it is efficient for scalable IDS deployment. This further supports the idea that using mixtures of autoencoders and boosting can help current IDS get both high accuracy and low resource utilization.

In parallel, recent research has increasingly explored deep learning architectures based on spatial-temporal feature learning for IoT intrusion detection. CNN, RNN, LSTM, and Transformer-based models are examples of how to build machine learning models that can learn the temporal dependencies between variables as well as represent complex interactions between the features in the network traffic data. These types of models have been found to demonstrate strong detection capabilities, especially when the traffic patterns show either a sequential or temporal dependency. However, these architectures can have a high computational load, large model sizes, and relatively long training and inference times; thus, they may not be suitable for use in a resource-constrained or real-time IoT environment. In contrast, XGBoost is an example of an ensemble model that is much smaller in size than many of the other model architectures, yet, still has strong performance on structured tabular data. Therefore, this motivates further exploration of the use of hybridisation between representation learning and efficient ensemble classifiers.

Besides intrusion detection solutions, the literature has also paid attention to preventive IoT security approaches that seek to detect vulnerabilities prior to their exploitation. In this context, the weaknesses of IoT devices are analyzed at different levels—from devices to protocols [22], specifically focusing on the authentication phase and communication.

Indeed, recent studies have examined issues related to replay attacks or authentication abuse to assess the resilience of IoT devices when faced with hostile attacks. The proposed approach requires an active analysis of the devices to detect potential security weaknesses that cannot be easily identified through passive analysis alone [23].

While these preventive solutions can complement IDS approaches in detecting security risks, the current study adopts a data-driven detection strategy for evaluating intrusion threats.

Building upon these prior efforts, our work contributes to this research direction by developing a hybrid intrusion detection system for IoT environments. The proposed model combines the capabilities of autoencoders for unsupervised feature extraction from high-dimensional and largely unlabeled data with the strengths of XGBoost for multi-class classification. Unlike many existing approaches that treat feature engineering and classification as separate steps, the proposed framework leverages deep representation learning to transform high-dimensional IoT traffic data into compact and informative latent features, which are subsequently used for accurate classification. The model was evaluated on multiple IoT datasets, including N-BaIoT and CICIoT2023, and demonstrated high performance in accurately distinguishing between various attack types and benign traffic. The results underscore the efficacy of the proposed methodology in tackling significant difficulties in IoT intrusion detection, particularly multi-class classification and generalization across diverse attack scenarios.

To further contextualize the existing research landscape, Table 1 summarizes representative studies in IoT intrusion and anomaly detection, highlighting the datasets used, adopted methods, and their key contributions.

Table 1. Comparison of ML/DL-based intrusion detection approaches in IoT environments.

Reference	ML/DL Technique	Dataset Used	Key Contribution
[24]	ADSBAN	BoT-IoT	Evaluated multiple ML models for BAN anomaly detection, achieving high accuracy with Logistic Regression and XGBoost on healthcare IoT data.
[25]	PCA-RF	FDIA	Proposed a stacked autoencoder-based framework to handle data imbalance and evaluated ML models under dimensionality reduction. Random Forest and XGBoost maintained high accuracy and robustness.
[26]	TOCA-IoT	CICIoT2023	Presented the TOCA IoT framework integrating causal inference (LiNGAM) with explainable AI to improve the accuracy, interpretability, and robustness of IoT anomaly detection.
[27]	CNN-BiLSTM	UNSW-NB15	Developed a CNN-BiLSTM hybrid intrusion detection model for IoT network traffic and benchmarked it against several machine learning classifiers.
[28]	DAE-BiLSTM	CICIDS2017, BoT-IoT	Introduced a hybrid IDS using Deep Autoencoder and BiLSTM optimized with the Sparrow Search Optimization Algorithm (SSOA) for enhanced detection accuracy and diminished latency in fog-based IoT environments.
[29]	AE + EL	WSN-DS, UNSW-NB15, KDD CUP99	Proposed a lightweight NIDS combining one-class Bi-GRU autoencoder and ensemble learning to detect unknown attacks with high accuracy.
[30]	CNN-LSTM-AE	None	Presented a real-time deep anomaly detection framework using CNN and dual LSTM-AE optimized for edge devices and IIoT environments.
[31]	CSAE	CICIoT2023, DS2OS	Developed a cost-sensitive autoencoder-based ensemble IDS addressing class imbalance and dynamic IoT conditions, attaining near-optimal accuracy in binary and multi-class intrusion detection.
[32]	SKM-XGB	NSL-KDD, UNSW-NB15	Introduced an imbalance-aware IDS combining hybrid sampling (SMOTE + K-means), denoising autoencoder feature selection, and XGBoost with SHAP explainability.
[33]	XGB-RF	CICIDS2017, CSE-CICIDS2018, CICDDoS2019	Introduced a lightweight intrusion detection framework using optimized feature selection with Bayesian-tuned XGBoost for real-time IIoT detection.
[34]	Stacking Ensemble (XGBoost + RF + ANN)	UNSW-NB15	Proposed a real-time anomaly detection framework utilizing a stacking ensemble approach for enhanced performance across multiple attack types in IoMT scenarios.
[35]	HADTF	Self-generated RPL attack dataset	Proposed a hybrid autoencoder–decision tree framework for detecting RPL routing attacks with improved feature extraction and reduced false positives.
[36]	IGWO optimized stacking ensemble	NSL-KDD, UNSW-NB15, MQTTset, CICIDS2017	Presented an optimized stacking ensemble for IoT edge anomaly detection achieving high accuracy and strong cross-dataset generalization.
Our Study	AE + XGBoost	N-BaIoT, CICIoT2023	Introduced a hybrid IDS integrating an autoencoder for unsupervised feature extraction with XGBoost for supervised multi-class classification, evaluated on multiple IoT datasets (N-BaIoT and CICIoT2023), demonstrating high accuracy and robust generalization across diverse and complex attack scenarios.

3. Methodology

The structure illustrated in Figure 1 describes a hybrid learning architecture designed to accurately identify and detect multiple types of intrusion in an IoT ecosystem. This process begins with raw traffic ingestion from IoT datasets, including the N-BaIoT and CICIoT2023 datasets, followed by a data preprocessing stage where duplicate and irrelevant records are removed, and feature standardization is conducted using StandardScaler to guarantee that all features contribute equally during training. To make sure that the evaluation is fair and consistent, the same preprocessing and feature extraction pipeline is used on all datasets.

Figure 1. Workflow of the Proposed Framework.

To combat the curse of dimensionality that often accompanies the learning of higher-level representations of network behaviors, we employ a deep autoencoder in this capacity for unsupervised feature extraction. The encoder component of the autoencoder compresses the high-dimensional input to form a compact latent representation. The decoder subsequently reconstructs the original input while minimizing reconstruction loss via Mean Squared Error (MSE) [37]. This enables the system to extract more informative features to leverage while disregarding or minimizing redundancies or noise.

Next, a multi-class XGBoost classifier will take the encoded latent features. As stated, XGBoost is a scalable and regularized gradient boosting framework that can learn to differentiate benign traffic from ten different IoT attack classes. One of the benefits of XGBoost is its class imbalance management capabilities and its ability to learn complex decision boundaries. Therefore, it is appropriate for heterogeneous IoT intrusion data.

The performance of the models will be evaluated by using standard classification evaluation metrics such as accuracy, precision, recall and F1-score. Classifier accuracy will be evaluated overall and individually for each of the classes in order to measure model performance. In addition, hyperparameter tuning for all classifier types will be documented and include all metrics, as well as visual representations of classifier performance, and evaluations of both on multiple configurations of the hyperparameters.

3.1. Dataset Description

This study employed the N-BaIoT dataset [14], a widely recognized benchmark designed to evaluate intrusion detection systems in IoT environments. The dataset was produced by observing nine consumer-grade IoT devices with some monitoring activities occuring in benign and others in a malicious state. Malicious traffic was created with the use of two widely known botnets, Mirai and Gafgyt, which resulted in using ten different types of attacks that included distinct variants of TCP and UDP flooding, scanning, and combo attacks.

Each entry in the database has 115 statistical features distilled from network flows that included features on the packets that monitored metrics such as packet length, inter-arrival time, and connection state (host and destination). The N-BaIoT dataset supports both binary classification (benign vs. malicious) and multi-class classification (ten attack classes + one benign class) where the evaluations of the dataset allow for measuring fine-grained intrusion detection effectiveness.

In addition to over 1.8 million labeled instances, the N-BaIoT offers a corpus rich in diversity and volume as a suitable dataset for training and validating machine learning models in the context of real-world IoT attack scenarios. The richness of the dataset and the volume presented allow for evaluating not only the detection accuracy but also, how effective machine learning models can scale and generalize for other classes of devices and attack behavior.

As indicated in Figure 2, the dataset has a skewed distribution across the various attack classes, which reflects an important imbalance among the classes. This imbalance presents a persistent issue in the performance of conventional classifiers. In order to tackle the issue of class imbalance, our approach employs two oversampling techniques, such as SMOTE, to maintain an even distribution of classes in training to improve generalization.

Figure 2. Distribution of Attack types in the N-BaIoT Dataset.

Furthermore, in order to validate the generalizability ability of the developed framework, experimental analysis was performed on the CIC IoT Dataset 2023 as well [38]. Specifically, the CICIoT2023 dataset constitutes an up-to-date benchmark created for the task of IoT IDS that involves attacks simulated in the real-time IoT network, which consists of 105 interconnected devices.

The dataset comprises a collection of 33 types of attacks divided into seven groups such as DDoS attacks, DoS attacks, Reconnaissance attacks, Web-based attacks, Brute Force attacks, Spoofing attacks, and Mirai attacks. Notably, all the attacks listed above are initiated by malicious IoT devices directed against other devices connected to the same network.

In this research, a subset of 11 different types of attacks from the initial dataset was chosen for the sake of a more realistic approach while ensuring computationally efficient experimental setup. Such a decision ensures the right balance between the computational complexity and variability of data.

Unlike the N-BaIoT dataset, the latter one includes a greater number of various attacks that occur in a bigger-scale network environment. Therefore, it can be considered a more appropriate benchmark to verify the quality of the presented solution.

3.2. Data Preprocessing

To make sure that the feature extraction and classification model delivers better accuracy and reliability in the results, the raw data collected from both N-BaIoT and CICIoT2023 datasets went through a few data processing steps that aim to improve the quality of the data and avoid any kind of bias in the modeling process. This same data processing pipeline was also used for both datasets for a fair comparison.

First, to remove redundant data in the dataset, all duplicate records were found and deleted. In the case of the N-BaIoT dataset, the deletion of these duplicate records led to the elimination of around 322,367 data entries, resulting in a reduction of the overall dataset size from 1,854,174 to 1,531,807.

The StandardScaler normalization method was used in order to bring all features into similar distributions by centering and scaling the data to unit variance. In order to balance out the dataset because of the presence of class imbalance, the SMOTE technique (Synthetic Minority Over-sampling Technique) [39] was used to generate additional data instances from minority classes in the training set. The following Table 2 shows the class distribution of the training dataset before and after using the SMOTE technique on the N-BaIoT dataset. A similar technique was used for the CICIoT2023 dataset.

Table 2. Class distribution before and after applying SMOTE.

Class	Before SMOTE	After SMOTE
benign	8072	21,450
gafgyt_combo	10,451	21,450
gafgyt_junk	10,174	21,450
gafgyt_scan	10,447	21,450
gafgyt_tcp	9574	21,450
gafgyt_udp	11,029	21,450
mirai_ack	17,889	21,450
mirai_scan	11,307	21,450
mirai_syn	21,450	21,450
mirai_udp	16,636	21,450
mirai_udpplain	15,494	21,450
Total	142,517	235,950

To visually illustrate the impact of the SMOTE technique [39], Figure 3 shows the class distribution before and after oversampling. The original dataset exhibited significant imbalance among classes, which was effectively mitigated after applying SMOTE, resulting in a more uniform distribution.

Figure 3. Visualization of Class Distribution Before and After SMOTE Application.

To detect linear dependencies between the features and possible redundancies within them, a Pearson correlation analysis was performed on the normalized feature set. This step plays an essential role when dealing with the multidimensional space of IoT traffic features, since multicollinearity within the feature set may adversely affect the model stability and performance during classification.

Figure 4 illustrates the correlation matrix of the feature set, with both high positively and negatively correlated features depicted. A high positive correlation suggests that there exist some redundant features in the dataset, especially in the case of similar behavior measurements obtained from various sensors. On the other hand, negatively correlated pairs (blue) show inverse associations between the traffic features, thereby enabling distinction between benign and malicious traffic.

Figure 4. Heatmap of Feature Correlation in N-BaIoT Dataset.

Based on the findings above, the use of autoencoders in this work appears justified because autoencoders are able to encode non-linear relationships into the latent space of the model. By removing the redundancies while retaining the necessary information, the autoencoder improves the performance of the downstream classifiers like XGBoost.

3.3. Feature Selection

To increase the input data quality and avoid redundancy in the initial feature space, a preprocessing stage consisting of feature selection was conducted before deep features were extracted [40]. Network traffic datasets are often characterized by high dimensionality since numerous statistical features related to network activity at various time intervals are used to describe IoT devices. Although such a representation provides valuable information, it may also contain some redundant and irrelevant elements, thus affecting classification accuracy.

In the current research, two high-dimensional IoT network traffic datasets were used, namely N-BaIoT and CICIoT2023. Applying all features provided by the datasets may complicate the training procedure for algorithms due to noise presence and increased dimensionality.

This problem was addressed with the help of the random forest-based approach to determining the importance of features [41]. The idea behind using random forest consists of the construction of several decision trees based on selected features to classify data samples. The feature importance is measured based on the Mean Decrease in Impurity metric.

The Random Forest algorithm was applied using the training data set that had been balanced through the SMOTE method in order to avoid bias in determining the feature importances in favor of the majority classes. All feature importances were determined and 20 most important features were selected.

The selection of 20 features was guided by an empirical analysis of the importance distribution. It was observed that a small subset of highly ranked features captures most of the discriminative information, while significantly reducing the dimensionality of the input space. Increasing the number of selected features beyond this threshold did not yield noticeable performance improvements, whereas reducing it led to a decline in classification accuracy.

Figure 5 illustrates the importance ranking of the selected features. This selection reduced the dimensionality of the feature space while preserving the most relevant characteristics of network traffic. The selected features are then used as input to the autoencoder, which further extracts compact latent representations for classification.

Figure 5. Top 20 Important Features Identified by Random Forest.

By combining feature selection with deep feature extraction, the proposed framework reduces noise and redundancy while improving computational efficiency and overall detection performance in IoT intrusion detection tasks.

A similar feature selection process was implemented to the CICIoT2023 dataset, confirming that a compact subset of top-ranked features is sufficient to preserve the most discriminative information across different IoT datasets.

3.4. Feature Extraction

After completing the feature selection, a deep autoencoder network will be used in unsupervised learning of new features. An autoencoder will be performed using the reduced number of features generated in the feature selection stage by selecting the top 20 most important features. This is important since the selected subset of features will eliminate some redundant features, which makes the autoencoder task easier to perform without increasing computational complexity.

Network traffic in IoT networks usually presents high dimensional feature spaces with complex relationships between features; therefore, it is necessary to learn compressed latent representations of the data in order to reduce computational complexity while keeping the relevant features in the representation of network traffic behavior.

In our research, a deep autoencoder will be used to compress selected network traffic features learned from the N-BaIoT and CICIoT2023 datasets. Autoencoders are neural networks that learn compressed efficient encodings of data in order to reconstruct the input data; this type of architecture helps to learn non-linear relationships and perform dimensionality reduction of the data features.

Latent variable representation obtained from the autoencoder of the selected features contributes to the quality of feature representation, thereby improving the performance of the subsequent classifiers such as XGBoost.

Autoencoder Architecture:

The autoencoder consists of two primary components:

• Encoder: Maps the high-dimensional input into a concise latent representation via a sequence of completely connected layers. In our architecture, the encoder comprises three hidden layers with 64, 32, and 16 neurons, respectively, using the ReLU activation function. This design enables the model to capture non-linear dependencies and abstract patterns in the data.
• Decoder: Reconstructs the original input from the encoded representation using a symmetric structure with two hidden layers (32 and 64 neurons) followed by a linear output layer matching the original 115-dimensional input size. This symmetry ensures the model learns a meaningful embedding space aligned with the reconstruction objective.

Training Details:

The autoencoder was trained unsupervised to reduce the Mean Squared Error (MSE) between the input and its reconstruction. The Adam optimizer was employed with a learning rate of 0.001 and early halting implemented with a patience of 5 epochs to alleviate overfitting. The model underwent training for a maximum of 50 epochs, utilizing a batch size of 256. Upon completion of training, we retained solely the encoder component of the autoencoder to transform the raw feature vectors into latent representations. The processed lower-dimensional embeddings, that maintain the fundamental characteristics of the original input data, were utilized as input for the classifier in the multi-class intrusion detection task following the deep learning process. This process of combining deep unsupervised learning with supervised classifiers, is the idea of utilizing the best of both methods as a scalable and accurate solution for high-dimensional, imbalanced IoT datasets.

3.5. Multi-Class Classification

The next step involved the evaluation of several different machine learning classification algorithms that would yield the most suitable approach for the multi-classification intrusion detection problem. The classifiers considered in this paper comprise the following: Logistic Regression (LR), Naïve Bayes (NB), Support Vector Machine (SVM), XGBoost, and a Multi-layer Perceptron (MLP).

LR served as the first benchmark classifier since it is straightforward and easy to understand. The algorithm was trained on the training data using L2 regularization with lbfgs solver. The maximum number of iterations for convergence was set to 1000.

NB was selected based on the assumption of feature independence. It is implemented via the Gaussian Naïve Bayes algorithm, which can serve as a good baseline model for comparison purposes.

SVM served as the second non-linear margin-based classifier that could capture more complicated relationships among the encoded features.

In addition, a Multilayer Perceptron (MLP) was employed as a deep learning-based classifier operating on the latent feature space. The MLP architecture consists of three fully connected layers with 128, 64, and 32 neurons, respectively, each followed by ReLU activation functions. Dropout regularization with rates of 0.3 and 0.2 was applied between layers to mitigate overfitting. The output layer uses a softmax activation function for multi-class classification. The model was trained using the Adam optimizer with a categorical cross-entropy loss function, a batch size of 256, and a maximum of 50 epochs. was implemented based on validation loss with a patience of 5 epochs to mitigate overfitting.

Finally, XGBoost (Extreme Gradient Boosting) was employed as a tree-based ensemble learning algorithm known for its strong performance on structured data. The model was trained using 100 estimators, a learning rate of 0.1, and a maximum depth of 5. All classifiers were trained using the latent feature representations generated by the autoencoder to ensure a fair comparison.

3.6. Evaluation Metrics

In order to properly and effectively evaluate the effectiveness of the proposed models of intrusion detection, we have chosen certain evaluation parameters that are applicable to classification problems, and these are Accuracy, Precision, Recall, and F1 Score. These evaluation parameters will give us a general idea of the effectiveness of the intrusion detection model in classifying the different types of intrusions and normal traffic correctly.

Accuracy quantifies the overall ratio of properly anticipated instances across all classes, providing a general indication of classification success.

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(1)

Precision denotes the ratio of accurately detected attack occurrences to the total instances expected as attacks, indicating the classifier’s reliability in minimizing false positives.

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(2)

Recall measures the ratio of real attack occurrences that were successfully detected, highlighting the model’s completeness in capturing threats.

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(3)

F1-score is a measure that offers a balanced representation between precision and recall, in addition to representing both metrics, so it is particularly useful when dealing with value imbalances; when considering either precision or recall individually.

$$F1\text{-}score=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(4)

3.7. Analytical Justification of the Proposed Model

The rationale for combining an autoencoder with an XGBoost classifier is grounded in the complementary strengths of unsupervised representation learning and supervised gradient-boosted decision trees.

• Dimensionality Reduction with Autoencoder:
  High-dimensional IoT traffic data can contain redundant or noisy features, which hinder the performance of traditional classifiers. An autoencoder takes the input space and compresses that information into a lower dimension latent representation, allowing for a reduction of the original data’s dimension while preserving the important structure and eliminate noise. This is achieved by minimizing the reconstruction loss function $L(x,\widehat{x})={\parallel x-\widehat{x}\parallel }^2$, that quantifies the disparity between the original input x and its reconstruction $\widehat{x}$.
• Class Discriminability via XGBoost: XGBoost is chosen for its capacity to model non-linear relationships and efficiently handle multi-class classification. It constructs an ensemble of weak learners $h_t\left(x\right)$, optimizing the following objective function:

  $$\mathcal{L}=\sum _{i=1}^{n}l(y_i,{\widehat{y}}_i^{\left(t\right)})+\sum _{t=1}^T\mathrm{\Omega }\left(h_t\right)$$

  (5)

  where l is the loss function (e.g., softmax for multi-class classification), and $\mathrm{\Omega }$ is the regularization term that controls overfitting.
• Complementarity of the Components: By first extracting robust, denoised features from raw traffic via the autoencoder, we alleviate the burden on the classifier to perform both noise filtering and classification. The downstream XGBoost model thus benefits from improved feature separability, leading to higher predictive performance.
• Robustness to Class Imbalance: Autoencoders learn global representations independent of class distribution, while XGBoost supports class-weighting and handles imbalanced data effectively via tailored loss functions. This makes the hybrid architecture suitable for intrusion detection where normal traffic often dominates.
• Computational Efficiency: Reducing dimensionality before classification reduces the number of operations and memory consumption during both training and inference—an essential requirement for deployment in IoT edge environments with constrained resources.

4. Results and Discussion

The efficacy of the proposed intrusion detection framework was assessed using multiple IoT datasets, including the N-BaIoT and CICIoT2023 datasets, which comprises network traffic generated by multiple IoT devices under both normal and malicious scenarios. In addition, the prevalent issue of class imbalance that is common in the IoT dataset was addressed using the SMOTE method, which ensured that all the classes of attacks were adequately represented during the training phase. However, the dataset was first normalized using the StandardScaler method prior to the training phase.

All experiments were performed on a machine equipped with an Intel Core i7 processor, 16 GB of RAM, and running a 64-bit operating system. No GPU acceleration was used during training and evaluation. The reported computational metrics, including training time and inference latency, depend on this hardware configuration and are primarily intended for relative comparison between the evaluated models.

Following the preprocessing phase, the proposed random forest-based method was employed to select the most relevant features from the total number of features, which is 115, extracted from the network traffic. The top 20 features with the highest importance values were identified based on the computed values and then used as the input to the deep autoencoder that was employed to perform unsupervised feature extraction using the extracted features.

To identify the optimal classifier for the intrusion detection problem, different machine learning models were tested using the defined feature representation methods, as follows: logistic regression, naive Bayes, SVM, and XGBoost. It is worth mentioning that all the classifiers were evaluated under identical conditions to facilitate an equitable comparison.

Figure 6 presents an overview of the comparative performance metrics for all classifiers, including accuracy, precision, recall, and F1-score. As illustrated in Figure 6, XGBoost outperforms all the classifiers in all the evaluation metrics, indicating its ability in effectively modeling complex patterns in IoT networks.

Figure 6. Comparative Performance Metrics Across Different Classifiers.

Table 3 presents a summary of the classification performance results for all the classifiers. From Table 3, it is very clear that the accuracy achieved by XGBoost is significantly much higher than that achieved by the other classifiers used in this study. XGBoost achieved a much higher accuracy, precision, recall, and F1-score, 99.63%, 99.64%, 99.63%, and 99.63%, respectively, compared to the other classifiers. This is an indication that XGBoost is very effective in learning or fitting complex patterns, as presented by the encoded feature representation generated by the autoencoder.

Table 3. Performance comparison of machine learning models.

Metric	XGBoost	MLP	SVM	LR	NB
Accuracy	99.63%	91.97%	89.74%	86.93%	71.02%
Precision	99.64%	95.16%	94.40%	83.48%	75.22%
Recall	99.63%	91.97%	89.74%	86.93%	71.02%
F1-Score	99.63%	89.62%	87.22%	84.45%	66.87%

The Multilayer Perceptron (MLP) classifier also demonstrated competitive performance, achieving an accuracy of 91.97%, precision of 95.16%, recall of 91.97%, and an F1-score of 89.62%. While the MLP is capable of modeling non-linear relationships within the latent feature space, its performance remains lower than that of XGBoost, suggesting that tree-based ensemble methods are more effective for structured tabular representations of IoT traffic in this context.

On the other hand, the Support Vector Machine classifier obtained a lower accuracy of 89.74%, a higher precision of 94.40%, a lower recall of 89.74%, and a lower F1-score of 87.22%. Although SVM obtained higher precision in this study, its overall performance was lower than XGBoost since it was difficult for SVM to model complex multi-class patterns in IoT networks.

The Logistic Regression model achieved an accuracy of 86.93%, precision of 83.48%, recall of 86.93%, and an F1-score of 84.45%. As a linear classifier, Logistic Regression is limited in its capacity to identify non-linear correlations among features, which explains its relatively lower performance compared to tree-based ensemble methods.

The Naive Bayes classifier obtained the lowest performance among the evaluated models, with an accuracy of 71.02% and an F1-score of 66.87%. This result can be attributed to the strong independence assumption inherent in Naive Bayes, which rarely holds for complex network traffic data where multiple features interact in non-linear ways.

An ablation study was carried out for determining the impact of the autoencoder in this regard by training the XGBoost classifier using only the selected features without any use of the autoencoder.

XGBoost performs well in terms of accuracy, precision, recall, and F1-Score without the help of autoencoder, having scores of 98.76%, 98.75%, 98.76%, and 98.76%, respectively. However, adding the autoencoder increases all these metrics to 99.63% for each metric.

These improvements suggest that using the autoencoder for better feature representation, where non-linearity and redundancies are reduced, improves the performance of the model to some degree. This implies that although the gain is not very significant, still the learned latent features have been helpful in improving the detection capabilities of the model.

According to the findings of the research, the use of an autoencoder as a part of feature extraction coupled with XGBoost as a classifier proves to be an efficient means for intrusion detection in IoT systems. The use of the autoencoder allows obtaining concise and relevant representations of data, which are then analyzed by the ensemble machine learning algorithm, XGBoost. The additional ablation test shows that despite the high efficiency of using XGBoost individually, the combination with the autoencoder improves the efficiency of intrusion detection.

In order to acquire more insight into the proposed model’s classification behavior, a review of the complete confusion matrix was performed. The confusion matrix contained in Figure 7 illustrates all class-by-class predicted values using the test data, which show how well the hybrid Intrusion Detection Model predicts attack types and benign traffic.

Figure 7. The confusion matrix of our model (AE + XGBoost).

The findings displayed in the confusion matrix illustrate the general efficacy of the examined model in accurate classification; it is evident that the majority of outcomes align with the diagonal of the confusion matrix. The hybrid Intrusion Detection Model can distinguish between benign and malicious traffic with near-perfect accuracy, a crucial attribute for any intrusion detection system. This capability mitigates alert fatigue and enhances the operational efficiency of the system when implemented in a live environment.

The confusion matrix also provides evidence of solid performance in regards to the detection of multiple attack classes (mirai_syn, gafgyt_udp, and mirai_scan) representing complex threats associated with IoT devices, which consistently exhibit high accuracy rates and very little overlap. This shows that the model is capable of detecting complex patterns related to changes in network behaviors, as a result of various types of attacks.

However, minor confusions were observed between classes of attacks that share common traffic patterns, such as in the case of mirai_udp and mirai_udpplain classes of attacks. These classes of attacks frequently share common packet formats and temporal transmission patterns, making it challenging for advanced models to differentiate between these classes of attacks. The minor number of misclassifications between these classes of attacks also presents another avenue for further refinement of these classes of attacks using attention-based models or feature augmentation in future studies. Overall, the results obtained from the confusion matrix confirm that the proposed conclusion can be supported that the proposed model has the capability to perform fine-grained multi-class discrimination with low misclassification rates, which is essential in the application of IoT intrusion detection systems that require the accuracy and interpretability of the classes to be significant.

The overall classification performance indicates the effectiveness of the suggested framework. However, the class-level evaluation of the proposed model will provide additional insights into the effectiveness of the suggested framework in discriminating different types of attacks. Table 4 presents the class-level evaluation metrics obtained using the XGBoost classifier for the different classes of the traffic data.

Table 4. Per-class evaluation metrics for the proposed model.

Class	Accuracy (%)	Precision (%)	Recall (%)	F1-Score (%)
benign	99.77	100.00	99.77	99.88
gafgyt_combo	95.89	99.58	95.89	97.70
gafgyt_junk	99.63	98.04	98.85	98.45
gafgyt_scan	100.00	99.73	99.96	99.84
gafgyt_tcp	99.95	99.90	99.95	99.93
gafgyt_udp	100.00	99.92	100.00	99.96
mirai_ack	100.00	99.90	100.00	99.95
mirai_scan	99.83	100.00	99.83	99.92
mirai_syn	100.00	100.00	100.00	100.00
mirai_udp	99.86	100.00	99.86	99.93
mirai_udpplain	100.00	100.00	100.00	100.00

From the results provided, it is evident that the framework has the capability to attain high performance on almost all the classes of the traffic. In particular, the attack types of “mirai_syn” and “mirai_udpplain” have been observed to attain the best classification performance with 100% precision, recall, and F1-score. This is because the attack types have distinctive characteristics that can be captured through the learned representations. In addition, the attack types of “gafgyt_udp,” “mirai_ack,” and “mirai_scan” have been observed to attain near-perfect detection rates with F1-score values above 99.9%.

Good performance was also seen in the detection of benign traffic, with precision at 100% and recall at 99.77%. This is important in intrusion detection systems because it helps to reduce false alarms. A small number of classes shows slightly lower performance than the others. For example, gafgyt_combo showed a slightly lower F1-score of 97.70%. This is because of similarities in traffic behaviors with other Gafgyt attacks.

Overall, the performance of the proposed hybrid framework in the evaluation of each class has been verified in the accurate detection of benign traffic as well as different types of IoT attacks. TThe findings have demonstrated the effectiveness of the proposed approach in the accurate detection of discriminative traffic patterns for different types of attacks using the combined process of feature selection, autoencoder-based feature extraction, and classification using XGBoost.

Along with observing the classification accuracy, another important metric for evaluating an intrusion detection model is its computational efficiency in a resource-constrained environment like IoT. Besides how well the intrusion detection model can detect intrusions in real time, intrusion detection also has a significant impact on how much memory is used, how long it takes to use the model to make predictions, and how long it takes to train it. These four metrics are increasingly relevant as deployment is likely to occur either as part of an edge deployment, or on devices that have limited processing capacity.

Table 5 displays the outcomes of the comparison between the four machine learning classifiers (XGBoost, Logistic Regression, Naive Bayes, SVM, and MLP) and the computational footprint of the classifiers. The results include several metrics, including model size (in Bytes), total processing time (in seconds), and average prediction time (in microseconds).

Table 5. Performance Comparison of ML Models: Memory Usage, Execution Time, And Inference Delay.

ML Classifier	Model Size (MB)	Duration of Processing (s)	Mean Prediction (ms/Samples)
XGBoost	1.68	20.08	0.189
LR	0.002	132.03	0.0007
NB	0.0036	0.13	6.65
SVM	21.15	386.35	11,793.12
MLP	0.073	1.80	0.059

The XGBoost model has an equally distributed computational cost, characterized by a reasonable size of 1.68 MB, execution time of 20.08 s, and a minimal prediction time of 0.0189 ms per test sample. This makes XGBoost ideal for implementing intrusion detection in real-time since the algorithm is capable of achieving very high levels of precision without sacrificing its predictive performance. The use of decision trees and the boosting method helps XGBoost capture the underlying relations in structured IoT data.

The Multilayer Perceptron (MLP) demonstrates moderate computational efficiency, with a model size of 0.073 MB, a processing time of 1.81 s, and an average prediction latency of 0.0592 ms per sample. Although the MLP remains relatively lightweight and faster to train than more complex deep learning models, its inference latency is higher than that of XGBoost, and its overall detection performance is lower in this study.

Logistic Regression proves to be very efficient when it comes to consuming computer memory since its model is rather small (0.002 MB), and the inference latency of this algorithm is exceptionally low (0.0007 ms). On the other hand, Logistic Regression takes an exceptionally long training period (132.03 s), thus proving inefficient for frequently changing environments.

Naive Bayes possesses the least computational complexity among all the algorithms under consideration. It is characterized by a rather small model (0.0036 MB), a short processing time (0.13 s), and a prediction latency that is comparatively low (0.00665 ms). As can be seen from above, Naive Bayes is a good choice for constrained environments, but it performs badly at detecting.

SVM presents the highest computational overhead, with a large model size of 21.15 MB, a long processing time of 416.05 s, and a very high prediction latency of 11.793 ms per sample. This is mainly due to the kernel-based computations required to model non-linear decision boundaries. As a result, SVM is less appropriate for real-time intrusion detection in IoT environments where low latency is critical.

Further tests for evaluating the generalization ability of the suggested method were carried out by employing the CICIoT2023 dataset. It is worth mentioning that this dataset represents a set of attacks of more diversity and complexity than the previously used N-BaIoT one, and therefore, this test case provides reliable results concerning the robustness of the created model. The analysis shows that the suggested AE + XGBoost classifier reaches the accuracy rate of 98.94%. Moreover, high precision, recall, and F1-score measures were achieved. As follows, the developed model retains its high effectiveness when applied to other datasets with different features. It should be noted that in per-class testing, high values were obtained, with the F1-score exceeding 0.90 for all the categories. Some changes in performance might be noticed due to higher complexity and similarity between some attacks. Overall, such findings indicate the robustness and high generalization potential of the suggested framework. To receive detailed information about the classification process of the proposed Intrusion Detection Model, it is necessary to analyze its confusion matrix. The confusion matrix illustrated in Figure 7 demonstrates all the predicted class-to-class values on the test dataset and reveals the efficiency of attack detection.

In order to assess whether the proposed hybrid model is more effective than existing state-of-the-art approaches that were evaluated on the N-BaIoT dataset, we conducted a comparison with other similar approaches using the same data set. Table 6 presents key performance metrics obtained from previous studies (BiLSTM-CNN [42], FL-CNN [43], AE + RF [44], and DNN [45]) and indicates that our model, which combines an autoencoder and XGBoost, exceeded all other competitors on every metric used to evaluate them, demonstrating superior performance for classification accuracy, precision, recall and F1-score. In addition, this comparison further supports the ability of our approach to successfully withstand different types of attacks within an IoT network environment and generalize well across multiple attack types.

Table 6. Comparison of IDS approaches on the N-BIoT dataset.

Reference	Method(s)	Dataset Used	Performance Metrics
[42]	BiLSTM-CNN	N-BIoT	Accuracy: 89.79%; Precision: 93.92%; Recall: 85.73%; F1-score: 89.11%.
[43]	FL-CNN	N-BIoT	Accuracy: 89.97%; Precision: 94.35%; Recall: 94.45%; F1-score: 87%.
[44]	AE + RF	N-BIoT	Accuracy: 96.2%; Precision: 89%; Recall: 97%; F1-score: 98%.
[45]	DNN	N-BIoT	Accuracy: 97.21%; Precision: 91.41%; Recall: 87.31%; F1-score: 88.48%.
Our Study	AE + XGBoost	N-BIoT	Accuracy: 99.63%; Precision: 99.64%; Recall: 99.63%; F1-score: 99.63%.

For practical IoT intrusion detection, it is important to consider where and how the proposed model can be deployed. Owing to the constrained computational resources of many IoT end devices, the AutoBoost-IoT framework is not intended to run directly on individual sensors or embedded devices. Instead, it is more suitable for deployment at the IoT gateway, edge server, or a cloud-assisted monitoring layer, where sufficient computational resources are available. Under normal circumstances in a deployment process, the data created by the IoT devices will be first collected at the edge node or gateway node. The extracted data will undergo analysis using the pre-trained autoencoder in order to create the latent representations which will be used for classification through the XGBoost algorithm. For ensuring sustained performance in the future, there should be an inclusion of performance monitoring techniques for measuring parameters like detection precision, false positives, and class-wise results. As changes occur in the network behavior, called concept drift, the performance of the algorithm might be affected in the future. For that reason, the model must be periodically retrained using new data. The training phase can be executed in an off-line manner using advanced computation resources. Once trained, the new model can then be uploaded to the monitoring node. Overall, this deployment strategy allows the proposed framework to balance detection performance, computational efficiency, and practical feasibility in real-world IoT environments.

5. Conclusions and Future Research Directions

In this study, a hybrid intrusion detection framework for IoT networks was developed by combining autoencoder-based unsupervised feature learning with XGBoost for multi-class classification. The autoencoder enables the transformation of high-dimensional network traffic into compact latent representations while maintaining fundamental behavioral characteristics, thereby reducing input dimensionality. These latent features are subsequently used by the XGBoost classifier to effectively distinguish between benign traffic and multiple attack types. The efficacy of the suggested framework has been analyzed using several IoT datasets, such as N-BaIoT and CICIoT2023. The results show that the accuracy reached 99.63% and 98.94% on N-BaIoT and CICIoT2023 datasets correspondingly. In addition, good precision, recall, and F1-score were observed among other measures, indicating excellent results. This fact highlights the reliability and generalizability of the proposed model for different IoT devices. The presence of inference latency analysis and memory footprint measurement proves the low weight and applicability of the model for real-time IoT threat detection due to the lack of resources. Moreover, the use of the suggested hybrid approach is justified from the viewpoint of efficiency since it demonstrates better results compared to complex neural network architectures with high computational costs. In the future, it may be beneficial to develop the suggested model using XAI methods, applying online learning algorithms, and testing the model in real-world distributed environments.