Hybrid Deep Architectures in Contrastive Latent Space: Performance Analysis of VAE-MLP, VAE-MoTE, and VAE-GAT for IoT Botnet Detection

Abstract

The rapid proliferation of Internet of Things (IoT) devices has significantly expanded the attack surface of modern networks leading to a surge in IoT-based botnet attacks. Detecting such attacks remains challenging due to the high dimensionality and heterogeneity of IoT network traffic. This study proposes and evaluates three hybrid deep learning architectures for IoT botnet detection that combine representation learning with supervised classification: VAE-encoder-MLP, VAE-encoder-GAT, and VAE-encoder-MoTE. A Variational Autoencoder is initially trained to learn a compact latent representation of the high-dimensional traffic features. Subsequently, the pretrained VAE-encoder component is employed to project the data into a lower-dimensional embedding space. These embeddings are then used to train three different downstream classifiers: a multilayer perceptron (MLP), a graph attention network (GAT), and a mixture of tiny experts (MoTE) model. To further enhance representation discriminability, supervised contrastive learning is incorporated to encourage intra-class compactness and inter-class separability. The proposed architectures are evaluated on two widely studied benchmark datasets—the CICIoT2022 and N-BaIoT dataset—under both binary and multiclass classification settings. Experimental results demonstrate that all three models achieve near-perfect performance in binary attack detection, with accuracy exceeding 99.8%. In the more challenging multiclass scenario, the VAE-encoder-MLP model achieves the best overall performance, reaching accuracies of 98.55% on CICIoT2022 and 99.75% on N-BaIoT. These findings provide insights into the design of efficient and scalable deep learning architectures for IoT intrusion detection.

1. Introduction

The Internet of Things (IoT) is now an indispensable component of contemporary civilization, underpinning a multitude of applications at both individual and enterprise levels. However, despite the widespread adoption, the IoT network remains highly susceptible to cyberattacks due to inherent device constraints such as limited computational resources, weak security configurations, and heterogeneous deployment environments. In particular, IoT botnet-driven attacks, most notably DDoS attacks, continue to present major security risks to enterprises and institutions across diverse industry verticals. To mitigate these challenges, a broad spectrum of artificial intelligence (AI)-based attack detection strategies have been introduced and extensively evaluated to strengthen the resilience of IoT ecosystems. For example, prior studies have explored advanced AI-driven techniques based on federated learning [1,2,3], graph neural networks [4,5,6], transformer-based architectures [7,8,9], autoencoder and variational autoencoder models [10,11,12], latent space arithmetic and alignment-based methods [13,14,15], and cost-sensitive learning strategies designed to address class imbalance in IoT security datasets [16,17,18,19].

However, despite its demonstrated potential for improving computational efficiency in domains such as NLP and LLMs [20,21,22], Mixture of Experts (MoE) [23,24] methods remain relatively unexplored in the context of IoT botnet detection. In the MoE design, computation is dynamically routed such that only a small number of expert networks are utilized for each data instance. To ensure feasibility on low-resource IoT edge devices, we adopt the mixture of tiny experts (MoTE) [25], a variant that leverages lightweight expert models for per-instance classification. This mechanism substantially reduces inference latency, energy consumption, and memory overhead, while preserving strong representational capacity through expert specialization.

This work introduces the VAE-encoder-MoTE framework in which a VAE encoder first transforms high-dimensional IoT botnet traffic features into a compact latent embedding space. The MoTE model is then trained on these low-dimensional embeddings, enabling efficient learning while retaining the critical structural information necessary for accurate attack detection. This integration of latent representation learning with conditional expert activation achieves a strong trade-off across detection accuracy and computational cost, positioning the method as well-suited for deployment on IoT systems with limited resources.

To further improve the discriminative quality of the learned representations, supervised contrastive learning [26,27] is incorporated during model training. This training approach explicitly promotes instances from the same attack category to form tight groupings in the latent representation, while ensuring that instances from different categories are separated by larger distances. As a result, the learned feature space exhibits improved intra-class compactness and inter-class separability, which enhances generalization capability, robustness to class imbalance, and resilience to evolving IoT traffic patterns. In addition to the proposed VAE-encoder-MoTE architecture, two alternative models, namely VAE-encoder-MLP and VAE-encoder-GAT, are also trained and evaluated. A comparative analysis of the three architectures is conducted using standard performance metrics.

The main contributions of this study are:

• A hybrid framework that combines VAE-based latent representation learning with an MoTE architecture for IoT botnet detection is introduced. The approach compresses high-dimensional IoT traffic data into compact embeddings and applies conditional expert routing to enable efficient and scalable detection.
• The study incorporates supervised contrastive learning to enhance the discriminative power of the latent feature space. By encouraging embeddings of samples from the same attack class to cluster together while separating different classes, the approach improves intra-class compactness and inter-class separability.
• A systematic empirical evaluation is conducted comparing the proposed VAE-encoder-MoTE model against the VAE-encoder-MLP and VAE-encoder-GAT models. The comparison assesses their effectiveness under both binary and multiclass IoT botnet detection scenarios. Performance is analyzed using multiple metrics providing a detailed understanding of the strengths and limitations of each architectural design.
• Through extensive empirical validation on benchmark IoT botnet datasets, the study provides insights into how architectural choices and dataset characteristics influence detection performance in IoT intrusion detection systems.

The remainder of this paper is organized as follows. Section 2 reviews the relevant literature on supervised contrastive learning, mixture-of-experts architecture, and GNNs for attack detection. Section 3 presents the proposed framework, including the VAE-based representation learning approach, the MoTE architecture, and the experimental setup. Section 4 reports the experimental results obtained under both binary and multiclass classification settings. Section 5 provides an in-depth analysis and interpretation of the findings. Section 6 summarizes the key contributions and outlines prospects for future work.

2. Related Work

This section surveys existing literature relevant to three key areas: (1) supervised contrastive learning, (2) mixture of experts and tiny expert architectures, (3) graph neural networks for intrusion detection.

2.1. Supervised Contrastive Learning

Contrastive learning is a representation learning framework where models are optimized to reduce the separation between related samples (positive pairs) while maximizing the separation between unrelated samples (negative pairs) in an embedding space. Supervised contrastive learning broadens this framework by leveraging label information to establish richer semantic relationships between samples, unlike purely unsupervised approaches that rely only on data augmentation without considering class labels [26,28]. By encouraging examples from the same category to be drawn closer together in the learned representation space, supervised contrastive objectives often produce features that are more discriminative and transferable for classification tasks [26,29]. In comparison with traditional supervised objectives such as cross-entropy, supervised contrastive learning has yielded better resilience to data augmentation, label noise, and dataset biases [26,30]. Furthermore, contrastive objectives tend to yield smoother and better-structured representation spaces, which are advantageous for transfer learning and downstream applications [28,31].

The study in [27] investigated the geometric structure of representations learned using supervised contrastive loss in comparison with those obtained through conventional cross-entropy training. The authors analyze how both loss functions encourage the formation of compact class clusters within the embedding space and theoretically demonstrate that optimal representations under both objectives converge to vertices forming a regular structure positioned over a high-dimensional sphere. Under this configuration, samples from the same class become tightly grouped while samples from different classes remain maximally separated. The study also provided empirical evidence linking this geometric structure to improved generalization performance.

The work presented in [32] proposes a weakly supervised contrastive learning framework designed to bridge instance discrimination with supervised information. Rather than treating each sample as an independent class, the proposed method introduces two projection heads: one dedicated to conventional instance discrimination and another that utilizes graph-based similarity relationships to assign weak labels across samples. Samples sharing these weak labels are considered positive pairs under a supervised contrastive objective, thereby encouraging the learning of semantically meaningful representations. To further enrich positive sample diversity, the framework incorporates a k-nearest neighbor multi-crop strategy. Experiments conducted on standard computer vision benchmarks demonstrated improvements in representation quality and competitive semi-supervised classification performance when only limited labeled data are available.

With respect to cybersecurity applications—particularly intrusion detection—the study in [33] introduced FeCo (Federated Contrastive Learning), a framework that leverages contrastive objectives to align feature embeddings of network traffic collected from distributed IoT devices. The approach coordinates multiple local models so that they learn shared representations capable of distinguishing normal network behavior from malicious activity by leveraging both feature and label similarities during contrastive training. Through the federated contrastive loss, the encoder learns an embedding space where semantically related network traffic patterns are positioned closer together, thereby improving the effectiveness of intrusion detection across heterogeneous deployment environments.

The research in [34] further extends contrastive learning principles to sequence modeling and transformer-based architectures. In this approach, sequences of network events are encoded using transformer models, and contrastive learning is employed to differentiate benign from malicious patterns. Positive pairs are constructed from similar event sequences, whereas negative pairs represent dissimilar sequences. This supervised contrastive framework improves the discriminative capacity and robustness of intrusion detection systems, particularly in settings where annotated data is limited, highlighting the adaptability of contrastive approach beyond traditional computer vision domains.

2.2. Mixture of Experts and Tiny Expert Architectures

The MoE framework represents a form of conditional computation in which a neural network is partitioned into multiple expert sub-networks that are orchestrated by a routing or gating mechanism. For any sample, a subset of experts is activated, enabling the model to increase its representational capacity while avoiding the full computational overhead associated with dense model inference. This strategy has been widely adopted to address scalability and efficiency challenges in deep learning, particularly in applications requiring large model capacity or dealing with heterogeneous input distributions [35,36].

Recent developments have introduced the concept of tiny experts, where each expert module is deliberately designed to be extremely lightweight. These experts often consist of minimal parameter structures such as shallow multilayer perceptrons or even simple single-layer transformations. The work in [25] formalizes this concept through the Mixture of a Million Experts framework, demonstrating that a very large collection of small experts can be efficiently accessed using parameter-efficient retrieval strategies while activating only a limited subset for each input. Related research has further explored fine-grained expert decomposition and sparse activation mechanisms to minimize redundancy and enhance parameter efficiency [37,38,39].

Tiny expert architectures provide several advantages compared with conventional MoE models, particularly in environments with constrained computational resources such as IoT edge devices. Traditional MoE designs typically employ moderately large expert networks, which can result in increased memory consumption and higher inference latency, making them unsuitable for resource-limited edge hardware. In contrast, architectures composed of numerous tiny experts enable highly selective activation, lower computational cost per expert, and greater flexibility in adapting to dynamic traffic patterns. These characteristics make MoTE particularly suitable for lightweight intrusion and botnet detection systems [40,41,42,43,44]. Such properties are critical in IoT security applications, where real-time threat detection, low power consumption, and resilience against evolving attack strategies are essential.

2.3. Graph Neural Networks for Intrusion Detection

Recent research has focused on combining graph-based neural methods with transformer-style attention mechanisms to strengthen intrusion detection. Such combined frameworks leverage the distinct yet complementary advantages of both approaches: graph neural models are well-suited for learning intricate relationships between interconnected nodes, whereas transformer architectures are highly effective at capturing long-range dependencies within data. Bringing these techniques together has been found to boost both the accuracy and resilience of detection systems. For example, the study in [45] introduced a graph attention network-driven intrusion detection framework specifically designed for diverse IoT settings, achieving encouraging results on the NSL-KDD dataset. Similarly, [46] developed a hybrid architecture that integrates a GNN with a transformer to jointly capture structural and contextual dependencies, while [47] combined graph convolution with attention mechanisms to detect IoT botnets through device interaction modeling. In the context of electric vehicle charging networks, [48] demonstrated that integrating transformer components with GNNs improves the modeling of complex feature relationships for cyber-attack detection.

Several studies have concentrated on developing sophisticated GNN architectures tailored for IoT security. For instance, the EGAT-LSTM model proposed in [49] integrates an improved graph attention mechanism alongside a LSTM component, enabling it to learn both structural dependencies and evolving traffic dynamics over time. This integration significantly improves malicious traffic classification compared with approaches that rely solely on individual flow features. Similarly, [50] proposed AJSAGE, an attention-enhanced GraphSAGE model designed to improve anomalous node detection in graph-structured network attack datasets, particularly for complex and evolving threats.

Dimensionality reduction techniques have also been examined as complementary strategies for enhancing GNN-based detection performance. The study in [12] compared AE-encoder, VAE-encoder, and PCA methods for reducing the high-dimensional CICIoT2022 into a low dimensional space before graph construction. Among these approaches, the VAE-encoder produced the best results. In addition, the 3-euclidean n_neighbors-metric configuration achieved the strongest performance for kNN-based graph generation, emphasizing the importance of both high-quality latent representations and appropriate graph construction parameters.

Despite these advances, comparative evaluations suggest that GNN-based models still require further refinement to achieve competitive performance. In [4], the authors compared VAE-GCN and VAE-GAT against VAE-MLP and ViT-MLP models utilizing the 8-dimensional latent vectors derived from the N-BaIoT dataset. Although VAE-GAT demonstrated better performance than VAE-GCN, both GNN models generally underperformed in comparison with the alternative learning frameworks. These findings highlight the need for improved architectural designs and more effective integration strategies when applying GNNs to intrusion detection systems.

3. Methodology

This study proposes the VAE-encoder-MoTE model and systematically evaluates its detection performance against two other advanced hybrid deep learning architectures—VAE-encoder-MLP and VAE-encoder-GAT—for IoT botnet detection under both binary and multiclass classification settings. To boost the discriminative quality of the learned representations, supervised contrastive learning is employed to promote intra-class compactness and inter-class separability, thereby enhancing detection performance.

To tackle the complexity of high-dimensional IoT traffic data, a VAE with a latent space of dimension k (set to k = 8 [4,9,12] in this study) is initially trained on the original feature space. After training, only the encoder part of the VAE is preserved. For each model architecture, this pretrained encoder maps the inputs into compact, lower-dimensional vectors. The resulting reduced-dimension training data is subsequently used to train an MLP, a GAT, and an MoTE model under a supervised contrastive learning framework.

3.1. Variational Autoencoder for Dimensionality Reduction

Dimensionality reduction has become a key strategy for building efficient detection systems. Common approaches include feature selection methods (filter, wrapper), PCA, and deep learning techniques such as Auto-encoders (AEs) and Variational Auto-encoders (VAEs). Filter methods apply statistical measures to score and prioritize features, while wrapper methods iteratively evaluate model performance with different feature subsets. PCA reduces dimensionality by projecting data onto directions of maximum variance. However, these techniques have limitations: filter methods may produce suboptimal subsets, wrapper methods are computationally expensive, and PCA ignores class labels and struggles with nonlinear data. AEs offer an alternative by learning compact latent representations, but their lack of explicit regularization can lead to poorly structured latent spaces, reducing generalization performance and interpretability. Therefore, the VAE was selected for this task.

The VAE, originally proposed in [51], is a generative model that introduces a structured and regularized latent space, distinguishing it from traditional autoencoders. Given a dataset $\mathbf{X}$, the generative process assumes a variable z drawn from a prior $p_{\theta }\left(z\right)$, followed by sampling from the conditional distribution $p_{\theta }\left(x\right|z)$. Because direct optimization of $\log p_{\theta }\left(x\right)$ is computationally intractable, the true posterior is approximated using a variational distribution $q_{\varphi }\left(z\right|x)$, parameterized by an encoder network. The model is then trained by maximizing the evidence lower bound (ELBO), as expressed in Equation (1).

$$\mathcal{L}(\theta ,\varphi ;x)={\mathbb{E}}_{q_{\varphi }\left(z\right|x)}\left[\log p_{\theta }\left(x\right|z)\right]-D_{KL}(q_{\varphi }\left(z\right|x)\parallel p_{\theta }\left(z\right))$$

(1)

Here, the Kullback–Leibler (KL) divergence term $D_{KL}(q_{\varphi }\left(z\right|x)\parallel p_{\theta }\left(z\right))$ acts as a regularizer, encouraging the learned latent distribution to stay close to the prior. As a result, the VAE simultaneously learns a compact latent representation and a probabilistic generative model, supporting both accurate reconstruction and data generation under uncertainty.

In this study, both the encoder and decoder networks comprised four fully connected hidden layers. The encoder employed a decreasing architecture with 128, 64, 32, and 16 neurons to transform the input into the latent representation, whereas the decoder adopted a symmetric increasing structure with 16, 32, 64, and 128 neurons to reconstruct the input from the latent representation. The LeakyReLU activation function was applied across all hidden layers.

3.2. Contrastive Learning

To promote tighter grouping of instances belonging to the same class and clearer separation across different classes in the learned embedding space, supervised contrastive learning is adopted. Given a batch B of feature embeddings ${\left\{z_i\right\}}_{i=1}^B$, where $z_i\in {\mathbb{R}}^k$, ($k\to \mathrm{latent}\mathrm{space}\mathrm{dimension}$), the embeddings are first ${\mathcal{l}}_2$-normalized using Equation (2):

$${\tilde{z}}_i={\displaystyle \frac{z_i}{\parallel z_i{\parallel }_2}}$$

(2)

The pairwise cosine similarity between samples ${\tilde{z}}_i$ and ${\tilde{z}}_j$ is computed using Equation (3):

$$\mathrm{sim}({\tilde{z}}_i,{\tilde{z}}_j)={\displaystyle \frac{{\tilde{z}}_i^{\top }{\tilde{z}}_j}{\tau }}$$

(3)

where $\tau$ is a temperature scaling parameter.

The supervised contrastive loss for a sample i is obtained using Equation (4):

$${\mathcal{L}}_{\mathrm{SupCon}}^{\left(i\right)}=-{\displaystyle \frac{1}{\left|\mathcal{P}\right(i\left)\right|}}\sum _{p\in \mathcal{P}\left(i\right)}\log {\displaystyle \frac{\exp \left(\mathrm{sim}\right(i,p\left)\right)}{{\sum }_{a\ne i}\exp \left(\mathrm{sim}(i,a)\right)}}$$

(4)

where

$\mathcal{P}\left(i\right)$ denotes the set of indices corresponding to samples that share the same class label as sample i.

$sim(i,p)$: similarity between sample i and positive sample p.

a: any sample in the batch (both positives and negatives).

The final supervised contrastive loss is obtained by averaging over the batch using Equation (5):

$${\mathcal{L}}_{\mathrm{SupCon}}={\displaystyle \frac{1}{B}}{\displaystyle \sum _{i=1}^B}{\mathcal{L}}_{\mathrm{SupCon}}^{\left(i\right)}$$

(5)

3.3. Tiny Experts Architecture

Each tiny expert takes the form of a lightweight MLP featuring two dense layers (24 and 16 units) with ReLU activations. This compact architectural design reduces computational overhead while preserving sufficient representational capacity for effective feature transformation. The transformation performed by an expert is given by Equation (6):

$$\mathrm{Expert}\left(x\right)=\sigma (W_2\sigma (W_{1}x+b_1)+b_2)$$

(6)

where $W_1$, $W_2$ are weight matrices, $b_1$ and $b_2$ are bias terms, $\sigma (·)$ denotes the ReLU activation function.

3.4. Mixture of Tiny Experts (MoTE)

The MoTE model introduces conditional computation through a trainable routing mechanism between a set of tiny experts. Given an input x, the router produces a probability distribution over E experts:

$$g\left(x\right)=\mathrm{softmax}(W_{r}x+b_r)$$

(7)

where $g\left(x\right)\in {\mathbb{R}}^E$, and $E\to$ number of experts.

For computational efficiency, activation is limited to the k experts with the highest routing probabilities for any given input. The aggregated representation results from weighting and summing the outputs of the selected experts as in Equation (8):

$$z=\sum _{i\in \mathcal{K}\left(x\right)}{g}_i\left(x\right)·{\mathrm{Expert}}_i\left(x\right)$$

(8)

where $\mathcal{K}\left(x\right)$ denotes the set of top-k selected experts.

3.5. Classification Head and Joint Optimization

The aggregated embedding z is passed to a linear classifier to obtain class predictions:

$$\hat{y}=\mathrm{softmax}(W_{c}z+b_c)$$

(9)

The supervised classification objective is defined using the cross-entropy loss:

$${\mathcal{L}}_{\mathrm{CE}}=-{\displaystyle \sum _{c=1}^C}{y}_c\log \left({\hat{y}}_c\right)$$

(10)

To jointly optimize representation learning and classification performance, the total training objective was formulated as follows:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{SupCon}}+{\mathcal{L}}_{\mathrm{CE}}$$

(11)

Figure 1 provides an illustration of the complete architectural design of the VAE-encoder-MoTE model. A supervised contrastive learning layer is incorporated to enhance class separability within the latent space by drawing samples of the same category nearer to each other, while simultaneously pushing samples from different categories farther away from each other.

3.6. The MLP Architecture

The MLP architecture comprises four hidden dense layer (128 → 64 → 32 → 16 units) with ReLU activation. The output layer utilized the softmax activation function to produce normalized class probability distributions. To protect the model from overfitting, regularization was introduce via a 0.1 dropout rate after the third and fourth layers. The output from the VAE-encoder component was passed through the contrastive learning layer and then fed into the MLP model as shown in Figure 2.

3.7. The GAT Architecture

GAT is a GNN model that incorporates the self-attention mechanism to learn the relative contribution of neighboring nodes. Because GNNs operate on graph-structured inputs, the low-dimensional embeddings of the IoT traffic instances were first transformed into a graph structure. To construct the graph and determine node neighborhoods, the approach proposed in [4] was adopted. Specifically, the k-NN algorithm with $n\_neighbors=3$, and $euclidean$ distance metric was used to establish connections between nodes. The graph was constructed on the embeddings produced from the contrastive learning layer which was placed between the VAE-encoder and the kNN algorithm as shown in Figure 3.

3.8. Datasets

The three model architectures were trained and evaluated using two well-studied benchmark datasets—the N-BaIoT dataset [10] and the CICIoT2022 dataset [52].

3.8.1. N-BaIoT Dataset

To overcome the limited availability of real-world IoT botnet traffic, the N-BaIoT dataset was introduced by [10] as a comprehensive benchmark derived from operational IoT devices. The dataset contains 115 features, computed as 23 statistical descriptors across five temporal windows, extracted from NetFlow traffic captured in a controlled yet realistic testbed environment comprising of nine devices. The devices were configured to run two prominent IoT malware families, Mirai and BashLite. Following preprocessing to remove duplicate records, the dataset was reduced from 6,331,884 to 2,482,470 instances and subsequently employed to train and evaluate each of the three model architectures under binary classification, and fine-grained ten-class classification settings, with percentage class distributions of {“Normal”: 21.52%, “mirai_udp”: 23.30%, “mirai_syn”: 13.29%, “mirai_ack”: 11.74%, “mirai_scan”: 10.74%, “gafgyt_udp”: 4.51%, “gafgyt_combo”: 2.61%, “gafgyt_junk”: 1.31%, “gafgyt_scan”: 1.30%, “mirai_udpplain”: 9.66%}.

3.8.2. CICIoT2022 Dataset

In contrast to N-BaIoT, which focuses on a limited number of devices, the CICIoT2022 dataset [52] provides a substantially broader representation of IoT ecosystems by incorporating traffic from 60 heterogeneous devices. This increased device and protocol diversity facilitates a more comprehensive characterization of IoT traffic behavior under benign and malicious conditions. Feature extraction was performed on the released .pcap files using the revised CICFlowMeter (https://github.com/GintsEngelen/CICFlowMeter accessed on 23 August 2025). The resulting dataset was composed of more than 3.21 million data samples and 84 training features. The dataset was annotated using directory-based labels, resulting in five traffic classes. The sample distribution across the five classes is {“Normal”: 80.870%, “HTTP flood”: 17.130%, “TCP flood”: 1.418%, “Brute force”: 0.379%, “UDP flood”: 0.203%}.

3.9. Data Preprocessing

Data preprocessing constituted refining the datasets through the removal of outliers, duplicate records, invalid samples, and missing values. In addition, non-informative attributes were excluded, including “Flow ID”, which served as a unique ID for each NetFlow record; “Src IP” and “Dst IP”, which associated traffic with specific source and destination addresses; and “Timestamp”, which linked NetFlows to temporal information. Subsequently, the training variables were scaled to take on values in the range of 0 to 1.

3.10. Experimental Setup and Model Training

The training process spanned 30 epochs, utilizing the Adam optimizer with a learning rate set to ${10}^{-3}$. Via empirical analysis the final value of temperature parameter $\tau$ in the supervised contrastive loss was $0.1$. The architecture employs six experts with top-2 expert selection per input sample. The dataset was divided into train (80%) and test (20%) sets while a batch size of 128 was used during model training for all model architectures.

To mitigate the risk of performance overestimation associated with a single train–test split, all experiments were conducted over 10 independent runs for each model and classification task. In each run, the dataset was randomly shuffled prior to splitting, with a different random seed applied to ensure variability across splits. The set of seeds used was 10, 42, 116, 17, 37, 1412, 73, 100, 98, 3200. For every experimental configuration, the average and variance of each performance metric were computed and reported.

The training procedure was conducted in two stages. First, the VAE encoder and the contrastive learning layer were trained to learn a structured latent representation. Subsequently, each classifier was trained independently using the learned embeddings. During the classifier training phase, the parameters of the pretrained VAE encoder and the contrastive learning layer were frozen, ensuring that only the classifier parameters were updated.

3.11. Performance Evaluation Metrics

The performance was analyzed based on typical multi-class classification measures: accuracy, precision, recall, and F1-score, while barplots were utilized to provide visual insights of how each model performs in comparison with other models. A key point to consider is that for multiclass classification macro averages are reported for precision, recall and F1-score. These metrics provided a comprehensive assessment of overall performance as well as class-wise behavior, which is critical for security-sensitive applications such as IoT botnet detection.

4. Results

This section presents the results from a comprehensive evaluation of the proposed approach across binary and multiclass classification tasks on the CICIoT2022 and N-BaIoT datasets. The performance of the three models—VAE-Encoder-MoTE, VAE-Encoder-MLP, and VAE-Encoder-GAT—is reported before and after the application of contrastive learning. All results represent the mean performance obtained across several independent experiments conducted using varying data partitions and initialization seed values, and the outcomes are reported using mean score ± square root of variance. The highest values of accuracy, precision, recall, and F1-score obtained for each experimental setting are presented in boldface in the corresponding tables.

4.1. Binary Classification on the N-BaIoT Dataset

Table 1. Binary classification performance on the N-BaIoT dataset before and after application of contrastive learning.

	Accuracy (%)		Precision (%)		Recall (%)		F1-Score (%)
Model	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL
VAE-MLP	$99.91\pm 0.07$	$99.94\pm 0.02$	$99.90\pm 0.04$	$99.92\pm 0.02$	$99.90\pm 0.04$	$99.92\pm 0.03$	$99.88\pm 0.01$	$99.91\pm 0.01$
VAE-GAT	$99.50\pm 0.12$	$99.80\pm 0.08$	$99.17\pm 0.11$	$99.82\pm 0.09$	$99.01\pm 0.10$	$99.84\pm 0.08$	$99.08\pm 0.06$	$99.75\pm 0.05$
VAE-MoTE	$\mathbf{99}.\mathbf{92}\pm \mathbf{0}.\mathbf{01}$	$\mathbf{99}.\mathbf{97}\pm \mathbf{0}.\mathbf{01}$	$\mathbf{99}.\mathbf{94}\pm \mathbf{0}.\mathbf{02}$	$\mathbf{99}.\mathbf{96}\pm \mathbf{0}.\mathbf{03}$	$\mathbf{99}.\mathbf{91}\pm \mathbf{0}.\mathbf{04}$	$\mathbf{99}.\mathbf{95}\pm \mathbf{0}.\mathbf{01}$	$\mathbf{99}.\mathbf{92}\pm \mathbf{0}.\mathbf{03}$	$\mathbf{99}.\mathbf{95}\pm \mathbf{0}.\mathbf{01}$

summarizes the empirical findings for all models on the N-BaIoT dataset for the binary classification task, evaluated before and after the application of contrastive learning. The comparative outcomes are presented in Figure 4a and Figure 4b, showing performance prior to and after the incorporation of contrastive learning, respectively.

The results indicate that the application of contrastive learning consistently improves performance across all models, with the VAE-Encoder-MoTE model achieving the best overall performance.

4.2. Binary Classification on the CICIoT2022 Dataset

Table 2. Binary classification performance on the CICIoT2022 dataset before and after application of contrastive learning.

	Accuracy (%)		Precision (%)		Recall (%)		F1-Score (%)
Model	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL
VAE-MLP	$\mathbf{99}.\mathbf{84}\pm \mathbf{0}.\mathbf{12}$	$99.86\pm 0.06$	$99.50\pm 0.11$	$99.79\pm 0.07$	$99.63\pm 0.13$	$99.81\pm 0.06$	$99.57\pm 0.11$	$99.79\pm 0.06$
VAE-GAT	$99.20\pm 0.10$	$99.24\pm 0.04$	$99.00\pm 0.11$	$99.72\pm 0.03$	$99.10\pm 0.10$	$99.36\pm 0.04$	$99.05\pm 0.09$	$99.29\pm 0.04$
VAE-MoTE	$99.81\pm 0.11$	$\mathbf{99}.\mathbf{93}\pm \mathbf{0}.\mathbf{02}$	$\mathbf{99}.\mathbf{77}\pm \mathbf{0}.\mathbf{06}$	$\mathbf{99}.\mathbf{87}\pm \mathbf{0}.\mathbf{06}$	$\mathbf{99}.\mathbf{69}\pm \mathbf{0}.\mathbf{07}$	$\mathbf{99}.\mathbf{82}\pm \mathbf{0}.\mathbf{04}$	$\mathbf{99}.\mathbf{73}\pm \mathbf{0}.\mathbf{06}$	$\mathbf{99}.\mathbf{80}\pm \mathbf{0}.\mathbf{04}$

summarizes the results on the CICIoT2022 dataset. The corresponding performance comparisons are illustrated in Figure 5a and Figure 5b, which depict the results prior to and after the incorporation of contrastive learning, respectively.

4.3. Multiclass Classification on the N-BaIoT Dataset

Table 3. Multiclass classification performance on the N-BaIoT dataset before and after application of contrastive learning.

	Accuracy (%)		Precision (%)		Recall (%)		F1-Score (%)
Model	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL
VAE-MLP	$\mathbf{99}.\mathbf{51}\pm \mathbf{0}.\mathbf{07}$	$99.84\pm 0.04$	$\mathbf{99}.\mathbf{53}\pm \mathbf{0}.\mathbf{22}$	$99.87\pm 0.06$	$99.42\pm 0.12$	$99.85\pm 0.03$	$\mathbf{99}.\mathbf{42}\pm \mathbf{0}.\mathbf{08}$	$\mathbf{99}.\mathbf{86}\pm \mathbf{0}.\mathbf{08}$
VAE-GAT	$95.12\pm 0.13$	$96.87\pm 0.08$	$93.11\pm 0.12$	$96.13\pm 0.09$	$94.82\pm 0.10$	$96.79\pm 0.08$	$94.92\pm 0.13$	$96.75\pm 0.05$
VAE-MoTE	$99.38\pm 0.01$	$\mathbf{99}.\mathbf{87}\pm \mathbf{0}.\mathbf{02}$	$99.36\pm 0.04$	$\mathbf{99}.\mathbf{88}\pm \mathbf{0}.\mathbf{03}$	$\mathbf{98}.\mathbf{59}\pm \mathbf{0}.\mathbf{04}$	$\mathbf{99}.\mathbf{86}\pm \mathbf{0}.\mathbf{01}$	$98.36\pm 0.03$	$99.85\pm 0.01$

reports the multiclass classification results on the N-BaIoT dataset. The corresponding performance comparisons are illustrated in Figure 6a and Figure 6b, which depict the results prior to and after the incorporation of contrastive learning, respectively.

The gains from contrastive learning are also evident in the multiclass setting, with improved classification consistency across all models.

4.4. Multiclass Classification on the CICIoT2022 Dataset

Table 4. Multiclass classification performance on the CICIoT2022 dataset before and after application of contrastive learning.

	Accuracy (%)		Precision (%)		Recall (%)		F1-Score (%)
Model	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL	W/o CL	With CL
VAE-MLP	$\mathbf{98}.\mathbf{21}\pm \mathbf{0}.\mathbf{25}$	$\mathbf{99}.\mathbf{03}\pm \mathbf{0}.\mathbf{12}$	$\mathbf{98}.\mathbf{78}\pm \mathbf{0}.\mathbf{27}$	$\mathbf{99}.\mathbf{35}\pm \mathbf{0}.\mathbf{13}$	$93.53\pm 0.26$	$96.40\pm 0.12$	$\mathbf{95}.\mathbf{22}\pm \mathbf{0}.\mathbf{24}$	$96.35\pm 0.12$
VAE-GAT	$97.93\pm 0.12$	$98.20\pm 0.10$	$97.25\pm 0.13$	$98.18\pm 0.11$	$83.47\pm 0.17$	$90.15\pm 0.10$	$88.91\pm 0.21$	$92.53\pm 0.10$
VAE-MoTE	$95.30\pm 0.23$	$98.90\pm 0.11$	$95.45\pm 0.24$	$98.70\pm 0.12$	$\mathbf{95}.\mathbf{10}\pm \mathbf{0}.\mathbf{23}$	$\mathbf{97}.\mathbf{83}\pm \mathbf{0}.\mathbf{11}$	$95.05\pm 0.22$	$\mathbf{97}.\mathbf{75}\pm \mathbf{0}.\mathbf{11}$

summarizes the multiclass performance outcomes on the CICIoT2022 dataset. The corresponding performance comparisons are illustrated in Figure 7a and Figure 7b, which depict the results prior to and after the incorporation of contrastive learning, respectively.

Consistent with previous observations, contrastive learning enhances model performance in the multiclass scenario, demonstrating its robustness across datasets and classification settings.

5. Discussion

The findings highlight the strong performance achieved through the integration of VAE-based latent representation learning with supervised contrastive learning for IoT botnet detection across both datasets and classification settings.

5.1. Impact of Contrastive Learning

A consistent improvement is observed after incorporating supervised contrastive learning across all models and tasks. In the binary setting (Table 1 and Table 2; Figure 4 and Figure 5), performance gains are modest due to already-saturated accuracy levels. However, reduced standard deviations indicate improved stability. In contrast, the multiclass setting (Table 3 and Table 4; Figure 6 and Figure 7) shows more substantial improvements, confirming that contrastive learning enhances class separability in complex classification scenarios.

5.2. Model Comparison

The VAE-encoder-MoTE model demonstrates a competitive performance across all tasks, achieving the highest or near-highest accuracy in binary classification (Table 1 and Table 2). Its low variance across runs indicates stable learning. In the multiclass setting, MoTE benefits significantly from contrastive learning, suggesting that expert specialization is more effective when the latent space is well-separated.

The VAE-encoder-MLP model achieves the best overall performance in multiclass classification, particularly on the CICIoT2022 dataset (Table 4). This suggests that well-structured latent representations can enable a strong performance even with relatively simple classifiers.

The VAE-encoder-GAT model consistently underperforms in multiclass scenarios (Table 3 and Table 4). This may be attributed to sensitivity to graph construction and the fixed k-NN strategy used, which may not adequately model intricate relationships within the dataset. Performance degradation is more evident on the CICIoT2022 dataset, indicating limited robustness under heterogeneous and imbalanced conditions.

5.3. Binary vs. Multiclass Performance

All models achieve near-perfect results in binary classification, indicating that detecting attack traffic from normal samples is relatively straightforward in the learned latent space. However, multiclass classification remains more challenging, with noticeable performance gaps between models. This highlights the importance of discriminative representation learning, particularly for fine-grained attack categorization.

5.4. Dataset Effects

Performance is consistently higher on the N-BaIoT dataset than on CICIoT2022. This suggests that N-BaIoT exhibits more separable class structures, whereas CICIoT2022 introduces greater heterogeneity and class imbalance, leading to reduced recall and F1-scores (Table 4). The MoTE model shows improved recall under these conditions, indicating its potential for handling diverse traffic patterns.

An investigation on which instances of the N-BaIoT dataset were misclassified in the multiclass classification setting, revealed that 684 instances of the “gafgyt_combo” attack category were wrongly classified as “gafgyt_junk” by the VAE-Encoder-MoTE model. Also, similar patterns were recorded using the other two models. Following this, t-SNE was utilized to obtain a graphical representation of the spatial distribution of three categories: correctly classified “gafgyt_combo” instances, correctly classified “gafgyt_junk” instances, and “gafgyt_combo” instances misclassified as “gafgyt_junk”. The test dataset was passed through the VAE-Encoder-MoTE model pipeline and the representations immediately preceding the output layer were extracted and mapped into a two-dimensional space using t-SNE. The t-SNE output is presented in Figure 8. A similar pattern was observed when investigating the misclassified instances on the CICIoT2022 dataset.

The t-SNE visualization reveals that the misclassified “gafgyt_combo” instances tend to concentrate in intermediate or overlapping regions between the primary clusters. These observations suggest two plausible causes of misclassification: (1) feature ambiguity—where the misclassified samples exhibit attack patterns, traffic statistics, or payload characteristics more closely aligned with “gafgyt_junk” than with “gafgyt_combo”, thereby necessitating additional feature engineering or the adoption of a more sophisticated detection model to effectively discriminate between the two classes; (2) traffic labeling errors [53]—where some traffic instances are incorrectly assigned to the wrong categories. This finding underscores the importance of evaluating deep learning models on multiple datasets to ensure more reliable and generalizable outcomes.

5.5. Practical Implications

From a deployment perspective, the findings reveal a compromise between model accuracy and resource utilization. Although the MLP model achieves the highest accuracy, the MoTE architecture offers an effective equilibrium between detection capability and computational overhead due to its conditional computation mechanism. This makes it a strong candidate for implementation in IoT settings with limited computational resources.

Overall, the findings confirm that combining VAE-based dimensionality reduction with supervised contrastive learning provides a robust framework for IoT botnet detection, with model selection depending on task complexity and deployment constraints.

5.6. Computational Cost Analysis

VAE encoder and contrastive layer time complexity: Both the VAEs and contrastive layer architectures in this study used dense layers as the core computational component. Therefore, the time complexity of a dense layer is with $x_{in}$ inputs and $y_{out}$ neurons is $\mathcal{O}(x_{in}·y_{out})$. For a VAE encoder with a dense layers, and constant sampling $\mathcal{O}\left(1\right)$, the time complexity is $\mathcal{O}(a·x_{in}·y_{out})$. If the contrastive layers deploys b dense layers, the total time complexity of the VAE encoder and contrastive learning layer is $\mathcal{O}(c·x_{in}·y_{out})$, where $c=a+b$.

MLP and MoTE Time Complexity: Like the VAE and Contrastive learning layer, the MLP and MoTE in this work constituted dense layers with the MoTE using fewer layers (only two layes in this case) and far fewer computational units per layer to make more lightweight. In addition, the MoTE has a routing operation of constant time complexity $\mathcal{O}\left(1\right)$. Therefore, the time complexity of MLP and MoTE can be expressed as $\mathcal{O}(d·x_{in}·y_{out})$ and $\mathcal{O}(E·k·x_{in}·y_{out})+\mathcal{O}\left(1\right)$ which is simple $\mathcal{O}(E·k·x_{in}·y_{out})$; where d and k with ($k<<d$) are the number of layers in MLP and MoTE while E is the number of tiny experts in MoTE, respectively. Therefore the overall time complexities of the VAE-Encoder-MLP and VAE-Encoder-MoTE can be expressed as $\mathcal{O}(P·x_{in}·y_{out})$ and $\mathcal{O}(Q·x_{in}·y_{out})$ where $P=c+d$ and $Q=c+E·k$, respectively.

GAT time complexity: This incorporates a time complexity $\mathcal{O}(V{l}_z^2+E{l}_z)$ for constructing graph with V vertices (each of latent dimension $l_z$) and E edges and the time complexity $\mathcal{O}\left(n(V{l}_{z}K+HEK)\right)$ for a GAT with n GATConv layers, attention heads H, and output feature size K per head.

Table 5. Computational cost for each of the three models.

Framework	Time Complexity
VAE-Encoder-MLP	$\mathcal{O}(P·x_{in}·y_{out})$
VAE-Encoder-MoTE	$\mathcal{O}(Q·x_{in}·y_{out})$
VAE-Encoder-GAT	$\mathcal{O}(c·x_{in}·y_{out})$ + $\mathcal{O}(V{l}_z^2+EV{l}_z)$ + $\mathcal{O}\left(n(V{l}_{z}K+HEK)\right)$

summarizes the computational complexity associated with using each of the detection frameworks.

5.7. Comparison with Benchmark Studies

Table 6. Performance evaluation contrasting the proposed VAE-Encoder-MoTE framework with existing advanced techniques on the CICIoT2022 and N-BaIoT datasets.

	N-BaIoT Dataset
Study	Methods	Classification Type	Acc (%)	Prec	Recall	F1-score
[54]	Random Forest, MLP	4-class	99.98	0.999	0.999	0.999
[55]	23 models (Best: Random Forest)	3-class	99.92	-	-	-
[56]	6 models with PCA (Best: Extra Trees)	multiclass	99.94	0.999	0.999	0.999
[57]	Autoencoder, MLP	3-class	99.98	0.990	0.990	0.990
		multiclass	88.89	0.900	0.860	0.850
[58]	LSTM	multiclass	86.38	0.829	0.795	-
[59]	3 DL models with FL (Best: CNN)	multiclass	90.90	0.940	0.900	0.870
[60]	spiking neural networks	multiclass	71.00	0.692	0.634	0.652
VAE-MoTE	VAE encoder, supervised contrastive learning, MoTE	binary	99.98	0.999	0.999	0.999
		multiclass	99.85	0.999	0.998	0.999
	CICIoT2022 Dataset
Study	Methods	Classification Type	Acc (%)	Prec	Recall	F1-score
[61]	Hierarchical Matrix mapping, Transformer-ResNet	multiclass	95.81	-	-	0.957
[62]	FL with QDVTA and ResNet18	multiclass	95.46	0.965	0.955	0.955
[63]	Golden sine with crystal structure algorithm, GRU	binary	99.97	0.999	0.999	0.999
VAE-MoTE	VAE encoder, supervised contrastive learning, MoTE	binary	99.97	0.999	0.999	0.999
		multiclass	98.90	0.989	0.988	0.985

presents a comparative analysis of the proposed VAE-Encoder-MoTE method against approaches reported in prior studies. To ensure a fair and consistent comparison, only methods evaluated on the same datasets (CICIoT2022 and N-BaIoT) and performance measured using identical metrics were considered. In this comparison, we have considered the highest recorded scores from each study across the different evaluation metrics and compared with the highest recorded scores by the VAE-Encoder-MoTE model. As indicated by the boldface values in Table 6, the proposed model achieved competitive performance across both datasets and outperformed sate-of-the-art methods.

6. Conclusions and Future Work

This study presented a comparative evaluation of three hybrid deep learning architectures for IoT botnet detection: VAE-encoder-MLP, VAE-encoder-GAT, and VAE-encoder-MoTE. The proposed framework utilized a variational autoencoder to compress high-dimensional IoT traffic features into a compact latent representation, which was subsequently used to train downstream classifiers under a supervised contrastive learning objective. This training strategy encouraged intra-class compactness and inter-class separability in the embedding space, thereby improving the discriminative quality of the learned representations. In addition, the Mixture of Tiny Experts (MoTE) architecture was investigated as a lightweight conditional computation strategy designed for resource-constrained IoT environments.

Experimental results on the CICIoT2022 and N-BaIoT datasets demonstrated that the proposed architectures achieve near-perfect performance in binary attack detection, with accuracy exceeding $99.80\%$ across both datasets. In the more challenging multiclass scenario, the VAE-encoder-MLP model achieved the best overall performance, particularly on the N-BaIoT dataset, indicating that well-structured representations can enable highly accurate intrusion detection even with relatively simple classifiers. The MoTE architecture achieved competitive results and demonstrated improved recall on the CICIoT2022 dataset, highlighting the benefits of expert specialization and conditional routing. In contrast, the GAT-based model showed lower performance in multiclass settings, suggesting sensitivity to the graph construction method and dataset imbalance.

Future work will focus on improving graph construction strategies for GNN-based models, enhancing expert routing mechanisms in MoTE architectures, compression via quantization, and incorporating temporal modeling techniques to better capture evolving IoT attack patterns.