Robust and Noise-Resilient Botnet Detection Framework Using Heterogeneous Radial Basis Function Neural Network

Abstract

The rapid evolution of botnet attacks poses a critical challenge facing cybersecurity, necessitating the development of intrusion detection models that are both highly accurate and computationally efficient. This paper proposes a heterogeneous radial basis function neural network structure that employs non-uniform RBF kernels to enhance discriminative capability between normal and botnet activities, leveraging flow-level packet length distribution features derived from the CTU-13 dataset, which encompasses 30 distinct botnet types, to ensure comprehensive detection across several botnet behaviors. The model was accurately evaluated across several dimensions, including training stability, robustness to noise, and overall detection accuracy and generalization performance. Experimental results demonstrate that the proposed model achieves a superior accuracy of 97.86%, with an AUC of 0.9968 and a notably low false-positive rate of 0.02. The model effectively mitigates class-imbalance bias, with an average detection rate of 94.62% even for minority botnet classes. Furthermore, inference-time evaluation showed a latency of approximately 1.0118 microseconds, confirming that the model is well-suited for high-speed networks. In addition, robustness analysis under controlled noise injection revealed a smooth degradation in performance, with accuracy remaining at 96%, highlighting the structural resilience of the proposed model and making it a robust solution for detecting modern botnet attacks.

1. Introduction

The rapid evolution of network infrastructure has enabled the emergence of diverse and pervasive botnets, which remain a significant threat to cybersecurity [1]. In these attacks, controllers (botmasters) coordinate networks of hijacked devices to execute large-scale cyberattacks via command-and-control servers [2]. Figure 1 illustrates the typical stages of these harmful activities. These threats are becoming more sophisticated, particularly as modern botnet architectures transition to more decentralized and increasingly employ evasion techniques [3].

Figure 1. Botnet lifecycle.

According to the 2024 Data Threat Report [4], produced by Thales, a cybersecurity organization, there is a critical shift in botnet development architectures, as shown in Figure 2.

Figure 2. Botnet sophistication in 2024.

Despite a high volume of straightforward attacks, botnet attacks remain prevalent. On the other hand, advanced and moderate botnet attacks account for 55% of all botnet activity, as attackers increasingly employ sophisticated techniques to generate large volumes of malicious traffic that emulates human traffic. These botnets are designed to operate stealthily with high precision, posing a greater challenge for detection.

From an economic perspective, the financial impact of botnet activity has reached unpredictable levels. According to the Cybercrime Report, produced by Cybersecurity Ventures, global cybercrime costs associated with advanced botnet-driven activities exceeded $10.5 trillion annually by 2025 [5]. These botnet risks are exacerbated in high-frequency trading enterprise network environments, where even a microsecond of latency caused by an undetected botnet can result in millions of dollars in losses. Consequently, investing in ultra-low-latency detection is no longer merely a security preference but an urgent necessity for safeguarding digital assets and protecting all sectors from botnet threats.

As long as botmasters develop new attack methods, sophisticated solutions for botnet detection are essential. Despite notable advancements in botnet detection techniques, which are now highly accurate and based on deep learning frameworks, many of these architectures often face significant operational challenges, making them impractical in real-world environments. This limitation is primarily attributable to their reliance on numerous deep hidden layers, which require many trainable parameters for stable generalization, as in convolutional neural networks (CNNs) and long short-term memory (LSTM) networks [6]. In addition to considerable computational resources, these architectures with multiple deep layers typically incur high training and inference times. This computational overhead increases detection latency, which is particularly problematic in high-speed infrastructure where decision-making must occur at the microsecond scale [7]. Consequently, such approaches may struggle to maintain a balance between structural depth and provide a timely detection of sophisticated botnet activities.

Motivated by these considerations, this work addresses this gap by proposing a novel deep heterogeneous radial basis function neural network (RBFNN) architecture to detect various types of botnet behaviors. Specifically engineered for sophisticated attacks, our approach introduces structural heterogeneity at the neuron level, comprising four hidden layers, each with a different kernel, with relatively low complexity compared to other deep structures, thus bridging the trade-off between efficiency and accuracy. By exploiting the strengths inherent in the RBFNNs, including fast learning, localized receptive fields, and strong nonlinear approximation capabilities with a low number of trainable parameters [8], the model establishes a robust baseline for capturing complex botnet behaviors under practical computational constraints.

Current studies on RBFNNs have evolved in two distinct directions, either focusing on a single hidden layer with multiple kernels or emphasizing deep architectures with a uniform kernel type. By proposing a deep heterogeneous RBFNN, we integrate several radial basis functions simultaneously across four distinct hidden layers. This structure ensures that each kernel type identifies a distinct area of the input space, thereby optimizing detection depth for complex botnet patterns and increasing feature sensitivity compared to traditional homogeneous or single-layer hybrid approaches.

Heterogeneous RBFNN architectures extend homogeneous designs in which all radial basis neurons share an identical kernel; the heterogeneous structure enables individual radial basis neurons to operate with distinct parameters provided by different kernels. This architecture enables the model to represent diverse and evolving botnet behaviors, providing a practical, robust, and effective instrument for improving detection capability without relying on deep or highly parameterized architectures. To ensure the robustness and operational effectiveness of the proposed heterogeneous RBFNN architecture, we conducted a series of experiments to validate our hypothesis regarding structural diversity. The paper contribution is summed up as follows:

• A novel heterogeneous radial basis function neural network structure is proposed for botnet detection, employing non-uniform RBF kernels to map complex, high-dimensional botnet patterns and enhance discriminative capability under class-imbalance conditions.
• The proposed model uses hierarchical features based on the packet-length distribution to capture multi-level traffic behavior patterns. Thus, this approach ensures a high representation of network traffic, significantly enhancing the model’s ability to distinguish between dominant and rare botnet families that are often overlooked by traditional feature sets.
• Unlike studies limited to specific botnets, the proposed model was evaluated against a comprehensive dataset comprising 30 distinct botnet types, enabling the learning of diverse botnet communication patterns and demonstrating its robustness and high generalization capability rather than being overfitted to specific botnet behaviors.
• The proposed structure achieves a superior balance between high accuracy (97.86%) and computational efficiency, maintaining a very low inference latency on the order of a sub-microsecond per sample, which is critical for high-speed intrusion detection systems in real-time scenarios.
• The model achieves an optimal configuration that balances convergence stability with predictive accuracy, achieving a near-perfect AUC of 0.996, considered excellent classification performance, with a notably low false-positive rate of 0.02, ensuring high-precision detection with minimal false alarms, which is vital for operational network stability.
• Systematic Gaussian noise injection was conducted to confirm model robustness across different noise levels, with accuracy remaining at ~96% without an abrupt performance collapse.
• The proposed model mitigates the bias towards the majority of botnet classes and achieves a high average detection rate of 94.62% for minority botnet classes, ensuring effective performance across imbalanced datasets.

In summary, while existing intrusion detection system literature often relies on shallow and homogeneous architectures, the scientific novelty of this work lies in the synergistic integration of heterogeneous kernels within a hierarchical RBFNN framework. This approach not only enhances the mapping of complex, high-dimensional botnet patterns under class-imbalance conditions but also ensures mathematical transparency and microsecond-level latency. By explicitly addressing the trade-off between robustness to real-world network impairments and computational efficiency, this study provides a unique, scalable solution that distinguishes itself from standard deep RBFNN implementations.

2. Related Work

Botnet detection has been an active research area, particularly in recent years, with the academic focus shifting significantly toward deep learning architectures for identifying advanced botnet activities. This significant attention to deep-learning-based methods stems from their ability to automatically extract hierarchical feature representations from large volumes of network traffic. In this context, many researchers have integrated deep learning models with machine learning techniques to enhance the detection performance, robustness, and generalization capability of classifier models when dealing with complex and evolving botnet traffic patterns. Table 1 highlights the most recent state-of-the-art studies for detecting and classifying botnet activities along with their key characteristics.

Table 1. Summary of state-of-the-art studies based on botnet detection.

Refs.	Year	Objective	Dataset	Algorithm/Model	Key Features	Result
[9]	2025	Enhance botnet detection efficiency and robustness against adversarial botnets.	ISCX and ISOT	A hybrid CNN–LSTM model that combines spatial features with temporal dependency learning for botnet behavior analysis.	Employs 15 flow-based features extracted from TCP headers without deep packet inspection.	Accuracy: 99.33%, precision: 99.71%, FPR: 0.43%
[10]	2025	Detects botnet behavior while hiding the privacy of information in network traffic.	ISCX-2014 And CTU-13	(CNNs) for feature extraction, (Bi-LSTM) for traffic pattern analysis, the softmax classifier was used for the last classification. And federated learning to maintain privacy.	Traffic statistical features (DNS) traffic features The first 1024 bytes of raw traffic are converted into 32 × 32 images.	Accuracy: 98.44%, F1-score: 98.95%
[11]	2025	Enhance the detection of centrally controlled botnet nodes, especially within DNS query traffic.	TI-16 DNS-labeled benchmark dataset	Four heterogeneous graph neural network (GNN) architectures were built and evaluated.	Client features Domain features.	Accuracy: 95.45%, F1-score: 0.95%, recall: up to 98.55%
[12]	2024	enhancing the model’s ability and generality to detect complex patterns.	N-BaIoT2018 UNSW-NB15	combining long short-term memory autoencoders with multilayer perceptrons.	Extracting the features based on the time series, such as packet information and connection-level statistics.	Accuracy: 99.67%, precision: 99.62% recall: 99.78%
[13]	2024	Detects anomalies by capturing contextual semantic relationships and spatial attributes.	CTU-13 and ISCX-2014	Integrates semantic relationships (transformer-based deep neural network) and depicts traffic as a graph (CNN + Bi-LSTM).	Header information is represented as grayscale images. Payload data were used as semantic vectors to return context meaning, and low-level TCP/IP session features were extracted.	F1-score: 99.8% on the CTU-13 and 91.45% on the ISCX-2014 dataset
[14]	2024	Building a robust botnet detection system and introducing noise to examine botnet attack scenarios ensures the model maintains high accuracy even with noise samples.	Kaggle Microsoft Malware Classification Challenge (BIG 2015) dataset	Deep convolutional generative adversarial network generates a synthetic botnet and discovers adversarial attacks. Also, it integrates GANs with differential privacy to enhance information privacy	Grayscale images are created by converting malware binary code into a matrix shape. The pixel weights in grayscale images range from zero to 255, where 255 denotes “white,” and zero denotes “black.”	accuracy = 97.4%
[15]	2024	To enhance botnet detection by minimizing adversarial attack vulnerabilities and improving robustness using conformal prediction.	ISCX Botnet And ISOT Botnet	Decision trees, random forests, and six-layer neural networks (NNs) are used. The genetic algorithm (GA) is also optimized using particle swarm optimization (PSO).	A total of 32 features were extracted from raw network traffic to represent botnet behaviors.	ISOT has higher accuracy (99%). (GA) has the best performance for ISCX
[16]	2024	Uncover network third parties that are believed to be botnets by assessing relevant network activity.	three datasets CTU-13, NCC-1, and NCC-2	Integrate sequential pattern mining to identify correlated activities in network traffic and classify them according to their sequential patterns.	Time-based features. Network flow features. Statistical features. Derived features such as segment ID.	Accuracy = 97.71% Recall: 94.42%
[17]	2024	Enhance real-time detection and accuracy against two different types of Botnet C2 and P2P.	ISCX 2014 CTU-13, Real dataset, which is traffic from 90 dubious IPs	An ensemble learning algorithm called Extra Tree Classifier (Extremely Randomized Trees).	Features include average communication flow duration and number of sent and received bytes using (GCN)	98.85% and 99.10% for the synthetic dataset and 83.41% for the real dataset

Despite demonstrating strong detection performance, deep learning approaches rely on deep, multi-layered architectures with extensive parameterization, which introduce notable challenges in computational costs, training time, and real-time applicability, creating a trade-off in which high-performance accuracy often comes at the expense of increased latency and reduced real-time operational efficiency. In network monitoring environments, particularly those that require rapid response and persistent traffic analysis, there is a growing need for learning models that achieve competitive detection performance while maintaining reasonable depth and requiring minimal computational resources. In this context, radial basis function neural networks (RBFNNs) have emerged as a powerful and effective alternative to deep learning architectures that use extensive layers, offering strong nonlinear approximation capabilities through a compact architecture.

In this direction, our research builds on the established mathematical and biological foundations of radial basis function neural networks (RBFNNs) and focuses on developing a deep, heterogeneous approach by employing a multi-kernel RBF neural network for sophisticated botnet detection rather than relying on a shallow network with a single kernel type. The architectural design of the radial basis function neural network is deeply rooted in biological concepts, specifically mimicking the local receptive fields of neurons observed in the human visual cortex [18]. In biological neural systems, certain neurons respond only to stimuli within a specific region of the input space. This phenomenon, known as localized response mechanisms, allows the brain to process sensory information with extreme efficiency by activating only the relevant specialized neurons. Broomhead and Lowe introduced radial basis function neural networks in 1988 [19], translating the biological concept into a computational framework to formalize a model that mimics localized response behavior.

Approximation theory [20] provides a mathematical basis for the biological inspiration and effectiveness of RBFNNs. Specifically, it has been formally proven that an RBFNN with sufficient hidden neurons in a single layer can approximate any continuous nonlinear mapping between input and output with arbitrary accuracy. This direct link explains why RBFNNs can represent nonlinear relationships and be effective in complex domains such as traffic analysis. Therefore, RBFNNs have several distinctive characteristics. One of the most significant is localized activation: each hidden-layer neuron is highly activated only when the input vector lies near its center, ${\mu }_i$. As a result, each neuron models a local region of the input space, forming a receptive field [21], a defined region of influence centered on the neuron. This localized sensitivity enables RBFNNs to capture subtle variations in input data, making them robust to highly nonlinear and overlapping traffic. Empirical studies have shown that RBF networks require fewer epochs to achieve superior accuracy than other neural networks, especially when the number of hidden units is appropriately chosen [22,23]. RBFNNs are well-suited to applications that require frequent retraining or adaptive learning, such as detection models, because only a small subset of neurons is active for each input vector. At the same time, the remaining neurons remain nearly inactive, thereby minimizing interference among neurons. Consequently, learning becomes more efficient and stable, leading to faster convergence than in other neural network architectures with global activation.

A key benefit of the RBFNN architecture is its robustness against outliers and random noise. As a distance-based neural network using Euclidean distance between input features and hidden neuron centers [24], RBFNN reduces the influence of data points that deviate significantly from learned clusters on the final output. This advantage is particularly useful, since background noise often causes false positives in less robust, globally trained models. Owing to these characteristics, RBFNNs have been widely adopted in recent years across a broad range of disciplines, including real-time facial regression [25], air pollution prediction [26], signal processing [27], computer vision [28], cybersecurity [29], and many others, demonstrating their adaptability and effectiveness across diverse problem domains.

The structure of the classical radial basis function neural network is shallow, which has a relatively simple topology. Typically, an RBFNN consists of three main layers: an input layer, a single hidden layer composed of radial basis neurons, and a linear output layer, as shown in Figure 3. In this structure, the input layer serves as a transparent interface that receives feature vectors without performing any computational transformations. These inputs are then projected into the hidden layer, the core of the RBFNN. Within this layer, the input space is mapped to a higher-dimensional field via a nonlinear activation function. Based on the receptive field, each hidden neuron serves as a localized detector, responding only to patterns that fall within its region, defined by its center and spread factor. The output layer implements a linear combination of hidden-layer activations, with learnable weights, to efficiently produce the final output [30,31]. Despite shallow RBFNNs offering efficient nonlinear modeling, which often requires an exponentially large number of hidden neurons, their capacity becomes insufficient for highly complex data. To address this, recent research has increasingly shifted toward multilayer deep RBFNN architectures, stacking multiple hidden radial-basis layers to combine the localized learning benefits of RBF kernels with the representational power of deep learning, thereby enhancing the ability to discern more abstract patterns in data [32].

Figure 3. Classical structure of RBFNN.

In the literature, a significant trend has been the shift from shallow RBFNNs with a single hidden layer to multilayer or deep RBFNN architectures to enhance feature abstraction and enable more effective learning of complex nonlinear patterns, yielding lower prediction errors with fewer neurons per layer. In addition, it has been demonstrated that a deep neural network is less likely to become trapped in local minima than a shallow neural network [33]. Various studies [34,35,36,37,38] have highlighted that stacking multiple radial basis layers can achieve competitive performance in complex scenarios.

However, a critical observation across these diverse deep architectures is their reliance on a homogenous design, in which the same type of radial basis kernel is employed consistently across all layers, typically a Gaussian radial basis function [39]. While this uniformity kernel selection simplifies network architecture design and improves optimization, it can sometimes limit neural networks’ ability to capture simultaneously fine-grained and rare signatures that characterize modern botnets, which exhibit highly diverse statistical, temporal, and complex behavioral patterns.

These limitations naturally motivate a transition to a heterogeneous RBFNN, in which different types of radial basis functions are employed by the hidden-layer neurons. It is stated that, from both theoretical and practical perspectives, using multiple kernel types better aligns with data heterogeneity and exhibits stronger learning ability than a traditional single kernel type. Several studies, as shown in Table 2, suggest that combining different kernel types is more efficient than using a single kernel type in the hidden layer.

Table 2. Example of multi-Kernel RBFNNs by a single hidden layer.

Ref.	Kernel Types	Key Contribution
[40]	Gaussian and Wavelet	Introduced combinations of multiple kernels to create a more flexible model to capture complex patterns.
[41]	Gaussian and Cosine	improve pattern recognition and overall classification accuracy.
[42]	Euclidean and Cosine	Automatically learn kernel fusion weights by applying gradient descent to eliminate manual tuning.
[43]	Gaussian and Wavelet	improving convergence and resilience to local minima by assigning individual weights to each kernel to achieve local flexibility.
[44]	Gaussian and Cosine	By combining two kernels, the model can capture both magnitude and angular relationships to extract directional features, particularly in signal-detection tasks.
[45]	Gaussian, cosine, and inverse multiquadric kernels	The system was evaluated on three problems: pattern classification, system identification, and function approximation, achieving superior results.

In the context of RBFNNs, the use of both deep and multi-kernel approaches remains underexplored and has received limited attention in the literature. A recent novel study [38] proposed a deep RBFNN structure comprising multiple stacked RBF hidden layers and employing two kernels: the Gaussian and Mahalanobis kernels. The study aimed to investigate the impact of different dynamics, convergence rates, and representational capabilities of deep RBFNNs, thereby demonstrating their effectiveness relative to conventional deep neural network architectures. The results demonstrate that RBFNNs with different kernels can be successfully trained and remain stable when stacked in multiple layers.

In summary, the current literature on RBFNNs has developed in two distinct directions: either focusing on a single hidden layer by employing multiple kernel types or emphasizing deep RBFNN design while maintaining a homogeneous kernel type across all hidden layers. Our work addresses this gap by proposing a heterogeneous RBFNN that is relatively deep and incorporates several multi-kernel radial basis functions, which are rarely combined simultaneously. This novel framework is designed to optimize both detection depth for complex botnet patterns and increase feature sensitivity.

3. Methodology

In this section, we present the methodology details used to design our proposed deep heterogeneous radial basis function neural network developed for botnet detection. The proposed framework is designed to exploit both the depth of architectural and kernel heterogeneity by stacking multiple RBF hidden layers, each layer employing a different radial basis function. The details of the mathematical formulation of each RBF layer are presented in the following subsections. Figure 4 illustrates the generic pipeline methodology adopted in this study.

Figure 4. Generic pipeline of the proposed deep heterogeneous RBFNN botnet detection mode.

3.1. Dataset Description

This study uses a comprehensive subset from the CTU-13 repository [46]. CTU-13 has a wide-ranging collection of real network traffic captured at the Czech Technical University, comprising both botnet and normal traffic. The selected subset comprises 30 distinct botnet types to provide a high degree of traffic heterogeneity and cover a sufficient diversity of botnet communication behaviors, ensuring that the model is exposed to various botnets. Table 3 summarizes the distribution of flow samples across botnet and normal traffic classes used in this study.

Table 3. Distribution of normal and botnet traffic.

Category	Total Flow	Distribution (%)
Botnet	120,871	49.92%
Normal	121,256	50.08%
Total sample	242.127

The statistical breakdown of 30 distinct botnet classes is presented in Figure 5, detailing the flow counts within the total traffic samples of 242,127. In real-world network environments, botnet types are unevenly distributed: some types account for a larger proportion of samples than others; sophisticated botnet types often generate very low volumes of traffic to evade detection [47]. Accordingly, we intentionally selected botnet types with a broader distribution than others. This uneven contribution enables evaluation of the model not only on dominant botnet behaviors but also on minority botnets with low-intensity traffic that often bypass traditional detection systems.

Figure 5. Distribution of botnet traffic.

It is important to note that while the overall binary distribution between normal and botnet classes is balanced, as shown in Table 3, the individual botnet sub-categories exhibit a natural and significant imbalance, as shown in Figure 5. This diversity in sample sizes across different botnet types reflects the heterogeneous nature of real-world network threats. Consequently, the proposed framework’s resilience is evaluated against these minority types using imbalance-robust metrics, as will be explained in Section 4.8, ensuring that detection performance remains consistent even for low-volume, stealthy attack patterns.

3.2. Traffic Filtration Phase

The methodology incorporates selective data-trace filtering as a preliminary step. The first filter employs unidirectional flows rather than bidirectional ones, thereby simplifying analysis by focusing on either incoming or outgoing data streams and enabling more straightforward interpretation of flow behavior. Following unidirectional filtering, the data is further refined to include only transmission control protocol (TCP) traffic; the second filter excludes non-TCPs such as UDP and ICMP. The decision to focus exclusively on TCP traffic is motivated by several considerations: most modern botnets carry out high-risk malicious activities, including financial credential theft and secondary malware delivery; rely on TCP-based communication channels for command-and-control operations and payload delivery to ensure reliable, persistent connectivity; and receive instructions from the botmaster without loss [48].

3.3. Features Extracted Process

To preserve a high level of user privacy, the feature extraction process is based solely on the packet-length distribution, with features extracted from the packet header retained and those from the payload ignored. Moving beyond conventional statistical analysis, which assumes that botnet behavior is uniformly distributed across the flow duration. This study adopts a different approach, employing a granular flow analysis technique to construct high-level, abstract, hierarchical features based on the packet-length distribution [49,50], thereby enabling deeper insight into the traffic structure. The network flow duration has been divided into smaller segments, referred to as zones/categories, based on packet length. This extraction is conducted at two levels: First, eight primary variables, including count, average, maximum value, minimum value, rate, percent, burst rate, and burst start, are computed over the entire flow duration. Second, the same eight variables are extracted non-sequentially from each of the ten partitioned zones of the flow duration. This granular process yields a total of 88 features (8 + (10 × 8) = 88), defining a high-dimensional feature space (R88). The hierarchical arrangement of these zones and the resulting feature vector are further illustrated in Figure 6.

Figure 6. Captures a Tshark/Lua snapshot illustrating the packet-length extracted features calculated over the entire traffic flow and corresponding to each individual packet-length category.

A summary description of each extracted feature is provided in Table 4. Then, the aggregated features of all zones are used to generate the final feature vector. The motivation for adopting this approach stems from the empirical observation that botnet behavior often occurs within specific packet-length ranges, whereas it exhibits minimal activity outside these ranges [38]. Consequently, the resulting representation captures a distinctive “fingerprint” of how traffic behaves in different zones/categories of the flow duration.

Table 4. Feature extracted descriptions.

Count	The Total Number of Packets That Were Observed Within a Specific Category
Maximum	The average packet size belongs to the respective category.
Minimum	The smallest packet size observed in the category.
Rate	The Largest packet size observed in the category.
Average	Transmission Packet rate per second within each category.
Percent	The percentage of packets belonging to each category compared to the total number of packets.
Burst Rate	The number of packets transmitted within a short period of network activity.
Burst Start	The time at which the first burst in a flow begins.

In real-world deployment, particularly against adversarial evasion tactics such as packet padding, most methods aim to conceal detection by mimicking normal traffic distributions. However, the multi-level granularity of our hierarchical design, which partitions flow duration into 10 zones and extracts 88 distinct features, substantially raises the bar for adversaries. Unlike traditional systems that rely on global flow averages, our non-sequential extraction technique captures a unique ‘fingerprint’ of traffic behavior. For an attacker to successfully evade detection, they would need to manipulate packet-length distributions not just globally, but across 10 synchronized intervals, preserving the intricate dependencies and bursts of activity inherent in our 88-dimensional feature space. Consequently, a failure mode remains theoretically possible only if a botnet perfectly mimics these complex multi-zonal distributions, a challenge that defines the current boundary for feature-centric detection systems.

3.4. Preprocessing Extracted Features

Following feature extraction, a series of preprocessing steps, as shown in Figure 7, are applied to prepare the dataset for training the proposed heterogeneous RBFNN model, ensuring data integrity, eliminating bias, and optimizing the numerical representation of each feature.

Figure 7. Flow diagram of the preprocessing stage.

To transform raw feature vectors into supervised learning instances within the RBFNN framework, each extracted flow is assigned a ground-truth label based on the CTU-13 reference, where traffic patterns are categorized into two classes: ${\mathit{F}}_i^{normal}=1$ and ${\mathit{F}}_i^{Botnet}$ = 2. After the labeling step, both normal and botnet traffic samples were merged into a unified dataset. This step is a prerequisite for the next step, Fisher–Yates shuffling, a statistically unbiased permutation method for the dataset before the training phase, in which every possible ordering is equally likely. Hence, the RBF units are exposed to a balanced variety of input features, leading to better generalization [51].

The last and most critical step in the preprocessing stage is normalizing the feature vectors. This study employed Z-score normalization rather than Min–Max scaling. The mathematical properties of radial basis function kernels and the choice of normalization type are fundamentally important, as the activation functions of RBFNNs are essentially distance-based; thus, inappropriate scaling can adversely affect kernel responses. The benefit is that the Z-score normalization approach preserves the relative distribution of the data by centering it at a mean of zero and unit variance, which is particularly advantageous when employing heterogeneous RBF kernels. In the context of botnet traffic, extreme outliers are often observed in bursty network traffic, making Z-score normalization more appropriate than Min–Max scaling, which scales features to a bounded range that may reduce discriminative capability, especially in the presence of outliers [52].

3.5. Deep Heterogeneous RBF Neural Network Structure

The core contribution of this study is the development of an architectural design of the deep heterogeneous radial basis function neural network. Unlike deep homogeneous RBFNNs, which rely on a uniform kernel function type across all hidden layers, typically Gaussian RBFs, the proposed architecture employs a deep architecture comprising four distinct hidden layers to enhance nonlinear capability. The mathematical validity of selecting different kernels for the RBF family is supported by the landmark work of Micchelli (1986) [53] on radial basis function interpolation. According to Micchelli’s Theorem, a wide class of radial functions satisfying conditional positive definiteness yields non-singular interpolation matrices for distinct centers. This theorem formally establishes Gaussian, inverse, power, and square root type functions as members of the RBF family. By leveraging this mathematical proof, our proposed structure transitions from a deep homogeneous to a deep heterogeneous framework, ensuring that each kernel type in the hidden layer identifies a distinct area of the input space. At the same time, preserving the convergence and approximation properties. The four kernels were selected strategically based on their distinct structure to capture multi-scale behavioral characteristics of modern botnet traffic. Figure 8 shows the general structure of the deep heterogeneous radial basis function neural network.

Figure 8. General structure of the deep heterogeneous RBFNN.

The design of the RBFNN architecture is heavily influenced by the choice of radial basis functions and their respective shape parameters, a task that poses significant challenges for numerical stability and approximation precision [54]. To address the limitations of single-kernel RBF configurations, this section introduces a deep hierarchical architecture that integrates four heterogeneous kernels. By cascading these functions, the proposed model effectively navigates the kernel selection problem, ensuring stable feature transformations and robust classification across the network’s different stages, as detailed in the following subsections.

3.5.1. First Hidden Layer (h1): Gaussian RBF Kernel

The first hidden layer in our proposal model employs a Gaussian radial basis function to establish an initial feature representation, yielding a localized activation response: a neuron becomes significantly activated, approaching 1, only when the input features lie near its center. Conversely, the response decays rapidly to zero as the distance from the center increases. This high sensitivity to local data variations allows the model to establish dense cluster centers for common network behaviors. This localized sensitivity is a prerequisite for the subsequent layers, as it provides an effective means of modeling smooth decision boundaries. To achieve this for localized clusters, the output of each neuron in the first hidden layer can be calculated using the Gaussian RBF Equation (1), where $X_i$ input feature vector, $C_j$ is the spread factor, ${\mu }_j$ neuron’s center. and the Euclidean distance between the input vector and the neuron’s center can be represented as $r={\parallel X_i-{\mu }_j\parallel }^2$.

$$GaussainRBF=e^{{\displaystyle \frac{{-r}^2}{C^2}}}$$

(1)

3.5.2. Second Hidden Layer (h2): Inverse RBF Kernel

While the first layer focuses on localized responses, the second hidden layer introduces global responses by employing the inverse radial basis function, which plays a critical role in our heterogeneous approach by mitigating the Gaussian kernel’s rapid decay. The inverse RBF is characterized by a slower decay rate compared to the Gaussian RBF, which means it does not vanish quickly when moving away from the center; instead, it maintains a high degree of sensitivity over a much broader range of the feature space, allowing neurons to respond to inputs farther from their centers, resulting in broader receptive fields, especially for outliers or noisy traffic, and this interprets the robustness of the heterogeneous RBFNN against different levels of noise that deviate from the dense cluster centers, as we will discuss later in Results Section. On the other hand, the global response enables the network to recognize low-density botnet patterns, thereby enhancing the detection of stealthy botnets. To achieve this global response, the activation function of the inverse RBF can be computed as defined in Equation (2).

$$InverseRBF={\displaystyle \frac{1}{{\left(c^2+r^2\right)}^{\alpha }}}$$

(2)

$\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\alpha >0$.

3.5.3. Third Hidden Layer (h3): Power RBF Kernel

The heterogeneous RBF architecture is designed to transition from local boundary detection to global analysis and, subsequently, to broader feature mapping. While the Gaussian and inverse layers focus on isolating dense clusters of botnet traffic and both will eventually decay, the power RBF layer is essential for establishing unbounded global correlations. By placing it third, we enable the network to capture activity patterns that preceding layers overlook and maintain a persistent influence across the entire feature space, ensuring that the model captures long-range correlations and identifies stealthy, sparse botnet activities that other kernels might fail to detect. Accordingly, the power RBF layer produces highly flexible, potentially twisted decision boundaries that adapt to complex deviations in the feature space. This boundary ensures that isolated anomalies in botnet behavior are successfully encapsulated, thereby mitigating the risk of misclassifying outliers that lie far from the dense clusters.

From another perspective, the architecture of heterogeneous RBFNNs is specifically designed to optimize the network’s loss landscape during backpropagation, with the sequential placement of Gaussian and inverse RBF layers yielding sharply localized, bell-shaped activation regions. These kernels generate steep peaks in the decision landscape, as shown in Figure 9a,b, thereby enhancing discriminative sensitivity to subtle behavioral similarities among botnet behaviors. However, highly sharp activation surfaces may also introduce oscillatory gradient behavior, leading to unstable error propagation. The power RBF in the third layer introduces convexity into the representation space, as shown in Figure 9c, thereby creating smoother transition regions in the error landscape. This geometric modification enables the optimization process to navigate flatter regions, preventing sudden increases in error, thereby improving training stability while preserving discriminative sharpness and ensuring more stable convergence to the global minimum. The power RBF is formulated to bridge the gap between local and global precision, as shown in Equation (3).

$$PowerRBF={\left(c^2+r^2\right)}^{\beta }$$

(3)

$\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}0<\beta <1$.

Figure 9. Characteristics of the four heterogeneous RBF kernels.

3.5.4. Fourth Hidden Layer (h4): Square Root RBF Kernel

The architectural design of a heterogeneous RBFNN concludes with the square root RBF layer. This adoption of the square root RBF kernel as the final hidden layer serves as a simplification, intentionally following the high complexity generated by the preceding layers. While the inverse and power RBF layers are critical for generating a highly expressive decision boundary, they can twist to encapsulate complex and sparse botnet behaviors and capture all data variations. However, this extreme flexibility may lead to overfitting, thereby reducing the model’s generalization ability. The square root RBF serves as a stabilizer for the network’s output by introducing a sublinear growth rate, thereby limiting the model’s sensitivity to extreme data variation and yielding a smoother, more generalizable boundary decision, which enhances the system’s predictive accuracy on unseen data. The sublinear growth rate is mathematically inherent to the square root operator within the kernel function defined as Equation (4):

$$SquareRootRBF={\left(c^2+r^2\right)}^{\gamma }$$

(4)

$\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}0<\gamma <1$.

In summary, the proposed heterogeneous architecture works as a sequential filter, where each layer progressively refines the neural network’s ability to recognize traffic. The first two layers act as high-resolution filters, identifying specific botnet behaviors with sharp precision. The third layer extends this representation to include additional regions of feature space, capturing sophisticated, twisted patterns of stealthy botnets. Finally, the square root layer acts as a stabilizing filter, leveraging its sublinear growth to yield a smoother decision boundary and mitigate overfitting. Using this cascading approach ensures the proposed model achieves superior performance while maintaining the stability required for reliable detection.

Comparative empirical tests during the design phase confirmed that this specific sequence achieves faster convergence and more stable training gradients compared to alternative arrangements, such as placing power or square root kernels in the initial position.

To empirically validate the distinct contribution of each heterogeneous kernel, we conducted an ablation study by removing the third layer, the power RBFNN. As illustrated in Figure 10, removing this specialized nonlinear mapping stage results in a significant drop in detection accuracy to approximately 91.3%. Furthermore, the ablation leads to highly oscillatory training behavior and a non-monotonic loss convergence, confirming that the synergistic integration of all four heterogeneous kernels is critical for capturing complex, nonlinear botnet patterns. This justifies the proposed hierarchical architecture.

Figure 10. Convergence characteristics of the modified heterogeneous RBFNN during ablation: Removing the third (power RBFNN) layer decreases accuracy to 91.3% and causes increased oscillation in loss convergence.

3.6. Kernel Parameters Initialization

The power and effectiveness of radial basis function neural networks depend strongly on the accurate determination of the initial state of centers (${\mu }_i$) of hidden neurons and spread factors ($C_i$), which are also known as the width of the radial basis functions. These two trainable parameters govern how the hidden-layer neurons represent the input space. The centers determine the locations of the radial basis functions, while the spread factor defines the extent of their influence in the input space. An optimal configuration of these parameters is a crucial design challenge that ensures the network achieves nonlinear mapping and stable optimization [20]. Determining the placement of the RBF centers is critical, as this choice directly affects each layer’s ability to capture the geometry of the input data. A straightforward random approach to selecting centers offers simple computations; however, it may lead to redundant or poor selections. Placing centers in low-density regions yields suboptimal coverage of the input space and, consequently, instability during the learning stages [39].

In our proposed model, the K-means clustering approach was adapted to initialize the centers of all neurons in the hidden layers. K-means ensures that centers are strategically positioned near the highest-density regions of the data traffic in feature space, thereby maximizing the kernel function. by generating representative Euclidean distances, reducing redundancy among neurons, and facilitating efficient convergence of the model’s optimization process.

Layer-wise hierarchical initialization was used to align centers with evolving feature representations throughout the deep architecture. While the centers for the first hidden layer were derived directly from the input data distribution using K-means, the centers for subsequent layers were dynamically initialized based on clustered activation patterns from the preceding layer. This iterative clustering strategically places RBF neurons in deeper layers within high-density regions of the latent space, rather than randomly. This structured initialization reduces the risk of dead neurons. It also helps the model capture complex, nonlinear botnet fingerprints with fewer training iterations.

Unlike many other neural network models that degrade gradually, RBFNN performance can drop significantly when the spread factor changes slightly; therefore, selecting an optimal spread factor for the hidden neuron is critical. In this study, an extensive analysis was conducted to evaluate the impact of the spread factor ${\complement }_i$ on the kernel function across the four heterogeneous RBF layers, as shown in Figure 11. By varying ${\complement }_i$ from a highly localized value, ${\complement }_i$ = 0.1, to a broader value, ${\complement }_i$ = 6.8, we observed a significant transformation in the network’s receptive field.

Figure 11. Impact of the spread factor C on the kernel across all four heterogeneous layers. As C increases from 0.1 to 6.8, a clear transition from localized to broad responses is observed.

At lower spread values, the kernels, particularly the Gaussian and inverse RBFNN, exhibited sharp responses, which are essential for isolating precise botnet behaviors within dense data clusters. Conversely, as the spread factor increased, the kernels transitioned to smoother, broader distributions. This distributional shift was used to determine the optimal overlap between neurons, ensuring that no regions of the input space remain unclassified in the feature space while maintaining the degree of overlap, thereby reducing misclassification.

Since RBF activations are distance-dependent [55], particular attention was paid to initializing the spread factor for all kernels. Instead of allowing the spread factor parameters to evolve without a reference point, we impose a stable starting point for gradient-based learning, denoted as $\varphi \_target$, where ${\complement }_{initial}=\varphi \_target$. This initialization is not fixed; the parameters remain trainable and are refined via backpropagation, allowing the network to learn appropriate spreads dynamically during training and avoiding numerical instability.

Through extensive experimentation, this study identified ${\varphi }_{target}=0.8$ as the optimal threshold for the heterogeneous RBFNN. Technically, this high-activation target maximizes interneuron connectivity, eliminating dead zones in feature space between centers. By starting at an activation level of 0.8, the neurons achieve a superior balance: it provides appropriate overlap between neighboring basis functions, preventing excessively narrow receptive fields while maintaining sufficient discrimination, thereby capturing complex botnet patterns and ensuring stable gradient convergence without oscillatory range.

To evaluate the model’s sensitivity to ${\varphi }_{target}$, extensive experimental trials were conducted over a wide range of activation values, from 0.001 to 0.99. Notably, the model remained highly robust in the 0.75 to 0.85 range, where classification performance remained stable and optimal. However, deviations from this interval had noticeable effects. Specifically, values exceeding 0.9 led to over-saturation of the RBF units, resulting in a decline in accuracy and an increase in loss. Conversely, values below 0.5 led to sparse neuronal activation, which hindered the network’s ability to generalize across complex botnet patterns. Taken together, these empirical findings confirm that 0.8 serves as the optimal threshold for balancing global convergence with local feature sensitivity.

Unlike conventional multilayer neural network architectures, which insert decoupling linear layers between hidden stages, referred to as dense layers, such as in [56,57], our heterogeneous RBFNN architecture intentionally excludes intermediate linear summation layers between hidden stages. The proposed approach preserves the high degree of nonlinearity inherent in heterogeneous kernels, which is essential for capturing the overlapping feature distributions in network traffic, often reflecting complex botnet behaviors.

This layer performs a linear discriminative mapping that aggregates the contributions of the learned radial features into class-specific logits, mapping the final high-level feature to class-specific logits for the target classes, botnet and normal. The score layer computes class scores as $Z=(W\times h{4}_{output})+B$, where $W$ is the weight matrix, $h{4}_{output}$ is the output of the last hidden layer, and $B$ is the bias vector.

To ensure numerical stability and rapid convergence, the score layer weight parameters were initialized with a scaled random initializer $W=\sqrt{1e^{-3}/inputsize}$, so the initial state starts with a very small value and remains within a controllable range, preventing gradient explosion during learning. Furthermore, a marginal positive bias of $1e^{-24}$ was introduced to maintain numerical integrity during softmax computation.

Subsequently, the logit values were obtained from the score layer transform via the softmax function to produce a valid probability distribution over the target classes. This transformation ensures that the output values lie in [0, 1] and sum to 1, enabling probabilistic interpretation. Such a representation is particularly important for botnet traffic detection, where the model assigns a confidence score to each class, improving the reliability of probability predictions, even when behavioral similarities between botnet and normal traffic overlap.

Model training is guided to produce accurate results by using the cross-entropy loss function, which measures the divergence between the predicted probability distribution and the ground-truth labels.

The combination of the softmax activation function and cross-entropy loss provides a highly effective mathematical framework for classification tasks, such as botnet detection, due to its strong numerical stability and well-behaved gradients, which have been shown to enable reliable convergence during the training of a deep heterogeneous RBF-based architecture.

3.7. Hyperparameters and Model Configuration

Unlike the homogeneous architecture RBFNN, which relies exclusively on a single kernel type, often Gaussian RBF, across all hidden layers, thereby yielding a static internal behavior, the heterogeneous RBFNN architecture incorporates a diverse set of kernels, yielding dynamic internal behavior. This diversity introduces unique shape parameters, namely alpha, beta, and gamma, which are absent in homogeneous architectures. These hyperparameters are sensitive and govern the response and curvature of the activation functions within each layer. These values were determined through an extensive grid-search optimization process to balance detection sensitivity and model stability for detecting 30 distinct botnet types. For the heterogeneous layers, the optimal values for these parameters were identified by testing each across a predefined range, as summarized in Table 5.

Table 5. Empirical determination of the heterogeneous RBFNN optimal dynamics.

Hyperparameter	Associated Hidden Layer	Tested Range	Optimal Value
Alpha  $\mathit{\alpha }$	Inverse RBF	[0.1, 0.9] and [1, 15]	12
Beta  $\mathit{\beta }$	Power RBF	[0.1, 0.0001] and [0.1, 0.9]	0.2
Gamma  $\mathit{\gamma }$	Square Root RBF	[0.1, 0.0001] and [0.1, 0.9]	0.5

In addition to structural dynamics, Table 6 summarizes other hyperparameters used to optimize the performance of the deep homogeneous RBFNN model. Specifically, a learning rate of 0.001 was selected because it provided the best balance between training speed and preventing overshooting during the weight update process. The batch size was set to 64 to improve computational efficiency while maintaining sufficient gradient variance to escape local minima. Furthermore, the model was trained for 200 epochs, as empirical observations showed that the loss function plateaued beyond this point, with no signs of overfitting. These hyperparameter settings, summarized in Table 6, were consistently applied across all experimental runs to ensure a fair and reproducible benchmarking environment.

Table 6. Summary of hyperparameters for the heterogeneous RBFNN.

Parameter	Description	Specification
Learning Rate η	Step size to update the weight	η = 0.001
Number of Epochs	Training iterations	200
Batch Size	Number of training samples processed per gradient update	64
Neurons per Layer	Hidden-layer neuron distribution	{20, 20, 10, 10}

Furthermore, to ensure robust evaluation and prevent data leakage, the dataset was partitioned using a standard 80/20 split, with 80% of the samples dedicated to model training and the remaining 20% reserved strictly for independent testing. This ensures that the reported performance metrics reflect the model’s true generalization capability on unseen network traffic.

A series of experiments was conducted by varying the number of neurons per hidden layer to evaluate model performance. As shown in Table 7, the heterogeneous RBFNN model performance was evaluated across six different architectural configurations, with total neurons ranging from 30 to 300. The objective was to identify the configuration that provides optimal classification while maintaining behavioral stability and avoiding overfitting.

Table 7. Impact of the number of neurons on performance.

Network Architecture (Neuron Distribution Across Hidden Layers)		Total Neurons	Performance Matrices (%)	Performance Observations
Experiment_1	{10,10,5,5}	30	Accuracy = 0.902	° Stable Behavior
			Precision = 0.877
			Recall = 0.928
			F1 Score = 0.902
Experiment_2	{20,20,10,10}	60	Accuracy = 0.978	° Optimal Behavior ° Smooth Convergence and faster training.
			Precision = 0.975
			Recall = 0.981
			F1 Score = 0.978
Experiment_3	{30,30,15,15}	90	Accuracy = 0.925	° Stable Behavior ° Smooth Convergence
			Precision = 0.959
			Recall = 0.885
			F1 Score = 0.921
Experiment_4	{50,50,25,25}	150	Accuracy = 0.932	° Unstable Behavior ° longer training time
			Precision = 0.955
			Recall = 0.904
			F1 Score = 0.928
Experiment_5	{100,100,50,50}	300	Accuracy = 0.938	° Unstable ° Longer training time ° Oscillatory Convergence
			Precision = 0.9197
			Recall = 0.9612
			F1 Score = 0.940
Experiment_6	{20,20,20,20}	80	Accuracy = 0.941	° Stable Behavior ° Suboptimal Distribution
			Precision = 0.966
			Recall = 0.912
			F1 Score = 0.938

While larger densities, such as 300 neurons, increased computational training time and led to oscillatory behavior, the {20, 20, 10, 10} setup provided the most stable accuracy (0.978) and F1-score. Furthermore, the microsecond-level latency remains largely consistent across these configurations because the radial basis computations are parallelizable, making the model highly scalable for real-time engineering applications.

4. Experimental Results and Discussion

In this section, the performance of the proposed heterogeneous RBFNN model is rigorously evaluated through a series of extensive experimental scenarios to confirm its effectiveness in detecting complex botnet traffic. The following subsections detail the experimental component and its results, demonstrating that the heterogeneous design enhances the network’s discriminative capabilities and stability, even under challenging network conditions.

4.1. Fold Cross-Validation Evaluation of the Heterogeneous RBFNN

To verify the stability and generalization capability of the proposed heterogeneous RBFNN model to unseen data, a k-fold cross-validation procedure was implemented to ensure that the model’s performance is consistent across different subsets of the dataset and is not biased toward specific training instances. By partitioning the data into five equal folds, in each iteration, four folds were used for training while the remaining fold was reserved for testing. The performance metrics, including accuracy, precision, recall, F1-score, and false-positive rate, were computed for each fold and subsequently aggregated to calculate the mean and standard deviation, as illustrated in Figure 12. Looking at the final experimental results, the individual fold metrics reveal a remarkably stable performance across all data partitions. The model achieved a high mean accuracy and F1-score of 98.17%, with very low standard deviations of ±0.0028 and ±0.0027, respectively. Thus, the heterogeneous RBFNN is not overfit to a specific partition of the dataset and generalizes consistently across different folds.

Figure 12. Cross-validation metrics and standard deviation for a heterogeneous RBFNN.

Furthermore, achieving a recall value of 0.9839 demonstrates the model’s strong ability to detect botnet traffic with a low false-positive rate of 0.031%, which is critical in a real-time scenario where excessive false alarms can impact operational efficiency. The high and consistent F1-score across all folds indicates a balanced trade-off between precision and recall measures, underscoring the discriminative power of the hierarchical flow features when integrated with the heterogeneous RBF architecture.

4.2. ROC Curve and AUC Evaluation

To further validate the discriminative power of the proposed heterogeneous RBFNN model, a receiver operating characteristic (ROC) curve analysis was performed.

As illustrated in Figure 13, the model achieved an area under the curve (AUC) of 0.9968, indicating a near-perfect ability to distinguish between botnet attacks and normal network traffic, which means maintaining a high true positive rate and keeping false positives to a minimum.

Figure 13. Receiver operating characteristic (ROC) curve of the heterogeneous RBFNN model.

This high AUC value is particularly significant given the dataset’s complexity and the presence of several minority botnet classes. It confirms that the heterogeneous structure allows the network to establish highly precise decision boundaries, effectively minimizing the trade-off between sensitivity and specificity. Such results underscore the reliability and effectiveness of the heterogeneous RBFNN for deployment in real-time environments, where the costs of missed detections and false alarms require strict control.

4.3. Performance Metrics Evaluation

The classification performance of the proposed heterogeneous RBFNN was evaluated employing a comprehensive set of performance metrics, including accuracy, precision, recall, and F1-score. These metrics assess the classifier’s effectiveness in distinguishing between normal and botnet network traffic. The empirical results demonstrate superior discriminative power across all standard metrics, with high convergence and predictive reliability, as evidenced in Figure 14.

Figure 14. Performance metrics of the heterogeneous RBFNN.

The model achieved 97.86% accuracy, indicating that it correctly classifies most of the network traffic for both botnet and normal classes. More importantly, the recall value reaches 98.15%, confirming the strong capability to detect botnet activity with minimal missed detections. From a cybersecurity standpoint, high recall is critical, as undetected attacks can compromise networks. The precision reached 97.58%, indicating that the model generated few false alarms. Furthermore, the F1-score of 97.87% achieved by the proposed model confirms a well-balanced trade-off between detection sensitivity and prediction precision of the heterogeneous architecture across unbalanced data distributions.

4.4. Training Stability and Convergence Analysis of Heterogeneous RBFNN

Beyond final performance metrics, the dynamic behavior of the proposed heterogeneous RBFNN was analyzed, particularly in terms of stability and convergence behavior during both training and testing phases. Figure 15 illustrates the growth of the accuracy curves and the reduction in loss across 200 training epochs. It is important to note that the loss curve shows a smooth, gradual decline from the initial epochs to convergence. where a sharp reduction is observed during the early epochs, followed by a gradual stabilization phase, indicating efficient parameter updates and rapid learning of discriminative patterns. whereas the accuracy curve consistently ascends toward a stable peak above 97%, reaching convergence early in the training process. The absence of oscillation confirms that the learning process remains numerically stable throughout training. Moreover, the high overlap between the training and test sets in both the loss and accuracy plots provides empirical evidence that the proposed heterogeneous architecture effectively captures complex patterns without overfitting, thereby ensuring robust generalization to unseen network traffic.

Figure 15. Training behavior of the heterogeneous RBFNN.

From the other perspective, the proposed heterogeneous RBFNN demonstrates remarkable convergence efficiency, achieving initial training and testing baseline accuracies of 94.03% within the first epoch, as evidenced by the training logs in Table 8. This high-star performance suggests that the heterogeneous architecture effectively maps the feature space from the onset of the learning process. As training progresses, the model exhibits a steady, rapid learning curve, reaching 95.33% accuracy by the 10th epoch while incurring minimal computational overhead. The model’s ability to exceed the 96% accuracy threshold within a limited number of iterations (under 40 epochs) underscores its suitability for deploying the heterogeneous RBFNN architecture in diverse, high-velocity environments and in time-sensitive domains such as autonomous systems or real-time industrial monitoring, where low training latency and high initial reliability are essential.

Table 8. Early convergence and stability across selected training epochs.

Training Stage	Epoch	Training Accuracy (%)	Testing Accuracy (%)
Initial Steps	1	0.9403	0.9421
	2	0.9433	0.9451
Early Convergence	5	0.9491	0.9508
	10	0.9544	0.9533
	40	0.9599	0.9613

4.5. Confusion Matrix and FPR Analysis

To provide a detailed description of the model’s classification performance, the confusion matrix for the heterogeneous RBFNN was analyzed to assess its ability to categorize network traffic as normal or botnet attacks by comparing predicted class labels with the actual ground truth. As presented in Figure 16, the matrix displays the distribution of true positives (TPs), representing botnet traffic correctly identified as a botnet attack; true negatives (TNs), representing normal traffic correctly identified as normal; false positives (FPs), representing botnet incorrectly identified as normal traffic; and false negatives (FNs), representing botnet traffic incorrectly classified as normal traffic. The model correctly identified 23,821 attack instances and 23,569 benign instances, demonstrating high precision across both categories.

Figure 16. Confusion matrix of heterogeneous RBFNN.

Notably, the number of misclassifications remained low, with the model incorrectly flagging only 589 normal samples as attacks, resulting in a false-positive rate of approximately 0.022433%. Similarly, only 447 botnet attack samples were missed. confirms that the heterogeneous architecture effectively minimizes both types of errors, ensuring a reliable detection mechanism that balances high sensitivity with a low alarm rate, which is significant in the context of network security. These classification results align perfectly with the previously discussed AUC of 0.9968 and a high accuracy of 97.86%, providing further evidence that the low error rates in the confusion matrix are directly attributable to the model’s superior discriminatory power.

4.6. Computational Efficiency and Runtime Analysis

In practice, computational efficiency is a critical factor in evaluating detection models, particularly in real-time, high-speed network-monitoring environments. Therefore, the computational cost of the proposed heterogeneous RBFNN model was systematically evaluated in terms of total training time and classification accuracy across different batch sizes to assess its ability to converge in stable behavior within a reasonable timeframe.

As summarized in Table 9, the impact of batch sizes on the heterogeneous RBFNN performance was investigated. A notable observation regarding the relationship between computational speed and batch size: increasing the batch size from 64 to 512 significantly increases training duration, from 18.9904 to 35.939 min, while preserving robust accuracy above 96.6% across all configurations. Notably, the convergence behavior remained exceptionally stable throughout all experiments, with no visible oscillations in the loss or accuracy curves. Furthermore, consistent stability across all tested batch sizes was achieved with only 200 epochs, which is considered very efficient compared with other deep learning models that often require thousands of iterations to reach stability.

Table 9. Comparative analysis of batch size influence on heterogeneous RBFNN performance.

Batch Size	Accuracy	Total Training Time (Seconds)	Total Training Time (Minutes)
64	0.978	~ = 1139.4255	~ = 18.994
128	0.977	~ = 2156.3872	~ = 35.939
300	0.969	~ = 1275.13396	~ = 21.252
512	0.966	~ = 1376.8846(s)	~ = 22.948

A critical benchmark for deploying a botnet detection model in a real-time scenario is inference time, which measures the model’s ability to process traffic and deliver rapid responses with minimal latency promptly. The proposed heterogeneous RBFNN was tested across various batch sizes to measure the total inference duration and the per-sample latency. As summarized in Table 10, the proposed model demonstrates exceptional processing speed, maintaining microsecond-scale inference latency.

Table 10. Inference and latency time across different batch sizes.

Batch Size	Inference Time (s)	Per-Sample Inference Latency (µs)
64	0.0183645	3.7923 × 10−6
128	0.1844503	3.80896 × 10−6
300	0.049352	3.90896 × 10−6
512	0.0495723	3.77000 × 10−6

The results reveal that the per-sample inference latency is remarkably low and stable, around 3.7 to 3.9 microseconds. For example, at a batch size of 64, the latency was 3.79, while even at a larger batch size of 512, it remained highly efficient at 3.77, with minimal sensitivity to batch size. This consistency indicates that the heterogeneous architecture of RBFNN is highly optimized, ensuring that network traffic is classified almost instantaneously, which is considered vital for protecting the network from the propagation of botnet attacks. This level of performance is essential for modern high-speed networks, where even a millisecond of delay can cause significant data buffering or loss. Thus, by maintaining a microsecond-level response time, the proposed model ensures that malicious botnet activity is intercepted promptly, providing instantaneous protection without introducing any observable delay or bottleneck in the network environments.

The inference latency was calculated as the average time required to process a single network flow sample. To ensure high precision and mitigate system noise or JIT (Just-In-Time) compilation overhead, we used the @belapsed macro from the BenchmarkTools.jl suite in the Julia programming language. The total inference time for the entire test set T_total was measured over multiple iterations, and the per-sample latency $L_{Sample}$ was derived using the formula: $L_{Sample}={\displaystyle \frac{T_{total}}{N_{test}}}$, where $N_{test}$ represents the total number of samples in the test dataset. This rigorous benchmarking confirms that the heterogeneous RBFNN operates within the microsecond range, making it suitable for real-time line-rate network intrusion detection.

To further assess the computational efficiency of the proposed heterogeneous RBFNN model, we compared it with common deep learning architectures, including CNNs and LSTMs. A recent study [9] shows these deep models require long times—372.50 s for CNN and 455.19 s for LSTM—due to their complex structures and high parameter counts. Although they offer high accuracy, their iterative and convolutional operations result in millisecond-level inference latency. In contrast, our streamlined heterogeneous RBF architecture achieves microsecond-level latency through efficient radial basis function computations. allows it to outperform LSTM and CNN models at inference, significantly reducing computational requirements and meeting the strict real-time requirements of high-speed network monitoring.

4.7. Robustness Evaluation with Noise Injection

Often, the models were trained on clean features assumed to be noise-free to ensure training under ideal conditions. Therefore, to simulate realistic operating environments, additive noise was injected only into the test set to evaluate the robustness of the proposed heterogeneous RNFNN and generalization capability.

It is important to note that in intrusion detection systems, noise injection is a widely adopted research strategy for assessing model resilience under non-ideal operational data conditions. In practice, minor distortions often arise from packet loss, timing variations, feature-extraction inaccuracies, or packet delays. Accordingly, in this study, a noise level of $\sigma =$ 0.15 is used to simulate these common real-world network impairments, such as packet jitter and moderate synchronization errors. While many works limit noise injection to low levels (e.g., 1–5%) to represent minor feature distortions [58]. Other robustness-focused studies evaluate the model under higher noise ratios, reaching up to 15% or even 20%, to evaluate the classifier model under more challenging conditions and adverse scenarios [59].

In this study, additive Gaussian noise was used to investigate its impact on the performance of the heterogeneous RBFNN model. Controlled noise was injected into the test samples, with its magnitude gradually increased to simulate varying noise levels. Additive noise levels ranged from sigma equals 0.00 to $\sigma =0.00\mathrm{t}\mathrm{o}\sigma =0.15$ with a step size of 0.01, yielding 15 noise levels, including the baseline. The distribution of Gaussian noise is defined as $~\mathcal{N}(\mu ,\sigma )$, where the standard deviation $\sigma$ represents the noise scaling levels. Thus, for each sample vector in $X^{test}$ the additive noise $x^{noise}$ was added as: $X^{test}=X^{test}+x^{noise}$, where $x^{noise}\in ~\mathcal{N}(\mu ,0$) and $\sigma =$ {0.00, 0.01, 0.02, …, 0.15}. where $\sigma$ = 0.00 represents the scenario of a noise-free baseline. The mean $\mu$ is set to 0 to ensure that the noise is stochastic and does not introduce a systematic bias into the input features; it perturbs the feature values randomly without shifting their original means.

As illustrated in Figure 17, the proposed model exhibits high performance and stable behavior across a range of realistic noise levels, starting at the baseline noise level of 0.0, with an accuracy of approximately 0.978. As noise intensity gradually increases toward σ = 0.15, accuracy declines to nearly 0.960. Notably, the degradation follows a smooth curve with no evidence of abrupt cliffs or oscillations across the evaluated noise, indicating that the decision boundaries are not overly sensitive to moderate feature distortions and that the interpretation ability of heterogeneous structure can capture complex botnets that tend to add some noise to their structure to avoid detection. The overall reduction in accuracy remains around 2%, demonstrating the model’s robustness to noise. Both controlled and predictable degradation patterns confirm the structural stability of the heterogeneous RBF architecture and its ability to preserve discriminative capability under different conditions.

Figure 17. Performance and behavior of heterogeneous RBFNN across a range of realistic noise levels.

While conventional intrusion detection system robustness commonly evaluates noise intensity at moderate levels up to 0.15 or 0.20 to simulate a realistic network, in this study, our experiment intentionally extends the noise range from 0.0 to 1.0. Although these high noise levels exceed those commonly recorded in intrusion detection systems, this choice serves a broader academic objective. Specifically, this extended evaluation framework places the proposed heterogeneous RBFNN architecture beyond the scope of botnet attack detection alone. Demonstrating robustness across an expanded noise range indicates that the architecture is not task-specific and can be adaptable to other application domains with highly noisy inputs. Thus, adding noise up to 1.0 is not intended to reflect network traffic conditions, but rather to simulate extreme disruptions like total packet loss or severe electromagnetic interference (EMI), opening the door to further research in other applications.

As illustrated in Figure 18, a comprehensive series of experiments was conducted by progressively injecting Gaussian noise with a range from 0.0 to 1.0 to systematically observe the degradation of performance and determine the structural capability limits of the proposed model. The proposed model exhibits strong stability under low-to-moderate noise levels. For small noise levels (σ ≤ 0.2), the classifier retains high TNR, precision, recall, and F1-score, with only a slight reduction in overall accuracy. This behavior confirms that the heterogeneous RBF structure remains resilient to mild feature distortion.

Figure 18. Performance metrics of heterogeneous RBFNN under various noise injection levels.

As the noise level increases (0.3 ≤ σ ≤ 0.6), performance degrades but remains controlled. Although accuracy and F1-score decline, the model does not exhibit cliff collapse. Instead, the degradation exhibits a linear, smooth, gradual pattern, indicating stable functional-structural behavior.

Even under severe disruption, where noise levels approach the extreme σ = 1.0, the classifier model retains non-trivial discriminative capability. While the natural decline in TNR and increases in FPR are continuous, the model remains functionally operative, producing meaningful classification performance rather than totally collapsing into random predictions. This gradual degradation pattern is a key indicator of architectural robustness.

4.8. Detection Resilience Against Minority Botnet Types

To further evaluate the robustness of the heterogeneous RBFNN, we conducted a detailed analysis of its ability to identify rare botnet patterns. While the dataset in this study contains 30 botnet types, eight of them represent minority classes with significantly fewer samples than the majority of traffic. Detecting such low-frequency botnet attacks is particularly challenging, as many classifier models tend to be biased toward majority classes.

Our analysis concentrated on the False Negative Rate (FNR) for minority classes, as the failure to detect even a small number of botnet samples can lead to severe security breaches. Thus, FNR serves as a more reliable metric for evaluating the sensitivity of heterogeneous RBFNN to rare botnet variants.

Ultimately, the hierarchical structure of the proposed model successfully discriminates between subtle malicious patterns and legitimate traffic variations. To ensure these results are not an artifact of overfitting, especially for minority classes, the 5-fold cross-validation protocol (established earlier in Section 4.1) was strictly followed. As shown in Table 11, the model achieved a 100% detection rate across several minority botnet types, including Pony and Trojan.Snojan, across all folds. This stability in performance metrics, confirmed by high sensitivity and specificity, demonstrates that the non-uniform distribution of RBF centers enables the network to form highly accurate decision regions even for rare patterns. Consequently, the model maintains strong discriminative power without suffering from a biased majority-class orientation, thereby meeting the requirements for robust detection in underrepresented categories.

Table 11. Performance analysis of the heterogeneous RBFNN on minority botnet classes.

Minority Botnet Type	Total Samples	Detected Samples (TP)	Missed Samples (FN)	FN Rate (%)	Detection Rate (%)
Dr.Autoit	30	28	2	6.67%	93.33%
PUP.Plumbytes	748	711	37	4.95%	95.05%
Pony	6	6	-	0.00%	100.00%
Trojan.Snojan	25	25	-	0.00%	100.00%
Bundler.DownloadGuide	34	34	-	0.00%	100.00%
DownloadGuide	88	86	2	2.27%	97.73%
PUA	70	65	5	7.14%	92.86%
Ramnit	912	855	57	6.25%	93.75%
Total/Average	1913	1810	103	5.38%	94.62%

5. Computational Environment

This section outlines the setup of the computational environment used in this study to execute and evaluate the deep heterogeneous RBFNN model, which was maintained consistently across all experiments, ensuring reproducibility and facilitating objective comparison with existing botnet detection approaches. The summary of the Computational Environment is shown in Table 12.

Table 12. Summary of computational environment and system specifications.

Component	Specification
Processor	AMD Ryzen 9 5900HX with Radeon Graphics (3.30 GHz)
Number of Cores	8
Memory (RAM)	16.0 GB
Operating System	64-bit, Windows 11
Implementation Platform	Julia v1.10 (LTS)

6. Conclusions

This paper presented a heterogeneous radial basis function neural network for robust botnet detection based on flow-level packet length distribution features. Unlike homogeneous RBF configurations, the proposed architecture employs non-uniform RBF kernels, including Gaussian RBFNN, Inverse RBFNN, Power RBFNN, and Square RBFNN, to effectively capture the complex, nonlinear patterns of modern botnets, achieving high detection accuracy while maintaining low false-alarm rates and efficient computational performance, making it suitable for real-time scenarios. The experimental results demonstrated that the proposed model achieves a high detection performance of 97.86%, with strong generalization stability and consistent convergence across multiple experiments, as evidenced by k-fold cross-validation. A strong discriminative capability, with an AUC of approximately 0.9968, indicates excellent separation between botnet attacks and normal traffic.

In addition, the confusion matrix analysis further confirmed reliable detection performance, with a notably low false-positive rate of nearly 0.02, a critical requirement for practical deployment in operational network environments, while maintaining a low inference latency of 3.79 microseconds per sample. These findings confirm that integrating heterogeneous kernels provides superior computational efficiency.

Furthermore, the robustness of the proposed model under controlled Gaussian noise was evaluated, demonstrating resilience with only minor performance degradation, while maintaining an accuracy of approximately 96% even at a significant noise level of 0.15.

Ultimately, the heterogeneous RBFNN model demonstrated strong detection towards minority botnet types, not merely dominant classes. Several low-frequency botnet types achieved perfect detection rates despite their limited representation in the dataset, and the remaining minority categories maintained consistently high detection performance.

One of the core strengths of the proposed heterogeneous architecture is its high generalization capability. By leveraging multiple RBF kernels, the model captures a broader spectrum of traffic patterns, enabling it to detect evolved botnet variants.

However, we must acknowledge certain limitations of the current study that present avenues for future improvement. Firstly, manual optimization of heterogeneous kernel parameters alpha, beta, and gamma is empirically intensive and time-consuming. On the other hand, although the proposed model exhibits high detection accuracy, using multiple deep layers slightly increases training computational overhead compared to traditional homogeneous RBF architectures.

Overall, the empirical results confirm that the heterogeneous RBFNN with multiple RBF kernels provides an effective and efficient solution for botnet detection. Future work will focus on extending the proposed framework to include traffic with TCP, UDP, and ICMP protocols, thereby supporting a wider range of botnet families.

To address challenges from modern HTTPS encryption, we plan to use TLS handshake metadata, such as cipher suite distributions and certificate exchange patterns. By integrating these elements into spatial mapping, the model can then detect malicious C2 activities over encrypted channels. Importantly, this approach works without payload decryption, thereby ensuring both high detection resilience and data privacy.

From a practical perspective, the heterogeneous RBFNN model simplifies deployment in modern network security infrastructures, making it suitable for seamless integration with existing solutions such as SDN controllers or edge security gateways. In a real-world scenario, network traffic is captured via Tshark, and specialized Lua scripts extract the necessary flow features. Given its microsecond-level latency, the HRBFNN can evaluate these features in real-time, enabling the system to trigger automated mitigation rules the moment a botnet signature is identified, thereby protecting the network without degrading performance.