Shield-X: Vectorization and Machine Learning-Based Pipeline for Network Traffic Threat Detection ^†

Abstract

This paper presents an integrative methodology combining advanced network packet vectorization techniques, parallel processing with Dask, GPU-optimized machine learning models, and the Qdrant vector database. Our approach achieves a 99.9% detection rate for malicious traffic with only a 1% false-positive rate, setting new performance benchmarks for cybersecurity systems. The methodology establishes an average detection time limit not exceeding 10% of the total response time, maintaining high precision even for sophisticated attacks. The system processes 56 GB of PCAP files from Malware-Traffic-Analysis.net (2020–2024) through a five-stage pipeline: distributed packet processing, feature extraction, vectorization, vector database storage, and GPU-accelerated classification using XGBoost, Random Forest, and K-Nearest Neighbors models.

1. Introduction

Cybersecurity threats constantly evolve, making network security an increasingly complex challenge. Traditional detection methods, often based on known signatures and log records, cannot keep pace with the speed and sophistication of modern attack attempts and vectors [1], especially those using evasive techniques [2] or representing zero-day threats [3]. Given this scenario, more dynamic and intelligent approaches become necessary to identify malicious traffic in real-time [4]. The essence of our approach lies in transforming raw network packet data into information-rich vector representations [5], allowing the capture of subtle patterns and complex relationships in traffic that escape conventional analyses [6].

Recent studies highlight different approaches and architectures aimed at improving real-time detection, scalability, and accuracy of network security systems. Komisarek et al. [7] present a machine learning-based approach for anomaly and cyberattack detection in network traffic data transmitted in continuous streams using Big Data frameworks like Apache Kafka and Spark. Srivatsa et al. [8] introduce the LogSense architecture, a scalable solution for real-time log anomaly detection using Large Language Models and advanced stream processing frameworks, integrated with the Qdrant vector database for efficient retrieval. Yu et al. [9] propose LogMS, a log anomaly detection method based on multi-source information fusion and labeling probability estimation using Long Short-Term Memory models. These works highlight the importance of efficient and scalable methodologies for anomaly detection in computer networks but present significant gaps regarding adaptability in dynamic environments and efficiency in resource usage.

2. Proposed Method

The proposed methodology was developed with a focus on feasibility and validated through the implementation of a complete pipeline (Figure 1a). Data collection was performed from datasets from the Malware-Traffic-Analysis.net website, covering 2020 to 2024 and totaling 56 GB of PCAP files containing raw network data in packet captures initially processed to extract relevant characteristics. Network traffic data was collected using the Scapy tool to capture network packets in PCAP format containing information about network traffic, including source and destination IP addresses, ports, protocols, TCP flags, packet length, payload size, and content [10,11].

Feature extraction from network packets transforms raw data into a structured format, enabling the creation of a more informative dataset for classification, prediction, and clustering tasks. Selected features include timestamp, src ip, dst ip, src port, dst port, protocol, length, inter-packet time, tcp flags, payload len, among others [6,12,13,14,15,16,17,18]. The importance of specific features has been established through extensive research: packet size for traffic pattern identification [6], protocol differentiation for detecting specific patterns [17], IP addresses for tracking traffic origin and destination [18], ports for identifying service types [16], TTL for identifying potentially malicious traffic [14], TCP flags as critical indicators of connection state [12], packet length for differentiating traffic types [13], timestamps for temporal traffic analysis [15], and inter-packet time for understanding traffic behavior [19].

In order to prepare data for machine learning analysis, features are normalized using the Scikit-Learn library, ensuring all features are on the same scale, preventing larger values from dominating during model training (Figure 1b). IP addresses are vectorized using CountVectorizer, transforming addresses into numerical embeddings, while TCP flags are converted to numerical values using MultiLabelBinarizer, applying the one-hot encoding technique. The resulting vectors from normalization and vectorization are stored in Qdrant, optimized for vector search. The collection was configured with a specific vector size and a cosine distance metric for efficient retrieval [20]. Insertion was performed in batches using process chunk managed by a Dask client, allowing parallel processing [21]. Each point in Qdrant contains the vector and a payload with details of the original packet.

The malicious network traffic detection methodology includes training machine learning models using collected and processed data essential for identifying anomalous patterns. Three main models were trained using a Dask cluster with GPUs configured via LocalCUDACluster, leveraging parallel processing capacity [21]. K-Nearest Neighbors was implemented using the cuML library to leverage GPU acceleration with hyperparameter search, including the number of neighbors and the distance metric [22]. Random Forest was trained using cuML implementation configured for different numbers of estimators and maximum depths, providing robustness against overfitting through aggregation of multiple decision trees. XGBoost utilized the GPU-optimized version trained with different learning rates, numbers of estimators, and maximum depths [23]. Training used GridSearchCV for hyperparameter search, ensuring optimal parameters, with 10 billion stratified samples maintaining proportional class representation.

Real-time analysis of network data is essential for efficiently identifying and mitigating cybersecurity threats using Apache Kafka as a distributed streaming platform acting as a backbone for ingestion, processing, and continuous distribution of large data volumes [10]. The analysis flow operates through continuous ingestion where KafkaProducer captures network packets in real-time, extracts characteristics, and sends information to a dedicated Kafka topic. Distributed consumption and processing employ KafkaConsumer reading messages from topics with processing load distributed using Dask. Each received packet transforms into a feature vector by applying normalization and vectorization techniques from the preprocessing phase.

Generated vectors are compared with data stored in the Qdrant vector database using cosine similarity search to identify previously stored packets vectorially similar to the current packet. Score calculation based on cosine similarity, where 0.9 suggests a small angle between vectors, indicating high similarity [24], effective for detecting patterns and anomalies in high-dimensional data [25]. Results with high similarity indicate potential anomalies selected for the second verification layer using trained XGBoost, KNN, and Random Forest models, generating a final prediction if model confidence exceeds 80%. All packet processing is distributed using Dask with GPU integration via LocalCUDACluster, allowing parallel processing of multiple packets, optimizing available hardware resources, with garbage collection for efficient memory management.

3. Results

Experiments were conducted on hardware featuring Intel Xeon E5-2680 v4 @ 2.40 GHz with 28 cores, 189 GB DDR4 RAM, 2x NVIDIA P40 24 GB GPUs, 1x NVIDIA 3090 24 GB GPU, and NVMe SSD 2 TB storage. Software environment included Ubuntu 22.04 LTS, XGBoost 3.0.0, Rapids cuML 24.04, Dask 2024.5.0, Qdrant 1.12.6, Scapy 2.5.0, and Apache Kafka 3.4.0. The dataset consisted of 56 GB PCAP files from Malware-Traffic-Analysis.net totaling approximately 180 million packets for training, plus 10 GB controlled test data from an isolated environment executing recent malware samples not included in the training set.

All models presented exceptional performance with accuracy above 0.96, as shown in

Table 1. Comparative model performance.

Model	Accuracy	Macro Precision	Macro Recall	Macro F1-Score
XGBoost	0.9700	0.95	0.94	0.94
Random Forest	0.9600	0.98	0.95	0.96
KNN	0.9736	0.9787	0.9775	0.9773

. The KNN model stood out slightly with the highest accuracy (0.9736) and best values for macro metrics, evaluating average performance among all classes without considering size, which is particularly important in malware detection scenarios where minority classes are as important as majority classes [26]. Random Forest presented the best balance between precision and recall with a macro F1-score of 0.96, while XGBoost obtained slightly lower results in macro metrics but maintained excellent performance considering class weights. When considering weighted average by class size, all models converge to similar values (0.97), indicating consistent performance between classes of different sizes, suggesting robustness against class imbalance, a common challenge in malicious traffic detection [27].

The system achieved a detection rate of 99.9% for simulated malicious traffic using the tcpreplay tool (

Table 2. Summarizes the detection performance metrics.

Metric	Simulated Traffic	Real Traffic
Detection Rate	99.9%	97.8%
False Positives	1%	5%
Average Detection Time	8 ms	12 ms
Throughput (packets/s)	1.2 M	800 K

), highlighting the model’s ability to accurately identify anomalous patterns and cybersecurity threats. False-positive incidence was 1% with simulated traffic and 5% with monitored real traffic, which is considered low, demonstrating system robustness. The distributed Dask environment, combined with GPU computational power, enabled real-time analysis of billions of packets, maintaining low latencies and high throughput. Architecture based on Dask and GPUs optimizes resource usage, allowing efficient workload balancing and better hardware utilization with 85–95% GPU utilization during intensive processing, load balancing across 3 simultaneous GPUs, and 70% reduction in training time compared to CPU-only processing.

Compared to traditional intrusion detection methods, the proposed methodology presents significant advantages in precision, efficiency, and scalability. While traditional methods fail to detect zero-day attacks or unknown variants, vectorization and similarity approach in Qdrant identifies subtle deviations from normal behavior, achieving high detection rates in simulated scenarios, including diverse malicious patterns. Rule-based systems generate many alerts for legitimate traffic superficially resembling malicious patterns; a combination of vector search with ML classification resulted in only 1% false positives in simulations and 5% in real traffic, significantly lower than observed in many traditional IDS/IPS systems, alleviating the burden on analysts [19,28]. Architecture demonstrated capability to process billions of packets with low latency, a performance level difficult to achieve with traditional monolithic systems without massive investments in specialized hardware [29,30]. Adding more processing nodes allows scaling analysis capacity as traffic volume increases.

The methodology provides a powerful tool for early threat detection, significantly contributing to critical infrastructure protection. Real-time detection capability of malicious traffic patterns allows for proactive response, mitigating potential attacks before causing significant damage. Reducing false positives alleviates the burden on security analysts, allowing focus on real threats and improving operational efficiency [19,28]. Despite high accuracy, intensive memory usage by models with increasing network traffic presented significant challenges requiring high-performance machines and GPU usage to maintain system efficiency and scalability. Three GPUs were used simultaneously in model training; nevertheless, the implementation demonstrated a methodology that was highly effective in identifying anomalies and reducing false positives using parallel processing and GPU resources handling large data volumes in real-time.

4. Conclusions

This study presented an innovative approach for detecting malicious network traffic combining network packet feature vectorization with vector databases like Qdrant and advanced machine learning techniques. The methodology achieved a 99.9% detection rate with 1% false positives in simulated environments, establishing new benchmarks for cybersecurity systems. Future work includes memory consumption optimization, incremental learning implementation for adapting to new attack patterns, and integration with existing SIEM systems, enhancing operational workflows while maintaining the detection accuracy required for defending against sophisticated cyber threats.