RandomForestNN Classification for Adversarial AI Black-Box Techniques on MITRE ATT&CK Labeled Data

Abstract

Research examining the security of network intrusion detection systems is vital for protecting modern digital infrastructure from increasingly sophisticated threats. This study investigates how machine learning network security models, trained with tactical frameworks like MITRE ATT&CK, respond to adversarial examples crafted through black-box optimization techniques. Using three attack algorithms, HopSkipJump, Simultaneous Perturbation Stochastic Approximation Attack and the Square Attack algorithms, we demonstrate that the Random Forest model remains vulnerable despite tactical framework integration. For example, the HopSkipJump attack achieved a 92% success rate in causing malicious traffic to appear benign. Our analysis reveals which network traffic features are most susceptible to manipulation, with model performance metrics declining significantly under adversarial conditions. These findings highlight an important gap between theoretical security frameworks and practical implementation that must be addressed to develop more robust defense systems. By identifying specific vulnerabilities, this research contributes valuable insights that can inform improved adversarial robustness in operational network security environments.

1. Introduction

Adversarial machine learning (AML) represents a critical security domain focused on the vulnerabilities of machine learning (ML)/artificial intelligence (AI) systems to malicious exploitation [1]. At its core, AML examines adversarial examples of inputs deliberately modified with perturbations, often imperceptible to humans, designed to deceive ML/AI models into producing erroneous outputs. While modern interest often centers on deep learning, the field has a twenty-year history, originating with efforts to evade linear spam filters and biometric systems in the early 2000s. Today, AML has evolved into a multidisciplinary intersection of security, assurance, and policy, addressing threats that span from computer vision and natural language processing to generative AI and agentic systems. The landscape of adversarial threats is traditionally organized into a lifecycle-centric taxonomy that maps attack vectors to specific stages of the AI pipeline. During the data collection and preparation phase, systems are vulnerable to poisoning attacks, where an adversary injects malicious data to compromise the future model’s integrity. In the model training stage, threats include backdoor embedding and gradient manipulation. Model evaluation and testing is the primary theater for evasion attacks, where attackers query the model to find blind spots. Finally, during model deployment and monitoring, adversaries may employ model extraction to steal proprietary IP or use prompt injection to bypass the safety filters of large language models.

This research investigates the vulnerability of a machine learning (ML)-based network security model, specifically the Random Forest (RF) classifier, in the face of adversarial attacks, specifically those crafted through black-box optimization techniques. To measure performance degradation in classification of various MITRE ATT&CK tactics [2] including Exfiltration, Exploitation, Reconnaissance, and Initial Access [3,4], when adversarial data was used against the RF model, the following were used from the Adversarial Robustness Toolbox (ART) [5]: (i) the HopSkipJump [6] attack; (ii) CleverHans [7] to implement the Simultaneous Perturbation Stochastic Approximation Attack [8]; and (iii) Foolbox [9] to implement the Square Attack [10] algorithm. The final classification was performed using neural networks. Through comprehensive perturbation analysis, we identify which network traffic features are most susceptible to manipulation and examine how adversarial attacks affect classification performance across different attack categories.

Using a Multilayer Perceptron Classifier for Neural Networks, our results demonstrate alarming vulnerabilities in the trained RF model despite tactical framework integration; for example, model performance decreased dramatically under adversarial conditions, with precision falling from 98.7% to 13.9% and F1 scores dropping from 98.7% to 14.9%. This work provides actionable insights into how adversarial robustness can be measured, understood, and potentially improved in operational ML-based network security systems.

The rest of this paper is organized as follows. Section 2 presents the background and related works; Section 3 explains the datasets used in this study; Section 4 explains the three attacks from the Adversarial Robustness Toolbox that will be used in this study; Section 5 presents the results; Section 6 presents the conclusions; and Section 7 presents future works.

2. Background and Related Works

The evolution of adversarial ML in network security represents a critical challenge that has shaped the development of modern defense systems.

2.1. Adversarial Machine Learning Research Landscape

The MITRE ATLAS framework provides yet another standardized methodology for identifying and mitigating AML risks [1]. MITRE ATLAS is an adaptation of the globally recognized MITRE ATT&CK framework, specifically tailored to the unique technical and operational nuances of AI systems. It catalogs over fifty distinct adversarial tactics and techniques across the ML Kill Chain: reconnaissance, resource development, and exfiltration. By providing a common language for threat modeling and red teaming, MITRE ATLAS enables security practitioners to simulate end-to-end attack paths and correlate technical findings with specific defensive controls. In practice, MITRE ATLAS serves as a critical operational bridge for developers and security teams. It converts theoretical AML research into actionable guidelines, allowing organizations to move beyond mere patching toward comprehensive security by design. When used in conjunction with broader governance frameworks like the NIST AI Risk Management Framework (RMF) or the EU AI Act, MITRE ATLAS helps ensure that AI deployments are not only technically robust but also compliant with emerging regulatory standards for safety and trustworthiness. As AI becomes more integrated into safety critical environments, the systematic application of such frameworks is essential for maintaining the integrity of automated decision making systems.

2.2. Tagging Alerts to Adversaries: ML-Enabled Classification Using MITRE ATT&CK

Talpur et al. (2025) [11] addresses a critical bottleneck in modern enterprise security: the overwhelming volume of alerts generated by intrusion detection systems (IDSs). Security Operations Centers (SOCs) frequently suffer from alert fatigue due to the high rate of false alarms, which can lead to critical, high-risk events being overlooked or undetected. To mitigate this, the authors propose an automated system that uses machine learning to enrich IDS alerts with context from the MITRE ATT&CK framework, thereby enabling faster prioritization and a more comprehensive understanding of multi-stage attack campaigns. The methodology centers on leveraging the MITRE ATT&CK matrix for enterprise environments to uncover correlations between technical alerts and specific adversarial tactics. The researchers employ two distinct machine learning architectures: a classical Multilayer Perceptron (MLP) and a modern large language model (LLM) based on Bidirectional Encoder Representations from Transformers (BERT). By utilizing these models, the system can automatically tag alerts with relevant tactical information, which traditionally requires time-consuming manual analysis by human experts. The study further investigates the influence of various feature groups on classification performance, utilizing feature selection to optimize the training process and exclude irrelevant data. The effectiveness of this approach was validated through extensive experiments using both real-world and public datasets. The authors collaborated with an enterprise SOC to gain access to two months of authentic security logs, where their pretrained models achieved an impressive accuracy of 95% in automated alert classification. Additionally, the transformer-based model was tested against public datasets, including the Lateral Movement Dataset (LMD), Dataset for Advanced Persistent Threats (DAPT), and Unraveled, yielding an accuracy of up to 99%. These results suggest that integrating MITRE ATT&CK knowledge into automated security workflows can significantly enhance the precision of threat detection while reducing the manual burden on security analysts. Talpur et al. (2025) [11] is a defensive study focused on improving the utility and accuracy of IDSs through automated classification and not an adversarial robustness study focused on the vulnerabilities of such models. The latter investigates whether integrating tactical frameworks like MITRE ATT&CK into model training actually enhances resilience against malicious exploitation or if fundamental security gaps persist. A primary methodological difference lies in the application of adversarial machine learning (AML). Talpur et al. (2025) [11] use ML (i.e., MLP and BERT) as a tool for classification. In contrast, this research employs the Adversarial Robustness Toolbox (ART) to launch black-box optimization attacks, specifically the HopSkipJump, Simultaneous Perturbation Stochastic Approximation (SPSA), and Square Attack algorithms. Instead of aiming for a 95% detection accuracy, this study seeks to find the “attack success rate,” demonstrating how small perturbations in network traffic features can cause malicious activity to be misclassified as benign. The findings of the two papers present a stark contrast regarding the reliability of ML-based security. Talpur et al. (2025) [11] report high success and precision in classifying threats. Conversely, the present research reveals vulnerabilities, reporting that the HopSkipJump attack achieved a 92% success rate in evading detection, causing model precision to plummet from 98.7% to just 13.9%. Furthermore, the datasets utilized differ; while Talpur et al. (2025) [11] used enterprise logs and APT-specific public datasets, this research utilizes the UWF-ZeekData22 and UWF-ZeekData24 datasets to perform its perturbation analysis. Ultimately, while Talpur et al. (2025) [11] advocate for the adoption of ML/MITRE integration for enterprise efficiency, this paper serves as a critical warning that such integrations remain highly susceptible to adversarial manipulation.

2.3. AI-Based MITRE ATT&CK Detection System

Koutras et al. (2025) [12] explore the integration of large language models (LLMs) into the domain of detection engineering to address the rapidly evolving landscape of cyber threats [12]. The study’s primary objective is to evaluate the feasibility of using generative AI to automate the development of Kusto Query Language (KQL) queries for Microsoft Sentinel, thereby enhancing the efficiency of Security Operations Centers (SOCs). Traditional threat detection often relies on manual rule creation, a process that is inherently time consuming and prone to human error. By leveraging LLMs, the authors aim to bridge the gap between theoretical security analytics and practical, automated implementations that can identify hostile techniques in real-time. A central technical contribution of this work is the application of advanced prompt engineering techniques, specifically the Chain of Thought (CoT) methodology. The researchers progressed from naive prompting to CoT, which encourages the AI to generate structured, contextually accurate queries by following a logical reasoning path. This approach was rigorously tested across various open-source and closed-source models, with performance measured through a dual-scoring accuracy framework. The findings indicate that CoT significantly boosts the precision of AI-generated security content, and among the models tested, ChatGPT-4o-mini demonstrated superior capability in producing well-structured KQL queries. Beyond query generation, the system is designed to provide thorough explanations for its outputs, which aids security analysts in optimizing threat response and reduces the overall manual labor required for maintenance. By integrating real time intelligence from the MITRE ATT&CK framework with Microsoft Sentinel log analysis, the proposed technology offers a foundation for a more proactive and productive security posture. Ultimately, the study concludes that AI-driven automation is a viable path forward for speeding up detection and improving decision making in the face of sophisticated adversarial behavior. This paper presents a fundamental departure from the focus of the AI-based feasibility study. While Koutras et al. (2025)’s study [12] is inherently constructive and defensive, focusing on how AI can build better tools, this paper is evaluative and adversarial, focusing on how those very models can be defeated. The former treats AI as a solution for SOC efficiency, while the latter treats it as a security liability that requires rigorous robustness testing against black-box optimization attacks. Methodologically, the two papers utilize different branches of artificial intelligence. Koutras et al. (2025) [12] focus on generative AI and large language models to translate natural language descriptions of attacks into technical detection rules. This paper investigates discriminative machine learning, specifically a Random Forest classifier for network traffic analysis. Instead of prompt engineering, this paper employs the Adversarial Robustness Toolbox (ART) to execute sophisticated evasion techniques: the HopSkipJump, Simultaneous Perturbation Stochastic Approximation (SPSA), and Square Attack. Furthermore, the role of the MITRE ATT&CK framework differs significantly between the two papers. For Koutras 2025 [12], MITRE ATT&CK serves as the contextual source for query generation, helping the LLM understand what to look for in logs. However, this paper uses MITRE ATT&CK as a labeling standard for its training datasets, UWF-ZeekData22 and UWF-ZeekData24, and then demonstrates that even when a model is trained specifically on these tactical categories, it remains highly vulnerable to evasion. For instance, while Koutras 2025 [12] made improvements in query precision, this paper reports a catastrophic 92% success rate for the HopSkipJump attack in bypassing detection, with model precision dropping from 98.7% to a mere 13.9%. Ultimately, the primary difference lies in their underlying philosophy toward AI in cybersecurity. Koutras et al. (2025) [12] view AI as an essential assistant for automating detection engineering to close the gap between theory and practice. Conversely, this paper highlights a critical gap between theoretical model performance and practical security implementation, warning that the integration of tactical frameworks like MITRE ATT&CK does not inherently guarantee resilience against malicious manipulation. Together, they represent two sides of the same coin: the promise of AI for defensive automation and the peril of AI’s susceptibility to adversarial exploitation.

2.4. Adaptive Threat Modeling with MITRE ATT&CK

Maniyat et al. (2025) [13] address the critical limitations of traditional rule-based cybersecurity systems, which often struggle to detect evolving adversarial behaviors and lack operational interpretability. To overcome these challenges, the authors propose a novel ML framework designed for real-time detection and attribution by integrating structured telemetry with the MITRE ATT&CK matrix. This system moves beyond static defenses by utilizing a hybrid approach that combines supervised classification, anomaly detection, and temporal reasoning to identify complex attack chains. The architecture of this framework is built on a unified log to threat pipeline that ingests data from heterogeneous sources: Sysmon, Zeek, and OSQuery. This telemetry is mapped to specific MITRE ATT&CK techniques (TTPs) using STIX/TAXII enrichment and sigma-based rules, transforming raw events into feature rich representations suitable for ML inference. A standout feature of the methodology is its focus on explainable AI; the framework computes threat scores and provides contextual attribution through SHAP (SHapley Additive exPlanations)-based values. This approach enhances analyst trust by detailing exactly which features contributed to a specific detection, effectively bridging the gap between automated analysis and human understanding. Adaptability is a core component of the proposed system, specifically addressing the phenomenon of concept drift, where adversary tactics change over time to evade detection. The framework incorporates an entropy-based drift detection mechanism alongside a continual learning module. This allows the system to be retrained using synthetic attack simulations and direct analyst feedback, ensuring that the detection models remain effective against zero-day threats and polymorphic behaviors. By automating the update cycle, the framework reduces the manual burden on SOCs while maintaining high defensive standards. Experimental evaluations of the framework across various simulated attack scenarios demonstrate its operational feasibility and high performance. The researchers report an F1 score of 95.0% and robust MITRE ATT&CK coverage of approximately 89%. Furthermore, the system achieves subsecond detection latency, which is essential for real-world SOC integration where rapid response is vital. Ultimately, the study presents a foundation for future threat modeling that is not only data-driven but also adaptive and interpretable for security practitioners. While both the Maniyat et al. (2025) [13] study and this paper utilize the MITRE ATT&CK framework and machine learning, they occupy opposite ends of the cybersecurity spectrum. Maniyat et al. (2025)’s [13] study is fundamentally constructive and defensive, focusing on the creation of an advanced detection pipeline to improve SOC efficiency. In contrast, this paper is evaluative and adversarial, investigating the inherent vulnerabilities of such ML-based systems when faced with black-box optimization attacks. While the former seeks to prove that AI can solve detection gaps, the latter serves as a warning that AI itself introduces a new attack surface. The methodological focus of the two papers also differs significantly regarding the type of AI employed and the tools used for evaluation. Maniyat et al. (2025) [13] emphasize generative and hybrid ML, utilizing SHAP values for explainability and entropy-based mechanisms for drift detection. Conversely, this paper evaluates a discriminative ML implementation of Random Forest specifically to measure its resilience against evasion. Instead of building a detection rule generator, the researchers in this paper apply the Adversarial Robustness Toolbox (ART) to launch sophisticated black-box attacks, including HopSkipJump, SPSA, and Square Attack algorithms. The role of the MITRE ATT&CK framework varies between the two papers as well. In Maniyat et al. (2025) [13], MITRE ATT&CK is the operational standard for attributing malicious events and enriching telemetry. In this paper, MITRE ATT&CK is used as the standardized labeling scheme for the UWF-ZeekData22 and UWF-ZeekData24 datasets to determine if tactical alignment improves model robustness. The findings in this paper suggest that tactical alignment does not inherently protect models, as the HopSkipJump attack achieved a 92% success rate in causing malicious traffic to be misclassified as benign. This directly challenges the assumption that better context, as proposed in Maniyat et al. (2025) [13], equals better security against a determined adversary. Ultimately, the results of the two papers present a striking contrast in the perceived reliability of machine learning in security. Maniyat et al. (2025) [13] report a highly successful 95.0% F1 score as evidence that AI can modernize threat modeling. However, this paper reveals that under adversarial conditions, model precision can plummet from 98.7% to a mere 13.9%. This comparison highlights a critical gap in current research: while defensive innovations like those in Maniyat et al. (2025) [13] provide powerful tools for detection, the adversarial analysis in this paper demonstrates that these tools remain fragile and easily manipulated unless robustness is prioritized during the design phase.

2.5. Other Works on Adversarial Machine Learning

Goodfellow et al. (2015) [14] introduced the concept of adversarial examples and demonstrated how small, carefully crafted perturbations in input data could lead to significant misclassification in deep learning models. Their introduction of the Fast Gradient Sign Method (FGSM) laid the groundwork for a new field of research that would profoundly impact network security, particularly in the realm of NIDS.

Building on this theoretical foundation, researchers began exploring practical implementations in network security contexts. Verma et al. (2018) [15] pioneered network traffic obfuscation using adversarial ML approaches, while Usama et al. (2019) [16] extended this work by demonstrating effective black-box adversarial attacks on network traffic classification without requiring knowledge of the target system’s architecture. The field took another significant leap forward with Han et al. (2021) [17], who proposed a sophisticated two-step process using Generative Adversarial Networks (GANs) and Particle Swarm Optimization (PSO) to generate adversarial examples while maintaining protocol constraints, achieving remarkable evasion rates exceeding 97% in certain cases.

Some work has highlighted the critical importance of connecting theoretical capabilities with practical, industry-ready solutions. Subbaratinam (2022) [18] investigated the relationship between operational context and vulnerability assessment effectiveness through a comparison of the context-agnostic Common Vulnerability Scoring System version 2 (CVSSv2) model, threat intelligence data, and MITRE ATT&CK adversary tactics, techniques, and procedures. His research demonstrated that despite its popularity, one of the major shortcomings of CVSSv2 is that the model assesses vulnerabilities in isolation with little regard for the operating environments that contextualize severity, impact, and business criticality.

As the field matured, several comprehensive surveys emerged that helped systematize our understanding of these advances. Recent work by Malik et al. (2024) [19] provided a thorough categorization of adversarial ML attacks and defensive controls, while Khazane et al. (2024) [20] specifically examined ML adversarial attacks in Internet of Things (IoT) networks. Li (2024) [21] and Ennaji et al. (2024) [22] further contributed to this knowledge base by examining impacts, challenges, and mitigation strategies in cybersecurity and NIDS respectively.

The defensive landscape has evolved in parallel with these attacks. Paya et al. (2024) [23] made significant strides with Apollon, a robust defense system that utilizes multiple classifiers and Multi-Armed Bandit algorithms to protect ML-based intrusion detection systems (IDSs) from adversarial attacks. This work, combined with the theoretical frameworks proposed by Khan and Ghafoor (2024) [24] and the vulnerability assessments conducted by Alhajjar et al. (2024) [25], has enhanced our understanding of how to protect network security systems against adversarial threats.

Recent years have seen a focus on generating increasingly sophisticated and realistic adversarial examples. Kumar et al. (2024) [26] introduced a groundbreaking variational autoencoder-based approach to ensure generated adversarial traffic adheres to domain constraints. Building on this work, Sadeghzadeh et al. (2024) [27] developed Adversarial Network Traffic (ANT) attacks, introducing three novel methods—AdvPad, AdvPay, and AdvBurst—that demonstrate how universal adversarial perturbations can effectively manipulate various aspects of network traffic.

However, a crucial insight into the state of the field comes from Jedrzejewski et al. (2024) [28], whose systematic literature review highlights a significant disconnect between academic adversarial ML research and its practical application in industry. This observation points to a critical research gap: while the field has made remarkable progress in understanding and implementing adversarial attacks against network security systems, there remains a substantial divide between theoretical capabilities and practical, industry-ready solutions.

This disconnect between theory and practice is particularly evident in how current research approaches domain constraints in network traffic. While researchers have made significant progress in addressing technical constraints, there is limited understanding of how these constraints interact with actual adversary behaviors and procedures. Multiple systematic reviews indicate that current research focuses primarily on the mechanical aspects of generating adversarial examples without considering how these techniques map to real-world attack patterns and tactics [21,28].

The integration of tactical frameworks like MITRE ATT&CK with adversarial ML techniques is an unexplored area that could provide a more comprehensive understanding of network security evasion techniques. By combining structured attack patterns with advanced adversarial techniques, future research has the opportunity to develop approaches that address both the technical and tactical aspects of adversarial attacks in network environments, potentially closing the gap between academic research and practical implementation.

3. The Datasets Used

The two datasets used for this study, UWF-ZeekData22 [29] and UWF-ZeekData24 [30], are both available at [31].

3.1. UWF-ZeekData22

The UWF-ZeekData22 dataset [29] is a comprehensive network traffic dataset based on the MITRE ATT&CK framework, collected at the University of West Florida. This dataset represents the first network collection specifically created using Zeek and labeled using the MITRE ATT&CK framework, providing a unique resource for network security researchers studying adversary behavior leading up to attacks, developing user profiles of potential attackers, and identifying attack traffic.

This dataset contains network traffic data collected from 81 subnets, totaling approximately 208.62 GB of Zeek logs and PCAPs. The data includes 140,477,116 connection records across multiple Zeek log files. The dataset provides a balance of malicious (9,280,869) and non-malicious (9,281,599) traffic samples, collected within a controlled Cyber Range environment using Security Onion to capture traffic from real (not simulated) attack scenarios [29,32].

The dataset includes 16 different Zeek log files with various record counts, including conn (140,477,116 records), dns (191,049,652 records), dhcp (2,356,475 records), and notice (144,946 records). The malicious traffic is labeled according to the tactics from the MITRE ATT&CK framework, with Reconnaissance accounting for 99.97% of the attacks (9,278,722 records), followed by Discovery (2086 records or 0.022%), and other tactics such as Credential Access (31 records), Privilege Escalation (13 records), and Exfiltration (seven records) making up smaller percentages.

Network traffic in the dataset is distributed across three protocols:

• TCP: 33,987,569 records;
• UDP: 105,098,306 records;
• ICMP: 1,391,241 records.

With 254 unique source IP addresses and 4324 unique destination IP addresses, the dataset provides a rich environment for studying diverse network behaviors and attack patterns. What makes UWF-ZeekData22 particularly valuable is its ability to detect adversary behavior leading up to an attack and develop profiles of users or user groups intending to perform attacks, rather than just identifying attacks after they have occurred. This makes it an excellent resource for developing and testing NIDS and studying adversarial behavior patterns. The publicly available data represents a modern benchmark for NIDS research using the MITRE ATT&CK framework [29].

3.2. UWF-ZeekData24

The UWF-ZeekData24 dataset [30] is a modern, controlled network traffic dataset designed to address the limitations of existing repositories such as KDDCup99 [33] and NSL-KDD [34], specifically regarding the lack of comprehensive coverage for contemporary attack techniques. Developed by researchers at the University of West Florida, the UWF-ZeekData24 dataset is aligned with the Enterprise MITRE ATT&CK Framework and utilizes Zeek, an open-source network traffic analyzer [35], to capture detailed network logs. A distinguishing feature of this dataset is its construction through a controlled experiment involving automated attacks generated by bash scripts and cron jobs, rather than simulated or crowd-sourced traffic. This methodology employs “mission logs” to precisely label data alongside attack events, ensuring a high degree of accuracy for ML applications. This dataset encompasses fourteen attack families, with Credential Access constituting the majority (approximately 90.88%) of the recorded attacks, followed by Reconnaissance. By providing a clean, meticulously labeled environment, UWF-ZeekData24 [30] aims to serve as a benchmark for training and testing intrusion detection systems (IDSs) and Intrusion Prevention Systems (IPSs).

While both the UWF-ZeekData24 [25] and UWF-ZeekData22 [29] datasets utilize the Zeek network analyzer and the MITRE ATT&CK framework to label network traffic, they differ fundamentally in their data collection methodologies and signal clarity [29,30]. UWF-ZeekData22 is characterized as a crowd-sourced dataset derived from live wargaming exercises within a cybersecurity curriculum. In contrast, UWF-ZeekData24 is the result of a controlled environment utilizing automated, scripted attacks to ensure consistency. This difference in origin leads to significant divergences in data precision:

• Temporal Accuracy: UWF-ZeekData22 exhibits a “slop factor” of approximately 5 min, meaning the timestamps between mission logs and network traffic may vary significantly, introducing noise into the labeling process. Conversely, UWF-ZeekData24 demonstrates far greater temporal precision with accurate timestamps and distinct network “floors” and “peaks,” facilitating exact correlation for machine learning training.
• Noise and Anomalies: The crowd-sourced nature of UWF-ZeekData22 results in a dataset rich in anomalous behavior and noise, which effectively represents the chaotic nature of human-driven attack scenarios. However, UWF-ZeekData24 provides a “cleaner” signal with significantly less noise, making it potentially more suitable for establishing baseline ground truth for algorithmic training.
• Attack Composition: While UWF-ZeekData22 contains a diverse mix of attacks generated by students (red teams) against blue teams, UWF-ZeekData24 relies on specific automated scripts (e.g., Nmap scans) to generate reproducible attack patterns, such as the heavy concentration of Credential Access attempts.

Hence, while UWF-ZeekData22 offers insights into crowd-sourced, high-noise environments, UWF-ZeekData24 improves upon this foundation by offering a sanitized, strictly controlled dataset optimized for the precise training of AI and ML models.

4. Adversarial Attacks Used in This Work

The Adversarial Robustness Toolbox (ART) is a Python 3.14.6 library designed to defend ML models against adversarial threats by providing tools to build, deploy, and test defenses [5]. It supports various ML frameworks (e.g., TensorFlow, Keras, PyTorch) and includes implementations of state-of-the-art attacks, including the HopSkipJump attack. The HopSkipJump attack is a decision-based black-box attack that requires only the final class prediction to generate adversarial examples (

Table 1. Adversarial Attacks.

Tool	Attack Type	Primary Use Case	Key Function/Method
ART	HopSkipJump	Black-box decision-based	HopSkipJump(classifier).generate(x)
CleverHans	SPSA	Gradient-free/Obfuscated gradients	SPSA(model, …).generate(x)
Foolbox	Square Attack	Black-box score-based	SquareAttack() (model, x, y, epsilons)

) [6]. It is efficient in query usage and works by estimating the gradient direction using binary search at the decision boundary. CleverHans is a library that provides standardized reference implementations of adversarial attacks and defenses, primarily to benchmark model vulnerability [7]. While the provided technical report (v2.1.0) focuses on gradient-based attacks like the Fast Gradient Sign Method (FGSM) and Carlini–Wagner (C&W), newer versions of the library (v4.0.0+) include the Simultaneous Perturbation Stochastic Approximation (SPSA) attack [33]. SPSA is a gradient-free optimization method useful when the model’s gradients are unavailable or non-differentiable (obfuscated gradients). Foolbox is a Python toolbox designed to benchmark the robustness of ML models by determining the minimum perturbation needed to craft an adversarial example [9]. It emphasizes correct and accurate implementations of attacks to provide reliable baselines. The Square Attack is a score-based black-box attack that does not rely on local gradient information and is known for its query efficiency [10]. Note that the provided Foolbox report (v1/v2) predates the Square Attack (proposed in 2020); however, it is a core component of the modern Foolbox (v3) Native API.

4.1. The HopSkipJump Attack

The HopSkipJump attack represents a significant advancement in decision-based adversarial attacks, addressing a crucial gap in current adversarial ML research [5,6]. Developed by Chen, Jordan, and Wainwright (2020) [6], this attack method operates under a highly practical threat model where the attacker only has access to the final classification decisions of the target model, without requiring knowledge of model architecture, weights, or probability outputs.

During the boundary search, the algorithm uses binary search to efficiently locate the decision boundary between classifications. At the boundary, it employs an innovative Monte Carlo sampling technique to estimate the gradient direction, despite having no direct access to model gradients. The final step involves a geometric progression to determine the optimal step size for moving along the estimated gradient direction, guided by theoretical analysis suggesting step sizes proportional to the distance between the current example and the original image.

What makes HopSkipJump particularly notable is its query efficiency [5,6]. Experimental results demonstrate that it requires significantly fewer model queries than comparable decision-based attacks like Boundary Attack. For instance, on CIFAR-10 with a ResNet model, the un-targeted L₂-optimized version achieves a median distance of 0.56 with just 1000 queries, compared to 2.78 for Boundary Attack at the same query budget [6]. This efficiency makes it practical for real-world applications where query limitations often exist.

The algorithm’s hyperparameter-free design and theoretical convergence guarantees further enhancement of its practical utility [5,6]. By identifying which network traffic features are most susceptible to manipulation, the research provides valuable insights for developing more robust defense systems. The findings underscore a critical disconnect between theoretical model capabilities and practical security implementation, informing improved adversarial robustness strategies for operational network security environments.

4.2. The SPSA Attack

The Simultaneous Perturbation Stochastic Approximation (SPSA) attack addresses scenarios where traditional gradient-based methods prove ineffective or unavailable. As described in the CleverHans library documentation [7], SPSA is particularly useful when the model is non-differentiable, or more generally, when gradients do not point in useful directions for adversarial example construction [7,8]. The attack operates under the practical assumption that attackers can query the target model for predictions but cannot access gradient information, model architecture details, or internal representations. Unlike gradient-based attacks that require white-box access, SPSA employs stochastic approximation techniques to estimate gradient directions through finite difference approximations computed along random directions.

The attack functions through an iterative optimization process that leverages the fundamental insight that gradient information can be approximated using simultaneous perturbation of all parameters [7,8]. At each iteration, the algorithm generates a random perturbation vector and evaluates the objective function at two points: the current adversarial candidate plus and minus the scaled perturbation vector. As implemented in CleverHans, this simultaneous perturbation approach allows the algorithm to estimate the gradient direction using only two function evaluations per iteration, regardless of input dimensionality. This is a large efficiency gain compared to coordinate-wise finite difference methods. The estimated gradient guides the optimization toward adversarial examples through projected steps that maintain feasibility constraints.

What makes SPSA particularly effective for adversarial example generation is its robustness to optimization landscapes where traditional gradients may be misleading or unavailable [7,8]. The CleverHans implementation parameterizes the attack with epsilon (perturbation magnitude), the number of optimization steps, learning rate (step-size), and the perturbation size used for finite difference approximation. For NIDS applications, this translates to a versatile attack method that can navigate complex decision boundaries without requiring knowledge of how the model processes network features, protocol information, or temporal patterns. The method’s gradient-free nature makes it especially valuable for attacking models where gradients may not provide reliable adversarial directions, such as ensemble classifiers or models with non-smooth decision boundaries, enabling effective evaluation of MITRE ATT&CK-trained models through systematic exploration of the feature space.

4.3. The Square Attack

The Square Attack represents a score-based adversarial optimization, implemented as part of the comprehensive Foolbox adversarial robustness evaluation framework [9,10]. Foolbox provides standardized reference implementations of adversarial attack methods designed to ensure comparable benchmarks across different research efforts. The library addresses a critical need in adversarial ML research by providing consistent implementations that eliminate variations in attack strength that could otherwise confound experimental results. Square Attack, as implemented in Foolbox, operates under the realistic assumption that attackers can observe model confidence scores or probability distributions, information commonly available in operational systems that provide prediction confidence alongside classifications.

The attack employs a conceptually elegant random search strategy centered on square-shaped perturbations applied systematically across the input space [9,10]. This approach aligns with Foolbox’s design philosophy of providing attack implementations that perform internal hyperparameter tuning to find minimum adversarial perturbations, eliminating the need for manual parameter specification that could introduce bias. The algorithm begins by initializing with a starting point and then iteratively applies uniform perturbations to randomly selected square regions within the input. Each iteration involves choosing a square size, randomly positioning this square within the input dimensions, and applying uniform perturbations to all features within the selected region while evaluating improvements to the adversarial objective.

What distinguishes Square Attack within the Foolbox framework is its exceptional balance of simplicity, effectiveness, and query efficiency [9,10]. Foolbox’s implementation philosophy emphasizes that the most comparable robustness measure is the minimum perturbation needed to craft an adversarial example, and Square Attack excels in this regard by efficiently finding minimal perturbations through its geometric search strategy. The attack’s integration into Foolbox ensures compatibility with multiple deep learning frameworks including PyTorch, Keras, TensorFlow, and others, making it accessible for evaluating diverse model architectures. For network traffic classification, this translates to a standardized, framework-agnostic method for evaluating the robustness of MITRE ATT&CK-trained models.

5. Methodology

5.1. Research Flow

Figure 1 presents a flowchart that illustrates the complete research workflow, from initial model analysis through implementation and evaluation. The initial baseline Random Forest implementation was selected due to the high classification results in [36]. The migration from the legacy PySpark-based Random Forest model to PyTorch neural network classifier followed a systematic approach.

First, the original PySpark Random Forest implementation [36] was analyzed to understand its architecture, preprocessing steps, and classification methodology (Figure 2). This involved examining how the data was loaded, feature columns were processed (particularly the binning techniques for numeric, categorical, IP address, and port features), and how models were trained and evaluated.

Next, neural network architectures were designed in PyTorch v 2.12.0 that would classify the attack of the data coming from the Random Forest model, RandomForestNN, employing multiple hidden layers with appropriate dropout to prevent overfitting while maintaining similar predictive power (Figure 3). Figure 4 presents the data processing flowchart.

For the model implementation, flexible PyTorch Dataset and DataLoader classes were created to efficiently process batches of data, along with training and evaluation utilities that calculated the same metrics as the original PySpark implementation (accuracy, precision, recall, F1 score, and AUC) (Figure 5). The training process incorporated modern optimization techniques like Adam while faithfully reproducing the original model’s classification characteristics. To facilitate performance comparison, a training and evaluation script was built that could process the same data through both frameworks and produce side-by-side metrics and visualizations. Finally, the PyTorch implementation was enhanced with model saving and loading capabilities, ensuring that trained models could be persisted and deployed with all necessary metadata intact. This methodical approach ensured that the PyTorch implementation remained functionally equivalent to the original PySpark models while offering the advantages of greater portability, easier deployment, and more efficient resource utilization.

A determination had to be made about which dataset would be used for training and which dataset would be used for testing (Figure 6). The UWF-TestZeekData24 dataset [30] was selected over the UWF-ZeekData22 dataset [29] for several key reasons. While the UWF-ZeekData22 dataset contains a larger number of samples (326,622 true negatives and 151,493 true positives), it also exhibits a substantially higher false positive rate, with 28,398 false positives. In contrast, UWF-TestZeekData24, despite having a smaller overall sample size (14,080 true negatives and 14,171 true positives), demonstrates improved class balance and lower false positive rates.

The model trained on UWF-TestZeekData24 [30] also demonstrated improved precision and recall performance, particularly in reducing false negatives. With only 219 false negatives out of 14,390 attack samples (false negative rate of 1.52%), this dataset facilitates a higher sensitivity to attack detection (

Table 2. UWF-ZeekData24 PyTorch Random Forest confusion matrix (left) vs. UWF-ZeekData22 PyTorch Random Forest confusion matrix (right).

Classifier	UWF-ZeekData24 (Test)	UWF-ZeekData22
Random Forest	$\left[\begin{array}{cc}\mathrm{14,080}& 292\\ 219& \mathrm{14,170}\end{array}\right]$	$\left[\begin{array}{cc}\mathrm{326,622}& \mathrm{28,398}\\ 149& \mathrm{151,493}\end{array}\right]$

). In contrast, UWF-ZeekData22 [29], while having a lower absolute number of false negatives (149), exhibited a significantly higher proportion of false positives, leading to unnecessary alerts that reduce trust in the model’s predictions. Another factor favoring UWF-TestZeekData24 [30] is its standardized preprocessing and cleaner feature representations. The dataset employs consistent labeling methodologies, such as the use of “_zeek” suffixes for related columns, which simplifies preprocessing and ensures robust feature mapping. Ultimately, UWF-TestZeekData24 [30] was chosen for its ability to provide a balanced and representative dataset that minimizes false positives while maintaining strong attack detection capabilities. Its alignment with contemporary network traffic patterns and structured data preprocessing made it a more suitable foundation for training a realistic model.

To evaluate the robustness of the trained PyTorch model against adversarial attacks, we utilized the Adversarial Robustness Toolbox (ART) (Figure 7). The experiments were conducted in a Linux-based virtual machine (VM) environment configured with Ubuntu 22.04.4, running kernel version 6.5.0-27-generic. The environment was set up with Python3 and pip, with Jupyter Notebook version 4.0.11 used for execution and visualization of the adversarial attacks.

A modified version of the security_curve.ipynb notebook from ART was used to test the trained PyTorch Random Forest model against each of the respective attacks, HopSkipJump, Simultaneous Perturbation Stochastic Approximation Attack and the Square Attack. This attack was chosen for its ability to generate adversarial examples with minimal perturbation while targeting classifiers in a black-box setting, making it highly relevant for real-world adversarial resilience testing.

The same UWF-TestZeekData24 dataset [30] which was used for training served as the basis for the adversarial attack evaluations. The HopSkipJump attack, a decision-based adversarial attack, was employed to assess the model’s vulnerability in scenarios where attackers have limited information about the classifier (Figure 8), along with the Simultaneous Perturbation Stochastic Approximation Attack and the Square Attacks to assess the model’s vulnerability in scenarios where the attackers have limited information about the classifier.

Through these experiments, we aimed to answer the primary research question: “Is the model trained using the Random Forest model-based MITRE ATT&CK framework still vulnerable to misclassification?” Additional questions explored included:

• How do adversarial attacks (HopSkipJump, SPSA and Square) affect the classification performance of a network traffic model trained using the Random ForestNN MITRE ATT&CK framework?
• Does leveraging the MITRE ATT&CK framework for training reduce the effectiveness of adversarial attacks on a network traffic classification model?
• Is there a specific attribute that is weighed more than another to make misclassification more frequent (or likely)?
• Is using a combination of modified attributes more successful in misclassification?

By conducting these adversarial evaluations, we sought to determine whether training models using the Random Forest-based MITRE ATT&CK framework provided increased robustness against adversarial attacks compared to conventional training methodologies. The results of these experiments will inform future improvements to NIDS and contribute to the broader field of adversarial machine learning defense strategies.

5.2. Parameters Used

5.2.1. Spark Parameters

The Spark Parameters used were: driver cores = 4; dynamicAllocation.shuffleTracking.enabled; minExecutors = 10; maxExecutors = 30; executor instances = 20; cores = 2; memory = 5 g; and shuffle partitions = 72.

5.2.2. Random Forest

The following parameters were used for Random Forest:

• featuresCol: str = ‘features’; labelCol: str = ‘label’; predictionCol: str = ‘prediction’; probabilityCol: str = ‘probability’; rawPredictionCol: str = ‘rawPrediction’; maxDepth: int = 5; maxBins: int = 32; minInstancesPerNode: int = 1; minInfoGain: float = 0.0; maxMemoryInMB: int = 256; cacheNodeIds: bool = False; checkpointInterval: int = 10; impurity: str = ‘gini’; numTrees: int = 20; featureSubsetStrategy: str = ‘auto’; seed: Optional[int] = None; subsamplingRate: float = 1.0; leafCol: str = ‘’; minWeightFractionPerNode: float = 0.0; weightCol: Optional[str] = None; and bootstrap: Optional[bool] = True.

5.2.3. Neural Network Parameters

Multiple hidden layers were used with different dropout rates, based on the attack.

6. Results

6.1. Classification Performance Metrics

To assess model performance under both baseline and adversarial conditions, we report four primary metrics: accuracy, precision, recall, and F1 score. These are calculated from predicted and true labels over a binary classification task, where the positive class represents malicious traffic. Accuracy reflects the overall proportion of correct predictions and is computed as:

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(1)

where TP and TN represent the number of true positives and true negatives, respectively, and FP and FN denote false positives and false negatives.

Precision measures the proportion of positive predictions that are correct:

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}}$$

(2)

Recall, also referred to as the true positive rate, quantifies how many actual malicious samples were correctly identified:

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(3)

The F1 score balances precision and recall through their harmonic mean:

$$\mathrm{F}1 \mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\ast {\displaystyle \frac{(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n} \times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}{(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n} + \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}}$$

(4)

To account for potential class imbalance or class collapse during adversarial attacks, we also include macro-averaged versions of these metrics. These are calculated by computing the metric independently for each class and averaging the result. For example, macro precision is defined as:

$${\mathrm{Precision}}_{\mathrm{macro}}={\displaystyle \frac{{\mathrm{Precision}}_0+{\mathrm{Precision}}_1}{2}}$$

(5)

and similarly for macro recall and macro F1. Lastly, the Area Under the ROC Curve (AUC) is reported where applicable, offering a threshold-independent measure of the classifier’s ability to distinguish between benign and malicious traffic.

6.2. Baseline Analysis

The adversarial testing of our PyTorch Random Forest models revealed catastrophic vulnerabilities across three different black-box adversarial attacks, HopSkipJump, SPSA and Square, despite the models being trained on data structured according to MITRE ATT&CK framework principles. The baseline model results, as presented in

Table 3. Results for the PyTorch Random Forest models baseline training.

Attack Type	Accuracy	Precision (Macro)	Recall (Macro)	F1 (Macro)
Credential Access	1	1	1	1
Defense Evasion	0.969	0.955	0.975	0.964
Exfiltration	0.87	0.862	0.873	0.865
Initial Access	0.988	0.983	0.989	0.986
Persistence	0.966	0.951	0.973	0.961
Privilege Escalation	0.972	0.958	0.98	0.968
Reconnaissance	0.998	0.998	0.998	0.998

, demonstrated varying performance across different attack types, with Reconnaissance achieving near-perfect performance (99.8% accuracy, 99.8% macro precision, 99.8% macro recall, 99.8% macro F1) while exfiltration showed the weakest baseline performance (87.0% accuracy, 86.2% macro precision, 87.3% macro recall, 86.5% macro F1). However, all three adversarial attacks achieved devastating success in degrading model performance across all MITRE ATT&CK tactics.

6.3. Attack Perturbation Analysis

To evaluate how adversarial attacks manipulate input features, the perturbation frequency for each feature was computed, defined as the proportion of samples in which a feature’s value changes between the original input and its adversarial counterpart. For each feature j, perturbation frequency is calculated as:

$$j ={\displaystyle \frac{Number of samples where the feature was changed}{Total number of samples}}$$

(6)

This metric ranges from 0.0 (never changed) to 1.0 (always changed) and is computed separately for each attack method. High-frequency features indicate points of vulnerability most often exploited to evade detection. This analysis helps identify which network attributes such as IP addresses, ports, or byte counts are most susceptible to adversarial manipulation.

6.4. HopSkipJump Attack Performance by MITRE ATT&CK Tactic

The HopSkipJump attack demonstrated devastating effectiveness, achieving complete performance failure across six of seven attack types (

Table 4. HopSkipJump attack performance across MITRE ATT&CK tactics.

Attack Type	Accuracy	Precision (Macro)	Recall (Macro)	F1 (Macro)
Credential Access	0.000	0.000	0.000	0.000
Defense Evasion	0.000	0.000	0.000	0.000
Exfiltration	0.000	0.000	0.000	0.000
Initial Access	0.000	0.000	0.000	0.000
Persistence	0.000	0.000	0.000	0.000
Privilege Escalation	0.000	0.000	0.000	0.000
Reconnaissance	0.090	0.076	0.090	0.083

). A 0 percent accuracy was observed on attack detection for Credential Access, Defense Evasion, Exfiltration, Initial Access Persistence, and Privilege Escalation. Similarly, 0.09 percent accuracy was observed on attacks for Reconnaissance.

Confusion matrices show how a classification model’s predictions compare to the actual values, breaking down correct and incorrect predictions into four categories: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). Zero true positives show that no true attack was detected, while false positives show attacks that are not true attacks. The following results were observed from

Table 5. HopSkipJump attack confusion matrices.

TACTIC	NN
Credential Access	$\left[\begin{array}{cc}0& 50\\ 50& 0\end{array}\right]$
Defense Evasion	$\left[\begin{array}{cc}0& 50\\ 50& 0\end{array}\right]$
Exfiltration	$\left[\begin{array}{cc}0& 50\\ 23& 0\end{array}\right]$
Initial Access	$\left[\begin{array}{cc}0& 50\\ 50& 0\end{array}\right]$
Persistence	$\left[\begin{array}{cc}0& 50\\ 50& 0\end{array}\right]$
Privilege Escalation	$\left[\begin{array}{cc}0& 50\\ 50& 0\end{array}\right]$
Reconnaissance	$\left[\begin{array}{cc}9& 41\\ 50& 0\end{array}\right]$

:

• For Credential Access, a TP of 0 and FP of 0.
• For Defense Evasion, a TP of 0 and FP of 0.
• For Exfiltration, a TP of 0 and FP of 0.
• For Initial Access, a TP of 0 and FP of 0.
• For Persistence, a TP of 0 and FP of 0.
• For Privilege Escalation, a TP of 0 and FP of 0.
• For Reconnaissance, a TP of 0 and FP of 9.

These modification rates, affecting a majority of instances, suggest a high potential for inducing significant model misclassification. From

Table 6. HopSkipJump attack perturbations.

Feature	Credential Access	Defense Evasion	Exfiltration	Initial Access	Persistence	Privilege Escalation	Reconnaissance
history	0.2	0.64	0.8082	0.44	0.55	0.54	0.41
proto	0.52	0.5	0.6712	0.36	0.5	0.5	0.41
service	0.55	0.64	0.8493	0.44	0.54	0.54	0.41
orig_bytes	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
dest_ip_zeek	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
orig_pkts	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
orig_ip_bytes	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
local_resp	0.58	0.14	0.3013	0.09	0.09	0.48	0.4
dest_port_zeek	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
Duration	0.59	0.63	0.8630	0.42	0.53	0.52	0.4
conn_state	0.61	0.64	0.7534	0.24	0.55	0.54	0.29
resp_pkts	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
resp_ip_bytes	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
src_port_zeek	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
resp_bytes	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
src_ip_zeek	0.61	0.64	0.8630	0.44	0.55	0.54	0.41
local_orig	0.56	0.21	0.2191	0.08	0.3	0.11	0.36
missed_bytes	0.61	0.64	0.8630	0.44	0.55	0.54	0.41

, the following was observed:

• For Credential Access, the features orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_post_zeek, conn_state, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, src_ip_zeek, and missed_bytes were perturbed with a frequency of 0.61, reflecting a modification rate exceeding half of the total occurrences.
• For Defense Evasion, the features history, service, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_port_zeek, conn_state, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, src_ip_zeek, and missed_bytes were perturbed with a frequency of 0.64, reflecting a modification rate exceeding half of the total occurrences.
• For Exfiltration, the features orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_port_zeek, duration, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, and mised_bytes were perturbed with a frequency of 0.8630, reflecting a modification rate exceeding half of the total occurrences.
• For Initial Access, the features history, service, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_port_zeek, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, src_ip_zeek, and missed_bytes were perturbed with a frequency of 0.44, reflecting a modification rate under half of the total occurrences.
• For Persistence, the features history, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_port_zeek, conn_state, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, src_ip_zeek, and missed_bytes were perturbed with a frequency of 0.55, reflecting a modification rate exceeding half of the total occurrences.
• For Privilege Escalation, the features history, service, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_port_zeek, conn_state, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, src_ip_zeek, and missed_bytes were perturbed with a frequency of 0.54, reflecting a modification rate exceeding half of the total occurrences.
• For Reconnaissance, the features history, proto, service, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, dest_port_zeek, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, src_ip_zeek, and missed_bytes were perturbed with a frequency of 0.41, reflecting a modification rate under half of the total occurrences.

6.5. SPSA Attack Performance by MITRE ATT&CK Tactic

The SPSA attack showed more varied results across attack types, with some tactics maintaining moderate performance while others suffered severe degradation, as reflected in

Table 7. SPSA attack performance showing variable degradation across MITRE ATT&CK tactics.

Attack Type	Accuracy	Precision (Macro)	Recall (Macro)	F1 (Macro)
Credential Access	0.040	0.037	0.040	0.038
Defense Evasion	0.500	0.250	0.500	0.333
Exfiltration	0.658	0.338	0.480	0.397
Initial Access	0.490	0.247	0.490	0.329
Persistence	0.460	0.240	0.460	0.315
Privilege Escalation	0.490	0.247	0.490	0.329
Reconnaissance	0.500	0.250	0.500	0.333

. A 50 percent or below accuracy on attack detection for Credential Access, Defense Evasion, Exfiltration, Initial Access, Persistence, Privilege Escalation, and Reconnaissance was observed. The confusion matrices in

Table 8. SPSA attack confusion matrices.

TACTIC	SPSA
Credential Access	$\left[\begin{array}{cc}4& 46\\ 50& 0\end{array}\right]$
Defense Evasion	$\left[\begin{array}{cc}49& 1\\ 50& 0\end{array}\right]$
Exfiltration	$\left[\begin{array}{cc}48& 2\\ 23& 0\end{array}\right]$
Initial Access	$\left[\begin{array}{cc}50& 0\\ 50& 0\end{array}\right]$
Persistence	$\left[\begin{array}{cc}48& 2\\ 50& 0\end{array}\right]$
Privilege Escalation	$\left[\begin{array}{cc}48& 2\\ 50& 0\end{array}\right]$
Reconnaissance	$\left[\begin{array}{cc}50& 0\\ 50& 0\end{array}\right]$

also reflect this.

As per Table 8, the confusion matrices show how:

• For Credential Access, a TP of 0 and FP of 4.
• For Defense Evasion, a TP of 0 and FP of 49.
• For Exfiltration, a TP of 0 and FP of 48.
• For Initial Access, a TP of 0 and FP of 50.
• For Persistence, a TP of 0 and FP of 48.
• For Privilege Escalation, a TP of 0 and FP of 48.
• For Reconnaissance, a TP of 0 and FP of 50.

These modification rates, affecting a majority of instances, suggest a high potential for inducing significant model misclassification. From

Table 9. SPSA attack perturbations.

Feature	Credential Access	Defense Evasion	Exfiltration	Initial Access	Persistence	Privilege Escalation	Reconnaissance
history	0.45	0.58	0.7534	0.7	0.47	0.47	0.61
proto	0.52	0.48	0.6575	0.48	0.48	0.48	0.48
service	0.91	0.97	0.9178	0.78	0.98	0.98	0.99
orig_bytes	0.96	0.98	0.9726	0.58	0.98	0.98	0.6
dest_ip_zeek	1	1	1	1	1	1	1
orig_pkts	1	1	1	1	1	0.56	1
orig_ip_bytes	1	1	1	1	1	1	1
local_resp	0.5	0.5	0.3150	0.5	0.5	0.49	0.5
dest_port_zeek	0.5	0.5	0.6849	0.5	0.5	0.96	0.5
Duration	0.48	0.61	0.9726	0.56	0.51	0.51	0.95
conn_state	0.61	0.56	0.6849	0.4	0.45	0.45	0.49
resp_pkts	0.89	0.89	0.9589	0.89	0.89	0.89	0.89
resp_ip_bytes	0.89	0.89	0.8082	0.89	0.89	0.9	0.57
src_port_zeek	1	1	1	1	1	1	1
resp_bytes	0.91	0.91	0.904	0.56	0.91	0.91	0.81
src_ip_zeek	1	1	1	1	1	1	1
local_orig	0.89	0.5	0.3150	0.5	0.5	0.5	0.62
missed_bytes	0.5	0.5	0.6849	0.5	0.5	0.5	0.5

, it can be observed that:

• For Credential Access, the features dest_ip_zeek, orig_pkts, orig_ip_bytes, src_port_zeek, and src_ip_zeek were perturbed with a frequency of 1, reflecting a modification rate of all occurrences.
• For Defense Evasion, the features dest_ip_zeek, orig_pkts, orig_ip_bytes, src_port_zeek, and src_ip_zeek were perturbed with a frequency of 1, reflecting a modification rate of all occurrences.
• For Exfiltration, the features dest_ip_zeek, orig_pkts, orig_ip_bytes, src_port_zeek, and src_ip_zeek were perturbed with a frequency of 1, reflecting a modification rate of all occurrences.
• For Initial Access, the features dest_ip_zeek, orig_pkts, orig_ip_bytes, src_port_zeek, and src_ip_zeek were perturbed with a frequency of 1, reflecting a modification rate of all occurrences.
• For Persistence, the features dest_ip_zeek, orig_pkts, orig_ip_bytes, src_port_zeek, and src_ip_zeek, were perturbed with a frequency of 1 reflecting a modification rate of all occurrences.
• For Privilege Escalation, the features dest_ip_zeek, orig_ip_bytes, src_port_zeek, and src_ip_zeek were perturbed with a frequency of 1, reflecting a modification rate of all occurrences.
• For Reconnaissance, the features dest_ip_zeek, orig_pkts, orig_ip_bytes, src_port_zeek, and src_ip_zeek were perturbed with a frequency of 1, reflecting a modification rate of all occurrences.

6.6. Square Attack Performance by MITRE ATT&CK Tactic

The Square Attack demonstrated the most nuanced pattern of vulnerabilities, with Credential Access showing the strongest resistance while Initial Access proved most vulnerable (

Table 10. Square Attack performance revealing differential vulnerability patterns across MITRE ATT&CK tactics.

Attack Type	Accuracy	Precision (Macro)	Recall (Macro)	F1 (Macro)
Credential Access	0.610	0.781	0.610	0.540
Defense Evasion	0.500	0.250	0.500	0.333
Exfiltration	0.603	0.328	0.440	0.376
Initial Access	0.360	0.209	0.360	0.265
Persistence	0.390	0.219	0.390	0.281
Privilege Escalation	0.460	0.240	0.460	0.315
Reconnaissance	0.500	0.250	0.500	0.333

). A 50 percent or below accuracy for attack detection on Defense Evasion, Initial Access, Persistence, Privilege Escalation, and Reconnaissance was observed. But, Credential Access and Exfiltration had an accuracy of higher than 50 percent.

From the confusion matrices in

Table 11. Square Attack confusion matrices.

TACTIC	SQUARE
Credential Access	$\left[\begin{array}{cc}11& 39\\ 0& 50\end{array}\right]$
Defense Evasion	$\left[\begin{array}{cc}50& 0\\ 50& 0\end{array}\right]$
Exfiltration	$\left[\begin{array}{cc}48& 2\\ 23& 0\end{array}\right]$
Initial Access	$\left[\begin{array}{cc}36& 14\\ 50& 0\end{array}\right]$
Persistence	$\left[\begin{array}{cc}50& 0\\ 50& 0\end{array}\right]$
Privilege Escalation	$\left[\begin{array}{cc}50& 0\\ 50& 0\end{array}\right]$
Reconnaissance	$\left[\begin{array}{cc}50& 0\\ 50& 0\end{array}\right]$

, it can be observed that:

• For Credential Access, a TP of 50 and FP of 11.
• For defense evasion, a TP of 0 and FP of 50.
• For Exfiltration, a TP of 0 and FP of 48.
• For Initial Access, a TP of 0 and FP of 36.
• For Persistence, a TP of 0 and FP of 50.
• For Privilege Escalation, a TP of 0 and FP of 50.
• For Reconnaissance, a TP of 0 and FP of 50.

Credential access was the only model able to detect any of the true attacks.

These modification rates, affecting a majority of instances, suggest a high potential for inducing significant model misclassification. From

Table 12. Square Attack perturbations.

Feature	Credential Access	Defense Evasion	Exfiltration	Initial Access	Persistence	Privilege Escalation	Reconnaissance
history	0	0.14	0.1917	0.08	0.05	0.04	0
proto	0	0.08	0.1232	0.05	0.03	0.02	0
service	0	0.14	0.1780	0.08	0.04	0.04	0
orig_bytes	0	0.14	0.1917	0.08	0.05	0.04	0
dest_ip_zeek	0	0.14	0.2054	0.08	0.05	0.04	0
orig_pkts	0	0.14	0.1780	0.08	0.05	0.04	0
orig_ip_bytes	0	0.14	0.2054	0.08	0.05	0.04	0
local_resp	0	0.08	0.1095	0.05	0.03	0.02	0
dest_port_zeek	0	0.07	0.095	0.05	0.03	0.03	0
duration	0	0.13	0.1917	0.08	0.04	0.04	0
conn_state	0	0.12	0.0821	0.08	0.04	0.03	0
resp_pkts	0	0.14	0.1780	0.08	0.05	0.04	0
resp_ip_bytes	0	0.14	0.2054	0.08	0.05	0.04	0
src_port_zeek	0	0.14	0.2054	0.08	0.05	0.04	0
resp_bytes	0	0.14	0.1780	0.08	0.05	0.04	0
src_ip_zeek	0	0.14	0.2054	0.08	0.05	0.04	0
local_orig	0	0.07	0.095	0.03	0.01	0.01	0
missed_bytes	0	0.09	0.1369	0.04	0.02	0.01	0

it can be observed that:

• For Credential Access, all features were not perturbed, with a frequency of 0, reflecting a modification rate of all occurrences.
• For Defense Evasion, the features history, service, orig_bytes, dest_in_zeek, orig_pkts, orig_ip_bytes, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, and src_ip_zeek were perturbed with a frequency of 0.14, reflecting a modification rate of all occurrences.
• For Exfiltration, the features dest_ip_zeek, orig_ip_bytes, resp_ip_bytes, and src_ip_zeek were perturbed with a frequency of 0.2054, reflecting a modification of all occurrences.
• For Initial Access, the features history, service, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, duration, conn_state, resp_pkts, resp_ip_bytes, resp_bytes, and src_ip_zeek were perturbed with a frequency of 0.08, reflecting a modification rate of all occurrences.
• For Persistence, the features history, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, and src_ip_zeek were perturbed with a frequency of 0.05, reflecting a modification rate of all occurrences.
• For Privilege Escalation, the features history, service, orig_bytes, dest_ip_zeek, orig_pkts, orig_ip_bytes, duration, resp_pkts, resp_ip_bytes, src_port_zeek, resp_bytes, and src_ip_zeek were perturbed with a frequency of 0.04, reflecting a modification rate of all occurrences.
• For Reconnaissance, all features were perturbed with a frequency of 0, reflecting a modification rate of all occurrences.

6.7. Feature Vulnerability Analysis

The comprehensive feature perturbation analysis across all three adversarial attack methodologies (Table 6, Table 9 and Table 12) revealed critical insights into which network traffic characteristics are most susceptible to adversarial manipulation. Network identifiers consistently emerged as the primary targets across all attack methods, with destination IP addresses (dest_ip_zeek), source IP addresses (src_ip_zeek), and port information (src_port_zeek, dest_port_zeek) achieving the highest overall vulnerability scores of 0.551.

Attack-Specific Feature Targeting Patterns

Each adversarial attack method demonstrated distinct feature targeting strategies that reflect their underlying optimization approaches.

Table 13. Top ten most vulnerable network traffic features across all adversarial attack methods.

Feature	HSJ Freq	SPSA Freq	Square Freq
dest_ip_zeek	0.580	1.000	0.074
orig_ip_bytes	0.580	1.000	0.074
src_port_zeek	0.580	1.000	0.074
src_ip_zeek	0.580	1.000	0.074
orig_pkts	0.580	0.937	0.070
Service	0.568	0.935	0.068
resp_bytes	0.580	0.906	0.070
resp_pkts	0.580	0.900	0.070
orig_bytes	0.579	0.877	0.072
resp_ip_bytes	0.580	0.833	0.074

presents the top ten most vulnerable network traffic features across all adversarial attack methods: HSJ, SPSA, and Square Freq.

For the HopSkipJump Attack targeting, the boundary-based optimization approach demonstrated broad-spectrum targeting with relatively uniform perturbation frequencies around 0.580 across multiple features simultaneously. This balanced targeting approach suggests systematic boundary exploration, with the attack requiring moderate but consistent perturbation across network identifiers, traffic volume metrics, and packet count features.

For SPSA Attack targeting, the stochastic approximation method showed the most aggressive targeting strategy, achieving maximum perturbation intensities (1.000 frequency) on critical network identifiers including dest_ip_zeek, orig_ip_bytes, src_port_zeek, and src_ip_zeek. This approach highly focused on network addressing and communication endpoints reflects the algorithm’s efficiency in identifying the most vulnerable feature components.

For Square Attack targeting, the score-based random search employed the most selective targeting strategy, requiring minimal perturbation frequencies (~0.074) while targeting the same critical features as other methods. This efficient targeting approach suggests these features represent fundamental vulnerabilities that can be exploited with minimal modification.

6.8. Attack Type Specific Feature Vulnerabilities

The feature perturbation analysis revealed distinct vulnerability patterns across different MITRE ATT&CK tactics. Credential Access demonstrated extreme vulnerability to SPSA attacks with orig_bytes achieving maximum perturbation frequency (1.000) and average perturbation intensity of 0.782. However, this attack type showed complete resistance to Square Attacks with 0.000 average perturbation across all features. Exfiltration exhibited consistently high vulnerability across all attack methods, with orig_bytes reaching maximum perturbation under both HopSkipJump (0.863) and SPSA (1.000) attacks. The consistently high perturbation intensities (average 0.775 for HopSkipJump, 0.814 for SPSA) suggest inherent structural vulnerabilities in exfiltration detection mechanisms. Defense Evasion showed notable vulnerability to the history feature under the HopSkipJump attacks (0.640) while SPSA attacks focused on the dest_ip_zeek manipulation (1.000). The moderate Square Attack vulnerability (0.121 average) indicates persistent but manageable susceptibility. Reconnaissance demonstrated an interesting vulnerability pattern with complete Square Attack immunity (0.000 average perturbation) but substantial SPSA vulnerability (0.749 average perturbation), particularly targeting the dest_ip_zeek features.

6.9. Comparative Attack Performance and Vulnerability Analysis

A comprehensive evaluation across seven MITRE ATT&CK tactics revealed distinct vulnerability patterns and differential attack effectiveness, as illustrated in

Table 14. Overall performance comparison showing average metrics across all MITRE ATT&CK tactics and degradation from baseline performance.

Attack Method	Avg Accuracy	Avg Precision (Macro)	Avg Recall (Macro)	Avg F1 (Macro)	Avg Degradation
Baseline	0.966	0.958	0.970	0.963	-
HopSkipJump	0.013	0.011	0.013	0.012	98.7%
SPSA	0.448	0.230	0.423	0.302	53.6%
Square	0.489	0.318	0.466	0.359	49.4%

. HopSkipJump attacks proved catastrophically effective, achieving complete model failure (0.000 accuracy, precision, recall, and F1) across six of seven tactics, with only Reconnaissance retaining minimal performance (9.0% accuracy). This represents performance degradations exceeding 99.0% for most attack types, with Reconnaissance suffering a 91.0% accuracy reduction from its 99.8% baseline performance. SPSA attacks demonstrated more variable effectiveness that inversely correlated with baseline performance strength. Exfiltration, despite showing the weakest baseline performance (87.0% accuracy), demonstrated unexpected resilience under SPSA attacks (65.8% accuracy), representing only a 24.4% degradation. Conversely, credential access suffered the most severe SPSA impact, dropping from perfect baseline performance (100.0% accuracy) to just 4.0% accuracy a 96.0% degradation. This security paradox reveals that optimization for high baseline performance may inadvertently create brittle decision boundaries vulnerable to specific adversarial strategies. Square Attacks revealed the most nuanced vulnerability profile, with credential access demonstrating the strongest adversarial resistance (61.0% accuracy, 78.1% macro precision) despite starting from perfect baseline performance. This pattern suggests that while the model’s Credential Access detection mechanisms are highly effective under normal conditions, they also possess inherent robust characteristics that provide partial protection against certain adversarial optimization strategies. Conversely, Initial Access proved most vulnerable to Square Attacks (36.0% accuracy), representing a 63.6% degradation from baseline.

6.10. Feature Removal Experiment

To validate the critical role of network identifiers in adversarial vulnerabilities, we conducted a controlled experiment removing IP address features (src_ip_zeek and dest_ip_zeek) from the model and re-evaluating adversarial robustness across all three attack methodologies. This experiment directly tested whether the features identified as most vulnerable through perturbation analysis were indeed responsible for the observed adversarial failures. The IP removal experiment revealed striking improvements in adversarial robustness, with effectiveness varying significantly by attack methodology, as demonstrated in

Table 15. Adversarial attack success rates showing the percentage of attacks that successfully evade detection.

Attack Method	With IP Features	Without IP Features	IP Advantage
HopSkipJump	100.0% success	68.4% success	+31.6%
SPSA	100.0% success	99.7% success	+0.3%
Square	85.7% success	85.7% success	+0.0%

.

The perturbation analysis of the IP removal experiment revealed critical insights into how adversarial attacks adapted their targeting strategies when their primary vulnerability was eliminated. All three attack methodologies demonstrated significant reductions in overall targeting efficiency, with total perturbation intensity decreasing by approximately 30% across all methods despite attempts to redistribute targeting to alternative features. HopSkipJump attacks experienced a 29.9% reduction in total targeting intensity (from 67.1 to 47.1), SPSA attacks showed a 32.6% decrease (from 96.0 to 64.7), and Square Attack demonstrated a 30.2% reduction (from 7.7 to 5.4). This consistent pattern indicates that IP features served as disproportionately efficient attack vectors that could not be adequately compensated through increased targeting of alternative features. Despite IP features representing only 12.1–14.6% of original targeting intensity, their removal caused 30% efficiency losses across all attack methods. This amplified impact demonstrates that network identifiers provided unique vulnerability characteristics that enabled highly efficient adversarial manipulation compared to other feature categories.

6.11. Targeting Redistribution

The attack-specific targeting redistribution analysis revealed distinct adaptation patterns as each adversarial method redistributed targeting to alternative feature categories. The HopSkipJump boundary-based optimization method refocused on traffic volume metrics, with orig_bytes, resp_bytes, and packet count features (orig_pkts, resp_pkts) emerging as the primary targets with a 0.579 average frequency. This shift to volumetric features suggests that boundary-based attacks require quantifiable traffic characteristics for effective decision boundary manipulation. The SPSA stochastic approximation method maintained high-intensity targeting on packet metrics (orig_pkts at 0.937) and service classification (service at 0.935), while increasing focus on traffic volume features. This pattern indicates that SPSA attacks can effectively exploit behavioral and application-layer features when network identifiers are unavailable. The square score-based optimization showed the most distributed targeting approach, focusing on IP-layer bytes (resp_ip_bytes at 0.074) and connection history features (history at 0.072). The relatively low targeting intensities reflect Square Attacks’ inherent efficiency in exploiting minimal feature perturbations.

7. Conclusions

The evaluation of the PyTorch Random Forest model against adversarial attacks revealed significant vulnerabilities despite the integration of the MITRE ATT&CK framework in the training process. Our findings conclusively demonstrate that models trained using the MITRE ATT&CK framework remain vulnerable to misclassification through black-box optimization techniques. The HopSkipJump attack demonstrated a concerning ability to reduce model performance across all metrics, with precision falling from 0.987 to 0.139 and F1 scores dropping from 0.987 to 0.149. Our investigation showed that adversarial attacks dramatically affect classification performance, with the HopSkipJump attack successfully causing the model to misclassify 84.0% of attack samples as benign traffic, representing a severe degradation in detection capabilities. Contrary to expectations, leveraging the MITRE ATT&CK framework provides some structure for model training but does not significantly immunize the model against sophisticated black-box attacks. Feature perturbation analysis uncovered specific attributes more susceptible to manipulation, with source IP addresses (src_ip at 0.089) requiring minimal modification to contribute to successful evasion, while service-related features demonstrated greater resistance.

Regarding effective attack strategies, we found that attackers can achieve high success rates by targeting features requiring minimal perturbation while carefully balancing necessary changes to more robust features, with different attack types requiring distinct evasion approaches. Exfiltration attacks achieved the highest success rates (0.98) despite requiring substantial perturbation of certain features, while Exploitation attacks, though having higher precision and recall compared to other attack types, still maintained relatively low detection metrics. These results highlight a critical gap between theoretical robustness and practical security implementation, even when leveraging established tactical frameworks. Future work should focus on enhancing model resilience specifically for features with low perturbation requirements, potentially through adversarial training that emphasizes source and destination characteristics. As adversarial machine learning continues to evolve, bridging the documented disconnect between academic research and industry implementation remains essential for developing truly robust network intrusion detection systems.

8. Future Works

Future work should focus on training models using a more verbose and complete dataset that captures the full breadth of Zeek logs including DNS, HTTP, SSL/TLS, and other protocol-specific telemetry. Expanding beyond connection-level features would provide the model with richer context and allow for more realistic behavioral correlations, ultimately improving detection accuracy and adversarial resilience. In addition, further research is needed to identify which types of network data contribute most to model hardening. This includes analyzing cross-log correlations, timing patterns, and session-based behaviors that may be missed when relying solely on discrete, binned connection metadata. Incorporating these elements could help reduce overfitting and lead to more robust, generalizable detection strategies. Finally, deploying the system in a live, dynamic environment where it can continuously monitor traffic, adapt to new behaviors, and respond to real-world adversarial scenarios would offer critical insight into its practical viability. Such a live testbed would enable the evaluation of real-time defenses, adversarial retraining mechanisms, and operational response strategies bridging the gap between proof-of-concept robustness and applied network security.