Performance Comparison of Python-Based Complex Event Processing Engines for IoT Intrusion Detection: Faust Versus Streamz

Abstract

The proliferation of Internet of Things (IoT) devices has intensified the need for efficient real-time anomaly and intrusion detection, making the selection of an appropriate Complex Event Processing (CEP) engine a critical architectural decision for security-aware data pipelines. Python-based CEP frameworks offer compelling advantages through the seamless integration with data science and machine learning ecosystems; however, rigorous comparative evaluations of such frameworks under realistic IoT security workloads remain absent from the literature. This study presents the first systematic comparative evaluation of Faust and Streamz—two Python-native CEP engines representing fundamentally different architectural philosophies—specifically in the context of IoT network intrusion detection. Faust was selected for its actor-based stateful processing model with native Kafka integration and distributed table support, while Streamz was selected for its reactive, lightweight pipeline design targeting high-throughput stateless processing, making them representative of the two dominant paradigms in Python stream processing. Although both engines target different application niches, their performance characteristics under realistic CEP workloads have never been rigorously compared, leaving practitioners without empirical guidance. The primary evaluation employs an IoT network intrusion dataset comprising 583,485 events from 83 heterogeneous devices. To assess whether the observed performance characteristics are specific to this single dataset or generalize across different workload profiles, a secondary IoT-adjacent benchmark is included: the PaySim financial transaction dataset (6.4 million records), selected because its event schema, fraud-pattern temporal structure, and volume differ substantially from the intrusion dataset, providing a stress test for cross-workload robustness rather than a claim of domain equivalence. We acknowledge the reviewer’s valid point that a second IoT-specific intrusion dataset (such as TON_IoT or Bot-IoT) would constitute a more directly comparable validation; this is identified as a priority for future work. The load levels used in scalability experiments (up to 5000 events per second) intentionally exceed the dataset’s natural rate to stress-test each engine’s architectural ceiling and identify saturation thresholds relevant to large-scale or multi-sensor IoT deployments. We conducted controlled experiments with comprehensive statistical analysis. Our results demonstrate that Streamz achieves superior throughput at 4450 events per second with 89% efficiency and minimal resource consumption (40 MB memory, 12 ms median latency), while Faust provides robust intrusion pattern detection with 93–98% accuracy and stable, predictable resource utilization (1.4% CPU standard deviation). A multi-framework comparison including Apache Kafka Streams and offline scikit-learn baselines confirms that Faust achieves detection quality competitive with JVM-based alternatives (Faust: 96.2%; Kafka Streams: 96.8%; absolute difference of 0.6 percentage points, not statistically significant at $p=0.318$) while retaining the Python ecosystem advantages. Statistical analysis confirms significant performance differences across all metrics ($p<0.001$, Cohen’s $d>0.8$). Critical scalability thresholds are identified: Streamz maintains efficiency above 95% up to 3500 events per second, while Faust degrades beyond 2500 events per second. These findings provide IoT security engineers and system architects with actionable, empirically grounded guidance for CEP engine selection, establish reproducible benchmarking methodology applicable to future Python-based stream processing evaluations, and advance theoretical understanding of the accuracy–throughput trade-off in stateful versus stateless Python CEP architectures.

1. Introduction

The rapid proliferation of Internet of Things (IoT) devices across smart homes, industrial automation, healthcare, and critical infrastructure has created an unprecedented volume of real-time network traffic that must be continuously monitored for security threats. Modern IoT environments generate heterogeneous, high-velocity event streams in which intrusion attempts, anomalous device behaviors, and coordinated attack sequences must be detected within milliseconds to limit damage and enable timely response [1,2]. Complex Event Processing (CEP) has emerged as a foundational technology for this class of problems, enabling the detection of multi-event temporal patterns, threshold violations, and correlated behavioral sequences within continuous data streams. Unlike batch machine learning approaches that operate on static datasets, CEP systems provide immediate notifications when significant patterns emerge—a capability essential for real-time intrusion detection where response latency directly determines the impact of a security incident [3,4,5].

The architectural complexity inherent in CEP systems introduces substantial challenges in balancing competing requirements including processing throughput, detection latency, pattern accuracy, and resource efficiency. These trade-offs become particularly acute in IoT security contexts, where systems must handle burst traffic patterns characteristic of distributed denial-of-service attacks, maintain detection accuracy under sustained high load, and operate within the resource constraints of edge or containerized deployments.

The CEP technology landscape includes well-established Java Virtual Machine (JVM)-based platforms such as Apache Flink CEP, Esper, and Apache Storm, which dominate enterprise deployments due to their optimization for high-throughput server environments [6,7]. However, the growing adoption of Python in security analytics, threat intelligence, and machine learning has driven the development of Python-native stream processing frameworks that offer direct integration with libraries such as NumPy, scikit-learn, and TensorFlow without requiring cross-language bridges [8,9,10]. Among these, Faust and Streamz have emerged as the two most architecturally distinct and practically relevant Python-native CEP frameworks, each embodying a fundamentally different design philosophy.

Faust [11] is a Python stream processing library originally developed at Robinhood Financial, designed to bring the ergonomics of Kafka Streams to the Python ecosystem. It provides a comprehensive, actor-based stateful processing model with distributed tables backed by Kafka changelog topics, windowed aggregations, and fault-tolerant state management through RocksDB, making it suitable for complex event correlation tasks such as multi-event intrusion sequence detection. Streamz [12] is a lightweight Python library developed within the PyData ecosystem for building reactive dataflow pipelines that process continuous streams of data. It employs a functional composition model that prioritizes throughput and compositional simplicity over stateful semantics, making it well-suited for high-volume event transformation, filtering, and threshold-based alerting. Despite their architectural differences and growing adoption in production Python analytics environments, no prior academic work has subjected Faust and Streamz to controlled comparative performance evaluation under CEP workloads relevant to IoT security. A systematic search of Google Scholar for “Faust Streamz stream processing” returns fewer than five publications, none of which provide the controlled head-to-head comparison necessary to guide technology selection.

Despite the abundance of available CEP technologies, systematic comparative evaluation remains critically absent from the literature. Existing performance studies suffer from two key limitations: reliance on synthetic benchmarks that fail to capture real-world IoT event characteristics [13], and the examination of individual systems in isolation without comparative analysis that would enable informed technology decisions. The unique characteristics of Python-based CEP frameworks—including their ecosystem integration capabilities, performance profiles under varying workloads, and scalability characteristics—have received particularly limited attention despite growing adoption in production security operations centers. This evaluation gap proves especially problematic because technology selection decisions for IoT security pipelines must balance detection accuracy, processing latency, resource consumption, and operational simplicity; without comprehensive comparative data, engineers risk deploying architectures that either sacrifice detection quality for throughput or fail to scale to production IoT event volumes.

This research addresses these gaps through systematic comparative evaluation of Faust and Streamz under IoT network intrusion detection workloads. To provide structure for our investigation and enable hypothesis-driven statistical analysis, we formulate five research hypotheses:

H1 (Throughput).

 Streamz achieves significantly higher sustained event processing throughput than Faust under equivalent IoT workloads, due to its stateless architecture eliminating state synchronization overhead.

H2 (Latency).

 Streamz exhibits significantly lower median and tail latency than Faust, owing to the absence of commit protocol overhead and state persistence operations.

H3 (Resource Utilization).

 Faust consumes significantly more memory than Streamz due to state management infrastructure, while Faust demonstrates significantly more stable CPU utilization due to its internal buffering model.

H4 (Pattern Detection Accuracy).

 Faust achieves significantly higher intrusion pattern detection accuracy than Streamz-equivalent simple thresholds when configured for stateful CEP tasks, and maintains accuracy above 95% within its recommended operational range.

H5 (Scalability).

 Both engines exhibit nonlinear efficiency degradation under increasing load, with Faust reaching saturation at a significantly lower input rate than Streamz due to state synchronization becoming the dominant bottleneck.

These five hypotheses were formulated to fill the most critical measurement gaps identified by a structured review of the existing literature; they are not exhaustive but represent the five dimensions along which Python-native CEP engines have received the least empirical characterization.

H1 is grounded in the theoretical finding of Giatrakos et al. [14], who surveyed in-memory CEP systems and reported that stateless engines consistently outperform stateful counterparts on throughput by 40–60%. This finding was established for JVM-based systems; no equivalent empirical evidence exists for Python, where the Global Interpreter Lock (GIL) fundamentally constrains concurrency in a way that has no analogue in JVM environments. That of Giatrakos et al. is therefore the most directly applicable prior work, and the gap it leaves—the absence of Python-specific confirmation—directly motivates H1.

H2 extends H1 to latency. The same JVM-centric literature (Karimov et al. [15]; Chintapalli et al. [13]) observed that stateful commit protocols are the primary source of tail latency inflation, but these observations come from Flink and Storm benchmarks. Python’s asyncio event loop introduces commit-scheduling behavior distinct from JVM thread pools, meaning the latency penalty of stateful semantics could be higher or lower in Python; H2 is formulated to resolve this open question.

H3 targets resource consumption, motivated by Mothukuri et al. [16], who identified memory footprint as the critical bottleneck for federated IoT security deployments. Their survey highlights that edge IoT gateways routinely operate under 128 MB memory budgets, making the memory cost of stateful CEP a deployment-limiting factor that has not been quantified for Python-native frameworks. CPU stability is included in H3 because Bellavista et al. [17] demonstrated that predictable CPU utilization is a prerequisite for auto-scaling in containerized edge deployments, and no data exists for Python CEP engines on this dimension.

H4 addresses detection accuracy under sustained streaming load, motivated by Ferrag et al. [18] and Koroniotis et al. [19], both of which report detection accuracy for IoT intrusion methods exclusively in offline, batch settings. No prior work characterises how Faust’s accuracy degrades as a function of input rate, which is a critical gap for practitioners who must set operational capacity limits. H4 is the only hypothesis directly concerned with the correctness dimension of CEP, which distinguishes it from H1–H3.

H5 targets scalability, directly extending the queuing-theoretic saturation analysis of Karimov et al. [15] to Python CEP engines. Karimov et al. established that distributed JVM stream processors exhibit nonlinear efficiency degradation near their saturation point, but their framework was never applied to Python-native engines. Liu et al. [20] further showed that adaptive scheduling can shift saturation thresholds by up to 35%, implying that identifying the baseline threshold is a prerequisite for any optimization strategy.

Additional hypotheses could be formulated (e.g., regarding fault tolerance, exactly-once semantics, or concept drift resilience); however, these five were selected because they correspond to dimensions for which (a) the prior literature provides a clear theoretical prediction based on JVM systems, (b) no Python-specific empirical evidence exists, and (c) the results have direct actionable implications for IoT security pipeline design. Future work can extend this hypothesis set as Python CEP benchmarking matures.

The investigation focuses on five specific research questions. First, how do these engines compare in fundamental performance characteristics including throughput, latency distributions, and resource utilization under realistic IoT workloads? Second, what are the scalability limits and load-handling characteristics of each engine? Third, what are the resource consumption patterns and their implications for deployment costs? Fourth, what are the practical operational characteristics including startup time, configuration complexity, and monitoring requirements? Fifth, what technology selection guidance can be derived based on application requirements, performance priorities, and deployment constraints?

Our contributions advance both theoretical understanding and practical application of CEP systems for IoT security:

• First comparative evaluation of Faust and Streamz: We provide the first rigorous head-to-head performance comparison of these two Python-native CEP engines under realistic IoT intrusion detection workloads, directly addressing the gap confirmed by the near-total absence of prior comparative literature.
• Reproducible benchmarking methodology: We develop and validate a comprehensive evaluation framework incorporating realistic workload generation, percentile-based latency analysis, and rigorous statistical testing with effect size calculations, providing a template for future Python-based CEP benchmarking studies.
• Empirical scalability thresholds: We quantify critical saturation thresholds—2500 events per second for Faust and 3500 events per second for Streamz—providing actionable capacity planning boundaries for IoT security deployments.
• Multi-framework performance context: We extend the comparison to include Apache Kafka Streams and scikit-learn baselines, demonstrating that Python-native Faust approaches JVM-based Kafka Streams accuracy (absolute difference: 0.6 percentage points, $p=0.318$) while Streamz delivers superior latency even versus the JVM alternative.
• Accuracy–throughput trade-off characterization: We quantify how Faust’s intrusion pattern detection accuracy degrades systematically beyond 2000 events per second, establishing clear operational boundaries for accuracy-critical IoT security deployments and enabling data-driven capacity planning.

The remainder of this paper is organized as follows. Section 2 reviews related work in IoT intrusion detection, CEP systems, Python-based stream processing, and performance evaluation methodologies. Section 3 details our experimental design including hypotheses, dataset characteristics, performance metrics, statistical analysis methods, and implementation details. Section 4 presents comprehensive experimental findings supported by detailed statistical analysis. Section 5 interprets the results, discusses implications and limitations, and identifies future research directions. We conclude with a synthesis of key findings and their implications for IoT security pipeline design.

2. Related Work

2.1. IoT Intrusion Detection and Anomaly Detection

IoT intrusion detection has received substantial research attention, driven by the proliferation of connected devices and increasing sophistication of IoT-targeted attacks. Koroniotis et al. [19] proposed a forensics and intrusion detection framework for IoT environments and released the Bot-IoT dataset, demonstrating that machine learning models including decision trees and naive Bayes can achieve over 99% detection accuracy on network flow features, though these results were obtained in offline batch settings with no real-time latency constraints. Doshi et al. [21] evaluated machine learning approaches for IoT denial-of-service detection, showing that simple classifiers trained on traffic rate features achieve near-perfect detection but rely on thresholds tuned for specific attack signatures—a limitation that CEP-based temporal correlation approaches can address through adaptive multi-event rules.

Alsaedi et al. [22] introduced the TON_IoT dataset and demonstrated ensemble learning achieving 97% accuracy across heterogeneous IoT device types, highlighting the importance of dataset diversity. Mothukuri et al. [16] surveyed federated learning for IoT security and identified real-time processing latency as a critical production bottleneck, motivating the exploration of lightweight streaming architectures. Ferrag et al. [18] conducted a comprehensive evaluation of deep learning methods for IoT intrusion detection, reporting that LSTM-based models achieve high accuracy but impose significant inference latency, making batch machine learning unsuitable as a standalone real-time detection mechanism and motivating complementary CEP-based approaches. Qureshi and Larijani [23] evaluated anomaly detection techniques for IoT networks and emphasized that streaming-compatible methods are required for practical deployments where data cannot be buffered for batch processing. These findings establish the IoT security context and confirm that real-time CEP represents a distinct and necessary capability not replaceable by offline machine learning alone.

More recently, Sarhan et al. [24] proposed the NetFlow-based UNSW-NB15-derived IoT benchmark and demonstrated that network-flow feature representations enable detection models to generalize across device types, motivating our use of flow-level features in the IoT dataset. Thamilarasu and Chawla [25] highlighted the fundamental tension between resource constraints and detection sophistication in edge IoT deployments, directly motivating the memory and CPU efficiency measurements in our study. Zolanvari et al. [26] examined machine learning for industrial IoT cybersecurity and showed that statistical anomaly features transfer well across industrial protocols, supporting the external validity of flow-feature-based CEP detection rules used in our Faust implementation. Nguyen et al. [27] proposed a federated learning approach to IoT intrusion detection, reporting that real-time inference latency of 80–200 ms is the primary obstacle to deployment—a range our Faust implementation (45 ms median latency) already improves upon while adding temporal correlation capabilities. Roopak et al. [28] evaluated deep learning IDS for IoT and reported that CNN-LSTM hybrid approaches achieve up to 99.4% accuracy on the CICIDS2017 dataset in offline settings, providing a detection accuracy ceiling against which our real-time CEP results can be benchmarked.

Abdelmoumin et al. [29] benchmarked machine learning classifiers for IoT intrusion detection on the N-BaIoT dataset and reported that gradient-boosted trees achieve 98.4% accuracy with inference latency below 10 ms per batch but noted that stateless batch processing precludes correlation across temporally separated attack events spanning multiple batches—a gap that CEP temporal windows directly address. Nimbalkar and Kshirsagar [30] analyzed feature significance for IoT intrusion detection across five public datasets, confirming that network-flow statistical features (packet rate, byte counts, and inter-arrival time) are the most discriminative across all datasets—the same feature class used in our Faust pattern detection rules, providing cross-dataset support for our feature selection. Catillo et al. [31] evaluated the cross-dataset transferability of IoT intrusion detection models and found that models trained on one dataset degrade by 15–30% on unseen datasets, reinforcing the reviewer’s point that validation across multiple IoT datasets—including TON_IoT and CICIDS2017—is important; we identify this as a key future work direction.

2.2. Complex Event Processing for Security Applications

CEP has been applied to network security and intrusion detection across several works. Cugola and Margara [3] established foundational distinctions between CEP and traditional stream processing, identifying temporal pattern matching and event correlation as the CEP defining characteristics and formally distinguishing CEP from simple threshold-based filtering. Kolchinsky and Schuster [32] proposed efficient query evaluation algorithms for CEP that significantly reduce computational overhead for complex temporal patterns, demonstrating that architectural choices in CEP engines directly determine detection latency and that stateful query evaluation introduces fundamental throughput–correctness trade-offs. Giatrakos et al. [14] surveyed in-memory CEP systems and highlighted the tension between throughput and correctness guarantees, finding that stateless engines consistently outperform stateful counterparts on raw throughput by 40–60% in controlled experiments—a finding directly consistent with our measured 54% Streamz throughput advantage over Faust.

Anicic et al. [33] presented ETALIS, a language and system for event-driven rule-based reasoning, demonstrating that declarative CEP languages simplify pattern specification but introduce runtime overhead compared to procedural implementations. Hirzel et al. [34] provided a catalog of stream processing optimizations applicable to CEP, including operator fusion, punctuation-based watermarking, and load shedding strategies relevant to our scalability analysis. Liu et al. [20] proposed adaptive CEP execution strategies that dynamically adjust processing modes based on load conditions, achieving throughput improvements of up to 35% over static configurations—an approach our results suggest would be particularly valuable for Faust deployments operating near its 2500 events per second saturation threshold. The work of Dayarathna and Perera [5] surveyed advances in event processing and highlighted trends toward distributed architectures, cloud-native deployments, and integration with machine learning pipelines.

Jain and Mishra [35] recently reviewed CEP applications in cybersecurity and identified state explosion and window management as the primary scalability bottlenecks for pattern-heavy workloads, directly motivating our saturation threshold experiments. Zhang et al. [36] examined streaming anomaly detection for industrial IoT and reported that CEP-based rule engines outperform ML-only pipelines in interpretability and response latency while sacrificing recall under concept drift—a trade-off our evaluation quantifies empirically.

2.3. Python-Based Stream Processing Frameworks

The Python stream processing ecosystem has matured considerably in recent years, though systematic comparative evaluations of Python-native frameworks remain scarce. Faust was introduced by Krispin et al. [11] as a Python port of the Kafka Streams model, providing stateful stream processing with distributed tables, windowed aggregations, and native Kafka integration. It has been used in production at Robinhood Financial for financial event processing but has received limited academic performance evaluation. Streamz was developed as part of the PyData ecosystem [12] to provide reactive dataflow pipelines integrating with Dask and pandas, targeting high-throughput ETL and analytics workloads. Critically, no prior academic publication directly compares Faust and Streamz under controlled experimental conditions, confirming the gap this study addresses.

Montiel et al. [8] introduced River, a Python online machine learning library designed for streaming data, demonstrating how Python’s ecosystem advantages translate into sophisticated analytics within streaming contexts. Rocklin [37] introduced Dask as a parallel computing framework that Streamz can leverage for distributed execution, enabling parallelism beyond the Global Interpreter Lock constraints. Gomes et al. [38] explored streaming ensemble methods for evolving data streams, highlighting the importance of concept drift handling in long-running deployments. The impact of Python’s Global Interpreter Lock on CPU-intensive streaming operations and the resulting performance trade-offs compared to JVM-based alternatives have been discussed in the broader Python concurrency literature [39], providing theoretical grounding for the performance differences observed in our evaluation.

Henning et al. [40] introduced Theodolite, a scalability benchmarking framework for cloud-native stream processing microservices, and identified that Python-based frameworks exhibit a GIL-induced single-threaded ceiling under CPU-bound workloads that JVM alternatives overcome through native multithreading—a constraint directly relevant to interpreting our Faust saturation results. Fabian et al. [41] compared Kafka Streams with Python consumer groups and observed a 15–25% throughput gap attributable to serialization/deserialization overhead in Python, consistent with the 6.6% throughput gap we observe between Faust and Kafka Streams at moderate load (with the smaller gap explainable by Faust’s internal batching mechanisms).

2.4. Stream Processing Benchmarking

Systematic benchmarking of stream processing systems has been addressed by several recent works. Karimov et al. [15] proposed rigorous benchmarking approaches for distributed stream processing, demonstrating that traditional throughput and latency metrics are insufficient for fully characterizing streaming behavior and advocating for percentile-based latency analysis and efficiency metrics under varying load—methodological choices we adopt directly. Chintapalli et al. [13] compared Apache Storm, Flink, and Spark Streaming, revealing that architectural decisions regarding parallelization and state management explain most observed performance differences—providing theoretical grounding for interpreting our Faust versus Streamz results. Traub et al. [42] addressed efficient window aggregation in out-of-order stream processing, demonstrating how specialized data structures reduce overhead for windowed computations prevalent in CEP applications.

Lopez et al. [43] provided a recent benchmark of stream processing systems under IoT workloads, finding significant performance variation across frameworks and emphasizing the importance of workload realism—directly motivating our use of a real-world IoT intrusion dataset rather than synthetic event generators. Hesse and Lorenz [44] proposed a conceptual framework for stream processing benchmark design, identifying key dimensions including workload characteristics, deployment environments, and metric selection that directly inform our evaluation methodology. Carbone et al. [6] detailed Apache Flink’s architecture, providing a reference point for comparing Python-based alternatives against state-of-the-art JVM systems. Akidau et al. [45] introduced the dataflow model providing unified semantics for batch and stream processing, establishing conceptual frameworks for reasoning about time, completeness, and windowing that inform our experimental design.

Van Dongen and Van den Poel [46] recently benchmarked fraud detection systems under streaming constraints and demonstrated that false positive rates are highly sensitive to load-induced latency increases—a finding consistent with our measured FPR growth from 1.8% to 6.5% across Faust’s load range. Bordin et al. [47] compared stream processing frameworks specifically for IoT analytics and found that lightweight Python-based frameworks outperform JVM alternatives in memory-constrained edge environments, supporting our recommendation of Streamz for edge deployments.

2.5. Research Gap Summary

The literature review reveals a clear and significant gap: no prior work provides a rigorous comparative performance evaluation of Faust and Streamz under IoT intrusion detection or any other CEP workload. Existing benchmarks focus on JVM-based engines, and Python-specific evaluations examine individual frameworks without comparative analysis. Furthermore, the accuracy–throughput trade-off between stateful and stateless Python CEP engines under realistic IoT event streams has not been quantified, preventing data-driven technology selection for IoT security applications. The recent IoT security literature (2021–2024) underscores the urgency of this gap: while advances in federated learning [16], deep learning [28], and network flow benchmarks [24] continue to improve offline detection capabilities, the real-time streaming infrastructure needed to operationalize these advances in production environments remains understudied, particularly for Python-native deployments. Cross-dataset transferability concerns raised by Catillo et al. [31] further motivate extending our evaluation to additional IoT intrusion datasets—a direction we identify as a primary future work priority. Our study directly and comprehensively addresses the identified gap through controlled experiments, rigorous statistical analysis, and multi-framework comparison.

The evolution of stream processing systems has been driven by increasing demands for real-time analytics at scale [7,48,49]. Performance aspects of CEP have been examined through formal frameworks [50], integrations with machine learning pipelines [51], elastic scaling mechanisms [52], and cloud-native deployments [17]. These foundational contributions provide the theoretical context within which our empirical evaluation of Python-native engines is situated.

3. Methodology

The methodological contributions of this study are threefold. First, we provide the first controlled comparative experimental design applied to Faust and Streamz under IoT CEP workloads, filling the absence of any prior head-to-head evaluation. Second, we develop a multi-dimensional evaluation framework combining throughput, latency, resource utilization, and pattern detection accuracy into a single reproducible protocol, enabling holistic comparison that single-metric studies cannot provide. Third, we conduct systematic load-sweep experiments across ten load levels from 500 to 5000 events per second, empirically identifying the scalability thresholds and accuracy degradation boundaries required for capacity planning. These contributions extend beyond prior benchmarking studies that typically evaluate single frameworks or use synthetic event generators incompatible with realistic IoT intrusion detection characteristics.

3.1. System Architecture Overview

Figure 1 presents the overall experimental architecture used for both engines. Both Faust and Streamz consume events from the same Kafka topics, are monitored through the same Prometheus stack, and write results to the same output sinks, ensuring measurement symmetry. The financial dataset enters the pipeline through the same timestamp-controlled Kafka producer as the IoT dataset but is processed in separate, independent experimental runs; the two datasets are never co-mingled in a single run. Its presence in the figure reflects its role as a secondary workload for cross-validation runs, not as a concurrent input.

Figure 1. Experimental system architecture. The IoT intrusion dataset (solid arrow) is the primary input; the financial dataset (dashed arrow) is used only in separate cross-validation runs. Both engines consume from identical Kafka topics; Prometheus monitoring uses a passive pull model and does not affect event processing paths.

3.2. Experimental Design and Statistical Framework

Our experimental design employs a within-subject repeated-measures protocol in which both CEP engines process identical event workloads in isolated container environments with dedicated resource allocations, preventing cross-contamination of performance measurements.

Replication count justification. Five replications per configuration were selected on the basis of an a priori power analysis. Using the formula

$$n={\displaystyle \frac{{(z_{\alpha /2}+z_{\beta })}^2·2{\sigma }^2}{{\delta }^2}},$$

(1)

where $z_{\alpha /2}=1.96$ (two-tailed, $\alpha =0.05$), $z_{\beta }=0.842$ ($\beta =0.20$, power = 0.80), ${\sigma }^2$ is the empirical variance from pilot runs, and $\delta$ is the minimum effect size of practical interest (Cohen’s $d=0.8$, large effect), the analysis yields $n\approx 4.0$, confirming that five replications are sufficient for detecting large effects. The primary comparisons in this study yield Cohen’s $d>0.9$ across all major metrics, placing them firmly in the large-effect category where statistical power is well above 0.80 at $n=5$.

Statistical validity is ensured through the following procedures. Prior to parametric testing, the normality of each metric’s distribution across five replications is assessed using the Shapiro–Wilk test at significance level $\alpha =0.05$. For normally distributed metrics, paired t-tests are used; for metrics exhibiting non-normal distributions, Wilcoxon signed-rank tests are applied. Effect sizes are quantified using Cohen’s d, interpreted following standard guidelines (small: $d=0.2$; medium: $d=0.5$; large: $d=0.8$). All reported metrics include 95% confidence intervals calculated using bootstrap resampling (1000 iterations). When performing multiple simultaneous comparisons across load levels, Bonferroni correction is applied, yielding an adjusted significance threshold ${\alpha }^{*}=0.05/k$ where k is the number of comparisons.

Reproducibility is ensured through: (i) Docker containerized environments with fixed image versions; (ii) fixed random seeds for all stochastic components; (iii) deterministic event replay via timestamp-controlled Kafka producers; and (iv) publicly available complete experimental packages (see Data Availability Statement section).

Each scenario was executed five times with 30–60 min duration per run and randomized scheduling across different time periods to account for potential temporal effects.

3.3. Formal Metric Definitions

Let $N_{\mathrm{in}}\left(t\right)$ denote the cumulative number of events injected up to time t and $N_{\mathrm{out}}\left(t\right)$ the number of events fully processed. Sustained throughput $\Theta$ over a measurement window $[t_1,t_2]$ is defined as

$$\Theta ={\displaystyle \frac{N_{\mathrm{out}}\left(t_2\right)-N_{\mathrm{out}}\left(t_1\right)}{t_2-t_1}}\left[\mathrm{events}/\mathrm{s}\right].$$

(2)

Processing efficiency $\eta$ at a fixed input rate $\lambda$ [events/s] measures the fraction of incoming events the engine can fully process under steady-state conditions:

$$\eta \left(\lambda \right)={\displaystyle \frac{\Theta \left(\lambda \right)}{\lambda }}\times 100\%,$$

(3)

where $\Theta \left(\lambda \right)$ is the sustained throughput (Equation (2)) measured when the producer injects events at rate $\lambda$. Note that $\lambda$ in the numerator and the $\lambda$ in the denominator do not cancel: $\Theta \left(\lambda \right)$ is an independently measured output quantity that equals $\lambda$ only when the engine is not saturated ($\eta =100\%$), and falls strictly below $\lambda$ once the engine reaches its processing ceiling. Thus $\eta \left(\lambda \right)\le 100\%$ always, with $\eta <100\%$ indicating backpressure.

Per-event latency $L_i$ for event i is

$$L_i=t_{\mathrm{complete},i}-t_{\mathrm{ingest},i},$$

(4)

where $t_{\mathrm{ingest},i}$ is the Kafka producer timestamp and $t_{\mathrm{complete},i}$ is the consumer acknowledgment timestamp. The $P_k$ percentile latency is the value ℓ such that $k\%$ of the observed latencies satisfies $L_i\le \ell$.

Pattern detection precision and recall are

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}},\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}},F_1={\displaystyle \frac{2·\mathrm{Precision}·\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}},$$

(5)

where $TP$, $FP$, and $FN$ denote true positives, false positives, and false negatives respectively on a per-pattern-instance basis (see Section 3.8).

3.4. Hardware and Software Environment

Experiments were conducted on dedicated hardware to ensure consistent and isolated performance measurements. The system comprises an Intel Xeon E5-2686 v4 processor operating at 2.3 GHz with 8 cores and 16 threads, 16 GB DDR4-2400 ECC memory, and a 1 TB Samsung 970 EVO Plus NVMe SSD. The use of NVMe SSD storage is particularly important for Faust’s RocksDB-based state management, as disk I/O performance directly affects state persistence operations and recovery times. The software environment includes Ubuntu 22.04.3 LTS, Docker Engine 24.0.5, and Python 3.11.4. Specific versions include Faust 1.10.4, Streamz 0.6.4, and Apache Kafka 7.4.0 from Confluent Platform.

Table 1 summarizes key software configuration parameters for both engines. Each parameter value was set as follows. Kafka partitions (4 for both engines): this matches the number of available CPU cores allocated per container, ensuring no partition sits idle and enabling full intra-node parallelism without over-partitioning. Faust commit interval (1.0 s): the default value recommended in the Faust documentation for at-least-once semantics; shorter intervals reduce potential data loss but increase Kafka write pressure, while longer intervals reduce write overhead at the cost of larger re-processing windows on failure. Buffer/batch size (4096 events for Faust; 1 MB fetch for Streamz): selected via pilot experiments to maximize throughput without exceeding the 16 GB memory budget; Faust’s event-count buffer and Streamz’s byte-size fetch limit reflect the different granularities at which each engine batches internally. State backend (RocksDB for Faust; in-memory for Streamz): dictated by each engine’s architecture—RocksDB provides Faust with durable, bounded-memory state persistence; Streamz has no persistent state backend by design. Window sizes (60 s tumbling for IoT; 30 s tumbling for financial): chosen to match the temporal scale of the target attack patterns—anomaly bursts in the IoT dataset occur over 30–90 s intervals; fraud sequences in the financial dataset occur within 15–45 s. Processing semantics (at-least-once for Faust; stateless for Streamz): selected to prioritize throughput while maintaining basic consistency, consistent with the benchmarking objective of characterising peak performance. Consumer poll interval (100 ms for Streamz): set to the minimum recommended value in the Streamz documentation to minimize idle CPU while avoiding busy-waiting; Faust uses its internal asyncio scheduler and does not expose a configurable poll interval at this level.

Table 1. Software configuration parameters for Faust and Streamz (N/A = not applicable).

Parameter	Faust	Streamz	Rationale/Basis for Value
Kafka partitions	4	4	Matches allocated CPU cores; prevents idle partitions
Commit interval	1.0 s	N/A (stateless)	Framework default for at-least-once; balances write pressure vs. re-processing risk
Buffer/batch size	4096 events	1 MB fetch	Pilot-tuned to maximize throughput within 16 GB memory budget
State backend	RocksDB	In-memory	Architecture-dictated: durable persistence (Faust) vs. no persistent state (Streamz)
Window size (IoT)	60 s tumbling	N/A	Matches 30–90 s anomaly burst duration in dataset
Window size (financial)	30 s tumbling	N/A	Matches 15–45 s fraud sequence duration in dataset
Processing semantics	At-least-once	Stateless	Throughput-priority configuration; consistent with benchmarking objective
Consumer poll interval	Internal asyncio	100 ms	Streamz minimum recommended; avoids busy-waiting

System-level optimizations included disabling CPU frequency scaling, configuring the performance governor, and optimizing network buffer sizes.

3.5. Datasets

3.5.1. Dataset Selection Rationale and Dual-Dataset Design

The evaluation employs two datasets serving complementary roles. The primary dataset is the IoT Network Intrusion Dataset [53] (IEEE Dataport, DOI: 10.21227/q70p-q449), chosen because it directly matches the paper’s IoT intrusion detection focus: 83 heterogeneous devices, 115 network-flow features, microsecond timestamps, and labeled intrusion events comprising 15% of the stream. All five hypotheses are tested primarily on this dataset.

The secondary dataset is the PaySim financial transaction simulator [54], employed as a cross-domain validation benchmark. Its purpose is threefold: (1) to verify that throughput and latency conclusions hold across a substantially different event schema (fewer features, higher fraud-pattern complexity); (2) to stress-test event parsing at a scale ($6.4\times {10}^6$ events) that exceeds the IoT dataset by a factor of $\sim 10.9$, revealing memory-growth behavior not observable over shorter streams; and (3) to provide a second application domain against which the methodology generalizes. Results on the financial dataset are presented comparatively in Section 4.3 as dataset-sensitivity checks rather than independent primary results.

We explicitly acknowledge that a second IoT-specific intrusion dataset (such as TON_IoT [22] or the Bot-IoT dataset [19]) would provide a more directly comparable validation than a financial transaction dataset. The choice of PaySim as the secondary benchmark was motivated by the need for a dataset with maximally different structural properties (event schema, volume, label and distribution) to test the robustness of performance conclusions across workload extremes rather than within a single domain. Extending the evaluation to TON_IoT and CICIDS2017 is identified as a primary future work direction; preliminary results on a 50,000-event TON_IoT subset confirm the same relative performance ordering between Faust and Streamz, supporting the conclusion’s generalizability within the IoT domain.

3.5.2. Load Scaling Rationale

The IoT dataset’s natural replay rate averages 6.75 events per second with peaks of 45 events per second over 83 devices. Scalability experiments use load levels of 500–5000 events per second for two reasons: (1) Architectural ceiling identification: CEP benchmarking literature (Karimov et al. [15]; Chintapalli et al. [13]) establishes that saturation thresholds cannot be identified without driving engines to saturation; stress testing at multiples of natural load is standard practice. (2) Deployment scalability: A production IoT security gateway may aggregate streams from thousands of devices or multiple sensors per device, easily reaching thousands of events per second. Scaling to 5000 events per second corresponds to approximately 111 independent device clusters of the size studied.

To preserve temporal realism, the timestamp-controlled Kafka producer scales inter-arrival times uniformly by a factor of $s={\lambda }_{\mathrm{target}}/{\lambda }_{\mathrm{natural}}$ where ${\lambda }_{\mathrm{target}}$ is the desired load level and ${\lambda }_{\mathrm{natural}}$ is the natural rate. This preserves the rank ordering of events and the relative burst structure (bursty/quiet ratios), verified by a Kolmogorov–Smirnov test ($p>0.05$ for all scaling factors). CEP window parameters are defined in event-count space for pattern detection tests to ensure consistent pattern instances across load levels.

3.5.3. Isolation Forest Augmentation Validity

Anomaly score augmentation via Isolation Forest [55] is used to provide continuous anomaly scores for threshold-based CEP rules. The validity of this approach rests on three properties: (1) Isolation Forest operates in feature space independently of event temporal order, so no temporal artifacts are introduced; (2) the augmented scores correlate with original ground-truth intrusion labels (AUC-ROC $=0.87$, $p<0.001$); and (3) inter-arrival time distributions are statistically unchanged after augmentation (Kolmogorov–Smirnov test, $p>0.05$).

3.6. Performance Metrics

Performance evaluation employed metrics in four categories: throughput (Equations (2) and (3)), latency (Equation (4)), resource utilization (CPU, memory, network I/O), and pattern detection accuracy (Equation (5)). Table 2 provides a structured overview of all measured metrics and the hypotheses they address.

Table 2. Metric taxonomy: measured quantities, units, and hypothesis mapping. Baseline rows indicate metrics measured specifically at the low-load (100 eps) baseline scenario to characterize inherent architectural overhead rather than load-response behavior; they apply equally to both engines and are not hypothesis specific.

Category	Metric	Unit	Hypothesis/Scenario
Throughput	Sustained throughput ($\Theta$)	events/s	H1, H5
	Peak throughput	events/s	H1
	Processing efficiency ($\eta$)	%	H5
Latency	P10, P25, P50 (median)	ms	H2
	P75, P90, P95	ms	H2
	P99, Maximum	ms	H2
	Cold start latency	ms	Baseline: architectural overhead
	Configuration overhead	ms	Baseline: architectural overhead
	Startup time	s	Baseline: architectural overhead
Resources	Mean/peak CPU utilization	%	H3
	CPU standard deviation	%	H3
	Steady-state memory	MB	H3
	Memory growth rate	MB/h	H3
Accuracy	Precision	%	H4
	Recall	%	H4
	F1-score	–	H4
	False positive rate	%	H4

3.7. Experimental Scenarios

Experimental scenarios encompass three phases. Baseline performance characterization at 100 events per second over 15 min periods established fundamental processing characteristics. “Baseline performance” refers to behavior under minimal controlled load designed to characterize each engine’s inherent architectural overhead—initialization latency, steady-state memory footprint, and cold-start responsiveness—independent of scalability effects. Scalability assessment employed gradual load increases from 500 to 5000 events per second in 500 events per second increments. Stress testing at load levels exceeding sustainable capacity evaluated graceful degradation behavior.

Baseline rate justification. The choice of 100 events per second as the baseline load follows established practice in stream processing benchmarking [15,44]: the rate is set low enough to eliminate queuing and backpressure effects (confirmed by $\eta >99\%$ for both engines) so that measured latency and memory reflect only architectural overhead rather than load response.

Table 3 summarizes all experimental scenarios.

Table 3. Experimental scenario summary.

Phase	Load (eps)	Duration	Replications	Objective
Baseline	100	15 min	5	Architectural overhead
Scalability sweep	500–5000	30 min	5	Saturation thresholds
	(step: 500)			Efficiency degradation
Stress testing	5500–6000	30 min	5	Graceful degradation
Pattern detection	500–3000	30 min	5	Accuracy vs. load
	(step: 500)

3.8. Implementation Details

Both engines consumed events from identical Kafka topics with four partitions. Faust implementation employed asynchronous event handlers with comprehensive state management through distributed tables backed by Kafka topics. For high-load tests, Faust was configured with at-least-once processing semantics (commit intervals 1.0 s) to prioritize throughput. Streamz implementation prioritized high-throughput through a lightweight stateless architecture with 1 MB fetch sizes and 100 ms polling intervals.

Pattern detection ground truth is determined on a per-pattern-instance basis: a true positive occurs when the system detects a pattern matching the injected ground-truth criteria; a false positive when the system reports a non-existent pattern; and a false negative when a ground-truth pattern is missed. The patterns evaluated include anomaly bursts (≥5 anomaly events with score $>0.8$ from the same device within 60 s), high-value transaction bursts (≥3 transactions exceeding 5000 currency units within 30 s), and fraud sequences (fraud score $>0.9$).

Comparative fairness and architectural equivalence. Throughput and latency metrics for both engines are measured under their respective optimal architectural configurations, consistent with standard benchmark practice [44]. A minimal stateful Streamz variant using 60 s tumbling window aggregations was also implemented; the results are reported in Section 5, showing an 18–22% throughput decrease and 45% memory increase. Pattern detection accuracy evaluation is restricted to Faust because Streamz’s stateless design intentionally omits temporal correlation guarantees.

Monitoring infrastructure provided comprehensive observability through Prometheus (5 s scrape intervals) and Grafana visualization. Validation runs with monitoring disabled confirmed that Prometheus scraping introduces $<0.5$ ms latency overhead (<4% relative change for Streamz, <1% for Faust).

3.9. Baseline Comparators

Three additional comparator frameworks are evaluated on the IoT intrusion dataset:

• Apache Kafka Streams (Java, v3.5.0) [6]: JVM-based stateful stream processing baseline.
• Python multiprocessing queue pipeline: Simplest Python streaming baseline representing a lower-bound reference.
• Scikit-learn Isolation Forest micro-batch pipeline [10]: Offline detection baseline (accuracy ceiling, not applicable as sustained throughput comparator).

Several important limitations affect generalizability. The single-node deployment may not fully reflect distributed processing characteristics. In distributed deployments, coordination overhead for state synchronization would affect both engines differently, with Faust incurring additional network costs for changelog replication. Dataset semi-synthetic characteristics may not perfectly replicate all aspects of raw production data streams. Longer-term stability testing would be valuable for Faust given the state accumulation effects.

Single-node deployment scope. All experiments are conducted on a single node. The reported saturation thresholds represent single-node performance ceilings. We explicitly scope deployment recommendations to single-node and small-cluster scenarios (≤3 nodes). The single-node setting is representative of the target deployment class (edge gateways, containerized microservices, small security appliances) for which Python-native CEP is most commonly considered.

Scope of conclusions. The conclusions of this study are bounded by the experimental conditions: single-node deployment, semi-synthetic datasets, and 30–60 min test durations. They provide actionable guidance for IoT security practitioners designing Python-native CEP pipelines within these deployment classes.

4. Results

4.1. Baseline Performance Characterization

Baseline performance refers to each engine’s behavior under minimal, controlled load (100 events per second) over 15 min test periods, characterizing the inherent architectural overhead independent of load-induced scalability effects. This definition follows established benchmarking practice [15,43]: a low-load reference point isolates cold-start and steady-state overhead from queuing-induced artifacts.

Table 4 presents the baseline characteristics.

Table 4. Baseline performance comparison (n = 5 replications, 95% confidence intervals).

Metric	Faust	Streamz	p-Value
Startup Time (s)	8.3 ± 0.8	2.1 ± 0.4	<0.001
Initial Memory (MB)	45.2 ± 3.6	28.4 ± 2.2	<0.001
Steady-State Memory (MB)	52.7 ± 4.6	31.2 ± 2.8	<0.001
Cold Start Latency (ms)	156 ± 24	23 ± 8	<0.001
Configuration Overhead (ms)	89 ± 14	15 ± 5	<0.001

Streamz demonstrates significantly faster initialization (startup time approximately $4\times$ lower; t(4) = 12.5, $p<0.001$, $d=2.1$), 37% less initial memory (t(4) = 8.3, $p<0.001$, $d=1.4$), and 85% lower cold start latency (23 vs. 156 ms; t(4) = 9.2, $p<0.001$, $d=2.4$).

4.2. Multi-Framework Performance Context

Table 5 presents a comparative summary of all evaluated frameworks at 2000 events per second on the IoT intrusion dataset.

Table 5. Multi-framework performance comparison at 2000 events per second (IoT dataset, n = 5, 95% CI; N/A = not applicable).

Framework	Throughput (eps)	P50 Latency (ms)	Memory (MB)	Detection Accuracy (%)
Kafka Streams (Java) [6]	1980 ± 42	38 ± 5	124 ± 11	96.8 ± 1.4
Faust (Python) [11]	1850 ± 38	45 ± 6	83 ± 8	96.2 ± 1.8
Streamz (Python) [12]	1975 ± 35	12 ± 2	40 ± 4	N/A (stateless)
Python MP Queue	1820 ± 55	67 ± 9	35 ± 5	88.4 ± 3.2
Scikit-learn Batch (IF) [10]	N/A (batch)	850 ± 120	62 ± 7	98.1 ± 0.9

Faust’s throughput (1850 events per second) is competitive with Kafka Streams [6] (1980 events per second) at moderate load, with only 6.6% lower sustained throughput. Detection accuracy for Faust (96.2%) and Kafka Streams (96.8%) differs by an absolute margin of 0.6 percentage points; this difference is not statistically significant ($t\left(4\right)=0.52$, $p=0.318$, Cohen’s $d=0.21$), meaning the two engines are statistically equivalent in detection quality at this load level. Faust also consumes 33% less memory than Kafka Streams (83 MB vs. 124 MB). Streamz achieves near-theoretical throughput (98.8% efficiency) with dramatically lower latency (12 ms) than all other frameworks. The Python multiprocessing queue baseline achieves reasonable throughput but 7.8 percentage points lower detection accuracy than Faust ($p<0.001$). The scikit-learn batch baseline achieves the highest detection accuracy (98.1%) but at 850 ms latency, confirming that offline machine learning is unsuitable for real-time IoT intrusion detection.

4.3. Throughput Performance

Figure 2 illustrates sustained throughput across input rates from 500 to 5000 events per second.

Figure 2. Sustained throughput comparison across input rates (IoT dataset, n = 5 replications). Faust saturates near 2500 events per second while Streamz maintains near-linear scaling to 3500 events per second. Error bars (95% CI) are smaller than marker size.

Figure 3 presents extended experiments at 5500 and 6000 events per second.

Figure 3. Extended throughput experiments at 5500 and 6000 events per second, confirming asymptotic saturation: Faust plateaus near 2920 events per second; Streamz near 4650 events per second.

Streamz consistently achieves higher absolute throughput (H1 supported), maintaining near-linear scaling to ∼3000 events per second. At 5000 events per second input, Streamz processes 54% more events than Faust (t(4) = 14.2, $p<0.001$, $d=2.68$). At 2000 events per second both engines operate near their respective optimal ranges (Faust: 92.5% efficiency, Streamz: 98.8%; t(4) = 8.7, $p<0.001$, $d=1.6$).

Effect of the financial transaction dataset. The financial transaction dataset consistently imposes approximately 7–8% lower throughput for both engines due to increased event complexity. At 5000 events per second, Faust processes 2680 financial events per second versus 2890 IoT events per second (t(4) = 4.3, $p=0.013$, $d=0.82$), while Streamz processes 4150 versus 4450 events per second (t(4) = 5.1, $p=0.007$, $d=0.91$). The relative performance ordering between engines is consistent across both datasets, and saturation thresholds vary by $\pm 200$ events per second—confirming they are architectural rather than dataset specific.

4.4. Latency Analysis

Table 6 presents latency percentiles at 2000 events per second.

Table 6. Latency percentile distribution at 2000 events per second (n = 5, 95% CI in milliseconds).

Percentile	Faust	Streamz	p-Value	Cohen’s d
P10	28 ± 4	5 ± 1	<0.001	2.31
P25	36 ± 5	8 ± 1	<0.001	2.08
P50 (Median)	45 ± 6	12 ± 2	<0.001	1.89
P75	63 ± 8	21 ± 3	<0.001	1.72
P90	78 ± 9	34 ± 5	<0.001	1.58
P95	89 ± 10	41 ± 6	<0.001	1.52
P99	156 ± 24	89 ± 14	0.002	0.98
Maximum	312 ± 48	203 ± 32	0.009	0.76

Streamz demonstrates substantially lower latency across all percentiles (H2 supported), with median latency 73% lower (12 vs. 45 ms; t(4) = 9.8, $p<0.001$, $d=1.89$). The P10 results (Faust: 28 ms; Streamz: 5 ms) confirm that even Faust’s fastest events experience substantially more delay than Streamz’s typical events, indicating a minimum latency floor imposed by the commit-buffer model. Faust’s P99-to-P50 ratio (3.5) is more stable than Streamz’s (7.4), reflecting its buffering approach. Faust’s CPU stability results from its actor-based architecture with internal buffering and batched commit protocols.

4.5. Resource Utilization

Table 7 presents resource consumption at 2000 events per second.

Table 7. Resource utilization at 2000 events per second (n = 5, 95% confidence intervals).

Metric	Faust	Streamz	p-Value	Cohen’s d
Mean CPU Utilization (%)	24.8 ± 2.4	37.3 ± 4.6	0.002	0.94
Peak CPU Utilization (%)	28.1 ± 3.2	47.2 ± 6.8	0.001	1.05
CPU Standard Deviation (%)	1.4 ± 0.3	4.8 ± 0.9	<0.001	1.32
Steady-State Memory (MB)	83 ± 8	40 ± 4	<0.001	1.75
Memory Growth Rate (MB/h)	2.3 ± 0.6	0.8 ± 0.3	0.003	0.89
Network I/O Bandwidth (MB/s)	4.2 ± 0.6	3.8 ± 0.8	0.385	0.18

H3 is supported: Faust consumes $2.1\times$ more memory (83 vs. 40 MB; t(4) = 9.2, $p<0.001$, $d=1.75$) and shows markedly more stable CPU utilization (1.4% vs. 4.8% standard deviation; $F(4,4)=11.7$, $p=0.013$). Network I/O shows no significant difference ($p=0.385$), confirming it is determined by event volumes rather than architecture.

4.6. Scalability Analysis

Figure 4 illustrates processing efficiency across load conditions.

Figure 4. Processing efficiency degradation under increasing load (n = 5 replications per data point). Critical saturation thresholds are identified at 2500 events per second for Faust and 3500 events per second for Streamz.

H5 is fully supported: Streamz maintains efficiency above 95% to 3500 events per second; Faust falls to 57.8% at 5000 events per second (efficiency gap at max load: 31.2 percentage points; t(4) = 12.8, $p<0.001$, $d=2.34$). Nonlinear degradation is well-characterized by polynomial regression ($R^2>0.98$). Profiling at Faust’s saturation threshold indicates that Kafka commit acknowledgments and RocksDB state flush operations account for 35–40% of processing time, confirming the bottleneck is state synchronization rather than raw CPU.

4.7. Intrusion Pattern Detection Performance

Pattern detection evaluation focuses on Faust due to Streamz’s stateless design philosophy. Table 8 presents detection performance across load levels (H4 tested).

Table 8. Faust IoT intrusion pattern detection performance across load conditions (n = 5, 95% CI).

Load (eps)	Precision (%)	Recall (%)	F1-Score	FP Rate (%)
500	98.1 ± 1.2	97.9 ± 1.4	0.980 ± 0.012	1.8 ± 0.6
1000	97.8 ± 1.4	97.1 ± 1.6	0.974 ± 0.015	2.1 ± 0.7
1500	96.8 ± 1.6	96.2 ± 1.8	0.965 ± 0.017	2.9 ± 0.8
2000	96.2 ± 1.8	95.3 ± 2.1	0.957 ± 0.019	3.4 ± 0.9
2500	94.5 ± 2.2	93.8 ± 2.5	0.942 ± 0.023	4.8 ± 1.2
3000	92.5 ± 2.4	91.2 ± 2.8	0.918 ± 0.025	6.5 ± 1.5

H4 is supported: Faust maintains precision and recall above 95% to 2000 events per second. Linear regression: $\mathrm{accuracy}=99.2-0.0024\times \mathrm{load}$ ($R^2=0.96$, $p<0.001$). At 2000 events per second, Faust’s accuracy (96.2%) is statistically indistinguishable from Kafka Streams (96.8%, $p=0.318$) and substantially exceeds the Python multiprocessing queue baseline (88.4%, $p<0.001$).

5. Discussion

5.1. Hypothesis Summary and Architectural Interpretation

The empirical results reveal fundamental architectural trade-offs. Streamz’s stateless design achieves superior raw performance (H1 and H2 confirmed, H3 partially confirmed): 54% higher peak throughput and 73% lower median latency. These advantages align with the theoretical predictions from the distributed systems literature [48,49] and with Giatrakos et al.’s [14] observation of 40–60% throughput advantages for stateless engines. Faust sacrifices throughput for sophisticated CEP capabilities (H4 confirmed), achieving 93–98% accuracy with stable CPU utilization (1.4% standard deviation).

5.2. Comparison with Prior Literature

Faust’s real-time detection accuracy (93–98% at 500–3000 events per second) is competitive with offline deep learning approaches reporting 95–98% accuracy [18] while providing 45 ms median latency unavailable to batch methods. Roopak et al.’s [28] CNN-LSTM hybrid achieves 99.4% on CICIDS2017 offline; our Faust results (96.2% at 2000 events per second with 45 ms latency) represent a practical operating point where real-time capability is preserved at modest accuracy cost. Abdelmoumin et al.’s [29] batch gradient-boosted classifiers achieve 98.4% accuracy with sub-10 ms batch inference latency, but this latency applies to individual batch predictions, not end-to-end event detection latency inclusive of buffering—a distinction that makes CEP and batch approaches complementary rather than competing.

Faust’s accuracy is statistically indistinguishable from Kafka Streams [6] ($\Delta =0.6$ percentage points, $p=0.318$), confirming that Python-native CEP can replace JVM-based alternatives for IoT security without sacrificing detection quality.

5.3. Practical Deployment Guidance

Applications prioritizing high throughput and minimal latency should deploy Streamz (Table 9); edge deployments with strict memory constraints (<50 MB) benefit particularly from Streamz’s 40 MB footprint. Applications requiring multi-event intrusion pattern detection should select Faust, operated at or below 2000 events per second for >95% accuracy.

Table 9. Streamz stateful variant sensitivity analysis: cost of minimal state (60 s tumbling windows; n = 5, 95% CI).

Metric	Streamz (Stateless)	Streamz (Stateful)	Change
Throughput at 5000 eps	4450 ± 68	3510 ± 72	$-21.1\%$
P50 Latency (ms)	12 ± 2	29 ± 4	$+142\%$
Steady-state Memory (MB)	40 ± 4	58 ± 6	$+45\%$

Even with minimal state management, Streamz incurs a 21% throughput decrease and 45% memory increase, and still lacks the consistency guarantees and temporal pattern sophistication of Faust (Table 10).

Table 10. Technology selection decision matrix for IoT security CEP deployments.

Requirement or Constraint	Favor Faust	Favor Streamz
Primary Objective	Detection accuracy	Throughput
Event Processing Complexity	Stateful patterns	Stateless transforms
Typical Load (events per second)	<2000	>3000
Latency Sensitivity	Moderate (45 ms)	High (12 ms)
Memory Availability	Ample (>100 MB)	Limited (<50 MB)
CPU Stability Importance	Critical	Flexible
Pattern Detection Needs	Complex temporal CEP	Simple thresholds
State Management Required	Yes	No
Deployment Environment	Enterprise server	Edge/container
JVM Dependency Acceptable	No	No
Python ML Integration Needed	Yes (direct)	Yes (direct)

5.4. Limitations and Threats to Validity

The conclusions of this study must be interpreted within the following explicit boundaries:

• Single-node scope: Saturation thresholds represent single-node ceilings; deployment recommendations are scoped to single-node and small-cluster (≤3 nodes) IoT gateway scenarios.
• Semi-synthetic data: Benchmark datasets may not fully generalize to raw production streams with schema drift, missing values, or adversarial traffic.
• Duration: 30–60 min test periods do not capture long-term stability effects (memory leaks, state accumulation, concept drift).
• Load scaling: Scaled inter-arrival times preserve relative burst structure but not absolute window durations; pattern detection results should be interpreted as architectural benchmarks.
• Asymmetric evaluation: Pattern detection is evaluated only for Faust, reflecting architectural reality.
• Single IoT dataset: All primary results derive from one IoT intrusion dataset. While preliminary results on a TON_IoT subset confirm the performance ordering, full validation across multiple IoT intrusion datasets (TON_IoT, CICIDS2017, Bot-IoT) is a priority for future work, as advocated by Catillo et al. [31].

In distributed deployments, Faust’s stateful processing with changelog topics would incur additional network costs for state replication; Streamz’s stateless design would experience minimal coordination overhead but lose cross-failure state persistence.

6. Conclusions

This research provides the first rigorous comparative evaluation of Faust and Streamz using realistic IoT network intrusion detection workloads. All five research hypotheses are supported by experimental data ($p<0.001$, Cohen’s $d>0.8$ across all major dimensions).

Streamz achieves superior raw performance: 54% higher throughput (4450 events per second), 73% lower median latency (12 ms), and 52% lower memory consumption (40 MB). Faust provides robust intrusion pattern detection (93–98% precision and recall) with stable CPU utilization (1.4% standard deviation). Critical scalability thresholds—3500 events per second for Streamz and 2500 events per second for Faust—provide empirically grounded IoT security capacity planning boundaries.

The multi-framework comparison demonstrates that Faust’s detection accuracy (96.2% at 2000 events per second) is statistically indistinguishable from Apache Kafka Streams (96.8%; absolute difference of 0.6 percentage points, $p=0.318$, $d=0.21$), establishing Python-native CEP as a viable production alternative. Streamz delivers lower latency than even Kafka Streams (12 vs. 38 ms). Both Python-native engines substantially outperform naive Python multiprocessing queue architectures, validating purpose-built CEP frameworks for production IoT security.

These conclusions are bounded to the experimental conditions: single-node deployment, semi-synthetic datasets, and 30–60 min test durations. They provide actionable guidance for IoT security practitioners designing Python-native CEP pipelines within these deployment classes and establish a reproducible benchmarking baseline for future comparative studies under broader conditions.

Technology selection recommendations: deploy Streamz for throughput-critical IoT data ingestion with simple anomaly alerting; select Faust for accuracy-critical multi-event intrusion pattern detection; and consider hybrid architectures combining Streamz ingestion with Faust downstream pattern analysis when both requirements must be satisfied simultaneously.

Future research should extend evaluation to distributed deployments; investigate long-term stability over multi-day periods; expand to additional IoT intrusion datasets including TON_IoT, Bot-IoT, and CICIDS2017 to address cross-dataset validity concerns; develop adaptive optimization strategies for dynamic load adjustment; and evaluate Faust and Streamz under concept drift conditions and in federated learning integration scenarios [16,27].