Enhancing Security and Privacy in IoT Data Streams: Real-Time Anomaly Detection for Threat Mitigation in Traffic Management ^†

Abstract

The rapid expansion of IoT in smart cities has improved traffic management but increased security risks. Traditional IDS struggle with advanced threats, prompting adaptive solutions. This work proposes a framework combining machine learning (ML), Zero Trust Architecture (ZTA), and blockchain authentication. Supervised models (XGBoost, RF, SVM, LR) detect known anomalies, while a CNN Autoencoder identifies novel threats. Blockchain ensures identity integrity, and compromised devices are isolated automatically. Tests on the IoT-23 dataset demonstrate superior accuracy, fewer false positives, and better scalability than conventional methods. The integration of AI, Zero Trust, and blockchain significantly boosts IoT traffic system security and resilience.

1. Introduction

The Internet of Things (IoT) is transforming urban systems like traffic management, smart grids, and public safety through real-time data collection and decision-making. By 2030, over 75 billion IoT devices are expected to be in use globally, forming dense sensor networks that improve urban efficiency [1]. However, this connectivity raises serious security concerns, such as cyberattacks, data breaches, and system vulnerabilities [2,3]. Traditional intrusion detection often fails to catch zero-day and advanced threats, highlighting the need for real-time anomaly detection [4]. In IoT-based traffic systems, ensuring data integrity is vital. While cryptography offers protection, it can burden resource-limited devices [5,6], and concept drift in dynamic networks reduces detection accuracy [7]. ML and DL methods offer scalable, adaptive solutions for anomaly detection and threat mitigation [4], enabling real-time insights and adaptation to new threats. As a result, the potential of lightweight, real-time detection models is increasingly emphasized [8,9].

This paper presents a real-time anomaly detection framework for securing IoT traffic systems, combining ML techniques with scalability and resilience. The paper is structured as follows. Section 2 covers theoretical background and related work, Section 3 presents the system model and threat analysis, Section 4 introduces the proposed framework, Section 5 outlines the methodology, Section 6 provides evaluation results, and Section 7 concludes with future research directions.

2. Background and Related Work

The expansion of IoT has revolutionized industries such as traffic management and healthcare, but it has also brought about security challenges [1]. Traditional Intrusion Detection Systems (IDS) often struggle to address sophisticated threats, necessitating the development of adaptive solutions [8]. Researchers have investigated methods like machine learning and deep learning for anomaly detection [3], lightweight encryption protocols, blockchain-based authentication, and hybrid AI models to improve security while keeping overhead low [5,10]. This section examines the challenges associated with anomaly detection, cryptographic security, and the management of IoT traffic.

2.1. Anomaly Detection in IoT Networks

IoT growth demands real-time anomaly detection to counter threats and ensure data integrity [3]. Signature-based IDS miss zero-day attacks, so ML/DL methods are essential [8]. Clustering (k-means, DBSCAN) helps traffic classification but lacks scalability [4,7,11]. Adaptive models like QF_HST and IForestASD update in real time [6], while distributed ML (e.g., GANs) enables privacy-preserving detection [2]. DL models like autoencoders and CNNs improve traffic analysis [12,13]. Hybrids like D-PACK and GA-HDLAD enhance accuracy with low overhead [9,10,12].

2.2. Cryptographic and AI-Powered Security Mechanisms

Traditional cryptography (AES, RSA) is secure but heavy for IoT devices [5,14]. Lightweight ciphers (SPECK, PRESENT) and AES-128 offer better performance [5,15]. Homomorphic encryption supports privacy-preserving processing [3,16]. Blockchain adds tamper-proof data handling and smart contracts [1,17]. AI and federated learning aid distributed threat detection [2,10,18]. Combining lightweight crypto, blockchain, and AI ensures scalable IoT security [5,19].

2.3. Security Challenges in IoT-Enabled Traffic Management

IoT traffic systems face MITM, DoS, and data leaks [19,20]. Blockchain and FL enhance privacy but add overhead [2,17]. ML requires quality data and faces drift issues [16,18]. A lack of standard security protocols limits interoperability [20]. Solutions include the following elements:

• Cross-platform security standards [2,3];
• Lightweight encryption techniques for real-time data [14];
• Decentralized security frameworks that integrate blockchain and AI [17].

3. System Model and Threat Analysis

This section introduces an IoT system model combining blockchain, heterogeneous networks, ZTA, and data-centric security to ensure secure communication, authentication, and threat mitigation.

3.1. IoT System Architecture and Data Flow

Our multi-layered IoT system consists of interconnected components responsible for data generation, secure transmission, processing, and storage. The system follows a structured data flow to maintain confidentiality, integrity, and availability, as shown in Figure 1.

The proposed system model comprises several layers:

1—Device Layer (Edge IoT Devices and Sensors):

–

This layer consists of heterogeneous IoT devices, including smart sensors, industrial IoT devices, and healthcare monitoring systems;

–

Devices communicate using various IoT protocols (Zigbee, MQTT, CoAP, etc.);

–

Lightweight Zero Trust authentication mechanisms, such as one-time device verification, ECC-based credentials, or hardware trust anchors (e.g., TPM, TrustZone), are employed to reduce processing overhead while maintaining strong identity assurance.

2—Edge Computing and Gateway Layer:

–

IoT gateways aggregate, filter, and encrypt data before forwarding it to the cloud;

–

Blockchain stores authentication logs, ensuring tamper-proof access control;

–

AI-based real-time anomaly detection helps detect malicious behavior at the edge.

3—Network Layer (Communication and Secure Routing):

–

Uses both short-range (Wi-Fi, BLE, Zigbee) and long-range (5G, LPWAN, LoRaWAN) communication;

–

Implements Zero Trust micro-segmentation, preventing lateral movement of attacks;

–

Enforces lightweight end-to-end encryption (TLS, DTLS with PSK or ECC) for secure and efficient data transmission.

4—Cloud and Blockchain Security Layer:

–

Cloud servers analyze real-time data using machine learning-based anomaly detection;

–

Blockchain ledger stores transaction logs, ensuring data integrity and non-repudiation;

–

Implementing data-centric security ensures that data protection mechanisms travel with the data, reducing risks of exposure.

Below are the stages of data flow in the system model:

• IoT devices generate data and transmit it securely via gateways using lightweight authentication (e.g., ECC or hardware-based);
• Gateways preprocess, encrypt, and verify data through adaptive Zero Trust policies;
• Blockchain records access logs to ensure authenticity and traceability;
• The cloud performs real-time anomaly detection and threat analysis;
• Secure storage and fine-grained access control protect data from unauthorized access.

3.2. IoT Network Setup, Devices, and Protocols

A heterogeneous IoT network integrates diverse devices and applications across multiple domains, enabling seamless communication and supporting various protocols and technical requirements within a unified system. It supports a range of IoT applications, including the following:

–

Smart cities: Traffic monitoring, smart grids, surveillance.

–

Healthcare IoT (MIoT): Wearable devices, remote patient monitoring.

–

Industrial IoT (IIoT): Smart factories, predictive maintenance.

Overview of the communication protocols used in different IoT layers see

Table 1. Overview of the communication protocols used in different IoT layers.

Layer	Devices	Communication Protocols
Device Layer	Smart sensors, actuators, cameras	Zigbee, BLE, Wi-Fi
Gateway Layer	IoT gateways, edge nodes	MQTT, AMQP, Edge Computing Protocols
Network Layer	Routers, base stations	5G, LPWAN, LoRaWAN
Cloud Layer	AI-powered servers, blockchain nodes	MQTT, CoAP, HTTPS

.

The Zero Trust principle ensures that each device, regardless of location, must be authenticated before gaining access to resources.

3.3. Attack Surfaces

IoT networks are vulnerable to various cyber threats, classified based on system layers:

–

Device Layer: Vulnerable to malware injection, physical tampering, and botnet attacks, where infected IoT devices are used for large-scale DDoS attacks.

–

Gateway Layer: Exposed to unauthorized access, protocol exploitation (e.g., MQTT, AMQP), and firmware attacks.

–

Network Layer: Prone to Man-in-the-Middle (MitM) attacks, Denial-of-Service (DoS), and eavesdropping, where unprotected data exchanges expose sensitive information.

–

Cloud Layer: At risk of data breaches, unauthorized access, and API exploitation.

A multi-layered security approach combining encryption, anomaly detection, and strong access controls is essential to mitigate these risks.

3.4. Key Security Challenges and Threats

The proposed system addresses several critical IoT security challenges (

Table 2. Security challenges in IoT networks and their corresponding solutions.

Challenge	Solution
Unauthenticated IoT devices	Zero Trust authentication
Data tampering	Blockchain-based integrity verification
Unauthorized access	Role-based and attribute-based access control (RBAC/ABAC)
DDoS attacks	AI-driven anomaly detection
Secure communication	End-to-end encryption (TLS, DTLS)

):

This hybrid model boosts IoT trust by combining ZTA, blockchain, anomaly detection, and data-centric security. The next section details its real-time threat mitigation framework.

4. Proposed Framework: Real-Time Anomaly Detection and Threat Mitigation

The framework secures IoT smart cities via AI-based detection, blockchain, and ZTA, enabling real-time threat response. It includes four core layers (Figure 2):

• Data Collection and Preprocessing Layer: Gathers real-time data from IoT sources, preprocessing via feature extraction, normalization, and blockchain-based labeling.
• Anomaly Detection Layer: Uses ML (SVM, RF) and DL (CNNs, Autoencoders) for threat classification, adaptive learning for evolving attacks, and blockchain for integrity and verification.
• Security and Access Control Layer: Applies ZTA to verify entities, uses blockchain for device authentication, and RBAC/ABAC for fine-grained access control.
• Threat Mitigation and Response Layer: Automates threat response, isolates devices, logs incidents on blockchain, and reallocates resources adaptively.

The proposed framework ensures secure, scalable IoT operations in smart cities, integrating blockchain and AI-driven anomaly detection. Zero Trust Architecture (ZTA) strengthens security but may strain lightweight devices. Automated threat response and dynamic resource allocation enhance resilience and performance.

5. Methodology

This study develops a real-time anomaly detection model to enhance IoT network security, integrating blockchain for data integrity. It aims for a scalable framework to detect both known and unknown attacks in dynamic IoT environments.

5.1. Dataset and Data Collection

The IoT-23 dataset captures real IoT traffic, including 20 malware and 3 benign scenarios (DDoS, botnets, scans) using Wireshark. It provides labeled data with features like duration, packet size, protocols, and connection states, making it suitable for supervised learning in IoT intrusion detection.

5.2. Data Preprocessing and Feature Engineering

We preprocessed the IoT-23 dataset by cleaning data, removing missing values, and normalizing key numerical features (e.g., duration, orig_bytes, resp_bytes, conn_state) using min–max scaling. The dataset was split into 80% training and 20‘% testing, with 20% of the training data reserved for validation. These steps optimized the dataset for anomaly detection and traffic classification.

5.3. Anomaly Detection Models (Model Selection)

In this work, we evaluated Random Forest, SVM, Logistic Regression, XGBoost, and CNN Autoencoder on the IoT-23 dataset for traffic classification. Models were selected for their classification strength in anomaly detection. Using an 80/20 train–test split and 5-fold cross-validation, we assessed performance via accuracy, precision, recall, and F1-score, focusing on both frequent and rare attacks.

–

Random Forest (RF): The Random Forest classifier achieved 81.72% accuracy (standard deviation: 0.0047), effectively distinguishing benign traffic from malicious activities. Its ensemble approach, averaging predictions from multiple decision trees, ensured stable, high performance.

–

Support Vector Machine (SVM): The Support Vector Machine (SVM) model achieved 54.06% accuracy with a low standard deviation of 0.0001, showing poor performance in distinguishing legitimate from malicious traffic. This was likely due to dataset dimensionality, class imbalance, and challenges in selecting an optimal kernel function.

–

Logistic Regression(LR): The Logistic Regression model achieved 46.59% accuracy with a standard deviation of 0.0716. It struggled with traffic classification and showed convergence warnings, indicating incomplete optimization, possibly due to the dataset’s complexity.

–

Extreme Gradient Boosting (XGBoost): The XGBoost classifier achieved 89.89% accuracy and strong performance metrics, including 90.78% precision and 89.89% recall, with hyperparameters optimized using Optuna. Its scalability, regularization, and ability to handle high-dimensional, imbalanced data made it well-suited for real-time IoT anomaly detection.

–

CNN Autoencoder: The CNN Autoencoder, trained on benign traffic to detect anomalies via reconstruction error, achieved 95.16% accuracy with a standard deviation of 0.0061. It outperformed traditional models, excelling in detecting rare attacks by capturing complex spatial–temporal patterns, though at the cost of higher computational demands.

Figure 3 compares RF, LR, SVM, XGBoost, and CNN Autoencoder in attack detection, with CNN Autoencoder excelling at subtle anomalies via deep feature extraction. Classification reports for XGBoost and CNN Autoencoder reveal their strengths across traffic types.

XGBoost:

–

Attack: precision—1.00; recall—1.00; F1-score—1.00.

–

Benign: precision—0.89; recall—0.88; F1-score—0.88.

–

C&C: precision—0.70; recall—0.72; F1-score—0.71.

–

DDoS: precision—0.86; recall—0.89; F1-score—0.88.

–

Okiru: precision—0.95; recall—0.87; F1-score—0.91.

–

PartOfAHorizontalPortScan: precision—0.97; recall—0.98; F1-score—0.98.

CNN Autoencoder:

–

Attack: precision—1.00; recall—1.00; F1-score—1.00.

–

Benign: precision—0.91, recall—0.89; F1-score—0.90.

–

C&C: precision—0.75; recall—0.78; F1-score—0.76.

–

DDoS: precision—0.82; recall—0.88; F1-score—0.85.

–

Okiru: precision—0.97; recall—0.90; F1-score—0.93.

–

PartOfAHorizontalPortScan: precision—0.99; recall—0.99; F1-score—0.99.

XGBoost balanced performance, while the CNN Autoencoder excelled in detecting subtle anomalies, albeit with higher computational cost.

5.4. Blockchain Integration, Hyperparameter Optimization, and Threat Mitigation

Blockchain strengthens anomaly detection by verifying anomalies via consensus, reducing false positives. The CNN Autoencoder achieved 95.16% accuracy, followed by XGBoost (89.89%) and Random Forest (81.72%). Blockchain ensures tamper-proof logs and smart contracts automate responses, improving security and reducing misclassifications. Hyperparameter tuning is crucial; Simulated Annealing enhances model accuracy, generalization, and lowers false positives. Verified anomalies prompt AI-driven actions, isolating devices, updating policies, and adjusting firewall rules. Blockchain logging and AI-based access controls ensure traceability and rapid response.

6. Results and Discussion

This section evaluates the framework’s anomaly detection and threat mitigation performance in IoT-based traffic systems, focusing on accuracy, false positives, efficiency, and response time.

6.1. Anomaly Detection Performance

The framework was tested using the IoT-23 dataset, consisting of real-world IoT network traffic, including both benign and malicious activities. The machine learning models were evaluated based on classification accuracy, precision, recall, and F1-score, with the results summarized in

Table 3. Performance of machine learning models for anomaly detection.

Model	Accuracy (%)	Precision	Recall	F1-Score	FPR
CNN-Autoencoder	95.16	0.98	0.95	0.98	0.07
XGBoost	89.89	0.90	0.89	0.88	0.11
RF	81.72	0.85	0.78	0.81	0.12
SVM	54.06	0.55	0.49	0.51	0.34
LR	46.59	0.47	0.42	0.44	0.41

.

The CNN-Autoencoder achieved 95.16% accuracy with a low 0.07 FPR, leading in IoT anomaly detection. XGBoost (89.89%) and Random Forest (81.72%) performed well but had higher FPRs (0.11, 0.12). SVM and Logistic Regression showed weak recall and high FPR. Simulated Annealing tuning boosted CNN accuracy by 4.8% and Random Forest recall by 3.5%, improving performance and training efficiency.

6.2. Blockchain Integration and Security Enhancements

Blockchain improved data integrity and anomaly verification, reducing false positives by 11% via distributed consensus. Security events were immutably logged on a permissioned blockchain, aiding forensics. Smart contracts enabled rapid responses, including the following:

–

Revoking device access in 1.2 s;

–

Dynamically updating firewall rules;

–

Adjusting access policies based on verified threats.

Though effective, blockchain added overhead. Future work will focus on lightweight solutions to reduce latency at scale.

6.3. Threat Mitigation and System Efficiency

The automated threat mitigation mechanisms played a crucial role in minimizing system downtime and responding to security incidents in real time. Once an anomaly was detected and verified, the system enacted the following process:

–

Isolated compromised devices within 1.2 s, preventing malware spread;

–

Updated security rules and firewall policies dynamically, reducing attack propagation;

–

Logged security incidents in blockchain, ensuring non-repudiation and forensic traceability.

The AI blockchain system cut response time by 40% over manual methods. Minor delays from blockchain were offset by improved real-time protection.

7. Conclusions and Future Work

This paper presents a real-time IoT anomaly detection framework combining LR, SVM, RF, XGBoost, and CNN Autoencoders. CNN models normal traffic, while blockchain ensures data integrity with low overhead. Evaluation on IoT-23 showed high accuracy, fewer false positives, and real-time adaptability. Dataset imbalance may favor attack detection; future work includes rebalancing, federated learning, lightweight cryptography, and large-scale smart city deployment. Technologies like 5G, Edge AI, and Quantum Cryptography offer further enhancement opportunities.