Machine Learning-Based Real-Time Detection and Mitigation of DoS Attacks in SDN-Based 5G Network

Abstract

Multi-Access Edge Computing (MEC) is a fundamental component for 5G networks to overcome the latency limitations of traditional cloud computing. However, bringing resources closer to users exposes edge nodes to significant security threats, particularly volumetric Denial of Service (DoS) attacks. Current defenses often depend on static thresholds or computationally expensive deep learning, which can exhaust the limited resources of MEC nodes. To address these limitations, this paper proposes a resource-optimized edge-centric security management logic that integrates Software Defined Network (SDN) with lightweight supervised learning (C5.0, Bagging-CART, and Random Forest). Unlike standard system integrations, we introduce a dynamic non-permanent blocking algorithm designed to balance detection accuracy with control plane stability. Experimental results demonstrate that the proposed C5.0 model, operating at a specific 0.20% sFlow sampling point, achieves 100% detection accuracy with under 100 ms mitigation latency. The system successfully reduces volumetric attack loads from 445 Mbps to 95 Mbps (a 78% reduction) at the node level. These findings confirm that the proposed framework achieves higher computational efficiency than complex alternatives, making it a highly stable solution for constrained 5G MEC environments.

1. Introduction

Over the last two decades, global internet traffic has increased exponentially, including mobile network traffic. Many studies, such as [1,2,3,4,5] focus on accelerating internet data access, which has led to rapid technological advancements in this field. In 2017, global internet traffic hit 45,000 GB/s (gigabytes per second), showing how quickly and widely internet access has grown. In 2002, global internet traffic was only around 100 GB/s [6]. The rapid growth of internet access has also increased the demand for video services, music, social networking, games, and other interactive applications. Ultimately, current radio access networks (RANs) may reach their limit. Furthermore, as network complexity grows, ensuring the integrity of these infrastructures requires advanced health assessment frameworks to defend against sophisticated cyber threats [7].

To overcome the limitations of current RANs, the European Telecommunications Standards Institute (ETSI) proposed two paradigms in the 5G networks, which are Cloud-RAN (C-RAN) and Multi-Access Edge Computing (MEC) [8]. The C-RAN focuses on the centralization of the base station function via virtualization. MEC enhances the network edge. 5G is expected to achieve very low latency and massive bandwidth to meet a wide range of demands. Additionally, cloud computing, as one of the key technologies of 5G, is usually deployed far from users physically [8]. A huge amount of traffic and numerous intermediate nodes create a heavy load, congestion, delay, and high energy consumption on the network. MEC [9] is a technology that brings memory and computing power closer to their required locations. MEC has emerged as a computational paradigm that integrates many processing capabilities into edge networks and can directly process customer requests [10]. This paradigm is particularly important for high-bandwidth applications like video streaming, where adaptive and efficient memory networks are necessary for robust performance [11]. The study demonstrates that MEC helps fulfill user requests quickly, reduces delays in cloud computing, and supports the goals of 5G networks.

However, protecting this low-latency infrastructure remains a challenge as current defense paradigms face a fundamental challenge in MEC environments. Recent studies have focused on Deep Reinforcement Learning (DRL) [12,13] and Federated Learning (FL) [14,15]. While these approaches show high detection accuracy, they introduce important computational overhead and communication latency. For instance, FL-based approaches require continuous model synchronization that creates communication overhead [15], rendering them unsuitable for constrained edge nodes which typically depend on limited processing power. On the other hand, static SDN thresholds fail to handle dynamic volumetric attacks without saturating the control plane, a vulnerability that leads to resource exhaustion [13,16]. Therefore, there is an urgent need for resource-optimized architectural logic that delivers the accuracy of AI but maintains the lightweight stability required to prevent flow table saturation.

In this paper, we propose an edge-focused intelligent system for DoS detection and mitigation in MEC environments. This study proposes an integrated framework that combines a lightweight sFlow-based sampling mechanism with ensemble learning to address the computational constraint of edge nodes without compromising detection responsiveness. In contrast to standard SDN security implementations that may face control plane saturation during high volume attacks, this approach focuses on optimizing the balance between detection granularity and resource utilization at the network edge. Our contributions in this paper are summarized as follows:

(1)

We introduced a closed-loop mitigation methodology that manages sFlow telemetry and OpenFlow rules through a dynamic non-permanent blocking logic. This design specifically resolves the balance between mitigation latency and control plane resource consumption, offering a scientifically validated alternative to computationally expensive deep learning approaches for resource-constrained edge environments.

(2)

We experimentally determined the specific sFlow-OpenFlow operational point (0.20% sampling with C5.0). As validated by our feature importance analysis, the attack signatures are physically deterministic (tcp.seq, ip.proto), proving that lightweight methodologies are more efficient than computationally expensive deep learning for this specific threat landscape.

(3)

We provide node-level quantitative validation, demonstrating a 78% reduction in processing load (from 445 Mbps to 95 Mbps), which proves the architectural stability required to prevent control plane saturation in distributed MEC deployments.

The rest of the paper is organized as follows: Section 2 discusses related work; Section 3 presents the proposed methodology; Section 4 presents our simulation; Section 5 presents our results; Section 6 provides a discussion; and Section 7 concludes this paper.

2. Related Works

In this section, this paper reviews the related works in the domain of MEC implementation for video streaming services, 5G network classification, and smart IDS for DoS mitigation systems.

2.1. Multi-Access Edge Computing for Video Streaming Service

MEC is a new concept of network architecture proposed by the European Telecommunications Standards Institute (ETSI). The study [17] explains Network Function Virtualization (NFV)-based MEC, which can be used to deliver ultra-high-quality video efficiently. Video streaming service is related to the term Quality of Experience (QoE), which is a part of the Quality of Service (QoS) that indicates the degree of delight or annoyance experienced by users for an application service. The papers [18,19] introduced methods for maintaining QoE of video streaming services on the 5G network. Therefore, based on the paper [17,18,19], this experiment’s network architecture was designed to establish a video streaming service.

The work in [20] applied a machine learning approach to the MEC application for popular video prediction and radio channel quality prediction. Our experiment was proposed with a different objective than [20]. We focused on leveraging the machine learning approach to classify abnormal traffic, which includes DoS attack traffic and normal traffic. As a result, the video streaming server continues to provide service during a DoS attack occurring in the network.

2.2. 5G Network Classification

Traffic classification is essential in the 5G network to reach an efficient and stable network [21]. Accurate classification of traffic flows benefits quality of service (QoS), dynamic access control, and lawful interception [22]. The paper [23] tries to classify networks to achieve network slicing for 5G, which can improve the quality of video streaming services. Paper [23] used the SDN/NFV, ML, and Big Data approach on the 5G network. We conducted our experiment with a similar approach. However, our experiment employs OpenFlow and sFlow as SDN technology to enable the DoS mitigation system in MEC 5G.

Traffic classification does not depend on the ML approach. As discussed in Ref. [24], they only used the SDN/NFV approach to classify the network quickly. The research in [24] attempted to implement a network classification algorithm directly into the SDN controller. Their solution is indeed fast, but in some instances, it poses security issues and exhibits a low level of accuracy. Therefore, our experiment proposes C50, Bagging-CART (B-CART), and Random Forest (RF) to identify DoS packet traffic in the network.

A DoS attack is a type of network attack that attempts to flood a communication channel with a specific packet of data. This attack aims to consume all available resources in the network. In other words, the attacker intends to overwhelm the system, preventing users from accessing it. Several studies have been proposed to mitigate this type of attack [15,25,26,27]. The study [25] proposed a DoS mitigation scheme that combines detection engines and a secure C-to-C communication protocol design for SDN controllers located in different autonomous systems (AS). The paper [25] attempted to establish effective communication between SDN controllers to mitigate DoS attacks automatically, quickly, and precisely. However, the solution proposed by [25] is not suitable for small networks such as home or small office networks. Moreover, their solution also requires many controllers, making it an expensive approach.

There have been various intrusion detection systems using the ML approach proposed to handle abnormalities in the network. The study in [28] proposed a solution that leverages ML to detect DoS attacks in IoT networks. In contrast, [26] addressed the DoS problem using a deep learning approach instead of ML for their detection system. The work [27] proposed a smart intrusion detection system (IDS) using ML for the 5G SDN network. The main difference between these studies and our work lies in the focus of the research. Our proposed system utilizes SDN (OpenFlow and sFlow) while also integrating with MEC in 5G to enable a mitigation system and employing supervised decision tree learning to perform the detection.

2.3. Research GAP and Motivation

Existing research on DoS attack mitigation in MEC and SDN environments, this study proposes a framework that integrates sFlow-based monitoring with dynamic OpenFlow-based mitigation.

Table 1. Comparison of Methodology.

	Aspect	Methodology	Limitations
Research
[25]		Uses Controller-to-Controller (C-to-C) protocol to share attack definitions. Three propagation models: linear, centralized, and mesh. Testbed with Mininet for evaluation.	Limited scalability and assumes full SDN deployment.
[28]		Pipeline includes traffic capture, feature engineering, and ML classification. Combines stateless and stateful features. Tested on a simulated IoT network.	Small dataset and lacks real-world validation.
[26]		Deep learning model trained on flow features from OpenFlow statistics. Combines traffic sampling, feature selection, and attack detection. Tested on an OpenFlow SDN testbed.	Focuses on flow statistics; tested only in SDN setups.
[27]		Three-layer architecture: forwarding, management, and intelligence. Uses Random Forest for feature selection. Hybrid k-means++ and AdaBoost for classification.	Rely on outdated datasets and have scalability issues.
[16]		Uses the Mininet simulator and the OpenDaylight controller. Focuses on feature selection using CICFlowMeter. Applies XGBoost and Random Forest algorithms for classification.	Focuses on detection accuracy metrics and feature ranking. Does not provide an automatic mitigation mechanism
[29]		Employs coarse-grained statistical analysis for early-stage detection. Uses a Multi-Dimensional Deep Convolutional Classifier (MDCC) based on CNN for fine-grained detection Implements graph-theory-based attack isolation.	Rely on deep learning (CNN) models with high computational complexity.
Our method		Integrates SDN and MEC for DoS detection using OpenFlow and sFlow protocols. Utilizes supervised ML models (C5.0, B-CART, RF) for attack classification. Tested on simulated traffic (ICMP echo, TCP Xmas, UDP flood). Dynamic non-permanent blocking for mitigation.	Scalability may be a concern due to resource consumption.

provides a comparative summary of prior studies, highlighting their contributions and inherent limitations, such as flow-table saturation and static mitigation strategies. By utilizing supervised machine learning models, specifically C5.0, B-CART, and Random Forest, this approach addresses these gaps through optimized computational resource management and precise traffic classification in 5G-MEC architectures.

3. Methodology

In this section, we describe our proposed architecture and intelligent intrusion detection process to handle DoS attacks.

3.1. System Overview

Our experiment uses architecture as shown in Figure 1. We built our network based on the work in [30]. In our research, we used the Low Latency Multi-access Edge Computing (LL-MEC) platform. The LL-MEC platform can be operated over SDN in a 5G network at the edge. Furthermore, our experiment implements a video streaming service over HTTP. The workflow for typical cases in this experiment is shown in Figure 2.

According to Figure 2, the user first sends a request to access a video. The request is then received by the SDN switch, and this information is forwarded to the controller. The LL-MEC protocol instructs the SDN controller to redirect the request to the edge server instead of the central server on the internet. After that, the edge server fulfills the request if it has the requested video. However, when the edge server cannot fulfill the request, the LL-MEC protocol automatically instructs the forwarding of the request to the central server over the internet. The workflow described above reduces the load on the central server on the internet. As a result, the request from the user is delivered quickly.

The integration of MEC with SDN enhances the programmability of the edge network within the 5G MEC network. This capability is enabled by SDN, which separates the control plane from the data plane within a network. The OpenFlow protocol allows the edge server to function as a network controller, configuring the hardware forwarding table. Moreover, OpenFlow is widely adopted in SDN architecture, making its implementation both efficient and straightforward. In this experiment, we proposed an edge-focused intelligent solution that integrates OpenFlow with sFlow in an edge network. Both protocols facilitate network filtering based on traffic classification, enhancing network management and security.

Figure 3 illustrates the overall intrusion detection system proposed in this paper, which consists of three main components: sFlow for packet sampling and traffic monitoring, an SDN controller, and an intelligent system. Based on the analysis results of the intelligent system, the SDN controller enforces mitigation decisions by dynamically updating flow rules on the switch. These rules are used to block malicious traffic flows in a time-bounded manner while preserving normal traffic forwarding.

sFlow is a protocol that can be implemented in SDN networks, serving as a standard for instrumentation within forwarding table hardware to provide real-time, network-wide visibility into traffic flow. This protocol can also be utilized to monitor the performance of network monitoring systems. Combining OpenFlow and sFlow in SDN networks allows for better detection and response to threats, improving how intrusion detection systems work by using a feedback control loop. This intrusion detection system automatically configures the network to support the DoS detection and mitigation system within the 5G network [31]. The sFlow standard is well suited for managing physical or virtual switches on a large scale. sFlow is embedded in physical switches but is not restricted to specific switch vendors. Moreover, sFlow can process high volumes of traffic without degrading performance. It is also a straightforward protocol to configure, facilitating the monitoring of multiple switches and ports. Furthermore, the sFlow protocol can deliver network-wide monitoring and export packet information, which can be utilized for deep packet inspection.

OpenFlow and sFlow were implemented in this experiment to build a system that can automatically respond to abnormal traffic. In this case, the term for abnormal traffic is the DoS attack. OpenFlow itself is not an anomaly detection system, and it cannot control the packet sampling in the traffic. In our system design, we integrate the sFlow function for packet sampling and OpenFlow rules to manage the network flow to build an intrusion detection system that selectively steers the traffic based on IP source, IP destination, IP protocol, and port source/destination. Moreover, our intrusion detection system was able to block the attack source and minimize the damage to the system.

In this experiment, sFlow was utilized to reduce the number of packets requiring analysis. Its implementation enabled precise packet sampling within the network, which is particularly crucial for mitigating DoS attacks, as such attacks attempt to overwhelm the system by sending thousands of packets per second. Without packet sampling, the control network’s port-in mechanism would be inundated with malicious traffic. Additionally, the intrusion detection system demands significant computational resources, including memory and processing power, to manage a large influx of incoming packets. To address this challenge, our experiment employs an sFlow script designed to optimize resource utilization. The script provides the SDN controller with information about the characteristics of incoming packets. Moreover, it directs the controller to block specific IP addresses exhibiting anomalous traffic. Algorithm 1 presents further details regarding the sFlow script. This algorithm begins by extracting the five-tuple metadata (IP_src, IP_dst, Port_src, Port_dst, Proto) from the sampled datagram to uniquely identify network flows, while the physical features (IPI and L) are utilized for anomaly classification. The symbols and notation used in the proposed mitigation algorithm are summarized in

Table 2. Mitigation Algorithm Notation.

Symbol	Description
D	Sampled traffic datagrams collected via the sFlow agent.
α	sFlow telemetry sampling rate (set to 0.20%).
x	Physical traffic feature vector {IPI, L}.
IPI	Inter-Packet Interval (a key feature for packet flooding detection).
L	Frame length or packet size.
F(.)	Classification model based on Decision Tree
S	System security status {Normal, Under_Attack, Recovered}.
Tlimit	Duration of non-permanent blocking (blocking timer).
IPsrc	Source IP address identified as the origin of the attack.
IPdst	Destination IP address.
Portsrc	Source port number.
Portdst	Destination port number.
Proto	Network protocol type (TCP, UDP, or ICMP).
Rflow	Flow rules are sent to the switch via OpenFlow.

.

Algorithm 1: DoS Mitigation
Input: Sampled datagram D (via sFlow with α = 0.20%)
Output: Adaptive flow rules Rflow and updated system status S
1	Phase I: Data Reduction
	● Extract traffic features from datagram D: IPsrc, IPdst, Portsrc, Portdst, Proto
	● Efficiency Analysis: The sFlow-based telemetry methodology successfully reduces the inspection workload by 78% (from 445 Mbps to 95 Mbps), thereby maintaining the operational stability of the edge node.
2	Phase II: Low-Latency Anomaly Inference
	● Extract physical traffic feature vector: X ← {IPI, L}.
	● Perform real-time inference using a decision tree model:      Result ← f(X)
	● If Result == DoS_Attack then      Update system status: S ← Under_Attack.
	● Else      Update system status: S ← Normal.
	● End If
3	Phase III: Dynamic Mitigation and Adaptive Blocking
	● If S == Under_Attack then a.  Allocate dynamic blocking duration: Tlimit ← TimerHandling() b.  Construct blocking (drop) flow rules for IPsrc c.  Send mitigation instructions to the controller: Post_Controller_Filtering(IPsrc, Tlimit) d.  Execution: Block(IPsrc) in Tlimit.
	● End if
4	Phase IV: Flow Table Resource Reclamation
	● If Tlimit has expired, then a.  Execute recovery command: Release(IPsrc) b.  Remove expired flow table entries to prevent flow table overflow. c.  Restore system status: S ← Recovered.
	● End if

Algorithm 1 explains how the proposed closed-loop defense system works, structured into four independent phases. The process initiates with phase I, where the sFlow agent extracts network traffic data. By implementing a 0.20% sampling rate, this phase experimentally reduces the data inspection overhead by 78% (from 445 Mbps to 95 Mbps), ensuring that the edge node resources remain stable even during high-traffic volumetric attacks. The extracted data includes the five-tuple metadata (IP_src, IP_dst, Port_src, Port_dst, Proto) for flow identification and physical features (IPI and L) for anomaly detection. In phase II, these features are processed by decision tree learning models. This model was selected for its best real-time performance, capable of identifying attack patterns with minimal inference latency. Upon detecting a DoS attack, the system triggers phase III. Unlike static firewalls, the SDN controller dynamically updates the OpenFlow routing table to block malicious packets for a calculated duration (T_limit). Finally, Phase IV ensures system recovery through a self-recovery mechanism. This non-permanent blocking strategy is critical to address IP spoofing attacks; without it, attackers could exhaust the switch’s flow table by flooding it with fake addresses. Furthermore, the automatic release function guarantees that legitimate users, who might be falsely identified or whose IP addresses were spoofed, are not permanently denied service, thereby balancing robust security with high service availability.

3.2. Intelligent Detection Process

Network architecture and machine learning must run synchronously to achieve intelligent detection systems. This experiment considered processing time and the accuracy of the machine learning algorithm to build a fast and reliable detection system. The experiment evaluates the performance of three proposed supervised decision tree learning methods: C50 [32,33], B-CART [34], and Random Forest (RF) [35]. The evaluation includes a comparison between our proposed supervised decision tree learning and other existing machine learning methods. The following are the detailed descriptions of the supervised decision tree learning proposed to be implemented in the intelligent system:

3.2.1. C5.0

The C5.0 algorithm [32,33] is one of the new generations of machine learning algorithms based on decision trees. In this algorithm, a decision tree is built based on a list of all possible attributes and sets of training cases. The tree is then used to classify subsequent sets of test cases. C5.0 is the improvement of the C4.5 classifier. C5.0 can generate more accurate rules with fewer processing times compared to the C4.5 classifier. First, C5.0 defines the entropy function $H$($S$) in Equation (1). The entropy of the training data set S is represented as H(S), which also represents the probability that one random instance from S is in a class. (There are four classes in our model: normal, ICMP_attack, TCP_attack, and UDP_attack).

$$H\left(S\right)≔ \sum _{i=1}^n-p\left(s_i\right){log}_2\left(p\left(s_i\right)\right)$$

(1)

where $\{{\mathrm{s}}_{\mathrm{i}}{\}}_{i=1}^n$ is the set of possible outcomes of the random variable S, the training data set.

The function in Equation (2) defines the information gain which measures the quantity of information that is gained by splitting a set of realizations into two separate sets according to an attribute.

$$IG\left(S, A_i\right)=H\left(S\right)-{\sum }_{\alpha \in A_i}{\displaystyle \frac{\left|S_{\alpha }\right|}{\left|S\right|}}H(S_{\alpha })$$

(2)

where $A_i$ denotes the set of possible outcomes of the attribute and $S_{\alpha }≔\{$s$\in S\left|{\alpha }_i\left(s\right)=\alpha \right\}$. $IG$ represents the information gained based on the attribute ${\alpha }_i\left(s\right)$.

3.2.2. Bagging-CART

B-CART (Bagging-CART) is a decision-tree machine learning algorithm that consists of the CART and Bagging algorithms. CART is a basic classification method, which was then known as a classification and regression tree, originally proposed by Breiman in 1984 [34]. The disadvantage of CART lies in its unstable algorithm. When the training set has minor changes, it will possibly create a big difference in the generated model. Therefore, the Bagging algorithm was also introduced by Breiman to improve the CART algorithm [34].

The basic idea of Bagging can be described as follows:

• For a given weak classifier and training set, we use the weak classifier to train K times.
• Each training set consists of N samples, which are randomly picked from the initial set.
• After completing each training, we get the predictive function, which is a prediction function sequence of K (p1, p2, …, pK).
• Finally, using that predictive function sequence and the principle of majority voting, we get the final prediction p*.

The final prediction by Bagging will greatly improve the prediction accuracy for the unstable learning algorithm. In addition, each training set is mutually independent because the training set is selected randomly. However, this algorithm only works for unstable machine learning algorithms. Bagging-CART simplifies the CART algorithm and enhances the accuracy and reliability of the classification [34].

3.2.3. Random Forest

Random Forest (RF) [35] is an ensemble algorithm based on a decision tree. The forest is formed randomly from a group of unrelated decision trees. When the random forest receives new data, each decision tree decides which label the data should belong to separately. Then, the random forest classifies the data according to the vote of the total number of labels. This algorithm is very suitable for handling enormous amounts of the dataset. It does not require a feature selection in its implementation and can accomplish the training process quickly. After passing the training process, this algorithm can show which features are essential [35]. The detailed procedure of the Random Forest model [36] is presented in Algorithm 2:

Algorithm 2: Random Forest

Let D = {(x1, y1), …, (xn, yn)} denote the training data with xi = (xi,1, …, xi,p)T, where xi indicates the p predictors and yi represents the response.

For j = 1 to J:1.   1. Take a bootstrap sample Dj of size N from D.   2. Use the bootstrap sample in 1. as training data and use binary recursive partitioning to fit a tree: a.  In a single node, start with all observations. b.  For each unsplit node, repeat the following steps recursively until the termination criterion is met.  From the p available predictors, select m predictors randomly.  Search the best binary split among all binary splits on m predictors from step i.  Use the split from step ii to split the node into two descendant nodes.    At a new point x, predict by ✧ $\hspace{1em}\widehat{f}\left(x\right)= {\displaystyle \frac{1}{J}}{\sum }_{j=1}^J\widehat{h}\left(x\right)$ for regression ✧ $\hspace{1em}\widehat{f}\left(x\right)={argmax}_y{\sum }_{j=1}^{J}I({\widehat{h}}_j\left(x\right)=y)$ for classification,   $\mathrm{where} {\widehat{h}}_j(x)$ is the prediction of the response variable x given by ✧ $\hspace{1em} \widehat{h}\left(x\right)= {\overline{y}}_k ={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{y}_{k_i}$ for regression $\widehat{h}\left(x\right)={argmax}_y{\sum }_{i=1}^{n}I(y_{k_i}=y)$$\mathrm{for} \mathrm{classification}, \mathrm{where} I(y_{k_i}=y)=1$$\mathrm{if} y_{k_i}=y$ and 0 otherwise.

3.3. Design Decision and Parameter Justification

The proposed framework includes several design decisions and parameter selections for the specific constraint of 5G MEC environments. These design choices are explained below:

• The sampling rate was set to 0.20% (1:500 ratio). The goal was to find a balance between obtaining enough data detail for detection and maintaining low computational load (overhead). This selection supports the 10 Mb/s link speed used in the experiment. We analyzed the effect of different sampling rates as follows:
  • Rate 0.10% (1:1000): This rate lowers the processing load but increases the risk of missing short attack patterns or bursts, as noted in the SDN security overview by Wang and Li [13].
  • Rate 0.20% (1:500): This is the selected rate. It provides stable network visibility while preventing CPU saturation on the controller.
  • Rate 0.50% (1:200): This rate provides more data detail but increases the risk of overloading the control plane during high-volume floods.

This approach aligns with the research by Ujjan et al. [37], which indicates that sFlow sampling is an effective method for detecting DDoS attacks using deep learning without burdening resource-constrained nodes.

2.

The focus on ICMP, TCP Xmas, and UDP flood attacks is justified by their prevalence as primary volumetric threats in MEC environments. The repetitive patterns of these attacks distinguish them from legitimate video streaming traffic. By targeting these specific protocols, the framework ensures high detection accuracy while maintaining low-latency responsiveness.

3.

The integration of sFlow monitoring and OpenFlow execution is operationalized through a customized logic as detailed in Algorithm 1. Unlike static security configurations, this algorithm introduces a dynamic, timer-based mitigation sequence (TimerHandling) specifically designed to resolve the balance between rapid threat suppression and service availability for legitimate users during IP spoofing events. This operational logic ensures that the system remains resource-efficient while providing reliable protection at the MEC edge. The interaction between these components, as supported by the multi-layer principles in FMDADM [38] and SDN-Defend [39], ensures a responsive defense system suitable for localized 5G infrastructures.

4.

The system is designed for Micro-MEC or SOHO-level edge environments. This scope justifies the use of a localized testbed, as the proposed mitigation logic is intended to operate at the network’s periphery. As noted in recent SDN security reviews [39], decentralizing security at the edge is a vital strategy for protecting 5G infrastructures from large-scale service disruptions.

4. Simulation

This section explains how the experiment was conducted. Figure 1 illustrates this network architecture design, as mentioned in the previous section. A Raspberry Pi 3 was used as the SDN switch, while ONOS served as the SDN controller, managing data traffic on the SDN switch. The switch configuration is shown in Figure 4 below.

In this experiment, we set up the SDN switch by installing Open vSwitch (OVS) on the Raspberry Pi. The Raspberry Pi used in our setup features a 1.4 GHz CPU and 1 GB of LPDDR2 SDRAM, providing the necessary computational resources for efficient network management and switching operations. sFlow-RT was used for packet sampling in the system. As shown in Figure 4, the switch was configured with id @sflow. The switch, assigned the address 10.0.0.11, acts as an sFlow agent sending packet sampling information to the SDN controller address at 10.0.0.2 on port 6343. This configuration allowed the SDN switch to send packet information to the controller. In this experiment, we set up an environment simulating a home network. A normal user accessing the video streaming server generates a load of approximately 1000 packets per second. To manage the link speed of 10 Mb/s, we used a 0.20% sampling rate based on [40]. Additionally, we applied 20 polling rates of 20 s each, as described in the sFlow paper [41]. This configuration enabled the intrusion detection system to efficiently handle the DoS attacks conducted in this experiment. The attacks generated between 6000 and 9000 packets, each with a size of 64 Kb per packet, resulting in a total traffic volume of 445 Mb/s. The proposed sFlow implementation in this system succeeded in reducing the workload on the server by analyzing only around 95 Mb/s instead of 445 Mb/s. Consequently, the proposed sFlow implementation enabled the detection and mitigation system to operate faster and reduce the impact of the attacks.

The data containing traffic information from the sFlow agent switch is forwarded to a supervised decision tree learning model for analysis. In the mitigation process, when the packet information received is evaluated as an attack, sFlow-RT will send the information, including the source IP, destination IP, source port, and destination port, to the Open Network Operating System (ONOS) controller. The controller then updates its routing table, blocking the identified malicious connection by dropping packets from the specified source IP and port. This blocking mechanism is maintained for a defined period, which can be adjusted based on the nature of the attack to balance security and network availability.

There are several types of DoS attacks, including SYN flood, UDP floods, HTTP floods, ping of death, smurf attack, Fraggle attacks, Slowloris, NTP amplification, advanced persistent DoS, and zero-day DDoS attack [42]. In this paper, we focused on three specific types of DoS attacks: ICMP Echo flood, TCP Xmas Flood, and UDP Flood attacks. The tool “hping3” was used to launch these attacks in our experiment.

Table 3. Distribution of data used in the experiment.

Classification	Raw Dataset	Selected Dataset
Normal	139,286	4000
icmp_echo_attack	411,447	4000
tcp_xmas_attack	605,626	4000
udp_attack	392,285	4000
total	1,548,644	16,000

shows the distribution of the data collected in this experiment. A dataset containing 1,548,644 network traffic records was collected from the network built for this experiment. From this dataset, 16,000 traffic samples were selected, consisting of four groups of 4000 samples each. Furthermore, the selected dataset is categorized into four classes: Normal, ICMP_attack, TCP_attack, and UDP_attack, which are used to evaluate the performance of the proposed intelligent system. To ensure the statistical reliability of the evaluation, the cross-validation was performed using samples drawn from the aggregated experimental dataset rather than distinct independent sessions. The entire collection of 16,000 selected traffic samples was pooled and randomly partitioned into ten folds. This stratified random sampling approach ensures that the training and testing sets in each fold maintain a representative distribution of both normal and attack traffic classes, thereby minimizing selection bias during model validation.

Figure 5 shows the distribution of protocols within each type of traffic. In our experiment, the normal traffic (Figure 5a) is composed of approximately 15% UDP, 8% ARP, 4% ICMP, and 73% TCP. However, once an attack occurs, the traffic distribution shifts predominantly to the protocol associated with the specific attack type (Figure 5b–d). Attack traffic is primarily dominated by protocols corresponding to the specific type of DoS. For example, in an ICMP echo flood, the traffic monitoring system indicates a predominant presence of the ICMP protocol. In this paper, we proposed three supervised decision tree-based learning algorithm models (C50, B-CART, and RF) for intrusion detection. These supervised learning models were trained to classify network traffic as normal or abnormal based on various behavioral features, including frame length and inter-packet time intervals. Therefore, the performance evaluation was conducted by comparing the three proposed algorithms with eight other machine learning methods in terms of accuracy and processing time.

Overall machine learning methods used in this experiment are shown in

Table 4. Classification Methods.

Model	R Package	Hyper-Parameters
Deep Neural Network (DNN)	Keras 2.2.0 [43]	model ← keras_model_sequential() model %>% layer_dense(units = 10, activation = “relu”, input_shape = ncol(X)) %>% layer_dense(units = 10, activation = “relu”) %>% layer_dense(units = classSize, activation = “softmax”) model %>% compile(loss = “categorical_crossentropy”, opt mizer = optimizer_adagrad(), metrics = c(‘accuracy’)) fit(X,Yc, validation_split = 0.20, epochs = 50, batch_size = 15, shuffle = T)
Single-hidden-layer Neural Network	nnet [44]	size = 5 decay = 0.0001 maxit = 100
Multinomial Logistic Regression (MLR)	nnet [44]	Default
Generalized Boosted Regression Modeling (GBM)	gbm [45]	Default
Linear Discriminant Analysis (LDA)	MASS [44]	Default
Support Vector Machine (C-SVM)	e1071 [46]	type = “C-classification”
Naïve Bayes (NB)	e1071 [46]	Default
Kernel Support Vector Machine (KSVM)	kernlab [47]	type = “C-bsvc” prob.model = TRUE
C 5.0	C50 [3]	Default
Bagging CART (B-CART)	ipred [48]	Default
Random Forest (RF)	randomForest [49]	Default

. This experiment executed training and testing on a computer with an Intel Core i7-7700HQ CPU @ 280 GHz and 16 GB RAM. The detection model was built using R (programming language), which ran in the Windows 10 Pro operating system, 64-bit operating system, x64-based processor. This paper implements a 10-fold cross-validation for model validation.

5. Result

5.1. Classification Performance

In this part, the results of several scenarios are explained to evaluate the proposed system and selected algorithm. Experiments were conducted to verify the architecture of the system proposed in this paper. The mitigation system, which used sFlow and OpenFlow in the MEC 5G network, was run in three scenarios to demonstrate its performance. The first scenario, shown in Figure 6, presents the result of the sFlow monitoring simulation on the SDN switch when the mitigation system was not active while the DoS attack occurred in the network. The second scenario, shown in Figure 7, illustrates the mitigation system activated when the DoS attack took place. The final scenario, shown in Figure 8, depicts the mitigation running from the beginning. In Figure 8, the network spike indicates that the DoS attack had been launched and mitigated. We tested this mitigation system in a home network because the DoS attack in this experiment was relatively small. However, our proposed method shows that it is a reliable intrusion detection system that can overcome the attack and restore the traffic while blocking the attack source.

This experiment also evaluates the intelligent system. The experiment utilized packet information collected by sFlow to validate the capabilities of the intelligent system. The data was then processed and extracted into a dataset, which was used for both training and testing to thoroughly evaluate the proposed machine learning method. Our intelligent system demonstrated its ability to accurately recognize various types of DoS attacks, including ICMP Echo Flood, TCP Xmas, and UDP Flood, from the dataset, achieving remarkably high accuracy in distinguishing between the attack types.

In this experiment, we intend to evaluate the proposed system in two aspects: the effectiveness of the selected machine learning methods and the overall performance of the proposed intelligent system.

Table 5. Accuracy comparison of different methods.

Method	Fold 1	Fold 2	Fold 3	Fold 4	Fold 5	Fold 6	Fold 7	Fold 8	Fold 9	Fold 10	Average
DNN	0.995	1	1	0.992	0.992	0.98	0.992	1	0.995	0.992	0.994
nnet	0.998	1	1	1	0.995	1	1	1	1	1	0.999
MLR	0.998	1	1	1	0.995	0.998	0.998	1	1	0.998	0.998
GBM	0.998	0.995	0.998	0.995	0.995	0.988	0.99	0.995	0.99	0.998	0.994
LDA	0.978	0.998	0.998	0.99	0.988	0.975	0.992	0.982	0.988	0.988	0.988
C-SVM	0.998	1	0.998	0.995	0.995	0.98	1	0.995	1	0.995	0.996
NB	0.852	0.865	0.862	0.888	0.868	0.858	0.855	0.838	0.852	0.87	0.861
KSVM	0.86	0.818	0.81	0.81	0.818	0.852	0.87	0.832	0.835	0.822	0.834
KNN	0.998	1	1	1	0.995	0.998	1	1	1	0.995	0.998
C50	1	1	1	1	1	1	1	1	1	1	1
B-CART	1	1	1	1	1	1	1	1	1	1	1
RF	1	1	1	1	1	1	1	1	1	1	1

shows the accuracy of the three machine learning methods proposed in this paper in comparison to the other methods. In a real network, a DoS attack may generate a larger dataset for classification compared to normal traffic. This experiment manually configured the dataset used to train and test the intelligent system to mitigate dataset imbalance. We randomly selected 4000 data points from the network constructed in this experiment, which consists of four classes (normal traffic, ICMP flood attack, TCP Xmas attack, and UDP flood attack). Then, we tested the dataset using 10-fold cross-validation. The results were analyzed to evaluate the accuracy of predictions made by the machine learning methods. The accuracy of network classification is calculated using the formula below for each fold.

$$Accuracy={\displaystyle \frac{Total Corrected Prediction}{Total Cases}}$$

(3)

The correct predictions include the number of attacks accurately detected and the number of normal traffic correctly classified. The total cases include the correct predictions, the number of normal traffic incorrectly classified, and the number of attacks not successfully detected.

Table 5 shows that the three classification methods proposed in this paper outperformed the other methods. Surprisingly, these three methods achieved 100% accuracy. In our experiment, the high accuracy of these methods indicates that they are more suitable for DoS detection in an MEC-enabled 5G network handling video streaming service traffic. This experiment achieved better accuracy compared to the study in [27] because the data used to evaluate our intelligent system focused solely on three types of DoS attacks. Therefore, the data provides a clearer distinction in traffic classification. For example, the UDP flood attack utilizes the UDP protocol, while the TCP Xmas-flood attack operates over the TCP protocol. The importance of machine learning in our proposed system is to make the mitigation system more dynamic. A reliable intrusion detection system should be able to determine the permissible threshold dynamically.

To verify that accuracy results from feature learning rather than data imbalance artifacts, we analyzed the per-class performance using the confusion matrix in

Table 6. Confusion Matrix of Random Forest Model.

Actual Class/Predicted Class	Normal	ICMP Attack	TCP Attack	UDP Attack
Normal	4000	0	0	0
ICMP attack	0	4000	0	0
TCP attack	0	0	4000
UDP attack	0	0	0	4000

. The matrix shows a balanced distribution, with 4000 samples for each class (Normal, ICMP, TCP, UDP). This means that there are no false positives or false negatives in any of the categories. Feature importance analysis shows that the interval between packets and the length of the frame are the most important factors. In the tests, volumetric DoS attacks (ICMP, UDP, and TCP Xmas) have short, even inter-packet intervals. Additionally, attack tools generate fixed frame lengths, creating distinct signatures compared to dynamic packet sizes in normal user activity. These behavioral patterns allow the decision tree-based models to establish clear decision boundaries, resulting in high classification performance.

We measured the processing time for both the training and testing phases to evaluate the practical feasibility of different machine learning methods within a DoS mitigation context. The comparison of processing times for each method is shown in

Table 7. Comparison of training and testing time for each method.

Method	Training Time (s)	Testing Time (s)	Total Time (s)
DNN	30.63	0.134	30.764
nnet	0.632	0	0.632
MLR	0.345	0	0.345
GBM	0.74	0.002	0.742
LDA	0.033	0	0.033
C-SVM	0.143	0.003	0.146
NB	0.009	0.12	0.129
KSVM	0.598	0.032	0.63
KNN	0	0.033	0.033
C50	0.303	0.034	0.337
B-CART	0.356	0.013	0.369
RF	1.338	0.005	1.343

. For the training phase, KNN exhibited the shortest processing time, whereas LDA, MLR, and nnet performed more efficiently during testing. While training time does not impact real-time detection, it is included here as a comparative cost factor, offering insight into the resources needed for initial model development. This suggests that, for real-time detection in DoS mitigation systems, methods with faster testing times, such as LDA, MLR, and nnet, may be preferable, as rapid response times are essential to limit potential damage. Because DNN takes longer to test, it may not be the best choice for systems where real-time detection is very important.

To investigate the underlying factors contributing to this high detection accuracy, we analyzed the feature importance distribution using the random forest algorithm. As illustrated in Figure 9, the analysis identifies TCP sequence number (tcp.seq), IP protocol (ip.proto), and frame length (frame.len) as the most significant discriminators. The high mean decrease accuracy for tcp.seq scientifically confirms that volumetric attacks generated by standard tools exhibit deterministic sequence patterns that are distinct from the stochastic nature of legitimate traffic. This finding is crucial as it validates the architectural premise of this study: since the attack signatures are clearly definable by superficial header features, lightweight decision tree models are sufficient to achieve near-perfect accuracy, rendering computationally expensive deep learning models redundant for this specific threat landscape.

5.2. Comparative Analysis

Table 8. Performance comparison of the proposed framework and baseline methods.

Baseline	Model	Dataset	Accuracy (%)	F1-Score (%)	Time (s)
[26]	3LSTM	Custom-generated dataset	99.79	99	N/A
[25]	C-to-C Protocol	SDN-based Scenario	98.20	N/A	0.045
[27]	RF & AdaBoost	CICIDS2017 (Public)	92.62	92.25	0.450
[28]	ML Classification	IoT Traffic (Public)	99.00	N/A	N/A
[16]	XGBoost	CICFlowMeter/Generated (Mininet)	99.48	99.4	N/A
[29]	MDDCC	Private/Generated (Mininet)	99.00	99.00	N/A
Ours	C5.0, B-CART, RF	Generated dataset (5G-MEC sFlow)	100.00	100.00	0.012

presents a comparative analysis between the proposed framework and several baseline studies. The comparison evaluates performance across different models, dataset types, and execution metrics. While studies such as [25,27] utilize public datasets, both [26] and the proposed method employ custom-generated datasets tailored to their respective network architectures. The results show that the proposed models (C5.0 and Random Forest) have a classification accuracy of 100% and an F1-score of 100%. This is similar to the 99.79% accuracy reported in [26] using a 3LSTM architecture. The proposed method also shows a testing time of 0.012 s, which is shorter than the 0.450 s reported in [27]. This means that it has less computational latency. These results suggest that the proposed methodology maintains high detection precision while optimizing resource consumption for real-time 5G-MEC environments.

5.3. System Responsiveness Analysis

In addition to classification accuracy, the latency between attack detection and the execution of mitigation actions is an important metric for MEC environments. The total mitigation time (T_total) consists of the machine learning inference time (T_inference) and the controller mitigation latency (T_mitigation), which includes the generation of the OpenFlow rule and its installation on the switch. According to the results in Table 5, the Random Forest model yields an inference time of approximately 0.005 s (5 ms) per sample. Upon classifying a flow as malicious, the ONOS controller sends a FlowMod message to the sFlow agent to block the source IP. In the experimental testbed using a Raspberry Pi 3, the average total time from detection to the application of the blocking rule was observed to be less than one second. This response time indicates that the system can mitigate volumetric attacks before they saturate the network resources.

6. Discussion & Future Work

There have been several intelligent systems proposed to detect and prevent anomalies in networks with SDN environments, such as [25,26,27,28,50]. However few studies have specifically focused on trying to focus on types of DoS attacks. In addition, the integration of SDN and supervised learning to enable intrusion detection systems in MEC 5G is still a relatively emerging approach. Moreover, our experiment shows that in specifically the DoS types, our accuracy improved and even achieved perfect accuracy. A study by [27] used the RF approach with AdaBoost, and it achieved 99.95% accuracy for DoS detection. In our work, we improved the accuracy and reached 100% with our system design that identifies the DoS types. The perfect accuracy observed in our results confirms that the dataset accurately reflects the deterministic packet generation typical of standard volumetric attack tools. The clear distinction in classification is a direct consequence of these standardized attack signatures. This finding aligns with recent studies by [16,29], which demonstrate that Tree-based and Deep Learning models can achieve >99% accuracy on generated datasets due to these precise physical patterns. Furthermore, this result supports the “Lightweight Security” paradigm discussed by [44], who argue that high-complexity models can overburden SDN controllers. Our results indicate that while Deep Learning approaches can be utilized for this threat class, the substantial computational overhead is not justified by efficiency, making lightweight Decision Trees a more cost-effective solution for resource-constrained MEC environments.

6.1. Scalability and Real-Time Feasibility

Maintaining real-time performance under high-throughput conditions is important for MEC networks. This study confirms the system’s efficacy in a localized testbed with a single controller. However, the architecture is intended to accommodate larger environments. Our experimental results provide quantitative validation of this scalability. Under attack conditions generating 445 Mbps of traffic, the implemented 0.20% sFlow sampling rate effectively reduced the inspection load to approximately 95 Mbps, as detailed in Section 4. This 78% reduction in data processing overhead experimentally proves that the system can maintain stable operation and prevent control plane saturation even on resource-constrained hardware (Raspberry Pi). This node-level efficiency is an important prerequisite for large-scale deployments, as it ensures that individual edge nodes do not fail under volumetric stress. The system supports horizontal scaling across distributed MEC infrastructures without requiring expensive hardware upgrades.

Furthermore, the selected SDN controller, ONOS, supports a distributed core architecture that allows multiple controller instances to operate as a unified cluster. Benchmarks by Istikmal et al. [51] and Tello et al. [52] indicate that ONOS clustering effectively handles high-throughput environments and outperforms single-controller setups in terms of throughput and delay. In a large-scale MEC environment with multiple controllers, the proposed detection logic can be deployed across the cluster, allowing the control plane workload to be balanced dynamically. This architectural compatibility indicates that the system is suitable for extension to support multi-controller topologies without fundamental redesign.

6.2. Security Analysis Against IP Spoofing and Adversarial Attacks

To evaluate the system’s defense capability against advanced threat scenarios, we analyzed the architectural design regarding IP spoofing and adversarial manipulations. First, when it comes to IP Spoofing attacks, hackers often change the source IP addresses at random to get around static blacklists and fill up the flow tables of SDN switches. In our experimental validation, we used the hping3 tool to create high-rate traffic with random source addresses to test this situation. The Non-Permanent Blocking mechanism described in Algorithm 1 is a part of our proposed system that reduces this risk. The controller uses a temporary blocking rule with a countdown timer instead of permanently blocking a source IP. As validated by existing studies [53], dynamic mitigation strategies at the data plane effectively prevent resource exhaustion (flow table overflow) even when facing high-rate spoofed traffic. Additionally, this mechanism prevents legitimate users from being permanently denied service if their IP addresses are momentarily spoofed by an attacker.

Second, regarding adversarially crafted packet patterns, the system relies on robust physical traffic features, specifically inter-packet intervals. While some studies [54] suggest that ML-based IDSs can be susceptible to adversarial perturbations, in the context of Volumetric DoS, manipulating these features comes with a trade-off. If an attacker modifies the inter-packet interval to appear “normal” (slower) to bypass detection, the attack loses its flooding efficacy. Therefore, the system maintains robustness against the primary objective of DoS attacks. Further testing against gradient-based adversarial examples is acknowledged as a direction for future research.

6.3. Generalization Capability and Applicability to IoT Environments

To address concerns regarding the method’s performance under different data distributions, we analyze the generalization capability of the selected features. The proposed system relies primarily on inter-packet interval and frame length as the dominant discriminators (as detailed in Section 5.1). These features capture the fundamental physical characteristics of volumetric flooding, specifically the deterministic, constant-rate packet generation used by attack tools. This behavior contrasts sharply with the stochastic, bursty nature of legitimate traffic. Consequently, while the model is optimized for video streaming traffic, its reliance on fundamental volumetric features suggests potential applicability to other environments. However, rigorous testing on different background traffic profiles is required to confirm this generalization capability.

Furthermore, regarding applicability to IoT environments, the system is highly suitable for such deployments. IoT attacks, such as those launched by the Mirai botnet, primarily utilize volumetric flooding to flood targets. As noted by Antonakakis et al. [55], these botnets generate high-rate traffic spikes consistent with the attack signatures our model is trained to detect. Since our detection logic is based on generic flow features rather than payload content, the system can effectively identify these anomalies at the MEC edge regardless of the device type.

Conversely, regarding data theft and low-rate attacks, the current system has limitations. Data theft often involves “low-and-slow” transmission patterns designed to blend in with legitimate traffic or avoid threshold-based detection. Our use of sFlow sampling (0.20%), while efficient for high-speed flooding detection, may fail to capture the sparse packet sequences typical of data theft attempts. As analyzed by Brauckhoff et al. [56,57], packet sampling significantly reduces the fidelity of flow metrics required to detect low-volume anomalies. Detecting such non-volumetric threats would require a different feature set and likely a non-sampled inspection mechanism, which are distinct from the design goals of this study.

6.4. Future Work

In this study, we proposed three machine learning methods: C5.0, B-CART, and RF, which performed effectively in the experimental setup. As shown in the results, these methods achieved an average accuracy of 100% with a total mitigation time of less than one second. These findings demonstrate that the proposed machine learning approach is suitable for detecting volumetric DoS attacks. Additionally, the integration of sFlow and OpenFlow in the 5G-MEC network allows the system to block attack sources for a specified duration, preventing memory exhaustion in network devices. These results motivate several directions for future research. First, evaluating the system within a distributed SDN controller cluster is necessary to ensure performance across broader and more complex network environments. Second, to address low-rate attacks or data theft that are more difficult to detect, future work will focus on developing adaptive sampling mechanisms. Third, testing against sophisticated packet pattern manipulations is required to further enhance system robustness. Finally, utilizing larger and more diverse datasets from various IoT devices will help assess the system’s capability to handle a wider variety of 5G network traffic.

7. Conclusions

5G networks require secure and low-latency mechanisms to protect Multi-access Edge Computing (MEC) infrastructure from cyber threats. This paper presented an intelligent intrusion detection and mitigation system that integrates SDN (OpenFlow), sFlow telemetry, and supervised machine learning (C5.0, B-CART, and Random Forest). By simulating a realistic MEC environment with video streaming services, we evaluated the system’s performance against three prevalent volumetric attacks: ICMP Echo flood, TCP Xmas flood, and UDP flood.

The experimental results show that the proposed decision tree-based models achieved 100% detection accuracy for the tested volumetric signatures. In addition to classification performance, the system proved its practical feasibility for edge deployment through two key architectural contributions. First, the integration of sFlow sampling (configured at 0.20%) successfully balanced detection granularity with computational efficiency, preventing control plane saturation. Second, the system responsiveness analysis confirmed a rapid mitigation latency of approximately 50–100 ms, making it suitable for real-time 5G applications. Additionally, the implementation of a dynamic non-permanent blocking mechanism ensures protection against IP spoofing while maintaining service availability for legitimate users.

While the current system works efficiently in mitigating volumetric flooding, future work will focus on improving system defense against adversarial attacks and developing hybrid monitoring techniques to detect low-rate, non-volumetric anomalies. Overall, this study confirms that integrating lightweight telemetry with intelligent edge control provides a scalable and independent security layer important for the reliability of next-generation 5G networks.