CICIoT2023: A Real-Time Dataset and Benchmark for Large-Scale Attacks in IoT Environment

Nowadays, the Internet of Things (IoT) concept plays a pivotal role in society and brings new capabilities to different industries. The number of IoT solutions in areas such as transportation and healthcare is increasing and new services are under development. In the last decade, society has experienced a drastic increase in IoT connections. In fact, IoT connections will increase in the next few years across different areas. Conversely, several challenges still need to be faced to enable efficient and secure operations (e.g., interoperability, security, and standards). Furthermore, although efforts have been made to produce datasets composed of attacks against IoT devices, several possible attacks are not considered. Most existing efforts do not consider an extensive network topology with real IoT devices. The main goal of this research is to propose a novel and extensive IoT attack dataset to foster the development of security analytics applications in real IoT operations. To accomplish this, 33 attacks are executed in an IoT topology composed of 105 devices. These attacks are classified into seven categories, namely DDoS, DoS, Recon, Web-based, brute force, spoofing, and Mirai. Finally, all attacks are executed by malicious IoT devices targeting other IoT devices. The dataset is available on the CIC Dataset website.


Introduction
Nowadays, the Internet of Things (IoT) plays a pivotal role in society and brings new capabilities to different industries [1][2][3]. IoT projects in areas such as transportation and healthcare are becoming increasingly popular, and new applications are under development [4,5]. This new paradigm relies on an extensively connected sensors and actuators network with multiple devices producing network traffic [6][7][8]. Research and industrial communities have been evolving this concept for years, and these devices are becoming more present in our daily lives [9][10][11].
Several areas have been transformed by this technology. For example, in healthcare applications, patients can be regularly monitored using IoT technology [12][13][14]. In transportation, IoT devices have been used to detect and prevent accidents [15][16][17]. Industrial IoT (IIoT) has also brought different solutions, such as high reliability and low latency automated monitoring and collaborative control [18]. IoT applications have also been developed for areas such as education [19], aviation [20], and forestry [21]. In the last decade, society has experienced a drastic increase in IoT connections [22]. In fact, IoT connections will increase in the next few years across different areas [23]. This motivates the creation and development of business ideas and new concepts that rely on a highly distributed infrastructure. In addition, various strategies have been proposed to solve potential problems in IoT operations, i.e., the deployment of new services is leveraged by the scientific findings achieved in the past few years.
Conversely, despite these benefits, several challenges still need to be faced to enable efficient and secure operations (e.g., interoperability, security, standards, and server technologies) [24][25][26][27]. The development of new applications may also bring new requirements to the systems [28,29]. For example, the Internet of Vehicles (IoV) may require more restrictive response times than common IoT applications. Furthermore, detecting and mitigating attacks performed against IoT devices is challenging due to several factors. For example, distributed connections and light devices without security mechanisms may harden the process of detecting and mitigating attacks [30][31][32][33].
Furthermore, although efforts have been made to produce datasets composed of attacks against IoT devices, several possible attacks are not considered. In addition, most efforts do not consider an extensive network topology with real IoT devices. Finally, the attacks performed against IoT devices are executed by computer systems (i.e., non-IoT devices), highlighting the need for a dataset composed of attacks performed by malicious IoT devices. To enable the development of security analytics solutions for intrusion detection in real-world scenarios, the data produced need to (i) include a variety of attacks that can harm IoT operations, (ii) be collected from an extensive topology with real IoT devices of different types and brands, and (iii) include attacks performed by malicious IoT devices.
The main goal of this research is to propose a novel and extensive IoT attack dataset to foster the development of security analytics applications in real IoT operations. To accomplish this, 33 attacks are executed in an IoT topology composed of 105 devices. These attacks are classified into seven categories, namely DDoS, DoS, Recon, Web-based, brute force, spoofing, and Mirai. In addition, all attacks are executed by malicious IoT devices targeting other IoT devices. This dataset includes multiple attacks not available in other IoT datasets and enables IoT professionals to develop new security analytics solutions. Furthermore, the data are available in different formats, allowing researchers to use features extracted in our evaluation or engineer new features.
The main contributions of this research are: • We design a new realistic IoT attack dataset, CICIoT2023, using an extensive topology composed of several real IoT devices acting as either attackers or victims; • We perform, document, and collect data from 33 attacks divided into 7 classes against IoT devices and demonstrated how they can be reproduced; • We evaluate the performance of machine and deep learning algorithms using the CICIoT2023 dataset to classify and detect IoT network traffic as malicious or benign.
This paper is organized as follows: Section 2 presents an extensive comparison of the contributions of this research with other works present in the literature. Secondly, Section 3 introduces the CICIoT2023 dataset and presents the steps involved in the data collection. After that, Section 4 presents the feature extraction process and describes the data. Section 5 presents the machine learning (ML) evaluation in the classification of different attacks using the CICIoT2023 dataset. Finally, Section 6 presents the conclusion of this research.

Related Works
In the past few years, different contributions have been published regarding IoT security datasets. In fact, data have been produced with different goals and using different methods and resources. To better understand the characteristics of existing datasets, we review several initiatives present in the literature and compare them with the proposed CICIoT2023. The authors in [34] propose a novel network-based dataset for detecting botnet attacks in the IoT environment called N-BaioT (2018). Mirai and BASHLITE botnets were used to attack nine commercial IoT devices. Multiple features were extracted from the network traffic and used by a deep-learning autoencoder for attack detection. In [35], the authors introduce a host-based IoT dataset composed of data from real IoT devices. This dataset, called IoTHIDS (2018), is produced based on experiments considering a topology of three devices infected by Mirai, Hajime, Adira, BASHLITE, Doflo, Tsunami, and Wroba malware botnets.
IoT-SH (2019) [36] is a dataset composed of captures of twelve attacks (categorized into four classes) against eight different smart home devices. A three-layer Intrusion Detection System (IDS) is used considering various combinations of rule-based and machine learning approaches to classify the attacks. BoT-Iot (2019) is introduced in [37] as a realistic traffic dataset, produced considering heterogeneous network profiles. Multiple attacks are performed (e.g., DDoS, DoS, data theft, and scan) against five devices. In the evaluation process, a set of new features are selected and used based on correlation coefficient and joint entropy techniques. Various machine and deep learning models are trained to evaluate the attack detection accuracy.
The authors in [38] introduce the Kitsune (2019) dataset, which is composed of four different categories of attacks executed against nine IoT devices. In the experiments conducted, a security camera was infected by a real Mirai botnet sample. This dataset is intended to support the development of plug-and-play Network Intrusion Detection Systems (NIDS) to detect normal and malicious traffic. Similarly, IoTNIDS (2019) [39] represent an initiative focused on collecting data from a real-world IoT networking environment based on the interaction between two IoT devices (speaker and camera). Multiple attacks are analyzed in this effort, e.g., Mirai, MITM, DoS, and scanning. MedBIoT (2020) [40] is an IoT network architecture dataset based on using real and emulated devices. The authors evaluated multiple machine learning techniques using 100 statistical features extracted from the IoT network traffic. In [41], the authors propose the IoT-23 (2020) dataset. This contribution refers to a botnet dataset captured composed of real network environment captures of benign and malicious traffic.
IoTIDs (2020) [42] is proposed as a dataset composed of IoT-related flow-based features, selected and ranked by the correlation coefficients technique and the Shapiro-Wilk algorithm, respectively. In the experiments, the authors performed four different attacks against two IoT devices (speaker and camera) and recorded the data. Multiple machine learning methods were used in the evaluation process (e.g., SVM, G-NB, LDA, and LR) focusing on attack detection and classification. The authors in [43] present the MQTT (2020) dataset with the primary goal of providing realistic data that include a protocol dedicated to IoT network scenarios. Furthermore, eight IoT devices were connected to the MQTT broker and a set of 33 different features were extracted and provided to various machine learning algorithms. Similarly, MQTT-IoT-IDS (2020) [44] is another contribution focused on producing a dataset using a lightweight protocol, i.e., MQTT, which is used in IoT networks. The authors focus on replicating a realistic IoT network by using a camera feed, twelve MQTT sensors, and a broker. Five scenarios are considered based on the variation in the attacks performed. Several packet-based, uni-, and bi-flow features are used alongside six different machine learning algorithms in the evaluation phase.
In [45], the authors proposed a new telemetry-based data-driven IoT/IIoT dataset called TON-IoT (2020). This heterogeneous dataset comprises both normal and attack samples captured in different scenarios. Targeting the development of a realistic dataset, the authors include attack sub-categories, data recorded from operating system logs, and network traffic. Several machine learning and deep learning algorithms are used in the evaluation phase and the achieved results are reported in detail. Finally, the Edge-IIoTSet (2022) dataset is introduced as a realistic cybersecurity resource for IoT and IIoT applications to enable the development of Intrusion Detection Systems (IDS) in centralized and distributed applications [46]. Throughout the paper, an in-depth description of the testbed used is presented. In addition, the authors also describe the dataset generation framework. Regarding the machine learning evaluation process, centralized and federated learning considerations are presented.

The Proposed CICIoT2023
This section introduces the CICIot2023 dataset. We aim to present an in-depth description of all steps and resources involved in producing this dataset. First, we describe the CIC IoT Lab. Then, we focus on the IoT topology, listing all IoT and network devices used and how they are connected. Then, we present a discussion on all attacks that have been executed. Finally, we provide insights into how the data were collected for benign and malicious scenarios.

IoT Lab
The production of IoT security data that can be used to support real applications is challenging for several reasons. One of the main problems is having an extensive network composed of several real IoT devices, similar to topologies of real IoT applications. Many works adopt simulated or very few IoT devices due to costs, network equipment required (e.g., switches, routers, and network tap), and personnel dedicated to maintaining such an infrastructure.
Thereupon, the Canadian Institute for Cybersecurity (CIC) has a distinguished presence in the cybersecurity ecosystem and a history of high-impact contributions to industry and academia. Examples are datasets used to develop new cybersecurity applications and several partnerships with the industry to improve the cybersecurity practice and develop new solutions. This success enabled CIC to establish an IoT lab with a dedicated network to foster the development of IoT security solutions. In fact, by sharing the data collected from this extensive topology, we intend to foster the advancement of IoT security research and support several initiatives in different IoT security aspects. Figure 1 shows the IoT lab at the CIC and its devices. Indeed, IoT devices are distributed across the lab, in which some of them are placed on the table, others on the floor, and some on the walls. We adopt a local network topology and several power plugs are available in the lab. Additionally, there are racks and storage rooms in order to organize the IoT and network devices.

IoT Topology
The IoT topology deployed to produce the CICIoT2023 is illustrated in Figure 2 and comprises 105 IoT devices. A total of 67 IoT devices were directly involved in the attacks and other 38 Zigbee and Z-Wave devices were connected to five hubs.
This topology mimics a real-world deployment of IoT products and services in a smart home environment. The devices list includes smart home devices, cameras, sensors, and micro-controllers which are connected and configured to enable the execution of several attacks and capture the corresponding attack traffic. The lab is also equipped with various tools and software, which enable us to perform several attacks and capture both benign and malicious attack traffic.
This topology is divided into two parts. In the first part, an ASUS router connects the network to the Internet and a Windows 10 Desktop computer shares this connectivity. In addition, a Cisco switch is placed between this computer and a VeraPlus access point connecting 7 Raspberry Pi devices. These devices are responsible for executing the attacks and malicious activities in the experiments. Using IoT devices as malicious agents is a CICIoT2023 characteristic not found in other efforts. Then, the Cisco switch is connected to the second part through a Gigamon Network Tap. This network device collects all the IoT traffic and sends it to two network monitors, which are responsible for storing the traffic using wireshark [47]. In fact, a network tap is a hardware device that allows for monitoring and analyzing network traffic by connecting to a network cable and providing a copy of the traffic to other monitoring and security tools. Network taps are connected in a way so as not to affect the normal operation and provide a full-duplex, non-intrusive, and passive way of accessing network traffic, without introducing any latency or affecting the performance of the network. This device has two network and two monitoring ports and is placed between the attacking and legitimate devices, connecting one port to the attackers and the other to the victim networks. Using the monitor ports, we are able to capture the traffic to and from the IoT network. In the second part, a Netgear Unmanaged Switch is connected to five gateways and base stations to enable communication with IoT devices with protocols such as Zigbee and Z-Wave. Furthermore, another VeraPlus controller is connected to the switch. This controller is also connected to other two Zigbee/Z-Wave hubs and to several devices considered victims in the attacks performed. The list of all IoT devices used in this dataset is presented in Table 1. Note that Zigbee and Z-wave devices do not have a MAC address and are labeled as "Not Applicable" (N/A) for that particular column.

Data Collection of Benign and Malicious Scenarios
As described in Section 3.2, a network tap and two traffic monitors are dedicated to monitoring the network traffic. Every packet sent through the network is stored in separate computers. In fact, the network has two different interfaces, which are associated with two other monitoring ports that send incoming packets to these computers. Hence, the network traffic is monitored using Wireshark [47] and stored in pcap format. Since two data streams are stored, mergecap [48] is used to unify pcap files for each experiment.
For each attack, a different experiment is performed targeting all applicable devices. In all scenarios, the attacks are performed by malicious IoT devices targeting vulnerable IoT devices. For example, DDoS attacks are executed against all devices, whereas web-based attacks target devices that support web applications. Table 2 depicts the tools used to perform all attacks alongside the number of rows generated. In addition, Figures 3 and 4 illustrate the instances count for each attack and category. The values are also presented in Table 3.

Benign Data Generation
The benign data represent the legitimate use of the IoT network. In this sense, the main goal of the data-capturing procedure relies on gathering IoT traffic in idle states and with human interactions (e.g., sensor data, echo dot requests, and accessing video feeds from smart cameras).
In terms of hardware for capturing, we relied on a network tap combined with two network monitors. In terms of software used, we adopted Wireshark to capture the entire traffic. Furthermore, all IoT devices are configured with default parameters and without malicious or attacking scripts. In this sense, benign data traffic gathering happens when there are no attacks. This process was conducted over a period of 16 h.

Executing DoS and DDoS Attacks
These attacks refer to flooding threats to compromise the availability of IoT operations. In the case of Denial-of-Service (DoS) attacks, one Raspberry Pi is responsible for flooding IoT devices. Furthermore, multiple Raspberry Pis are used to execute Distributed Denialof-Service (DDoS) attacks through an SSH-based master-client configuration. The attacks executed are: • ACK Fragmentation: a relatively small number of maximum-sized packets is used to compromise the network operation. In many cases, these fragmented packets are successfully sent and handled by routers, firewalls, and intrusion prevention systems, given that fragmented packets recompilation is not performed [

Gathering Information from the IoT Topology
These attacks gather all possible information about the target. In addition, an attacker can use a reconnaissance (i.e., scan) attack as a preparation step for other attacks. There are multiple ways to perform these attacks, and some of the most popular and threatening variations are: • Host Discovery: A host discovery attack, also known as a host identification or host enumeration attack, is a type of reconnaissance attack that is used to identify active hosts on a network. It involves using various techniques to identify the IP addresses of devices that are connected to a network, and it is the first step in many cyber-attacks [77].

Exploiting Web-Based Vulnerabilities
When executing these attacks, web services running on IoT devices were targeted. Web-based attacks are concerned with targeting web services in several ways. These attack types include injection, hijacking, poisoning, spoofing, and DoS [78]. The web-based attacks executed in this research are: • SQL Injection: an attack that targets web applications by injecting malicious SQL code into the application's input fields. The goal of an SQL injection attack is to gain unauthorized access to a database, steal sensitive information, or execute arbitrary commands on the database server [79]; • Command Injection: an attack that targets web applications by injecting malicious commands into an input field with the ultimate goal of gaining unauthorized access to a system, stealing sensitive information, or executing arbitrary commands on the targeted system [80]; • Backdoor Malware: involves installing malware on a targeted system that allows the attacker to gain unauthorized access to the system at a later time. The malware, known as a "backdoor," creates a hidden entry point into the system that can be used to bypass security measures and gain access to sensitive information or perform malicious actions [81]; • Uploading Attack: targets a web application by exploiting vulnerabilities in the application's file upload functionality. The goal of an uploading attack is to upload malicious files, such as malware, to a targeted system and use them to gain unauthorized access or execute arbitrary code on the targeted system; • Cross-Site Scripting (XSS): allows an attacker to inject malicious code (e.g., a script) into a web page. The injected script can then be executed by the web browser of any user with access to the page, allowing the attacker to steal sensitive information (e.g., cookies, session tokens, and personal data) or to perform other malicious activities (e.g., traffic redirection) [82]; • Browser Hijacking: a type of cyber attack in which an attacker modifies a web browser's settings, such as the home page, default search engine, or bookmarks in order to redirect the user to a different website or display unwanted ads. The goal of a browser hijacking attack is to generate revenue through advertising or to steal personal information [83].

Spoofing Communication
Spoofing attacks enable malicious actors to operate under the identity of a victim system and gain illegitimate access to the network traffic. The main focus of such a procedure includes gaining access to systems, stealing data, and spreading malware [84]. Two of the most popular spoofing attacks are: • ARP spoofing: relies on the transmission of manipulated ARP (Address Resolution Protocol) messages to associate the MAC address of the malicious device with the IP address of some other legitimate device in the network. This enables attackers to intercept, modify, or block network traffic [85]; • DNS spoofing: relies on the alteration of DNS entries in a DNS server's cache, redirecting users to manipulated or malicious websites. This enables attackers to steal sensitive information, spread malware, and perform other malicious actions [86].

Brute-Force Threats
Brute-force attacks consist of the submission of data (e.g., passwords or passphrases) to eventually gain access to systems [87]. Among the several procedures that can be executed, a dictionary brute-force attack is a type of attack that attempts to guess a password or passphrase by repeatedly trying words from a pre-defined list of words obtained from various sources. The goal of the attack is to find the correct password by trying all the words in the dictionary [88].

Mirai as an IoT Threat
The Mirai attack is a large-scale DDoS that can target IoT devices. In this paper, we are conducting different variations of Mirai attacks by using five different raspberries, as illustrated in Figure 5, alongside the connections considered in the different IoT network layers. In order to connect to the Internet, a gateway uses a Windows 10 instance to provide and monitor Internet access. This access is possible through a Netgear unmanaged switch that connects attackers and general IoT devices. Several tools are used to perform the attacks and a special Mirai configuration is also adopted. An online IoT supervisor coordinates the operation of the multiple IoT devices in the topology (e.g., sensors, cameras, and smart speakers). Finally, some other works do not consider Mirai in their attack set. In fact, we focus on several attacks that can be executed against IoT devices, and we consider the analysis and execution of new IoT attacks in the future directions of this research (e.g., attacks using future protocols). This attack infected devices to form a botnet that can flood targeted victims. This threat can cause disruption in different contexts and some of its most popular variations are: • GREIP: Within the GRE packet, this attack floods the target system with encapsulated packets. The internal data comprise random IPs and ports, whereas the external layer contains actual IPs [89]; • GREETH: This attack presents a similar procedure to GREIP. However, the main focus is on the packet encapsulation approach, which is based on the ethernet header [89]; • UDP Plain: This threat focuses on flooding targeted victim systems with UDP packets considering a repeated packet segment. However, the payload sent is different for each packet [89].

Feature Extraction and Data Description
The CICIoT2023 dataset is available in two different file formats: pcap and csv. Pcap files comprise the original data generated and collected in the CIC IoT network in different scenarios. These files contain all packets sent and can be used to extract and engineer other features. Furthermore, csv files present a simpler way of loading and using the data. Those files are composed of features extracted from the original pcap files summarized by a fixed-size packet window. In other words, the features are extracted from a sequence of packets carrying information between two hosts.
The method adopted to produce the dataset is illustrated in Figure 6. Firstly, the data are generated (i.e., captured), extracted, and labeled. This refers to the initial step, in which the actual attacks are executed against IoT devices. Then, the data are processed in a way to enable researchers to access the data generated easily. Finally, we conduct a machine learning (ML) evaluation to show how classification capabilities can be leveraged by the proposed dataset.  Figure 7 illustrates how the data generation, extraction, and labeling are conducted for each attack scenario (and benign scenario). The first phase relies on the use of different tools presented in Table 2 to execute attacks against IoT devices in the network. After that, the network traffic is captured in pcap format using Wireshark. Finally, for each attack executed, the entire traffic captured is labeled as belonging to that particular attack. Regarding the data processing step, illustrated in Figure 8, the network traffic data composed of captures of all attacks alongside benign traffic are used. As it represents about 548 GB worth of traffic data, we split it into smaller chunks of 10 MB to perform the conversion in parallel. This process is conducted using TCPDUMP [90]. After that, a parallel procedure is executed to extract several features using the DPKT package [91] and store them in separate csv files. These features are described in Table 4. In this process, DPKT is used to enable a flexible feature extraction procedure considering important attributes of the IoT operation highlighted in previous works. Conversely, other tools can also be used to extract features, e.g., CICFlowMeter [92] and Nfstream [93]. In this stage, we also perform the data cleaning by removing incomplete packets (i.e., packets that present null features). In our experiments, we only remove the timestamp from the list since it does not illustrate the network behavior-instead, it is used for sorting. In this case, all other features are directly used to evaluate how different ML models perform in such circumstances. These features are extracted based on proposals present in the literature regarding IoT security [8,46]. In fact, although these features have been used and validated in other efforts, our main goal is to present a flexible approach to training ML models with multiple features. Thus, several other features can be extracted or engineered based on the scripts used in this research as well as the raw network traffic (i.e., pcap files).
With Greeth Flood, and Mirai UDPPlain) packets to mitigate data size discrepancy (e.g., DDoS and CommandInjection) and calculate their mean values using Pandas [94] and Numpy [95]. Finally, we combine all subfiles into a processed csv dataset using Pandas. Thereupon, the resulting csv datasets represent the combination of features of each data chunk.
Moreover, each attack conducted in this research presents different characteristics. For example, the network traffic generated by a DDoS attack tends to be larger than the network traffic generated by a spoofing attack. Indeed, these differences can also be observed in other features of the dataset. Table 4 lists all features provided in the dataset, which Table 5 presents the characteristics of these features. For each feature in the entire dataset, we present the mean, standard deviation (std), minimum (min), 25th percentile (25%), median (50%), 75th percentile (75%), and maximum (max) values.

Machine Learning (ML) Evaluation
In order to demonstrate how the CICIoT2023 dataset can be used to train machine learning (ML)-based attack detection and classification methods, Figure 9 illustrates the ML evaluation pipeline adopted in this research. Firstly, we combine all datasets produced following the procedure presented in Figure 8. In this sense, malicious and benign traffics are combined and shuffled into a single dataset (i.e., blended dataset) using PySpark [96]. Once the data are integrated, we evaluate ML performance from three different perspectives: (i) multiclass classification, focussing on classifying 33 individual attacks; (ii) grouped classification, considering 7 attack groups (e.g., DDoS and DoS); and (iii) binary classification (i.e., malicious and benign traffic classification). In each case, the dataset is divided into the train (80%) and test (20%) sets, which are normalized using the StandardScaler method [97] before the actual training process. Finally, the results obtained are summarized as integrated results.

Metrics
The evaluation of different ML models and configurations is conducted based on evaluation metrics. Given that TP represents the True Positives, TN the True Negatives, FP the False Positive, and FN the False Negatives, the metrics used in this research are [98]: • Accuracy: responsible for evaluating the classification models by depicting the proportion of correct predictions in a given dataset and is based on the following expression: • Recall: the ratio of correctly identified labels to the total number of occurrences of that particular label: • Precision: the ratio of correctly identified labels to the total number of positive classifications: • F1-Score: geometric average of precision and recall:

Evaluation
In the evaluation process, we adopted five ML methods that have been successfully used in different applications, including cybersecurity: Logistic Regression [99], Perceptron [100], Adaboost [101][102][103], Random Forest [104], and Deep Neural Network [105]. Figure 10 illustrates the performance of all methods when framing the classification problem as binary (i.e., malicious and benign), multiclass with 8 classes (i.e., benign and attack categories), and multiclass with 34 classes (i.e., benign and all individual attacks). These results are also depicted in Table 6. For the binary classification, the results show that all methods present high performance, whereas accuracy is a metric that all methods reach over 98%, and the F1-score highlights the difference among these approaches. For example, Perceptron achieves 81%, showing that it suffers since the minority class (i.e., benign) is misclassified more often. In the classification of attack groups (i.e., eight classes), the overall performance is degraded since the classification task becomes more challenging. The Logistic Regression, Perceptron, and Adaboost methods show a significant decrease in accuracy. This impact is even more perceptible for F1-score. However, both Random Forest and Deep Neural Network are able to maintain high accuracy and F-1 score. These methods also present a decrease in performance but are capable of achieving F1 scores of 70%. Table 6. Results obtained in the classification process conducted using different machine learning models (illustrated in Figure 10). Finally, the most challenging classification task is represented by a multiclass classification of individual attacks (i.e., 34 classes). In this scenario, both Random Forest and Deep Neural Network could maintain high accuracy with very similar results. The same applies to F1-score since a slight reduction was perceived (around 1%) compared to the eight-class challenge. Furthermore, this case study shows that the Logistic Regression, Perceptron, and Adaboost methods are not able to categorize attacks as efficiently, given that the average accuracy is below 80% and F1-score is less than 50% in all cases.

Metric
These results show how ML methods can be used to classify attacks against IoT operations. In fact, this is a starting point that can be considered in any ML-based cybersecurity solutions for IoT operations. This effort not only highlights that the use of other ML methods is possible (e.g., optimized methods), but also enables the adoption of similar strategies to solve IoT-specific problems. Finally, although we are focussing on 33 different attacks, future directions could also be tailored to address issues related to individual attacks or categories.

Discussion
To illustrate how these models are performing for each class, Tables 7 and 8 show the confusion matrix for Random Forest and Deep Neural Networks in the case of multiclass classification (eight classes). In both cases, it is possible to observe that some classes are very well classified, mainly those with a large number of occurrences in the dataset. For example, the misclassification rates for DDoS, DoS, and Mirai are very small, followed by Recon and spoofing.
However, these models face challenges in classifying other attacks. For example, web-based attacks are usually classified as benign, Recon, or spoofing. The same occurs in the brute force classification. Although the similarities in the data patterns lead the models to make these mistakes, the classification is successful in most cases, leading to the results depicted in Figure 10. In fact, the results show that the multiclass classification performance degrades for three classes (Benign, Recon, and spoofing). The underlying traffic for those scenarios can be similar, and we intend to explore this phenomenon in future works further.
Finally, Tables 9 and 10 compare all datasets reviewed with the proposed CICIoT2023 dataset. These tables focus on presenting an analysis of attacks executed in this research as well as its main contributions, i.e., these datasets may include attacks other than those shown in these tables. Table 9. Comparison CICIoT2023 with existing IoT security datasets.

Web-Based
Browser

Conclusions
Nowadays, IoT is becoming increasingly important for society. In this context, the development of security solutions is pivotal to enabling efficient, secure, and dependable IoT operations. This research introduced a novel and extensive IoT attack dataset to foster the development of security analytics applications in real IoT operations. In this process, 33 attacks are executed in an IoT topology composed of 105 devices. These attacks are classified into seven categories (i.e., DDoS, DoS, Recon, Web-based, brute force, spoofing, and Mirai) and all attacks are executed by malicious IoT devices targeting other IoT devices. Furthermore, this dataset includes multiple attacks not available in other IoT datasets and enables IoT professionals to develop new security analytics solutions using data in different formats. The dataset is available through the CIC Dataset website (https://www.unb.ca/cic/datasets/index.html, accessed on 19 June 2023).
Compared to the state-of-the-art publications, the CICIoT2023 dataset extends existing IoT security insights by using an extensive topology with a variety of IoT devices, executing several attacks never present in a single IoT security dataset, and analyzing how widelyused machine learning (ML) methods perform in different classification scenarios.
Finally, this work enables the development of several future works, e.g., the optimization of ML models, the analysis of features and how they influence different ML models, the interpretation of classifications, and the analysis of transferability based on the comparison to other datasets.