Mitigating Adversarial Attacks against IoT Profiling

Abstract

Internet of Things (IoT) applications have been helping society in several ways. However, challenges still must be faced to enable efficient and secure IoT operations. In this context, IoT profiling refers to the service of identifying and classifying IoT devices’ behavior based on different features using different approaches (e.g., Deep Learning). Data poisoning and adversarial attacks are challenging to detect and mitigate and can degrade the performance of a trained model. Thereupon, the main goal of this research is to propose the Overlapping Label Recovery (OLR) framework to mitigate the effects of label-flipping attacks in Deep-Learning-based IoT profiling. OLR uses Random Forests (RF) as underlying cleaners to recover labels. After that, the dataset is re-evaluated and new labels are produced to minimize the impact of label flipping. OLR can be configured using different hyperparameters and we investigate how different values can improve the recovery procedure. The results obtained by evaluating Deep Learning (DL) models using a poisoned version of the CIC IoT Dataset 2022 demonstrate that training overlap needs to be controlled to maintain good performance and that the proposed strategy improves the overall profiling performance in all cases investigated.

1. Introduction

Internet of Things (IoT) applications have been helping society in several ways [1,2,3]. IoT’s popularity in areas such as transportation and healthcare has widely fostered its adoption in other industries and IoT devices and new applications are under development [4]. This new concept establishes a globally connected sensor network with multiple devices producing large amounts of traffic [5,6]. As a result of the efforts made by the research community and industrial groups, these devices are becoming more present in our daily lives [7,8].

Various areas have been positively impacted by IoT applications. In healthcare, patients can be continuously monitored [9,10,11]. In transportation, accident detection and prevention have been supported by IoT [12,13,14]. In Industrial IoT (IIoT), reliability, low latency monitoring, and collaborative control are enhanced [15]. Finally, there are also applications in areas such as education [16], aviation [17], and forestry [18].

In the last few years, society has experienced a drastic increase in IoT connections [19], which is expected to be even more present in the next few years across different areas [20]. This fosters the creation and development of business models and new paradigms that rely on a highly distributed infrastructure. Various approaches have been proposed to solve potential IoT problems, i.e., the scientific contributions developed in the past few years support the deployment of new services. The benefits include interoperability, security, standards, and server technologies [21,22,23].

However, several obstacles still need to be overcome to enable efficient and secure IoT operations. New applications may also entail new requirements to the systems, e.g., Internet of Vehicles (IoV) applications might require more restrictive response times than usual IoT applications [24]. Besides, detecting and mitigating attacks in IoT environments is challenging due to several factors (e.g., distributed connections and light devices without security mechanisms) [25].

In this context, monitoring the IoT network traffic to classify devices, identify abnormal patterns, and detect malicious activities becomes a pillar for secure operations [26,27,28]. IoT profiling refers to the service of identifying and classifying IoT devices’ behavior based on different features [29]. One possible approach for IoT profiling and device identification relies on using Deep Learning (DL) trained on datasets collected from IoT network traffic.

Moreover, Deep Learning (DL) performance can be affected by different attacks. In this sense, data poisoning attacks represent one of the most challenging attacks to detect and mitigate [30]. Dealing with such a threat is difficult since the learning process assumes the training data can be used to approximate the hidden function, i.e., the training data would prepare the model to perform well in a realistic environment [31]. Thereupon, data poisoning makes it complex for a model to infer if the data used in the learning process is legitimate [32].

As a result of these attacks, the performance of a trained model can be degraded. For example, suppose noisy instances are included in the training dataset of different clients. In that case, the training process can lead the model weights to erroneous configurations and, consequently, to lower classification and regression performance scores. Furthermore, attackers could worsen the performance of such models for particular classes of interest, which could lead to hazardous scenarios (e.g., misclassification of speed limit readings for self-driving cars [33]). In fact, label flipping is one of the most challenging data poisoning attacks [34,35].

Although the adoption of a profiling mechanism is an important aspect of IoT security, this is not intended to be the only countermeasure to deploy. Identifying abnormal behaviors of impersonated devices enhances the defensive posture against advanced attacks (e.g., Advanced Persistent Threats-APT and zero-day attacks). Thereupon, attacks against profiling data can compromise security in different environments. For example, in smart cities, threat actors can take advantage of delayed and less accurate profiling identification to launch attacks against a dynamic topology [36,37]. In the Internet of Vehicles (IoV), impersonated devices can interact with intra-vehicle and inter-vehicle [38,39] networks and target specific protocol characteristics (e.g., CAN bus [40]) while undetected. In the case of the Internet of Medical Things (IoMT), the increased number of false positives can degrade the trust of profiling mechanisms, preventing the use of advanced analytical solutions in healthcare security [41,42]. Finally, the Internet of Industrial Things (IIoT) can be affected by delayed intrusion detection as impersonated devices remain undetected, leading to safety hazards and efficiency shortcomings [43,44].

Thereupon, the main goal of this research is to propose the Overlapping Label Recovery (OLR) framework to mitigate the effects of label-flipping attacks in Deep-Learning-based IoT profiling. OLR uses random forests as internal cleaners to recover labels based on their voting mechanism to accomplish this. After that, the dataset is re-evaluated and new labels are produced to minimize the impact of label flipping. In fact, OLR can be configured using different hyperparameters and we investigate how different values can improve the recovery procedure. Finally, the results are obtained by evaluating Deep Learning (DL) models using a poisoned version of the CIC IoT Dataset 2022 [29]. The main contributions of this research are as follows:

• A framework to mitigate the effects of label flipping attacks in Deep-Learning-based IoT profiling called Overlapping Label Recovery (OLR);
• A novel label recovery mechanism based on overlapping training and data sampling;
• An evaluation model based on the average area between performance curves considering label flipping and recovery procedure.

This paper is organized as follows: Section 2 presents the related works. Secondly, Section 3 presents important topics necessary for the understanding of the proposed strategy. Then, Section 4 depicts the Overlapping Label Recovery (OLR) framework for IoT profiling. Section 5 and Section 6 show the methods adopted in this research and the experiments performed. Finally, Section 7 presents the conclusion of this research.

2. Related Works

This Section reviews works that are related to this research’s proposal. First, we review efforts in IoT profiling, highlighting their main aspects and goals. Then, we focus on label-flipping mitigation.

2.1. IoT Profiling

The authors in [45] propose a strategy to generate fingerprints of IoT devices based on the different device manufacturers’ network system implementations. The authors emphasize that there is a lack of IoT device discovery applications on a large scale due to the massive number of device models (i.e., types, vendors, and products). This motivates the exploration of IoT feature spaces in three network layers: network, transport, and application. The prototype implemented demonstrated effectiveness in the experiments performed generating device class labels with a 94% precision and 95% recall.

In [46], the authors focus on IoT device behavioral fingerprinting to undertake strong device identification. The security challenges brought by IoT applications include the identification and authentication of several devices. The problem lies in the lack of robust approaches to identify and classify behaviors in IoT operations. Hence, the authors approximate the device’s behavior using features extracted from the network traffic and train a machine-learning model that can be used to detect similar device types. The results showed that the proposed approach can achieve an identification rate of 93–100% and a mean accuracy of 99%. Similarly, Thangavelu et al. [47] propose a distributed DEvice Fingerprinting Technique (DEFT) that develops and maintains fingerprint classifiers. This approach is designed to be scalable and dynamic, with intersections with Software-Defined Networking (SDN) [48,49] and network function virtualization [50].

In [51], the authors produce device fingerprints for 20 IoT devices based on 30 features using Wireshark and four Four supervised machine learning algorithms (i.e., Support Vector Machine-SVM-, Decision Tree-DT-, Ensemble Random Forest-RF-, and Gradient Boosting Classifier-GBC). The results showed that the proposed approach is effective and promising for more extensive topologies. In addition to that, Rose et al. [52] explore the potential of using dynamic and active network profiling and machine learning to secure IoT operations against tampering attempts and suspicious network transactions.

In [53], the authors build the Abnormal Behavior Profiling (ABP) of IoT devices to support ML-based abnormal behavior detection in IoT. The authors used the k-Means and SVM algorithms to detect data modification out of four possible data points from one sensor, and the results showed that the k-Means (92%) outperform the SVM (69.5%) in terms of detection accuracy. Furthermore, the first fingerprinting framework used to identify ZigBee and Z-Wave IoT device classes is introduced in [54]. This approach monitors idle network traffic to implement signature-based device-class fingerprinting mechanisms. The experiments conducted showed that the proposed strategy achieves excellent performance in identifying different classes of IoT devices without yielding overhead to IoT devices or network traffic.

The efforts presented in [55,56,57] focus on IoT Device fingerprinting using Machine Learning (ML) and Deep Learning (DL). The authors in [55] propose a novel idea of device fingerprinting based on graphs of Inter Arrival Time (IAT) for packets, while [56] focuses on training ML algorithms on selected features extracted from the encrypted IoT traffic. Finally, the authors in [57] introduce an IoT device identification platform to improve Internet of Things (IoT) security using different techniques: Deep Neural Network (DNN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN). In all cases, the authors achieved high accuracy and promising results for future applications (e.g., more extensive network topologies).

2.2. Label Flipping Mitigation

The authors in [58,59] evaluate the effects of label flipping in different scenarios. Xiao et al. [58] evaluate the performance of Support Vector Machines (SVMs) to adversarial label noise attacks considering attacks that maximize the SVM’s classification error by flipping a number of labels in the training data. The highlight of the proposed strategy can be used to support the development of more secure SVM learning algorithms. Furthermore, Zhang et al. [59] propose two novel label-flipping attacks to evaluate the robustness of the Naive Bayes (NB) algorithm under label noise. The authors’ primary goal is to increase the false negative rate without affecting normal mail classification in the spam classification domain. The evaluation conducted showed that the label-flipping attacks reduce the overall classification performance. Similarly, Lukasik et al. [60] investigate whether label smoothing effectively mitigates the effects of label noise. The findings presented in this research indicate that label smoothing can be beneficial and that other aspects of DL training can be considered in future works (e.g., gradient clipping) [61].

In [62], the authors propose (i) an approach to perform optimal label-flipping poisoning attacks and (ii) a mechanism to detect and recover suspicious data points. The paper discusses details on the problem faced and the proposed solution. The experiments demonstrated a significant degradation of the models’ performance produced by the proposed attack and the effectiveness of the proposed method in mitigating the effect of label-flipping attacks. In addition to this effort, Ortego et al. [63] mitigate the effects of label noise by proposing a Multi-Objective Interpolation Training (MOIT) approach that exploits contrastive learning and classification. The experiments conducted showed that state-of-the-art results are achieved when training DNNs with different noise distributions and levels.

Authors in [64,65,66] focus on the effect of data poisoning in Federated Learning (FL) applications [67]. The general idea is to develop mechanisms capable of defending, preventing, detecting, and mitigating such attacks in a federated environment. All efforts presented promising results and pointed out the extension of their approach (e.g., in terms of techniques used) as their future direction.

The authors in [68] solve the optimization problem of minimizing the number of labels flipped using an approximate linear programming algorithm and provide theoretical guarantees on how close its result is to the optimal solution in terms of the number of label flips. Multi-class classification and regression tasks are highlighted as future directions of this research. In addition to this effort, Sharma et al. [69] propose a method to identify the poisoned training samples, a label-flipping attack based on a stochastic hill-climbing search, and a CatBoost-based defense mechanism to detect and recover malicious training samples.

Some recent cybersecurity efforts have also focused on mitigating data poisoning attacks. Yang et al. [70] address the issue of training federated models for IoT security considering label flipping. In addition, Jiang et al. [71] introduce a method to detect malicious clients in Federated Learning (FL) and mitigate label flipping through the recovery of feature characteristics. Finally, Taheri et al. [72] proposes a defensive mechanism to mitigate label flipping in the context of Android malware classification relying on Convolutional Neural Networks (CNNs).

3. Background

This section presents relevant topics in support of this research’s proposal. We discuss the main aspects of IoT profiling and how it can be affected by label flipping. After that, the main aspects of Deep Learning (DL) and Random Forest (RL) are presented with a focus on the context of this research.

3.1. IoT Profiling & Label Flipping

IoT profiling is the process of gathering and analyzing information about individual IoT devices and traffic that may include packet characteristics, behaviors, connections and interactions [23,53]. IoT devices are often targeted by malicious procedures due to the limited processing power and security mechanisms [73]. Thus, network-level security is pivotal to identifying and mitigating attacks in heterogeneous IoT networks. In fact, IoT profiling is an essential approach to identifying and monitoring connected devices, specific behavior, and abnormal traffic within the network. Adopting efficient IoT profiling solutions can prevent malicious network activity by promptly detecting and isolating compromised devices for further investigation.

Moreover, the success of Deep Learning (DL) is possible due to the massive amount of data available. The same applies to IoT profiling, where data can be used to identify devices and abnormal behaviors. One of the main goals during the training of a DL model is to establish a context as close as possible to the real-world application. This entails that the dataset used to train the model is expected to represent the real world and be trustworthy. However, attackers can poison the dataset in a way to transform it into a misleading representation of reality.

One of the most challenging attacks to detect and mitigate is label flipping. This malicious procedure controls labels assigned to training data to significantly diminish the performance of the DL classifiers [72]. Once this attack is executed, it is difficult to recover original labels since it usually requires domain knowledge and massive data.

3.2. Deep Learning (DL) & Random Forest (RF)

In the past few years, Deep Learning (DL) has been applied to several problems across many areas. This subarea of Artificial Intelligence (AI) has attracted the research community’s attention due to its extraordinary success in performing tasks considered complex to computational systems due to their reasoning requirements. For example, there are solutions in computer vision [74], Natural Language Processing (NLP) [75], and generative solutions [76].

Thereupon, Deep Neural Networks (DNNs) consist of processing units named neurons and are divided into multiple layers [77]. These processing units are connected to synaptic weights and output the result of an activation function [78]. The goal of DNN is to approximate a hidden function that maps the inputs (features) to the outputs to reduce the estimator error [79].

Although DNNs can be trained using different methods, the combination of the back-propagation and the Stochastic Gradient Descent (SGD) algorithms are used to optimize the weights. Figure 1 illustrates a standard DNN. Indeed, a different number of nodes and layers can be adopted and the flexibility of this model has been vital in its success in solving several problems.

As a very successful Machine Learning (ML) method, Random Forest (RF) is a tree-based ensemble that depends on a collection of random variables [81,82]. This method generates random subsets of training configurations and makes decisions based on the evaluation from several standpoints [83]. RF has been successfully used in data cleaning efforts [84,85]. Finally, its flexibility enables the OLR framework to adopt RF in the internal label recovery process. Figure 2 illustrates a simple RF model.

4. Overlapping Label Recovery for IoT Profiling

This Section introduces the main contribution of this research, i.e., the Overlapping Label Recovery (OLR) framework. The primary objective of this strategy is to recover original labels perturbed by label-flipping attacks based on the use of multiple Random Forest (RF) models, referred to as cleaners, training on overlapping samples.

Assume an IoT profiling dataset is attacked in a way that multiple labels are flipped. In this scenario, the dataset used to train models becomes deceiving since misleading data points are introduced. For example, data related to an IoT speaker can be wrongfully associated with a smart camera. This attack happens due to several reasons and can have a profound impact on classification performance. A framework that can mitigate this problem is vital for IoT profiling applications.

Furthermore, Figure 3 illustrates how the OLR framework supports the training of accurate Deep Learning (DL) methods for IoT profiling. Since the IoT profiling dataset has been compromised, OLR acts as a label recovery strategy to improve the overall data quality. The output dataset, namely the recovered IoT profiling dataset, is then used to train the Deep Learning (DL) model to improve its performance.

Moreover, the internal OLR process is divided into two phases as illustrated in Figure 4. This framework adopts multiple cleaners that enable a probabilistic profiling label recovery capability with the ultimate goal of mitigating attacks against data integrity. First, mechanisms to recover the flipped labels need to be initialized. In this sense, Figure 5 illustrates the process of training multiple cleaners to recover flipped labels. Firstly, The dataset is used as a baseline to train all cleaners. However, each cleaner is trained in a random sample of the original dataset, with a size equal to a factor $\varphi$ of the original dataset’s size ($\varphi$%, $0<\varphi <1$). In fact, these are overlapping samples, as they are randomly collected from the same dataset.

The second OLR phase refers to the actual label recovery, as illustrated in Figure 6. For each data point present in the dataset, all cleaners are used to calculate the probability of this data point belonging to each class defined in the dataset. All cleaners, previously trained with a random sample, produce a vector $Vp$ to describe all probabilities. Cleaners are instances of Random Forest (RF), i.e., the probability of a data point belonging to a class refers to the internal voting mechanism used in this method. Once all outputs are generated, the average probabilities are calculated, and the data point is assigned to the class with the greatest average probability.

This process is formally described in Algorithm 1. This approach requires the vector of cleaners and its length ($\mathsf{\Omega }$ and ${\mathsf{\Omega }}_s$), the IoT profiling dataset (D), the sampling factor ($\varphi$), and the number of classes (i.e., unique labels) present in the dataset (n) as inputs. Line 1 initializes an empty probabilities vector $Vgp$ to be populated in a $for$ loop (Lines 2 to 10). Then, for each cleaner $\omega \in \mathsf{\Omega }$, a randomly generated sample $d_{\omega }$ is used in the training process. After that, $\omega$ is used to calculate the probabilities of all data points present in D belonging to each class. Since these probabilities are iteratively stored in the local empty vector of probabilities $Vlp$, Line 10 adds such vector into the global vector P.

Algorithm 1 Overlapping Label Recovery (OLR) for IoT Profiling

Require:  $\mathsf{\Omega }$: vector containing cleaners; Require:  ${\mathsf{\Omega }}_s$: length of $\mathsf{\Omega }$; Require:  D: IoT Profiling dataset; Require:  $\varphi$: Sampling factor; Require:  n: Number of classes present in the dataset;  1: Initialize P a global empty vector of probabilities;  2: for $\omega$ in $\mathsf{\Omega }$ do  3:     Randomly sample as a factor $\varphi$ of D, producing $d_{\omega }$;  4:     Train cleaner $\omega$ using $d_{\omega }$;  5:     Initialize $Vlp$ a local empty vector of probabilities;  6:     for i in D do  7:         Calculate probabilities $p^i\leftarrow [p_0^{\omega },p_1^{\omega },...,p_n^{\omega }]$ of instance i belonging to each class;  8:         Add $p^i$ to $Vlp$;  9:     end for  10:     Add $Vlp$ to P;  11: end for  12: Initialize $A{P}_D$ as an empty vector of average probabilities;  13: for i in D do  14:     $ap\leftarrow \left[\begin{array}{c}\frac{P_{i,1}^1+P_{i,1}^2+\cdots +P_{i,1}^{{\mathsf{\Omega }}_s}}{{\mathsf{\Omega }}_s}\\ \frac{P_{i,2}^1+P_{i,2}^2+\cdots +P_{i,2}^{{\mathsf{\Omega }}_s}}{{\mathsf{\Omega }}_s}\\ \cdots \\ \frac{P_{i,n}^1+P_{i,n}^2+\cdots +P_{i,n}^{{\mathsf{\Omega }}_s}}{{\mathsf{\Omega }}_s}\end{array}\right]$  15:     Add $ap$ to $A{P}_D$  16: end for  17: Initialize empty vector $R_l$ of recovered labels;  18: for $ap$ in $A{P}_D$ do  19:     $labe{l}_{ap}\leftarrow \underset{x}{argmax}f\left(x\right)|f\left(x\right)=a{p}_x\forall x\in \{1,2,\dots ,n\}$;  20:     Add $labe{l}_{ap}$ to $R_l$;  21: end for  22: Return $R_l$;

Once P is populated with the probabilities of all instances belonging to all classes calculated by all cleaners, these results are combined from Lines 13 to 16. This process comprises the calculation of average probabilities of a given instance i belonging to a specific class ($1,2,\dots ,n$) according to all cleaners ($1,2,\dots ,{\mathsf{\Omega }}_s$) and, after that, defining labels based on the classes with the highest probabilities (Lines 17 to 21). Finally, the procedure returns the set of recovered labels $Rl$.

To compare the performance of the different training sample sizes ($\varphi$), we adopt the evaluation model shown in Equation (1). The main idea is to produce a DL configuration that maximizes the area between the curves generated by the results with and without OLR support. In this model, $\theta$ represents the metrics used to measure the DL performance, i.e., in this research, accuracy, precision, recall, and f1-score. Also, ${\gamma }_{min}$ and ${\gamma }_{max}$ represent the minimum and maximum label flipping factors adopted in the experiments. Given that D is the IoT profiling dataset used, ${\mu }_{\gamma ,\varphi ,\theta \left(D\right)}$ represents the DL performance for metric $\theta$ using the OLR framework to recover labels with training sample size $\varphi$ and label flipping factor $\gamma$. Similarly, ${\sigma }_{\gamma ,\theta \left(D\right)}$ represents the DL performance for metric $\theta$ without label recovering mechanisms. Finally, the outcome of this summation is multiplied by ${\theta }_t^{-1}$ as an approach to average the area between these two curves in all metrics.

$$f\left(\varphi \right)={\theta }_t^{-1}\sum _{i=1}^{{\theta }_t}{\int }_{{\gamma }_{min}}^{{\gamma }_{max}}({\mu }_{\gamma ,\varphi ,\theta \left(D\right)}-{\sigma }_{\gamma ,\theta \left(D\right)})d_{\gamma }$$

(1)

5. Evaluation Method

To demonstrate the applicability of the proposed approach and show the improvements achieved, we adopt the evaluation methods depicted in this section. Figure 7 shows how the evaluation is conducted. Three processes are initiated to (i) generate baseline results using the original data; (ii) generate results using the poisoned version of the profiling dataset and (iii) generate results using the same poisoned version of the profiling dataset, but considering the recovery capabilities of OLR. After that, a comparison is conducted to analyze the improvements of the proposed strategy regarding accuracy, recall, precision, and F1-score.

Figure 8 illustrates the method adopted to produce baseline results. Firstly, the IoT profiling dataset is loaded (train split, i.e., 80% of the dataset). Then, it goes through the preprocessing phase, which comprises normalization using the Min-Max approach [87]. After that, a Deep Learning (DL) model is trained and evaluated using the test split (i.e., 20% of the original dataset). Finally, the results are reported.

Figure 9 depicts the method adopted to produce results when the train set suffers a label-flipping attack. Firstly, the IoT profiling dataset is loaded (train split, i.e., 80% of the dataset) and an attack is performed in a factor of $\gamma$ of the data ($0<\gamma <1$). Then, the data are normalized and a Deep Learning (DL) model is trained and evaluated. Finally, the results are summarized.

Finally, Figure 10 describes the method adopted to produce results when the train set suffers a label-flipping attack and the OLR approach is used to recover original labels. Firstly, the IoT profiling dataset is loaded (train split, i.e., 80% of the dataset) and an attack is performed. After that, the data go through a recovery process conducted by OLR and is normalized using the MinMax strategy. Then, a DL model is trained and evaluated, and the results are reported.

6. Experiments

This Section describes the details and results of the experiments performed. The primary goal is to show how the proposed approach can mitigate label-flipping attacks in different scenarios. Three experiments are conducted varying the internal sampling factor $\varphi$ adopted by OLR, where in Experiment I, $\varphi =0.15$, in Experiment II, $\varphi =0.25$, and Experiment III adopts $\varphi =0.35$. In terms of models used, a Deep Learning (DL) composed of three hidden layers of 64 nodes is used in each experiment. Besides, Random Forests (RFs) are used as cleaners based on 100 internal classifiers. The results of using 5, 10, and 15 cleaners are discussed in each experiment. Finally, to show the applicability of the proposed approach, we used the CIC IoT Dataset 2022 dataset [29], which comprises network traffic produced by IoT devices to enable the analysis of IoT devices’ behavior exhibited operating under different constraints. Additionally, we consider the IoT profiling problem from the perspective of classifying groups of devices, namely audio (18774 traffic readings of Echo Spot, Nest mini, Echo Dot, Sonos, Echo Studio), home automation (19808 traffic readings of Amazon Plug, Roomba, Heim vision lamp, globe lamp, Eufy home base, Atomi cofee maker, Smartboard, Yutron, Phillips Hue, and Teckin), and cameras (77361 traffic readings of Amcrast, Arlo base cam, Arlo qcam, Borun cam, and DLink cam). In all cases, we consider label flipping factors of 0.1, 0.2, 0.3, 0.4, 0.5, and 0.6, i.e., ${\gamma }_{min}=0.1$ and ${\gamma }_{max}=0.6$. Finally, several features were used in the process, e.g., “L4_tcp”, “L4_udp”, “L7_http”, “L7_https”, “ethernet_frame_size”, “ttl”, and “protocol”. The graphs present multiple lines of different colors rather than symbols to simplify the visualization in overlapping areas.

6.1. Experiment I

This experiment considers $\varphi =0.15$ as the sampling factor used in the internal cleaning procedure of the OLR framework. Adopting the baseline, compromised, and recovered methods illustrated in Figure 8, Figure 9, and Figure 10, respectively, the results obtained are depicted in Figure 11. In fact, OLR is capable of improving the IoT profiling performance in all cases. For all metrics (i.e., accuracy, recall, precision, and F1-score) and all OLR configurations, the model performance using the recovered labels lies between the baseline and the flipped label performance. Furthermore, as the label flipping factor increases, it becomes more difficult for labels to be recovered and the performance decreases.

Figure 12 compares the performance of IoT profiling adopting different OLR configurations. All configurations present a very similar outcome in cases where the label-flipping factor is less than 0.3. However, increasing the number of cleaners improves overall performance in more challenging cases.

Finally, Figure 13 illustrates the capability of different OLR configurations in terms of label recovery. In all cases, several labels were recovered and used as stated in the original dataset. Furthermore, increasing the number of cleaners improves overall performance in most cases and the adoption of OLR mitigated the impact of label flipping, illustrated by the red line.

6.2. Experiment II

Adopting $\varphi =0.25$ as the sampling factor used in the internal cleaning procedure of the OLR framework, this experiment follows the methods illustrated in Figure 8, Figure 9 and Figure 10 to produce the results. Figure 14 shows that OLR can improve the IoT profiling performance in all cases. The performance using recovered labels lies between the baseline and the flipped label performance, although the performance decreases as the label flipping factor increases.

A comparison of the IoT profiling performance using different OLR configurations is illustrated in Figure 15. Once again, increasing the number of cleaners improves overall performance in more challenging cases, although all configurations present a very similar outcome when the label flipping factor is less than 0.3.

The label recovery capability of different OLR configurations is depicted in Figure 16. Several labels were successfully recovered since increasing the number of cleaners improves overall performance in most cases, mitigating the effects of label flipping illustrated by the red line.

6.3. Experiment III

This experiment considers $\varphi =0.35$ as the OLR sampling factor. The results illustrated in Figure 17 are obtained based on the methods illustrated in Figure 8, Figure 9 and Figure 10. OLR increases accuracy, recall, precision, and F1-score of IoT profiling with labels flipped. Furthermore, it becomes more difficult for labels to be recovered as the label-flipping factor increases.

Figure 18 compares the performance of IoT profiling using different OLR configurations and shows that having 10 cleaners is slightly better in cases where the label flipping factor is less than 0.3. However, although all configurations can yield high performance, presenting similar outcomes, increasing the number of cleaners improves overall performance in more challenging cases.

Finally, Figure 19 illustrates the capability of different OLR configurations in terms of label recovery, presenting several labels recovered in all cases. Furthermore, increasing the number of cleaners improves overall performance in most cases, especially when the label-flipping factors vary from 0.2 to 0.4.

6.4. Evaluation

Figure 20 shows the performance of the different OLR configurations across all experiments. These graphs show that having more cleaners can be beneficial in many cases. However, as we increase $\varphi$, reducing the number of cleaners can improve the score and the overall performance. This is an insightful finding since it shows that training overlap needs to be controlled to maintain good performance. On the other hand, considering a few cleaners with small fractions of the training data can be problematic as the underlying learning process will not consider several aspects of the dataset.

7. Conclusions

In this paper, we introduced the Overlapping Label Recovery (OLR) framework to mitigate the effects of label-flipping attacks in Deep-Learning-based IoT profiling. The current security challenges faced by IoT operators can be mitigated by novel methods, such as Deep-learning-based IoT profiling. However, label flipping can compromise the performance of such methods by manipulating data. OLR uses Random Forests (RF) as underlying cleaners to recover labels and re-evaluates the training dataset to recover its labels. The results are obtained by evaluating Deep Learning (DL) models using the CIC IoT Dataset 2022 demonstrating that training overlap needs to be controlled to maintain good performance and that the proposed strategy improves the overall profiling performance in all the cases investigated. In the future directions of this research, the authors intend to investigate the efficiency of other recovery methods for IoT profiling and present a comprehensive comparison with the results presented in this paper. Also, we will investigate how OLR can improve IoT profiling in federated environments.