Evaluating Moving Target Defense Methods Using Time to Compromise and Security Risk Metrics in IoT Networks

Abstract

The Internet of Things (IoT) networks face an increasing number of cyber threats due to their heterogeneous, distributed, and resource-constrained nature. Conventional static defense mechanisms are often inadequate against sophisticated and advanced persistent threats. Moving Target Defense (MTD) is a dynamic proactive security method that increases system resilience by continuously changing the attack surface, thereby increasing uncertainty and complexity for attackers. In this paper, we evaluate the effectiveness of shuffling or diversity-based MTD methods using time-to-compromise and security risk metrics. We develop attack path-based mean time-to-compromise and security risk reduction metrics for assessing the effectiveness of MTD. These metrics provide a quantitative basis for evaluating how well MTD techniques delay successful compromises and lower overall security risk exposure. The performance of the deployed MTD mechanism is evaluated and discussed for different attacker skill levels and shuffling frequencies.

1. Introduction

The rapid growth of Internet of Things (IoT) technologies has enabled the creation of innovative applications across diverse domains [1,2]. However, the large-scale deployment of IoT devices and their inherent constraints on resources such as bandwidth and power pose challenges for the effective application of traditional security and defense mechanisms. As a result, IoT systems remain vulnerable to various security threats and attacks which can disrupt the seamless delivery and normal operation of IoT-based services [3]. Moving Target Defense (MTD) methods have emerged as proactive and adaptive security strategies for safeguarding IoT systems despite resource limitations and evolving dynamic threats [4]. MTD mechanisms aim to dynamically change the attack surface to increase uncertainty and confuse attackers by invalidating information collected during reconnaissance or other procedures carried out by attackers prior to launching an attack [5,6]. Shuffling or diversity-based MTD techniques such as virtual IP shuffling [7,8], random host mutation [9], IP randomization [10], multiple operating systems rotation [11], IP address rotation or hopping [12], cryptosystem or firmware mutation [13], code partitioning [14], and cyberdeception [15] are common MTD mechanisms that can be employed to enhance system security in resource-constrained IoT environments [14]. Several assessment methods for evaluating MTD methods have been proposed [16,17,18,19,20,21,22]. However, their application is limited to particular attack scenarios, threat models, or specific network environments, and they generally lack consideration of IoT network environments. In this paper, we propose a new suite of attack path-based time-to-compromise (TTC) and security risk (SR) metrics to evaluate the effectiveness of shuffling and diversity-based MTD methods in IoT networks. The key contributions of this work are summarized as follows:

• We propose a set of attack-path-based TTC metrics to estimate the mean time to compromise a smart device in an IoT network deploying the MTD defense mechanism. TTC-related metrics include mean TTC, minimum TTC, and maximum TTC and consider different skill levels on the part of attackers.
• We devise a set of risk-based security metrics taking TTC and attack cost into account. These risk-based metrics compute the risk reduction with MTD mechanisms in terms of shuffling rate and strategies.
• We conduct extensive simulation experiments to measure the effectiveness of the MTD methods deployed in IoT networks and identify key factors that significantly influence the performance of MTD methods.

The rest of this paper is organized as follows: Section 2 discusses the state-of-the-art of related work; Section 3 describes the network, threat, and defense models considered in this work; Section 4 presents the proposed metrics used to evaluate the MTD methods; Section 5 describes the numerical computation of the metrics; Section 6 discusses the experimental results and analysis; lastly, Section 7 concludes the paper and suggests potential future research directions.

2. Related Work

The rapid use of IoT smart devices introduces significant security and privacy risks due to their constrained resources and limited built-in protections [23,24]. IoT devices are extensively used in every part of our lives, including homes (smart homes), vehicles, education, healthcare, etc. The lack of proper security protection of IoT-connected systems (e.g., smart homes) can lead to severe cyberattacks, resulting in substantial financial and reputational losses. To address these security challenges, the IoT security research community has been motivated to develop advanced defense mechanisms to protect IoT systems against emerging security and privacy threats. Li et al. [25] proposed sequential XGBoost and hierarchical bag-of-words models for privacy risk protection of mobile apps against app fingerprinting attacks. This approach recognizes user activities from encrypted wireless traffic and addresses the challenges of app fingerprinting, including hidden destination, invisible boundary, app multiplexing, and open-world recognition. Similarly, Li et al. [26] introduced a fine-grained open-world Android app fingerprinting (FOAP) method for user action identification in Android apps under an open-world setting, addressing the limitations of traditional app fingerprinting attacks. Ni et al. [27] proposed an automated attack framework (AppListener) that recognizes fine-grained mobile app activities from harvested radio-frequency energy signals and uses them against side-channel attacks. AppListener automatically recognizes fine-grained app behaviors, revealing new privacy risks through RF energy harvesting. Sun et al. [28] presented a firmware updates over-the-air (FUOTA) system for LoRa networks that addresses energy efficiency, transmission reliability, and fairness using delta scripting, channel coding, and beamforming. This work was further enhanced in LoRa+ [29] through a joint differencing and compression algorithm, a concatenated coding scheme, and a beamforming strategy to address previous limitations in energy efficiency, reliability, fairness, and security. The security methods discussed above all address evolving threats in IoT networks; however, they neither incorporate MTD strategies nor provide security risk assessments for deploying MTD methods in IoT environments.

Various security assessment models and metrics have been proposed for evaluating the effectiveness of MTD methods. Picek et al. [30] proposed optimization-based metrics to assess the effectiveness of MTD techniques by using a network neighborhood partitioning algorithm to improve the precision of MTD influence measurement. Osei et al. [31] developed a game-theoretic approach to produce an optimal defense strategy. Zaffarano et al. [18] introduced a quantitative framework measuring productivity, success, confidentiality, and integrity for given missions and attack models. Carroll et al. [32] proposed a probabilistic model and analyzed the performance of the network address shuffling MTD technique. Similarly, Hong and Kim [17] developed a set of risk-based metrics to assess the effectiveness of shuffling, diversity, and redundancy in MTD. In addition, Hong et al. [33] proposed attack and defense effort-based dynamic security metrics for assessing the effectiveness of MTD techniques. Several entropy-based methods have also been introduced to quantitatively assess the security of MTD methods [34,35,36]. However, these approaches rely on probability distributions to estimate the likelihood of events, which can limit their precision in capturing real-time system dynamics. Sharma et al. [16] proposed a set of dynamic security metrics to evaluate the effectiveness of the deployed MTD methods in software-defined network (SDN) environments. These metrics capture the dynamic changes in network and host state information such as IP addresses, ports, software stacks, vulnerabilities, or network topology that are introduced by the MTD’s shuffling and reconfiguration methods. Alavizadeh et al. [22] utilized system risk, attack cost, return on attack, and reliability metrics to assess the effectiveness of combined MTD techniques in cloud systems. Using security and economic metrics, they mainly focused on the specific context of cloud models for e-health applications. However, these metrics do not offer the ability to capture and measure the effectiveness of the MTDs in IoT networks.

State-of-the-art security metrics for assessing MTD methods mainly focus on measuring the static or dynamic aspects of network addresses, networks, or systems. Most of these metrics were developed for enterprise network settings, and are not suitable for IoT environments. In addition, there are no attack path-based TTC or security risk metrics for shuffling and diversity-based MTDs. Thus, this work proposes a suite of attack path-based time-to-compromise and security risk metrics for evaluating MTD methods in IoT networks.

3. System Model

This section presents the network, threat, and defense models considered in this work. In addition, it outlines the assumptions and scenarios about the considered IoT network system, including the devices, vulnerabilities, attacker capabilities, and deployed defense mechanisms.

3.1. Network Model

In this study, we consider a smart home IoT networked system similar to the one described in [3]. This IoT network system consists of various interconnected IoT devices with wireless communication protocols. Figure 1 shows the smart home IoT network setup scenario with smart TVs, smart door lock, smart security cameras, thermostats, meters, a smart plug, a smart phone, and appliances all communicating over wireless protocols such as WiFi, Bluetooth, or Zigbee. IoT devices are managed through a central hub or mobile app, allowing users to control and monitor them remotely. Each of these devices has one or more vulnerabilities.

Table 1. Vulnerabilities of IoT devices with CVE IDs, and CVSS score, and description.

Device Type	Brand/Model	CVE ID	CVSS Score	Description
Smart TV	LG WebOS	CVE-2023-6317	9.8	Remote command execution
Smart door lock	Suleve 5-in-1 Smart Door Lock v1.0	CVE-2023-39843	2.4	Missing encryption
Smart speaker	Sonos Era 100	CVE-2024-5269	8.8	Use-after-free remote code execution
Smartphone	Android Devices	CVE-2023-40088	8.8	Memory corruption
Security Camera	Wyze Cam	CVE-2019-9564	9.8	Bypass login and control the devices
Smart Meter	Siemens 7KT PAC1200	CVE-2017-9944	8.8	Authentication bypass
Smart Thermostat	Ecobee3 lite	CVE-2021-27954	8.5	Buffer-overflow
Smart Plug	WSP080 v1.2 lite	CVE-2023-33768	6.5	Incorrect signature verification

shows example vulnerabilities of these devices along with their CVE ID, CVSS score, and description [37]. For example, the smart TV (e.g, LG WebOS Smart TV) and security camera (Wyze Cam) have high-severity vulnerabilities (CVSS 9.8) that allow for remote command execution and login bypass, respectively, potentially giving attackers full control of the devices. The smartphone is also affected by memory corruption, posing serious risks when connected to smart environments. The smart door lock has relatively less risk, as it has a low CVSS score. Additionally, the smart plug, smart meters, and thermostats suffer from incorrect signature verification, authentication bypass, and buffer overflow.

3.2. Threat Model

In this study, we model attackers based on their ability to scan and exploit vulnerabilities within a smart home IoT network. As shown in Figure 1, it is assumed that the attackers are external to the network and that they can attempt to identify and compromise vulnerable IoT devices through scanning and analysis. They can use common network scanning tools such as Nmap [38] or Nessus [39] to detect active hosts, open ports, and known vulnerabilities. We consider three levels of attackers based on their technical expertise and knowledge:

• Beginner: These attackers use default scanning configurations and have limited knowledge of the target network. Their ability to interpret scan results is minimal, and they require significant effort to identify vulnerabilities and find usable exploits to attack the system.
• Intermediate: These attackers have a moderate understanding of IoT network structures and known vulnerabilities. They can customize scan parameters to focus on specific devices or services, and are generally able to identify vulnerabilities; however, they may struggle to find or develop suitable exploits that effectively compromise the targets.
• Expert: Expert attackers possess deep knowledge of IoT systems and employ stealthy or adaptive scanning techniques. They can efficiently discover vulnerabilities and are capable of locating or creating effective exploits, enabling them to compromise a wide range of heterogeneous devices with ease.

3.3. Defense Model

In this work, we use shuffling and/or diversity-based MTD methods to protect the smart home IoT network. These methods dynamically change key elements of the system such as the network address, operating system rotations, or application software at regular intervals [4]. This constant change makes it difficult for attackers to scan, identify, and exploit known vulnerabilities. For example, if a smart device’s operating system version is changed, the vulnerability targeted by the attacker may no longer exist. Similarly, changing the network layout or IP address can disrupt the attacker’s ability to track or reach the device. MTD network shuffling mechanisms dynamically reconfigure the network topology by disconnecting existing links and/or establishing new connections/links [40]. This dynamic disconnection and reconnection mechanism alters the network structure, which reduces an attacker’s ability to perform reconnaissance and launch targeted attacks. Figure 2 shows the network structure before and after the MTD shuffling operation. A brief description of time-based shuffling and diversity-based defense methods is provided below:

• Shuffling-based MTDs: Shuffling methods periodically modify network attributes such as IP addresses, ports, or configurations in a fixed interval of time. These MTD methods hinder an attacker’s ability to maintain an accurate view of the system, invalidating reconnaissance efforts and narrowing the window for successful exploitation.
• Diversity-based MTDs: Diversity-based MTD methods dynamically change device characteristics such as operating systems (OS rotation), firmware versions, or application configurations to ensure that the vulnerabilities of the IoT devices differ over time, thereby reducing the risk of uniform exploitation.

4. Proposed Approach

This section presents the proposed time-to-compromise and security risk metrics for evaluating the effectiveness of IoT-based MTD methods. IoT-based MTD mechanisms dynamically change the attack surface of the IoT network; in turn, these dynamic network changes affect the reachability and vulnerabilities of IoT devices. Attack graph-based security metrics describe attack paths using the attack graph [41], which helps in analyzing the level of security [42]. Attack paths are sequences of devices or vulnerabilities that an attacker can compromise to reach a target. We assume that the target IoT network system has been deployed with any time-based shuffling MTD mechanisms that continuously change the attack surface network, devices, or software properties by shuffling the IP address, route, cryptosystem, firmware, operating system, service, application software, or code partitioning. The shuffling event invalidates security-related information (e.g., vulnerabilities and exploits) of the target networked systems collected by the attacker and consequently reduces the visibility of the attack surface to the attacker.

Shuffling or diversity-based MTD mechanisms dynamically alter network configurations, topologies, and vulnerability information of both individual devices and the overall IoT network [4,17]. To effectively capture and model these temporal changes, we utilize the Temporal–Hierarchical Attack Representation Model (T-HARM), which is specifically designed to reflect the dynamic transformations introduced by MTD operations [43,44]. Figure 3 shows a two-layer hierarchical representation of an IoT network using the T-HARM. The reachability layer captures the evolving network topology and communication paths between nodes, while the vulnerability layer encodes node security information such as vulnerability of device operating systems, exploitability, and risk levels.

4.1. Time-to-Compromise Metrics

Time-to-compromise (TTC) security metrics estimate the mean time to compromise of a host in a network deploying an MTD mechanism. We devise security metrics for the MTD by modifying the existing models of Byres and Leversage [45] and McQueen et al. [46] for incorporating the MTD shuffling rate in their estimation. Our proposed metrics estimate the mean TTC for different skill levels of attackers and with different shuffling rates of the MTD mechanism while taking attack paths into account. Let $A{P}_{t_k}$ be the set of all possible attack paths from an attacker to a target host or device $h_i$. The time to compromise using an attack path $a{p}_j\in A{P}_{t_k}$ at time $t_k$ is defined as follows:

$$ttc\left(a{p}_j\right)=\sum _{i}ttc\left(h_i\right),\forall h_i\in a{p}_j,$$

(1)

where $ttc\left(h_i\right)$ is a function that estimates the expected time to compromise of a host or device $h_i$. Now, the mean time to compromise (MTTC) on all the attack paths is

$$MTT{C}_{t_k}={\displaystyle \frac{{\sum }_{j=1}^{|A{P}_{t_k}|}ttc\left(a{p}_j\right)}{|A{P}_{t_k}|}},\forall a{p}_j\in A{P}_{t_k}.$$

(2)

Using Equation (1), the minimum and maximum times to compromise on the attack path at time point $t_k$ can be estimated as follows:

$$MinTT{C}_{t_k}=\underset{\forall a{p}_j\in A{P}_{t_k}}{min}ttc\left(a{p}_j\right),$$

(3)

$$MaxTT{C}_{t_k}=\underset{\forall a{p}_j\in A{P}_{t_k}}{max}ttc\left(a{p}_j\right).$$

(4)

Equations (3) and (4) can be used to find the minimum and maximum TTCs for a given MTD shuffling time window [$t_1$,$t_m$], as follows:

$$MinTTC={\displaystyle \frac{{\sum }_{k=1}^{m}MinTT{C}_{t_k}}{m}},$$

(5)

$$MaxTTC={\displaystyle \frac{{\sum }_{k=1}^{m}MaxTT{C}_{t_k}}{m}}.$$

(6)

The metrics presented in Equations (1)–(6) can be used to estimate the TTC of a target–host pair using attack paths. Attack paths are sequences of hosts/devices or vulnerabilities that an attacker can exploit and compromise to reach the target device or system. However, the TTC of a host $ttc\left(h_i\right)$ in Equation (1) is still unknown. To estimate it, we adapt a method Byres and Leversage [45] considering three different levels of attacker skill and knowledge about the target IoT network system with three attacking processes. Estimation of the TTC relies on the attacker’s skills and knowledge. Based on the different skill levels of an attacker, the attacking processes can be divided into the following three exclusive processes:

• Attack Process 1: In this scenario, the attacker knows one or more vulnerabilities and exploits ( i.e., known vulnerabilities and known exploits). The attacker has all of the required knowledge to attack the system.
• Attack Process 2: In this process, the attacker knows one or more vulnerabilities but does not have any exploits on hand ( i.e., known vulnerabilities and unknown exploits). Thus, the attacker has partial knowledge about the target IoT network system.
• Attack Process 3: In this scenario, the attacker does not know any vulnerabilities or exploits (i.e., unknown vulnerabilities and unknown exploits). This means that the attacker does not possess any knowledge about the target system. Thus, the attacker must scan the network, find vulnerabilities, and build an exploit before launching an attack.

Some of the above attack processes are mutually exclusive. For instance, Attack Process 1 and Attack Process 2 are mutually exclusive processes, as are Attack Process 2 and Attack Process 3. Therefore, we can multiply the probabilities and expected time to compromise a host $ttc\left(h_i\right)$ as follows:

$$ttc\left(h_i\right)=t_1\times p+t_2\times (1-p)\times (1-u)+t_3\times u\times (1-p)$$

(7)

where p and u are the probabilities of the attackers in Process 1 and Process 3, respectively, and $t_1$, $t_2$, and $t_3$ are the respective times required for completing the attack in each attack process. As MTD techniques change the attack surfaces, they can make vulnerabilities and other related information invisible to attackers. The visibility factor of a host $h_i$ can be related with the shuffling rate ${\theta }_i$ as ($1-{\theta }_i)$, where ${\theta }_i={\displaystyle \frac{1}{T_i}}$ for $T_i>0$. We relate this visibility factor with the vulnerability information using different skill levels of the attacker to estimate the probabilities p and u. The computations of the probabilities p and u are as follows:

$$p=1-e^{-(1-{\theta }_i)\times v_i\times m_i/K},$$

(8)

$$u={(1-s)}^{(1-{\theta }_i)\times v_i},$$

(9)

where:

• $v_i$ is the number of vulnerabilities that exist in a host $h_i$ or a component,
• $m_i$ is the number of exploits readily available for the vulnerabilities of host $h_i$,
• K is the number of total non-duplicate vulnerabilities in the vulnerability database,
• ${\theta }_i$ is the shuffling rate of the host $h_i$ with MTD interval time $T_i$, ${\theta }_i={\displaystyle \frac{1}{T_i}},\forall T_i>0$,
• s is the attacker’s skill level, $s\in [0,1]$, e.g., $s=0.3$ for beginner, $s=0.55$ for intermediate, and $s=1.0$ for expert.

We estimate the attack completion time of each attack process by adapting the estimation method and parameters used in the model from McQueen et al. [46]. Let $t_1$, $t_2$, and $t_3$ represent the amount of time for completing process 1, process 2, and process 3, respectively, provided as follows:

• $t_1=1$ day,
• $t_2=5.8\mathrm{days}\times ET$, where

  $$ET=s\times \left(1+\sum _{tries=2}^{v_E+1}\left[tries\times \prod _{j=2}^{tries}\left({\displaystyle \frac{v_E-j+2}{v_i-j+1}}\right)\right]\right),$$

  $$t_3=30.42\times \left({\displaystyle \frac{1}{s}}-0.5\right)+5.8\mathrm{days}.$$

  (10)

  where:
  • $ET$: Expected number of tries
  • $v_A$: Number of vulnerabilities for which exploits are available or can be created by the attacker at their skill level
  • $v_E$: Number of vulnerabilities for which no exploits are available at the attacker’s skill level
  • s: Attacker’s skill level.

Now, we can estimate the time-to-compromise of a given host or device using Equation (7), substituting p, u, $t_1$, $t_3$, and $t_3$ obtained from Equations (8), (9), and (10), respectively.

4.2. Security Risks Metrics

The security risk (SR) is an important metric that is used to evaluate and assess security methods, including MTDs [17]. The quantitative security risk of an attack incident on IoT-based system host $h_i$ for an attack duration time t can be defined as follows:

$$SR\left(h_i\right)=Pr(h_i,t)\times CoA\left(h_i\right)$$

(11)

where $Pr(h_i,t)$ is the probability of an attack happening on host $h_i$ for attack duration t and $CoA\left(h_i\right)$ is the loss or damage cost of the attack, typically measured in monetary value (e.g., USD). Here, the probability $Pr(h_i,t)$ can be defined in terms of the expected time-to-compromise of a host $TT{C}_{h_i}$, obtained as follows:

$$Pr(h_i,t)={\displaystyle \frac{t}{TT{C}_{h_i}}},\forall t\le TT{C}_{h_i}$$

(12)

where $TT{C}_{h_i}=ttc\left(h_i\right)$, which can be obtained using Equation (7).

The SR of a network (SRN) is the risk of compromising all hosts in the network with the MTD mechanism under test, and can be computed as

$$SRN=\sum _{i=1}^{n}CoA\left(h_i\right)\times {\displaystyle \frac{t}{TT{C}_{mtd}^{tot}}}.$$

(13)

We propose a security risk reduction with MTD metric for measuring the effectiveness of deploying the MTD mechanism. The proposed metric estimates the security risk reduction provided by the MTD mechanism in terms of TTC based on the quantitative risk model. To measure it, we consider the system with and without the MTD mechanism. Using Equation (11), the security risk of the system with no MTD mechanism and with the mechanism can be respectively defined as follows:

$$S{R}_{no-mtd}\left(h_i\right)=P_{no-mtd}(h_i,t)\times CoA\left(h_i\right)$$

(14)

$$S{R}_{mtd}\left(h_i\right)=P_{mtd}(h_i,t)\times CoA\left(h_i\right)$$

(15)

where $P_{no-mtd}(h_i,t)$ and $P_{mtd}(h_i,t)$ are the respective probabilities of an attack occurring on a system of host $h_i$ for an attack duration time t without the mechanism and with it. In addition, the $CoA\left(h_i\right)$ value is the cost of the attack, which is the same in both systems. This metric evaluates the effectiveness of MTD method(s) deployed in an IoT network by comparing the defended network to a baseline model without any MTD methods. In addition, it is possible to consider attack paths and define path-based SR metrics. The SR reduction and path-based SR metrics are as follows:

• Security Risk Reduction with MTD (SRRM): The SRRM of an attack on a system in host $h_i$ with the MTD security mechanism for an attack duration time t can be obtained by subtracting Equation (15) from Equation (14):

  $$\begin{array}{cc}\hfill SRRM\left(h_i\right)& =S{R}_{no-mtd}\left(h_i\right)-S{R}_{mtd}\left(h_i\right)\hfill \\ \hfill & =({\displaystyle \frac{1}{TT{C}_{no-mtd}\left(h_i\right)}}-{\displaystyle \frac{1}{TT{C}_{mtd}\left(h_i\right)}})\times t\times CoA\left(h_i\right).\hfill \end{array}$$

  (16)

  where $t\le TT{C}_{no-mtd}\left(h_i\right)$. Equation (7) can be used to compute $TT{C}_{no-mtd}\left(h_i\right)$ and $TT{C}_{mtd}\left(h_i\right)$ with ${\theta }_i=0.0$ and $0.0<{\theta }_i\le 1.0$, respectively.
• Security Risk Reduction Percentage (SRRP): The SRRP can is expressed as a percentage, and can be obtained as follows:

  $$SRRP\left(h_i\right)={\displaystyle \frac{SRRM\left(h_i\right)}{S{R}_{no-mtd}\left(h_i\right)}}\times 100\%=(1-{\displaystyle \frac{TT{C}_{no-mtd}\left(h_i\right)}{TT{C}_{mtd}\left(h_i\right)}})\times 100\%.$$

  (17)
• Security Risk Reduction Percentage of a Network (SRRPN): The SRRPN is the network’s risk reduction compromising all hosts in the network using the MTD, and can be obtained as follows:

  $$SRRN=(1-{\displaystyle \frac{TT{C}_{no-mtd}^{tot}}{TT{C}_{mtd}^{tot}}})\times 100\%.$$

  (18)
• Security Risk on Path (SRP): The SRP metric estimates the risk associated with a given attack path. It is the sum of the security risks of the hosts on a path $a{p}_j\in A{P}_{t_k}$. The SRP of an attack path $a{p}_j\in A{P}_{t_k}$ for an attack duration time t starting at time $t_k$ is obtained as follows:

  $$\begin{array}{cc}\hfill SRP\left(a{p}_j\right)& =\sum _{i}Pr(h_i,t)\times CoA\left(h_i\right),\forall h_i\in a{p}_j\hfill \\ \hfill & =t\times \sum _i({\displaystyle \frac{1}{TT{C}_{h_i}}}\times CoA\left(h_i\right)),\forall h_i\in a{p}_j\hfill \end{array}$$

  (19)

  where $t\le TT{C}_{h_i}$.
• Security Risk on Paths of a Network (SRPN): The SRPN is the maximum security risk across all the attack paths, which can be obtained as follows:

  $$SRPN=\underset{\forall a{p}_j\in A{P}_{t_k}}{max}SRP\left(a{p}_j\right).$$

  (20)

5. Numerical Computation and Analysis

This section presents numerical computations of the proposed metrics to demonstrate their computation in evaluating the effectiveness of MTD techniques. To illustrate the computation of the proposed metrics, we consider an example IoT network comprising six IoT devices $h_0,h_1,\dots ,h_5$, where $h_0$ is an entry point and $h_5$ is a target device. Figure 4a,b presents the reachability network graphs of the example IoT network before and after the MTD shuffling operating. In the figure, each node denotes an IoT device and its edge represents the time-to-compromise from the preceding device to that device. The time to compromise a device depends on its vulnerabilities, the attacker’s skill level, and how frequently shuffling is performed by the deployed MTD techniques. Similarly, Figure 5 shows two adjacency matrices of corresponding reachability graphs before shuffling (at time t) and after shuffling (at time $t+T$), where T is the MTD shuffling interval time. The metrics are computed using these matrices with TTC values from the initial host to the target host without the MTD mechanism and with it. We assume that the attack time duration t is one day and that MTD shuffling follows some constant rate, i.e, ${\theta }_i>0.0$. In addition, we consider some given values for the cost of attack, e.g., CoA of host $h_1$ = $3000.00.

Table 2. Computed metric values.

IoT Devices	$\mathit{CoA}$	${\mathit{TTC}}_{\mathit{no}-\mathit{mtd}}$	${\mathit{TTC}}_{\mathit{mtd}}$	${\mathit{SR}}_{\mathit{no}-\mathit{mtd}}$	${\mathit{SR}}_{\mathit{mtd}}$	$\mathit{SRRP}$
$h_0$ (start)	-	-	-	-	-	-
$h_1$	$3000.00	6 days	6 days	500.00	500.00	0.00%
$h_2$	$3000.00	4 days	4 days	750.00	750.00	0.00%
$h_3$	$2000.00	5 days	13 days	400.00	153.84	61.00%
$h_4$	$2000.00	8 days	8 days	250.00	250.00	0.00%
$h_5$	$5000.00	7 days	10 days	714.28	500.00	30.00%

presents the summary of the computed metrics for each host from the entry point host $h_0$.

Computation of the path-based metrics is as follows:

$$\begin{array}{cc}\hfill ttc\left(a{p}_1\right)& =ttc(A\to h_0\to h_1\to h_3\to h_5)\hfill \\ & =6+3+2\hfill \\ & =11\mathrm{days}\hfill \\ \hfill ttc\left(a{p}_2\right)& =ttc(A\to h_0\to h_2\to h_4\to h_5)\hfill \\ & =4+5+2\hfill \\ & =11\mathrm{days}\hfill \\ \hfill ttc\left(a{p}_3\right)& =ttc(A\to h_0\to h_1\to h_4\to h_5)\hfill \\ & =6+2+2\hfill \\ & =10\mathrm{days}\hfill \\ \hfill ttc\left(a{p}_4\right)& =ttc(A\to h_0\to h_2\to h_3\to h_5)\hfill \\ & =4+1+2\hfill \\ & =7\mathrm{days}.\hfill \end{array}$$

The time-to-compromise (TTC) of target host $h_5$ using the path $a{p}_4$: $A\to h_0\to h_2\to h_3\to h_5$ is 7 days. This path yields the minimum TTC among all possible attack paths to $h_5$. The total time to compromise all the IoT devices/hosts in a network is the sum of the TTCs using the minimum TTC value. The total times to compromise all the IoT devices of the network without the MTD method and with it are 15 days and 19 days, respectively. Similarly, the security risk reduction percentage for compromising all the hosts in the network using the MTD can be obtained as shown below.

$$\begin{array}{cc}\hfill SRRN& =(1-{\displaystyle \frac{TT{C}_{no-mtd}^{tot}}{TT{C}_{mtd}^{tot}}})\times 100\%\hfill \\ \hfill & =(1-{\displaystyle \frac{15}{19}})\times 100\%\hfill \\ \hfill & =21.05\%\hfill \end{array}$$

In this example, the deployed networking shuffling MTD method reduces the overall security risk of compromising all the IoT devices/hosts in the network by 21.05%.

6. Experimental Results and Analysis

This section presents the network setting and scenario description, simulation experiment setup, and analysis of the results for the evaluation of MTD methods. The proposed metrics are used to evaluate the effectiveness of deploying MTD methods and analyze results by varying the number of vulnerabilities, shuffling rates, and attacker skill levels.

6.1. Network Setting and Scenario Description

Simulation experiments were conducted considering the scenarios and settings discussed in the system model (Section 3). Figure 6 shows a network graph of the IoT network considered for the simulation experiments. The network consists of six different IoT devices interconnected through a gateway/router ($h_0$) along with an attacker externally connected to this router. The router is assumed to run the OpenWRT operating system, and represents entry point to the network. The smartphone ($h_2$) and smart camera ($h_1$) are directly connected to the router, and operate on Android and embedded Linux, respectively. The target device in this setup is the smart door lock ($h_5$), which runs on FreeRTOS and is accessible via two intermediate devices: the smart TV ($h_4$) and smart speaker ($h_3$). The smart TV uses WebOS, while the smart speaker is based on Google Cast OS. Initially, the network topology permits four distinct paths to reach the target device.

In this setting, the IoT network system (smart home system) is protected by one or more shuffling- and/or diversity-based MTD mechanism(s), which dynamically reconfigure or shuffle configurations of each host or device $h_i$, such as the IP address, operating system/application software, or network topology using a specific shuffling rate ${\theta }_i$. Our simulation experiments used real-world data from a public vulnerability database [47], considering 122,774 unique vulnerabilities and 4,333 known exploits [37]. Attackers were assumed to be external to the network and to attack the network using their skills along with any available tools and techniques, as discussed in Section 3.2. The effectiveness of deploying different MTD methods was evaluated while varying key parameters such as the distribution of vulnerabilities, MTD shuffling rate, and attacker skill levels.

6.2. Results and Analysis

Evaluating MTD methods by varying vulnerabilities: Figure 7 presents the performance evaluation of the MTD mechanism(s) in terms of time-to-compromise (TTC) under varying vulnerability distributions and with different shuffling rates for different attacker skill levels. The results indicate that the TTC increases as the number of vulnerabilities in the network increases irrespective of the attacker’s skill level. This is because a higher number of vulnerabilities introduces more uncertainty for the attackers. Furthermore, the shuffling rate has a significant impact on the effectiveness of the MTD strategy. Our simulations explored shuffling intervals of 2, 5, 7, and 15 days. The results indicate that more frequent shuffling (i.e., every two days) significantly increases the TTC compared to less frequent shuffling (e.g., every fifteen days). In particular, shuffling intervals of 15 days or less consistently improve the IoT system’s resilience, making it harder and more time-consuming for attackers to successfully compromise the network. These findings suggest that frequent reconfiguration through MTDs is critical to enhancing security posture, especially in IoT environments with a high number of vulnerabilities or that face skilled adversaries.

Evaluating MTD methods with different shuffling rates: Figure 8 shows a performance comparison of MTD mechanisms using the time-to-compromise (TTC) metric across varying attacker skill levels and shuffling rates. These results demonstrate that the TTC increases with higher shuffling frequencies, indicating the high impact of the reconfiguration rate. In addition, the results show that attacker skill level plays a critical role. Low-skill (beginner) attackers experience a substantial increase in TTC even with only moderate shuffling, while highly skilled attackers are only significantly affected when the shuffling rate is very high. This indicates that defending against advanced adversaries requires frequent shuffling in order to effectively delay or prevent successful compromise.

Evaluating MTD methods with different attacker skills levels: Figure 9 presents a comparative analysis of MTD performance with time-to-compromise (TTC) metrics for different attacker skill levels. The results show a clear distinction in TTC values based on attacker expertise. Specifically, beginner-level attackers require approximately 8 to 158 days to successfully compromise a device/host, while intermediate attackers take between 30 to 62 days. In contrast, expert attackers exhibit significantly shorter TTC values ranging from approximately 15 to 30 days. These findings highlight that attacker proficiency substantially influences the effectiveness of the MTD mechanism.

Evaluating MTD methods with SRRP metric: Figure 10 evaluates the effectiveness of the deployed MTD mechanisms using the SRRP metric when varying the shuffling rate and with different attacker skill levels. The results indicate a consistent decline in MTD effectiveness as the shuffling interval increases. Specifically, daily shuffling yields the highest performance, reducing security risk by approximately 90% for all attacker skill levels. However, this effectiveness drops significantly to approximately 25–35% when the shuffling interval increases from one day to two days. This sharp decline highlights the sensitivity of MTD performance to shuffling frequency, emphasizing the necessity of frequent shuffling or reconfiguration to protect IoT environments from attackers.

7. Conclusions

This work has proposed a set of metrics for measuring the effectiveness of shuffling- or diversity-based MTD methods that shuffle device configurations, protocols, links, topology, operating systems, or application software in IoT networks. The proposed metrics estimate mean time-to-compromise and security risk scores for smart devices in IoT network systems, considering both the attacker’s skill level and available attack paths. Attack path-based metrics capture the dynamic change of an IoT network’s vulnerabilities with and without MTD methods. Our simulation results show that the effectiveness of MTD techniques depends significantly on the frequency of shuffling and the attacker’s skill level.

In future work, we plan to (1) develop context-aware metrics that incorporate contextual information such as device mobility, usage pattern, etc., in order to provide more adaptive and dynamic assessment of MTD methods; (2) design metrics that evaluate the energy consumption, computation overhead, and communication cost introduced by deploying MTD methods; and (3) investigate how machine learning models can be utilized to analyze system behavior and evolving pattern by using historical attack data in real-time settings.