Energy Efficient Cooperative Computation Algorithm in Energy Harvesting Internet of Things

Abstract

The limited battery capacity of Internet of Things (IoT) devices is a major deployment barrier for IoT-based computing systems. In this paper, we propose an energy efficient cooperative computation algorithm (EE-CCA). In an EE-CCA, a pair of IoT devices decide whether to offload some parts of the task to the opponent by considering their energy levels and the task deadline. To minimize the energy outage probability while completing most of tasks before their deadlines, we formulate a constraint Markov decision process (CMDP) problem and the optimal offloading strategy is obtained by linear programming (LP). Meanwhile, an optimization problem of finding pairs of IoT devices (i.e., IoT device pairing problem) is formulated under the optimal offloading strategy. Evaluation results demonstrate that the EE-CCA can reduce the energy outage probability up to $78\%$ compared with the random offloading scheme while completing tasks before their deadlines with high probability.

1. Introduction

From the recent advancement of Internet of Things (IoT) devices with high computing power, complicated computation can be handled without remote servers [1]. However, the development speed of batteries for IoT devices is inferior to that of computing module, and thus the limited battery capacity of IoT devices is being a major deployment barrier for IoT-based computing systems. Therefore, there is an increasing interest on the energy harvesting technique that converts wasted energy to electricity [2,3]. With this technique, IoT devices do not need to recharge and/or replace their batteries anymore, and thus the operating expenditure of IoT-based computing systems can be reduced [4]. However, the energy that can be harvested from external energy sources is generally uncontrollable and intermittent. Moreover, the harvested energy volume has temporal and spatial variations. Therefore, it is difficult to provide a reliable power supply to IoT devices. In this situation, if an IoT device cannot harvest energy for a long time and it processes lots of tasks requiring high computing power, its energy can be depleted. To mitigate this problem and improve the energy efficiency of harvesting IoT devices, a number of works (e.g., sleep scheduling, CPU cycle adjustment, and so on) have been investigated in the literature [5,6,7,8,9,10]. One of the possible solutions is offloading tasks to nearby IoT devices [5,6,7]. IoT-based computing systems have advantages compared to remote servers-based offloading systems. For example, offloading to remote servers consumes huge resources in networks when IoT devices generate lots of tasks. In addition, longer latency is needed when offloading to remote servers. Especially when an energy-scarce IoT device offloads tasks to a nearby energy-abundant IoT device, energy depletion of the energy-scarce IoT devices probably does not occur. However, unplanned offloading can cause another energy depletion. For example, when an offloader (i.e., an energy-scarce IoT device) always offloads all tasks to an offloadee (i.e., an energy-abundant IoT device) having lots of own tasks and/or small harvesting rate, the energy of the offloadee can be depleted within a short duration. Moreover, tasks cannot be completed within their deadlines due to the high load of the offloadee. Therefore, a sophisticated offloading algorithm should be devised.

In this paper, we propose an energy efficient cooperative computation algorithm (EE-CCA). In an EE-CCA, each IoT device is paired to its partner and a pair of IoT devices conduct cooperative computing. Specifically, a centralized controller collects information such as the distribution about temporal and spatial variations of external energy sources, the task occurrence rates of IoT devices, and the energy levels of IoT devices. Based on this information, the controller can construct and distribute offloading decision tables to IoT devices. Then, when a task occurs in IoT devices, they decide whether to offload some parts of the task to the opponent by following the decision tables. To minimize the energy outage probability while completing most of the tasks before their deadlines, we formulate a constraint Markov decision process (CMDP) problem, and the optimal offloading strategy is obtained by linear programming (LP). Meanwhile, an optimization problem of finding pairs of IoT devices (i.e., IoT device pairing problem) is formulated under the optimal offloading strategy. Evaluation results demonstrate that the EE-CCA can reduce the energy outage probability up to $78\%$ compared with the random offloading scheme while completing tasks before their deadlines with high probability. In addition, the EE-CCA operates adaptively even when the operating environment (e.g., inter-task occurrence rate) changes.

The contribution of this paper can be summarized as follows: (1) we develop the cooperative computation algorithm called EE-CCA for IoT devices, while optimizing the EE-CCA by means of CMDP formulation; (2) optimal pairs of IoT devices are decided based on the optimization problem; and (3) extensive evaluation results are presented and analyzed under various environments, providing valuable guidelines for the design of cooperative computing in energy harvesting IoT.

The remainder of this paper is organized as follows. Related works are summarized in Section 2, and the EE-CCA is described in Section 3. The CMDP model for cooperative computing and the optimization problem for the IoT device pairing are developed in Section 4. Evaluation results are given in Section 5, and followed by the concluding remarks in Section 6.

2. Related Works

A number of studies on the computation offloading have been conducted to improve the energy efficiency of energy-constraint devices [11,12,13,14,15,16,17,18,19,20,21,22]. These can be categorized into: (1) framework design [11,12,13,14,15,16]; and (2) algorithm design [17,18,19,20,21,22].

Wang et al. [11] proposed an evolutionary mobile network architecture called MobiScud that integrates the cloud services into the mobile networks by means of software defined networking (SDN) and network function virtualization (NFV) technologies in a backwards compatible fashion. Tong et al. [12] organized edge cloud servers into a hierarchical architecture which enables aggregation of the peak loads across different tiers of cloud servers. Specifically, when the loads exceed the capacities of lower tiers of edge cloud servers, they can be aggregated and offloaded by other servers at higher tiers in the edge cloud hierarchy to maximize the amount of mobile workloads being served. Taleb and Ksentini [13] introduced a follow me cloud concept that enables mobile cloud services to follow their respective mobile devices by migrating services to the optimal cloud. Liu et al. [14] proposed convergence of cloud and cellular systems, abbreviated as CONCERT, based on a concept of control/data plane decoupling and hierarchically placement of the resources within the network to manage flexibly and elastically networks and cloud services. Puente et al. [15] presented a seamless approach for the deployment of edge clouds where conventional mobile traffic and computing related traffic are segregated and handled individually at base stations. However, since these works do not consider the device-to-device offloading, IoT devices with high computing power cannot be exploited efficiently. Shukla and Munir [16] proposed a computation offloading architecture where an IoT device first tries to offload tasks to another IoT device instead of directly offloading to the cloud to process the huge amount of data while guaranteeing the task completion before the deadline. However, they did not provide any optimization method.

Ko et al. [17] proposed a spatial and temporal computation offloading decision algorithm where an energy-constraint device decides where and when to process tasks by means of a Markov decision process (MDP) by considering the energy consumption and the transmission cost. Zhao et al. [18] developed an optimization problem whose objective function is to maximize the probability that task execution satisfies the given delay bound. The problem was proved to be concave, and an optimal algorithm was proposed. Tang and Chen [19] studied a social-aware computation offloading game and designed a distributed computation offloading algorithm to achieve the Nash equilibrium. Similarly, Chen et al. [20] modeled a multi-user computation offloading game and designed a distributed computation offloading algorithm that can achieve the Nash equilibrium of the game. Zheng et al. [21] formulated the mobile users’ offloading decision process under a dynamic environment as a stochastic game. Then, they proposed a multi-agent stochastic learning algorithm that can run in a fully distributed manner without any information exchange. Yu et al. [22] developed an optimal collaborative offloading strategy under a distributed caching scenario. Specifically, they formulated a problem of users’ allocation as a coalition formation game with the consideration of relationships between the offloading and caching and then proposed an optimal offloading with a caching-enhancement scheme. These works improve the performance of computation offloading; however, no previous studies optimize the performance of IoT-based computing systems.

3. Energy Efficient Cooperative Computation Algorithm (EE-CCA)

Figure 1 shows the system model of this paper. In our system model, there are N IoT devices with the energy harvesting capability. We assume heterogeneous IoT environment, where IoT devices have different computing power and current energy level. In real systems, some IoT devices may not have sufficient computing power and/or harvesting capability. To support this situation, a study on the robustness of the proposed algorithm (e.g., a resilient multiscale coordination control [23]) should be conducted, which is one of our future works. In addition, since they are installed at different spots and the condition of external energy sources is volatile, energy volumes that can be harvested at each IoT device are different from each other. In addition, with different rates, these IoT devices periodically generate tasks that can be abstracted into the input data and the completion deadline before which the task should be completed [24]. In addition, we consider an application where the input data do not have dependency, e.g., binomial classification that determines whether each input is larger than a given threshold or not, and therefore the input data can be partitioned and offloaded. When the task occurs in a particular IoT device, it decides whether to offload some parts of the task (i.e., input data) to a neighbor IoT device or not with the consideration of the energy level and the deadline of the task. Note that, even though the formulation of this paper is based on the assumption where IoT devices can offload all or half of the tasks, it can be easily extended to consider other portions of the task.

Intuitively, when an IoT device offloads some parts of the task to a neighbor IoT device, the task can be processed in a distributed manner, which can reduce the energy consumption of the task owner. Since IoT devices offload their tasks to nearby IoT devices by exploiting transmission technologies with low power consumption (e.g., Bluetooth), the energy consumption for transmission of the task can be neglected compared to that for processing the task. Moreover, if the neighbor IoT device does not have its own task, the task completion time can be shortened. However, if the task is offloaded to an energy-scarce IoT device, it causes the energy depletion of the IoT device, and then the offloaded task cannot be processed due to the energy depletion. In addition, when the neighbor IoT device has its own task, it should process its own task and the offloaded task simultaneously, and thus its processing time can increase. Then, both tasks may not be completed within their deadlines. To prevent these situations, we propose the EE-CCA and its flow chart is shown in Figure 2. First, the controller collects and/or maintains information such as the distribution about temporal and spatial variations of external energy sources, the task occurrence rates of IoT devices, and the energy levels of IoT devices (Step 1 in Figure 2). Based on this information, the controller determines appropriate partners of IoT devices for cooperative computation and transmits the pairing information to IoT devices (Step 2 in Figure 2). In addition, the controller constructs an offloading decision table consisting of the current status and the operation in a centralized manner. If some parameters (e.g., the task occurrence rates of IoT devices) are changed, the offloading decision table can be reconstructed by the controller and transmitted to IoT devices again. Therefore, if the parameters are frequently changed, extra signaling overhead can occur. To mitigate the signaling overhead, several techniques such as aggregation and delta encoding can be exploited [25]. Note that the offloading decision table can be obtained by CMDP, which will be elaborated in Section 4. After that, the controller transmits the optimal offloading decision table to IoT devices (Step 3 in Figure 2). On the basis of this table, IoT devices can conduct the cooperative computing (i.e., decide whether to offload some parts of the task or not) (Step 4 in Figure 2). By means of table deployments in IoT devices, the CMDP model can be applied to resource-constrained IoT devices without any high computation overhead in IoT devices [26].

4. Constraint Markov Decision Process (CMDP)

To obtain the optimal offloading strategy, we formulate a CMDP model with five elements. (Since the CMDP model that is a mathematical framework to model decision-making when outcomes need to be constrained, and they are partially random and under the control of the decision maker [27], it is suitable to construct the optimal offloading decision table.): (1) decision epoch; (2) state; (3) action; (4) transition probability; and (5) cost and constraint functions. Subsequently, we convert the CMDP model to an equivalent LP problem to obtain the optimal policy. After that, the IoT device pairing problem is formulated under the optimal offloading policy. Important notations for the CMDP model and IoT device pairing problem are summarized in

Table 1. Summary of notations.

Notation	Description
$S_t$	State at the decision epoch t
$A_t$	Action chosen at the decision epoch t
$\tau$	Duration of each decision epoch
$\mathbf{S}$	Overall state space
${\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}$	State for denoting the occurrence and processing status for the task of IoT device i
${\mathbf{T}}_{\mathbf{j}}^{\mathbf{M}}$	State for denoting the occurrence and processing status for the task of IoT device j
${\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}$	State for denoting the processing status for the offloaded task of IoT device i
${\mathbf{T}}_{\mathbf{j}}^{\mathbf{O}}$	State for denoting the processing status for the offloaded task of IoT device j
${\mathbf{E}}_{\mathbf{i}}$	State for denoting the energy level of IoT device i
${\mathbf{E}}_{\mathbf{j}}$	State for denoting the energy level of IoT device j
${\mathbf{D}}_{\mathbf{i}}$	State for denoting whether the timer for the deadline of the task of IoT device i expires or not
${\mathbf{D}}_{\mathbf{j}}$	State for denoting whether the timer for the deadline of the task of IoT device j expires or not
$\mathbf{A}$	Action space
${\mathbf{A}}_{\mathbf{i}}$	Action space for IoT device i
${\mathbf{A}}_{\mathbf{j}}$	Action space for IoT device j
$E_{MAX}$	Maximum battery capacity of IoT device
$r(S,A)$	Cost function on the energy outage
$c_i(S,A)$	Constraint function on the timer expiration of IoT device i
$c_j(S,A)$	Constraint function on the timer expiration of IoT device j
${\zeta }^E$	Energy outage probability
${\xi }_i^T$	Timer expiration probability of IoT device i
${\xi }_j^T$	Timer expiration probability of IoT device j
${\theta }_i^T$	Upper limit on the timer expiration probability of IoT device i
${\theta }_j^T$	Upper limit on the timer expiration probability of IoT device j
${\zeta }_{ij}^E$	Individual energy outage probability of IoT device i when it is paired with IoT device j
$x_{ij}$	Decision variable to denote whether IoT device i is paired with IoT device j or not

.

4.1. Decision Epoch

Figure 3 shows the timing diagram for the CMDP model. A sequence $T=\{1,2,3,...\}$ represents the time epochs when successive decisions are made [28]. $S_t$ and $A_t$ denote the state and the action chosen at the decision epoch $t\in T$, respectively. $\tau$ represents the duration of each decision epoch.

4.2. State Space

We define the overall state space $\mathbf{S}$ as (The state space is constructed based on the assumption where IoT devices can offload all or half of the task. However, it can be easily extended to consider other portions of the task to add elements to ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}$, ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}$, ${\mathbf{T}}_{\mathbf{j}}^{\mathbf{M}}$, and ${\mathbf{T}}_{\mathbf{j}}^{\mathbf{O}}$.)

$$\begin{array}{c}\hfill \mathbf{S}={\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}\times {\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}\times {\mathbf{E}}_{\mathbf{i}}\times {\mathbf{D}}_{\mathbf{i}}\times {\mathbf{T}}_{\mathbf{j}}^{\mathbf{M}}\times {\mathbf{T}}_{\mathbf{j}}^{\mathbf{O}}\times {\mathbf{E}}_{\mathbf{j}}\times {\mathbf{D}}_{\mathbf{j}},\end{array}$$

(1)

where ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}$ and ${\mathbf{T}}_{\mathbf{j}}^{\mathbf{M}}$ are the states for representing the occurrence and processing status for the task of IoT devices i and j, respectively. ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}$ and ${\mathbf{T}}_{\mathbf{j}}^{\mathbf{O}}$ denote the states for the processing status of the offloaded task of IoT devices i and j, respectively. ${\mathbf{E}}_{\mathbf{i}}$ and ${\mathbf{E}}_{\mathbf{j}}$ are the states for the energy level of IoT devices i and j, respectively. ${\mathbf{D}}_{\mathbf{i}}$ and ${\mathbf{D}}_{\mathbf{j}}$ represent the states for denoting whether the timers for the deadline of the task of IoT devices i and j expire or not, respectively.

${\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}$, ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}$, ${\mathbf{E}}_{\mathbf{i}}$, and ${\mathbf{D}}_{\mathbf{i}}$ are the states for IoT device i, and these states are defined as follows.

First, ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}$ is given by

$$\begin{array}{c}\hfill {\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}=\left\{0,1,2,3,4\right\},\end{array}$$

(2)

where $T_i^M$ represents the occurrence and processing status for the task of IoT device i. In other words, $T_i^M=0$ denotes that the task does not occur in IoT device i, whereas $T_i^M=1$ refers to the situation immediately after the task occurs in IoT device i. $T_i^M=2$ represents the situation where IoT device i processes all of the task by itself. Meanwhile, $T_i^M=3$ and $T_i^M=4$ represent the situations where half of the task and all of the tasks are offloaded to IoT device j, respectively. Note that, when $T_i^M=3$, IoT device i processes the remaining half of the task.

${\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}$ is represented by

$$\begin{array}{c}\hfill {\mathbf{T}}_{\mathbf{i}}^{\mathbf{O}}=\left\{0,1,2\right\},\end{array}$$

(3)

where $T_i^O$ represents the processing status of the offloaded task of IoT device i. Specifically, $T_i^O=0$ describes the situation where any task is not offloaded to its partner (i.e., IoT device j). Meanwhile, $T_i^O=1$ and $T_i^O=2$ represent the situation where half of the task and all of the tasks are offloaded to IoT device j, respectively, and it is being processed in IoT device j.

${\mathbf{E}}_{\mathbf{i}}$ is represented as [29]

$$\begin{array}{c}\hfill {\mathbf{E}}_{\mathbf{i}}=\left\{0,1,2,\dots E_{MAX}\right\},\end{array}$$

(4)

where $E_{MAX}$ is the maximum battery capacity of an IoT device.

$$\begin{array}{c}\hfill {\mathbf{D}}_{\mathbf{i}}=\left\{0,1\right\},\end{array}$$

(5)

where $D_i$ denotes whether the timer for the deadline of the task of IoT device i expires or not. In other words, $D_i=0$ and $D_i=1$ represent that the timer for the deadline of the task of IoT device i does not expire and expires, respectively.

${\mathbf{T}}_{\mathbf{j}}^{\mathbf{M}}$, ${\mathbf{T}}_{\mathbf{j}}^{\mathbf{O}}$, ${\mathbf{E}}_{\mathbf{j}}$, and ${\mathbf{D}}_{\mathbf{j}}$ are the states for IoT device j, and these states can be defined as similar with the states for IoT device i. These definitions are omitted in this paper due to the page limitation and for simple descriptions, which can be found in [30].

4.3. Action Space

When the task occurs in IoT devices, each IoT device can decide whether to offload to its partner or not and the portion to be offloaded based on the current state information. The action set is constructed based on the assumption where IoT devices can offload all or half of the task. However, it can be easily extended to consider other portions of the task to define additional actions. Therefore, the action set can be described by

$$\begin{array}{c}\hfill \mathbf{A}={\mathbf{A}}_{\mathbf{i}}\times {\mathbf{A}}_{\mathbf{j}},\end{array}$$

(6)

where ${\mathbf{A}}_{\mathbf{i}}$ and ${\mathbf{A}}_{\mathbf{j}}$ are the action spaces for IoT devices i and j, respectively, which can be defined as

$$\begin{array}{c}\hfill {\mathbf{A}}_{\mathbf{i}}=\left\{0,1,2\right\}\end{array}$$

(7)

and

$$\begin{array}{c}\hfill {\mathbf{A}}_{\mathbf{j}}=\left\{0,1,2\right\},\end{array}$$

(8)

where $A_i=0$ and $A_j=0$ represent that IoT devices i and j do not offload its task, respectively. $A_i=1$ and $A_j=1$ denote that IoT devices i and j offload half of the task to its partner, respectively. In addition, $A_i=2$ and $A_j=2$ are the actions where IoT devices i and j offload all of the task to its partner, respectively.

4.4. Transition Probability

The state transition probability of IoT device i is affected by the state of IoT device j. Specifically, the processing speed of the task occurring in IoT device i (i.e., the transition probability of ${\mathbf{T}}_{\mathbf{i}}^{\mathbf{M}}$) is dependent on whether the task occurring at IoT device j is processed in IoT device i or not (i.e., $T_j^O$). In addition, the transition probability of $T_i^O$ is affected by whether IoT device j processes its own task or not (i.e., $T_j^M$). Similar to that, the state transition probability of IoT device j is also influenced by the state of IoT device i (especially $T_i^M$ and $T_i^O$). Therefore, the transition probability with the chosen action A from the current state S to the next state $S^{\prime }$ can be described by

$$\begin{array}{c}\hfill P\left[S^{\prime }\right|S,A]=P\left[S_i^{\prime }\right|S_i,T_j^M,T_j^O,A]\times P\left[S_j^{\prime }\right|S_j,T_i^M,T_i^O,A],\end{array}$$

(9)

where $S_i^{\prime }$ and $S_j^{\prime }$ denote the next state of IoT devices i and j, respectively. In addition, $S_i$ and $S_j$ represent the current state for IoT devices i and j, respectively.

Meanwhile, $T_i^M$ and $T_i^O$ are influenced by the chosen action A, and these states are dependently changed with each other. In addition, $T_i^M$ is affected by $T_j^O$. For example, when the task of IoT device j is processed in IoT device i, the processing speed of the task of IoT device i can decrease. Similarly, $T_i^O$ is influenced by $T_j^M$. For example, when IoT device j does not process its own task, it can focus on processing the offloaded task from IoT device i, and therefore the offloaded task can be completed within a short duration. Meanwhile, when the task is processed in IoT device i, its energy level can decrease. That is, the transition of $E_i$ is influenced by $T_i^M$. The timer for the deadline of the task operates only when the task occurs, and therefore the transition of $D_i$ is affected by $T_i^M$ and $T_i^O$. Meanwhile, other states change independently of each other. Therefore, for the chosen action A, the transition probability from the current state of IoT device i, $S_i=[T_i^M,T_i^O,E_i,D_i]$, to the next state of IoT device i, $S_i^{\prime }=[T^{\prime }{}_i^M,T^{\prime }{}_i^O,E_i,D_i]$, can be described by

$$\begin{array}{c}\hfill P\left[S_i^{\prime }\right|S_i,T_j^M,T_j^O,A]=P\left[T^{\prime }{}_i^M\right|T_i^M,T_i^O,T_j^O,A]\times P\left[T^{\prime }{}_i^O\right|T_i^O,T_i^M,T_j^M,A]\times P\left[E_i^{\prime }\right|E_i,T_i^M]\times P\left[D_i^{\prime }\right|D_i,T_i^M,T_i^O].\end{array}$$

(10)

We assume that the inter-task occurrence time of IoT device i follows an exponential distribution with mean $1/{\lambda }_i$. Then, the probability that the task occurs in IoT device i during a decision epoch can be calculated as ${\lambda }_i\tau$ [27,31]. Therefore, $P\left[T^{\prime }{}_i^M\right|T_i^M=0,T_i^O=0,A]$ can be represented by

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=0,T_i^O=0,T_j^O,A]=\left\{\begin{array}{c}1-{\lambda }_i\tau ,\mathrm{if}{T}^{\prime }{}_i^M=0,\hfill \\ {\lambda }_i\tau ,\mathrm{if}{T}^{\prime }{}_i^M=1,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(11)

Before receiving the result of the offloaded task, IoT device i does not generate the task. Therefore, $P\left[T^{\prime }{}_i^M\right|T_i^M=0,T_i^O\ne 0,T_j^O,A]$ can be defined as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=0,T_i^O\ne 0,T_j^O,A]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^M=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(12)

Meanwhile, when the task occurs (i.e., $T_i^M=1$), IoT device i decides whether to offload to IoT device j or not and the offloaded portion (i.e., half of the task and all of the task). If IoT device i decides not to offload it (i.e., when $A=0$), the task state will change to 2 representing the situation where IoT device i processes all of the task by itself (i.e., $T_i^M=2$). On the other hand, when IoT device i decides to offload half of the task and all of the task (i.e., when $A=1$ and $A=2$), the next states of the occurrence and processing status for the task of IoT device i (i.e., $T^{\prime }{}_i^M$) become 3 and 4, respectively. Therefore, the corresponding transition probabilities can be represented as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=1,T_i^O,T_j^O,A=0]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^M=2,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(13)

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=1,T_i^O,T_j^O,A=1]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^M=3,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(14)

and

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=1,T_i^O,T_j^O,A=2]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^M=4,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(15)

respectively.

We assume that the processing time of IoT device i for its own task follows an exponential distribution with mean $1/{\mu }_i^{F,S}$ when it processes all of its own task and any task of IoT device j is not offloaded to IoT device i (i.e., $T_i^M=2$ and $T_j^O=0$). In this case, the probability that the task is completed during a decision epoch is ${\mu }_i^{F,S}\tau$ [27,31]. Then, the probability that a task is not completed during a decision epoch is $1-{\mu }_i^{F,S}\tau$. On the other hand, if some portion of the task of IoT device j is offloaded to IoT device i (i.e., $T_j^O\ne 0$), the processing speed of IoT device i for its own task decreases, and thus it is assumed that the processing time of IoT device i for its own task follows an exponential distribution with mean $1/{\mu }_i^{F,D}$ ($>1/{\mu }_i^{F,S}$). In this case, the probability that the task is completed (or not completed) during a decision epoch is ${\mu }_i^{F,D}\tau$ (or $1-{\mu }_i^{F,D}\tau$) [27,31]. Therefore, $P\left[T^{\prime }{}_i^M\right|T_i^M=2,T_i^O,T_j^O=0,A]$ and $P\left[T^{\prime }{}_i^M\right|T_i^M=2,T_i^O,T_j^O\ne 0,A]$ can be derived as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=2,T_i^O,T_j^O=0,A]=\left\{\begin{array}{c}1-{\mu }_i^{F,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=2,\hfill \\ {\mu }_i^{F,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=0,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(16)

and

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=2,T_i^O,T_j^O\ne 0,A]=\left\{\begin{array}{c}1-{\mu }_i^{F,D}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=2,\hfill \\ {\mu }_i^{F,D}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(17)

Meanwhile, when offloading half of the task to IoT device j (i.e., $T_i^M=3$), the remained task can be completed with shorter time. It is assumed that the processing time of the remained task follows an exponential distribution with mean $1/{\mu }_i^{H,S}$ when any task of IoT device j is not offloaded to IoT device i (i.e., $T_j^O=0$), and then the probability that the remained task is completed during a decision epoch is ${\mu }_i^{H,S}\tau$ [27,31]. On the other hand, when IoT device i offloads the half of its task to IoT device j and processes the offloaded task from IoT device j (i.e., $T_i^M=3$ and $T_j^O\ne 0$), the processing time of the remained task of IoT device i follows an exponential distribution with mean $1/{\mu }_i^{H,D}$. In this case, the probability that the remained task is completed during a decision epoch is ${\mu }_i^{H,D}\tau$ [27,31]. Thus, the corresponding transition probabilities can be represented as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=3,T_i^O,T_j^O=0,A]=\left\{\begin{array}{c}1-{\mu }_i^{H,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=3,\hfill \\ {\mu }_i^{H,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=0,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(18)

and

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^M\right|T_i^M=3,T_i^O,T_j^O\ne 0,A]=\left\{\begin{array}{c}1-{\mu }_i^{H,D}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=3,\hfill \\ {\mu }_i^{H,D}\tau ,\mathrm{if}{T}^{\prime }{}_i^M=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(19)

When the task does not occur, the processing status of the offloaded task of IoT device i does not change. Therefore, $P\left[T^{\prime }{}_i^O\right|T_i^O=0,T_i^M\ne 1,A]$ can be denoted as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=0,T_i^M\ne 1,T_j^M,A]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^O=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(20)

Meanwhile, when the task occurs (i.e., $T_i^M=1$), the processing status of the offloaded task of IoT device i changes according to the chosen action A. Therefore, corresponding transition probabilities can be represented as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=0,T_i^M=1,T_j^M,A=0]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^O=0,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(21)

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=0,T_i^M=1,T_j^M,A=1]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^O=1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(22)

and

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=0,T_i^M=1,T_j^M,A=2]=\left\{\begin{array}{c}1,\mathrm{if}{T}^{\prime }{}_i^O=2,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(23)

respectively.

When some portion of the task is offloaded (i.e., $T_i^O=1$ or $T_i^O=2$), it is processed by IoT device j. Meanwhile, the processing time of the offloaded task to IoT device j depends on the portion of the offloaded task (i.e., $T_i^O$) and whether IoT device j processes its own task or not (i.e., $T_j^M$). Specifically, when half of the task (or all of the task) of IoT device i is offloaded and IoT device j does not process its own task, we assume that the processing time of the offloaded task follows an exponential distribution with mean $1/{\mu }_j^{H,S}$ ($1/{\mu }_j^{F,S}$). On the other hand, if half of the task (or all of the task) of IoT device i is offloaded and IoT device j processes its own task, the processing time of the offloaded task follows an exponential distribution with mean $1/{\mu }_j^{H,D}$ ($1/{\mu }_j^{F,D}$). Then, the probabilities that the offloaded task is completed during a decision epoch for each case can be derived as ${\mu }_j^{H,S}\tau$, ${\mu }_j^{F,S}\tau$, ${\mu }_j^{H,D}\tau$, and ${\mu }_j^{F,D}\tau$, respectively [27,31]. Therefore, the corresponding transition probabilities can be denoted as

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=1,T_i^M,T_j^M=0,A]=\left\{\begin{array}{c}1-{\mu }_j^{H,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=1,\hfill \\ {\mu }_j^{H,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=0,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(24)

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=2,T_i^M,T_j^M=0,A]=\left\{\begin{array}{c}1-{\mu }_j^{F,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=1,\hfill \\ {\mu }_j^{F,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=0,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(25)

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=1,T_i^M,T_j^M\ne 0,A]=\left\{\begin{array}{c}1-{\mu }_j^{H,D}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=1,\hfill \\ {\mu }_j^{H,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=0,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(26)

and

$$\begin{array}{c}\hfill P\left[T^{\prime }{}_i^O\right|T_i^O=2,T_i^M,T_j^M\ne 0,A]=\left\{\begin{array}{c}1-{\mu }_j^{F,D}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=1,\hfill \\ {\mu }_j^{H,S}\tau ,\mathrm{if}{T}^{\prime }{}_i^O=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(27)

The IoT device can harvest energy only when its environments provide energy (e.g., when the wind blows above a certain speed). Therefore, the probability that IoT device i harvests one unit energy at an arbitrary decision epoch is modeled by a Bernoulli random process with the probability $p_i^H$ [32]. Then, when the IoT device i does not process any task (i.e., $T_i^M=0$ or $T_i^M=1$) and its battery is not fully charged (i.e., $E_i\ne E_{MAX}$), $E_i$ increases by one unit with the probability $p_i^H$. If the battery of IoT device i is full (i.e., $E_i=E_{MAX}$), it cannot harvest energy anymore. Therefore, the corresponding transition probabilities can be represented as

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i\ne E_{MAX},T_i^M=0]=\left\{\begin{array}{c}1-p_i^H,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ p_i^H,\mathrm{if}{E}_i^{\prime }=E_i+1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(28)

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i=E_{MAX},T_i^M=0]=\left\{\begin{array}{c}1,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(29)

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i\ne E_{MAX},T_i^M=1]=\left\{\begin{array}{c}1-p_i^H,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ p_i^H,\mathrm{if}{E}_i^{\prime }=E_i+1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(30)

and

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i=E_{MAX},T_i^M=1]=\left\{\begin{array}{c}1,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(31)

When IoT device i processes the task (i.e., $T_i^M=2$ or $T_i^M=3$) and it has energy (i.e., $E\ne 0$), it consumes one unit energy. On the other hand, if IoT device i does not have any energy, it cannot process for any task the sensed data, and thus no energy is consumed. In addition, its energy $E_i$ increases by one unit with the probability $p_i^H$. Therefore, the corresponding transition probabilities can be expressed by

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i\ne 0,T_i^M=2]=\left\{\begin{array}{c}1-p_i^H,\mathrm{if}{E}_i^{\prime }=E_i-1,\hfill \\ p_i^H,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(32)

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i=0,T_i^M=2]=\left\{\begin{array}{c}1,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(33)

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i\ne 0,T_i^M=3]=\left\{\begin{array}{c}1-p_i^H,\mathrm{if}{E}_i^{\prime }=E_i-1,\hfill \\ p_i^H,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(34)

and

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i=0,T_i^M=3]=\left\{\begin{array}{c}1,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(35)

Meanwhile, when all of the tasks are offloaded to IoT device j (i.e., $T_i^M=4$), IoT device i does not consume its own energy. Therefore, the corresponding transition probability can be denoted as

$$\begin{array}{c}\hfill P\left[E_i^{\prime }\right|E_i,T_i^M=4]=\left\{\begin{array}{c}1,\mathrm{if}{E}_i^{\prime }=E_i,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(36)

When the task does not occur (i.e., $T_i^M=0$ or $T_i^O=0$), the timer for the deadline of the task does not start, and therefore it does not expire. Therefore, $P\left[D_i^{\prime }\right|D_i,T_i^M=0,T_i^O=0]$ can be represented as

$$\begin{array}{c}\hfill P\left[D_i^{\prime }\right|D_i,T_i^M=0,T_i^O=0]=\left\{\begin{array}{c}1,\mathrm{if}{D}_i^{\prime }=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(37)

We assume that the timer for the deadline of the task of IoT device i follows an exponential distribution with mean $1/{\kappa }_i$ [33,34]. Then, when the task is not completed (i.e., $T_i^M\ne 0$ or $T_i^O\ne 0$), the probability that the timer expires during a decision epoch is ${\kappa }_i\tau$. Thus, the corresponding transition probabilities can be represented as

$$\begin{array}{c}\hfill P\left[D_i^{\prime }\right|D_i=0,T_i^M\ne 0,T_i^O]=\left\{\begin{array}{c}1-{\kappa }_i\tau ,\mathrm{if}{D}_i^{\prime }=0,\hfill \\ {\kappa }_i\tau ,\mathrm{if}{D}_i^{\prime }=1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(38)

and

$$\begin{array}{c}\hfill P\left[D_i^{\prime }\right|D_i=0,T_i^M,T_i^O\ne 0]=\left\{\begin{array}{c}1-{\kappa }_i\tau ,\mathrm{if}{D}_i^{\prime }=0,\hfill \\ {\kappa }_i\tau ,\mathrm{if}{D}_i^{\prime }=1,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(39)

Meanwhile, when the task is completed (i.e., $T_i^M=0$ and $T_i^O=0$), the timer is reset and does not operate, which means that there is no expiration. Therefore, $P\left[{D_i}^{\prime }\right|D_i=1,T_i^M=0,T_i^O=0]$ can be denoted as

$$\begin{array}{c}\hfill P\left[D_i^{\prime }\right|D_i=1,T_i^M=0,T_i^O=0]=\left\{\begin{array}{c}1,\mathrm{if}{D}_i^{\prime }=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(40)

If the timer expires (i.e., $D_i=1$) and the task is not completed (i.e., $T_i^M\ne 0$ or $T_i^O\ne 0$), the timer remains in the expired state. Therefore, the corresponding transition probabilities can be represented as

$$\begin{array}{c}\hfill P\left[D_i^{\prime }\right|D_i=1,T_i^M\ne 0,T_i^O]=\left\{\begin{array}{c}1,\mathrm{if}{D}_i^{\prime }=1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(41)

and

$$\begin{array}{c}\hfill P\left[{D_i}^{\prime }\right|D_i=1,T_i^M,T_i^O\ne 0]=\left\{\begin{array}{c}1,\mathrm{if}{D}_i^{\prime }=1,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(42)

The transition probability for the states of IoT device j can be defined as similar to that of IoT device i. These are omitted in this paper due to the page limitation and for simple descriptions, which can be found in [30].

4.5. Cost and Constraint Functions

4.5.1. Cost Function

To define the cost function, we consider the energy outage of IoT devices. The energy outage occurs when batteries of IoT devices are empty. Therefore, the cost function can be defined as

$$\begin{array}{c}\hfill r\left(S,A\right)=\left\{\begin{array}{c}1,\mathrm{if}{E}_i=0,\hfill \\ 1,\mathrm{if}{E}_j=0,\hfill \\ 0,\mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(43)

4.5.2. Constraint Function

To prevent the situation where the task cannot be finished before the timer expiration, the constraint functions for the timer expiration of IoT devices i and j can be represented by

$$\begin{array}{c}\hfill c_i\left(S,A\right)=\left\{\begin{array}{c}1,\mathrm{if}{D}_i=1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(44)

and

$$\begin{array}{c}\hfill c_j\left(S,A\right)=\left\{\begin{array}{c}1,\mathrm{if}{D}_j=1,\hfill \\ 0,\mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(45)

respectively.

4.6. Optimization Problem Formulation

Since $\sum _{t^{\prime }}^t\mathrm{E}\left[r\left(S_{t^{\prime }},A_{t^{\prime }}\right)\right]$ means the number of energy outages, the average energy outage probability ${\zeta }^E$ can be defined as

$$\begin{array}{c}\hfill {\zeta }^E=\underset{t\to \infty }{lim}sup\frac{1}{t}\sum _{t^{\prime }}^t\mathrm{E}\left[r\left(S_{t^{\prime }},A_{t^{\prime }}\right)\right],\end{array}$$

(46)

where lim denotes the value that a function approaches as the input approaches a specific value. In addition, sup (i.e., supremum) means the least upper bound.

Meanwhile, the average timer expiration probabilities of IoT devices i and j, denoted as, ${\xi }_i^T$, and ${\xi }_j^T$, respectively, can be defined as

$$\begin{array}{c}\hfill {\xi }_i^T=\underset{t\to \infty }{lim}sup\frac{1}{t}\sum _{t^{\prime }}^t\mathrm{E}\left[c_i\left(S_{t^{\prime }},A_{t^{\prime }}\right)\right]\end{array}$$

(47)

and

$$\begin{array}{c}\hfill {\xi }_j^T=\underset{t\to \infty }{lim}sup\frac{1}{t}\sum _{t^{\prime }}^t\mathrm{E}\left[c_j\left(S_{t^{\prime }},A_{t^{\prime }}\right)\right].\end{array}$$

(48)

Then, the optimization problem in the CMDP model can be formulated as

$$\begin{array}{c}\hfill \underset{\pi }{min}{\zeta }^E,\end{array}$$

(49)

$$\begin{array}{c}\hfill \mathrm{s}.\mathrm{t}.{\xi }_i^T\le {\theta }_i^T\mathrm{and}{\xi }_j^T\le {\theta }_j^T,\end{array}$$

(50)

where ${\theta }_i^T$ and ${\theta }_j^T$ are the upper limits on the timer expiration probabilities of IoT devices i and j, respectively.

The formulated optimization problem can be transformed into an equivalent LP problem [28]. That is, when $\varphi (S,A)$ represents the stationary probability of state S and action A, the solution of the LP problem ${\varphi }^{*}(S,A)$ can be mapped to that of the CMDP-based optimization model. The equivalent LP model can be expressed as

$$\begin{array}{c}\hfill \underset{\varphi (S,A)}{max}\sum _S\sum _A\varphi (S,A)r(S,A),\end{array}$$

(51)

$$\begin{array}{c}\hfill \mathrm{s}.\mathrm{t}.\sum _S\sum _A\varphi (S,A)c_i(S,A)\le {\theta }_i^T,\end{array}$$

(52)

$$\begin{array}{c}\hfill \sum _S\sum _A\varphi (S,A)c_j(S,A)\le {\theta }_j^T,\end{array}$$

(53)

$$\begin{array}{c}\hfill \sum _A\varphi (S^{\prime },A)=\sum _S\sum _A\varphi (S,A)P\left[S^{\prime }\right|S,A],\end{array}$$

(54)

$$\begin{array}{c}\hfill \sum _S\sum _A\varphi (S,A)=1,\end{array}$$

(55)

$$\begin{array}{c}\hfill \varphi (S,A)\ge 0.\end{array}$$

(56)

The objective function in (51) is to minimize the energy outage probability of IoT devices. Meanwhile, the constraints in (52) and (53) are to maintain the timer expiration probabilities of IoT devices i and j below ${\theta }_i^T$ and ${\theta }_i^T$, respectively. The constraint in (54) satisfies the Chapman–Kolmogorov equation. The constraints in (55) and (56) are for the probability properties.

The optimal policy ${\pi }^{*}(S,A)$, which is the probability of taking a particular action at a certain state, can be obtained from the solution of the above LP problem. The optimal policy can be derived from

$$\begin{array}{c}\hfill {\pi }^{*}\left(S,A\right)=\frac{{\varphi }^{*}\left(S,A\right)}{\sum _{A^{\prime }}{\varphi }^{*}\left(S,A^{\prime }\right)}\mathrm{for}S\in \mathbf{S},\sum _{A^{\prime }}{\varphi }^{*}\left(S,A^{\prime }\right)>0.\end{array}$$

(57)

Note that, if $\sum _{A^{\prime }}{\varphi }^{*}\left(S,A^{\prime }\right)=0$, which means that there is no solution to satisfy all constraints, IoT devices do not offload any task. LP problem can be solved in polynomial time [35,36,37]. Therefore, our proposed algorithm can be implemented to real systems without high computational power.

4.7. IoT Device Pairing Problem

The optimal offloading policy in the previous subsection is obtained given the paired IoT devices i and j. In this subsection, we formulate an IoT device pairing problem whose objective is to minimize the summation of energy outage probabilities of all IoT devices.

Let ${\zeta }_{ij}^E$ denote an individual energy outage probability of IoT device i when it is paired with IoT device j. ${\xi }_{ij}^E$ can be calculated as (We assume that the individual energy outage probability of IoT device i when it is paired with IoT device j is lower than that when it is not paired (i.e., ${\xi }_{ij}^E<E_{ii}^E$). This assumption is reasonable because paired IoT devices operate by following the optimal policy obtained by CMDP:

$$\begin{array}{c}\hfill {\zeta }_{ij}^E=\sum _S\sum _A{\pi }^{*}\left(\left[T_i^M,T_i^O,E_i=0,D_i,S_j\right],A\right).\end{array}$$

(58)

Then, the optimization problem for pairing IoT devices can be defined as

$$\begin{array}{c}\hfill \underset{x_{ij}}{min}\sum _i{\zeta }_{ij}^E{x}_{ij},\end{array}$$

(59)

$$\begin{array}{c}\hfill \mathrm{s}.\mathrm{t}.\sum _j{x}_{ij}=1,\forall i,\end{array}$$

(60)

where $x_{ij}$ is a decision variable that is 1 if IoT devices i and j are paired and 0 otherwise. The objective function in (59) is to minimize the summation of energy outage probabilities of all IoT devices. Meanwhile, the constraint in (60) to ensure that all IoT devices are paired with only one IoT device. This optimization problem is solved in the controller by using several algorithms (e.g., brute-force approach, LP relaxation, and branch-and-bound algorithm), and therefore there is no burden in IoT devices.

5. Evaluation Results

For performance evaluation, we compare the proposed algorithm, EE-CCA, with the following four schemes: (1) ALL where IoT devices always offload all of the tasks; (2) HALF where IoT devices always offload half of the task; (3) NON where IoT devices do not offload any task (i.e., process their tasks by themselves); and (4) RAND where IoT devices randomly offload their tasks at each decision epoch. For pair comparison, IoT devices are paired based on the solution of the optimization problem in Section IV-G. Meanwhile, the objective of this paper is to minimize the energy outage probability while maintaining the probability that the task is completed before their deadline above a certain level. Therefore, the average energy outage probability, ${\zeta }^E$, and the average probabilities that the task of IoT device i and j are completed before their deadline, ${\eta }_i$ and ${\eta }_j$, are used as performance measures of the EE-CCA. Note that ${\eta }_i$ and ${\eta }_j$ can be calculated by $1-{\xi }_i^T$ and $1-{\xi }_j^T$, respectively.

To improve the reliability of the simulation results, we have conducted over 10,000 simulation runs with different seed values independently. The default number of IoT devices is set to 6. The other default parameter settings are summarized in

Table 2. Default parameter settings.

$\mathit{\tau }$	$\mathit{\lambda }$	${\mathit{\mu }}^{\mathit{F},\mathit{S}}$	${\mathit{\mu }}^{\mathit{F},\mathit{D}}$	${\mathit{\mu }}^{\mathit{H},\mathit{S}}$	${\mathit{\mu }}^{\mathit{H},\mathit{D}}$	${\mathit{p}}^{\mathit{H}}$	$\mathit{\kappa }$	${\mathit{\theta }}^{\mathit{T}}$
1	[0.2 0.3]	[0.3 0.4]	[0.15 0.2]	[0.6 0.8]	[0.3 0.4]	[0.4 0.9]	[0.2 0.3]	0.99

, where $\left[ab\right]$ denotes a random value between a and b.

5.1. Effect of the Harvesting Probability

Figure 4 shows the effect of the harvesting probability $p_j^H$ of IoT device j on the average energy outage probability and the average probabilities that tasks of IoT devices i and j are completed before their deadlines. As shown in Figure 4, it can be found that the EE-CCA can reduce significantly the energy outage probability of IoT devices (see Figure 4a). For example, when $P_i^H$ is 0.1, EE-CCA can reduce the energy outage probability by $78\%$ compared to RAND. while maintaining the probability that the task is completed before the deadline (i.e., 0.99) (see Figure 4b,c). This is because IoT devices in the EE-CCA decide whether to offload some parts of the task to the opponent with the consideration of the energy harvesting probability, the task occurrence rate, and the current energy levels of IoT devices. For example, IoT device does not offload its task to the partner when the current energy level of the partner is low and predicted to be decreased due to low harvesting probability and high task occurrence rate.

From Figure 4a, it can be shown that the average energy outage probabilities of all schemes decrease as $p_j^H$ increases. This is because a high battery level of IoT device j can be maintained regardless of whether to offload or not when $p_j^H$ is high. Meanwhile, from Figure 4b,c, it can be seen that the probabilities that tasks of IoT devices i and j are completed before their deadlines increases with the increase of $p_H$ except specific cases (i.e., NON in Figure 4b and ALL in Figure 4c). This is because the tasks can be completed only when IoT devices have sufficient energy. In other words, IoT devices cannot complete their tasks within the deadline if they cannot harvest sufficiently energy. On the other hand, in NON, IoT device i does not offload its task to IoT device j. Therefore, the harvesting probability $p_j^H$ of IoT device j does not affect the probability ${\eta }_i$ that the task of IoT device i is completed within the deadline. Similarly, in ALL, all tasks of IoT device j are processed in IoT device i, and thus ${\eta }_j$ is not affected by $p_j^H$.

5.2. Effect of the Inter-Task Occurrence Rate

Figure 5 shows the effect of the inter-task occurrence rate ${\lambda }_i$ of IoT device i. From Figure 5, it can be found that the average energy outage probabilities of all schemes increase with the increase of the inter-task occurrence rate. This is because IoT devices consume more energy when tasks occur frequently. However, the incremental ratio of the EE-CCA is smallest among comparison schemes. This is because IoT devices in the EE-CCA operate adaptively even when the operating environment changes. Specifically, as the inter-task occurrence rate of IoT device i increases, it offloads more tasks to its opponent to avoid the energy depletion.

5.3. Effect of the Average Deadline

The effects of the average deadline of the task on the average energy outage probability are demonstrated in Figure 6. From Figure 6, it can be observed that the average energy outage probability of the EE-CCA decreases with the increase of the deadline. This is because, when a sufficient deadline is given, IoT devices in the EE-CCA can handle the task within the deadline by themselves even though they do not offload any tasks to energy-scarce opponents. On the other hand, the other schemes follow the fixed policy regardless of the deadline of the task, and thus their average energy outage probabilities do not change according to the deadline.

5.4. Comparison between the Optimal IoT Device Pairing and the Random Pairing

Figure 7 shows the average energy outage probabilities of the EE-CCA when pairing IoT devices based on the optimization problem (denoted by OPTIMAL) and pairing IoT devices randomly (denoted by RAND) as a function of the number of IoT devices. As shown in Figure 7, the average energy outage probability of OPTIMAL decreases as the number of IoT devices increases. This can be explained as follows: a large number of IoT devices means that there are lots of candidate IoT devices to be matched to a specific IoT device. In this situation, each IoT device can be paired to more appropriate IoT device. For example, an energy-scarce IoT device can be paired to more energy-abundant IoT device. On the other hand, since IoT devices are paired randomly in RAND regardless of the number of IoT devices, its energy outage probability is not affected by that number.

6. Conclusions

In this paper, we proposed an energy efficient cooperative computation algorithm (EE-CCA), in which a pair of IoT devices decide whether to offload some parts of the task to the opponent with the consideration of their energy harvesting probabilities, task occurrence rates, and current energy levels. The optimal offloading decision can be obtained by means of a constraint Markov decision process (CMDP). Moreover, an optimization problem for IoT device pairing is formulated under the optimal offloading strategy. The evaluation results demonstrate that the EE-CCA offloads tasks appropriately, and thus the energy outage probability can be reduced by up to $78\%$ compared to the random offloading scheme while providing the desired probability that tasks are completed before the deadline. Moreover, it can be seen that the EE-CCA operates adaptively even when the operating environment (e.g., inter-task occurrence rate) is changed. In our future works, we will investigate an incentive mechanism to encourage IoT devices to process tasks. In addition, a study for the robustness of the proposed algorithm will be conducted for supporting heterogeneous functionality of IoT devices.