Channel Quality-Based Optimal Status Update for Information Freshness in Internet of Things

Abstract

This paper investigates the status updating policy for information freshness in Internet of things (IoT) systems, where the channel quality is fed back to the sensor at the beginning of each time slot. Based on the channel quality, we aim to strike a balance between the information freshness and the update cost by minimizing the weighted sum of the age of information (AoI) and the energy consumption. The optimal status updating problem is formulated as a Markov decision process (MDP), and the structure of the optimal updating policy is investigated. We prove that, given the channel quality, the optimal policy is of a threshold type with respect to the AoI. In particular, the sensor remains idle when the AoI is smaller than the threshold, while the sensor transmits the update packet when the AoI is greater than the threshold. Moreover, the threshold is proven to be a non-increasing function of channel state. A numerical-based algorithm for efficiently computing the optimal thresholds is proposed for a special case where the channel is quantized into two states. Simulation results show that our proposed policy performs better than two baseline policies.

1. Introduction

Recently, the Internet of things (IoT) has been widely used in the field of industrial manufacturing, environment monitoring, and home automation. In these applications, the sensors generate and transmit new status updates to the destination, where the freshness of the status updates is crucial for the destination to track the state of the environment and to make decisions. Thus, a new information freshness metric, namely age of information (AoI), was proposed in [1] to measure the freshness of updates from the receiver’s perspective. There are two widely used metrics, i.e., the average peak AoI [2] and the average AoI [3]. In general, the smaller the AoI is, the fresher the received updates are.

AoI was originally investigated in [1] for updating the status in vehicular networks. Considering the impact of the queueing system, the authors in [4] investigated the system performance under the M/M/1 and M/M/1/2 queueing systems with a first-come-first-served (FCFS) policy. Furthermore, the work of [5] studied how to keep the updates fresh by analyzing some general update policies, such as the zero-wait policy. The authors of [6] considered the optimal schedule problem for a more general cost that is the weighted sum of the transmission cost and the tracking inaccuracy for the information source. However, these works assumed that the communication channel is not error-prone. In practice, status updates are delivered through an erroneous wireless channel, which suffers from fading, interference, and noises. Therefore, the received updates may not be decoded correctly, which induces information aging and energy consumption.

There are several works that considered the erroneous channel [7,8]. The authors in [9] considered multiple communication channels and investigated the optimal coding and decoding schemes. The channel with an independent and identical packet error rate over time was considered in [10,11]. The work of [12] considered the impact of fading channels in packet transmission. A Markov channel was investigated in [13], where threshold policy was proven to be optimal, and a simulation-based approach was proposed to compute the corresponding threshold. However, how the information of channel quality should be exploited to improve system performance in information freshness remains to be investigated.

Channel quality indicator (CQI) feedback is commonly used in wireless communication systems [14]. In block fading channels, the channel quality, generally reported by the terminal, is highly relevant to the packet error rate (PER) [15] or, namely, the block error rate (BLER). It is probable that a received packet fails to be decoded when the channel suffers from a poor condition. However, a transmitter with the channel quality information is able to keep idle when there is deep fading, thereby saving energy. The channel quantization was also considered in [12,13], where the channel was quantized into multiple states. However, the decision making was not dependent on the channel state in [12], while [13] did not consider the freshness of information. These motivate us to introduce the information of channel quality into the design of the updating policy.

In this paper, a status update system with channel quality feedback is considered. In particular, the channel condition is quantized into multiple states, and the destination feeds the channel quality back to the sensor before the sensor updates the status. Our problem is to investigate the channel quality-based optimal status update policy, which minimizes the weighted sum of the AoI and the energy consumption. Our key contributions are summarized as follows:

• An average cost Markov decision process (MDP) is formulated to model this problem. Due to the infinite countable states and unbounded cost of the MDP, which makes analysis difficult, the discounted version of the original problem is first investigated, and the existence of the stationary and deterministic policy to the original problem is then proven. Furthermore, it is proven that the optimal policy is a threshold structure policy with respect to the AoI for each channel state by showing the monotonic property of the value function. We also prove that the threshold is a non-increasing function of channel state.
• By utilizing the threshold structure, a structure-aware policy iteration algorithm is proposed to efficiently obtain the optimal updating policy. Nevertheless, a numerical-based algorithm which directly computes the thresholds by non-linear fractional programming is also derived. Simulation results reveal the effects of system parameters and show that our proposed policy performs better than the zero-wait policy and periodic policy.

The rest of this paper is organized as follows. In Section 2, the system model is presented and the optimal updating problem is formulated. In Section 3, the optimal updating policy is proven to be of a threshold structure, and a threshold-based policy iteration algorithm is proposed to find the optimal policy. Section 4 presents the simulation results. Finally, we summarize our conclusions in Section 5.

2. System Model and Problem Formulation

2.1. System Description

In this paper, we consider a status update system that consists of a sensor and a destination, as shown in Figure 1. Time is divided into slots. Without loss of generality, we assume that each time slot has an equal length, which is normalized to unity. At the beginning of each slot, the destination feeds the CQI back to the sensor. It is worth noting that the PER is different for different CQIs. Based on the CQI, the sensor decides in each time slot whether it should generate and transmit a new update to the destination via a wireless channel or keep idle for saving energy. These updates are crucial for the destination to estimate the states of the surrounding environment of the sensor and to make in-time decisions. Let $a_t$, which takes value from the action set $\mathcal{A}=\{0,1\}$, denote the action that the sensor performs in slot t, where $a_t=1$ means that the sensor generates and transmits a new update to the destination, and $a_t=0$ represents that the sensor is idle. If the sensor transmits an update packet in slot t, an acknowledgment will be fed back at the end of this time slot. In particular, an ACK is fed back when the destination successfully receives the update packet, and a NACK otherwise.

Figure 1. System model.

2.2. Channel Model

Suppose that the wireless channel is a block fading channel where the channel gain remains constant in each slot and varies independently over different slots. Let $z_t$ denote the channel gain in slot t which takes value from $[0,+\infty )$. We quantize the channel gain into $N+1$ levels which are denoted as $(z_0,z_1,\dots ,z_i,\dots ,z_N)$. The quantization levels are arranged in an increasing order where $z_0=0$ and $z_N=\infty$. Hence, the channel is said to be in state i if the channel gain $z_t$ belongs to the interval $[z_i,z_{i+1})$. We denote by $h_t$ the state of the channel in slot t, where $h_t\in \mathbb{H}\triangleq \{0,1,2\dots ,N-1\}$. With the aid of CQI fed back from the destination, the sensor has knowledge of the channel state at the beginning of each time slot.

Let $p_z\left(z\right)$ denote the distribution of the channel gain. Then, the probability of the channel being in state i is

$$p_i={\int }_{z_i}^{z_{i+1}}{p}_z\left(z\right)dz.$$

(1)

We assume that the signal-to-noise ratio (SNR) per information bit during the transmission remains constant. Then, the PER depends only on the channel gain. In particular, the PER for channel state i is given by

$$g_i={\int }_{z_i}^{z_{i+1}}{P}_{\mathrm{PER}}\left(z\right)p_z(z|i)dz,$$

(2)

where $P_{\mathrm{PER}}\left(z\right)$ is the PER of a packet with respect to the channel gain. The success probability $q_i$ of a packet transmitted over channel state i is $q_i=1-g_i$. According to [15], the success probability is a non-decreasing function of the channel state.

2.3. Age of Information

This paper uses the AoI as the freshness metric, which is defined as the time elapsed since the generation time of the latest update packet that is successfully received by the destination [1]. Let $G_i$ be the generation time of the ith successfully received update packet. Then, the AoI in time slot t, ${\Delta }_t$, is defined as

$${\Delta }_t=t-max\{G_i:G_i\le t\}.$$

(3)

In particular, if an update packet is successfully received, the AoI decreases to one. Otherwise, the AoI increases by one. Altogether, the evolution of the AoI is expressed by

$${\Delta }_{t+1}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}\mathrm{the}\mathrm{transmission}\mathrm{is}\mathrm{successful},\hfill \\ {\Delta }_t+1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

An example of the AoI evolution is shown in Figure 2, where the gray rectangle represents a successful reception of an update packet, and the mesh rectangle represents a transmission failure.

Figure 2. An example of the AoI evolution with the channel state $h_t$, the action $a_t$, and the acknowledgment ${\mathrm{ACK}}_t$. The asterisk stands for no acknowledgment from destination when the sensor keeps idle.

2.4. Problem Formulation

The objective of this paper is to find an optimal updating policy that minimizes the long-term average of the weighted sum of AoI and energy consumption. A policy $\pi$ can be represented by the sequence of actions, i.e., $\pi =(a_0,a_1,\dots ,a_t,\dots )$. Let $\Pi$ be a set of stationary and deterministic policies. Then, the optimal updating problem is given by

$$\underset{\pi \in \Pi }{min}\underset{T\to \infty }{lim\; sup}\frac{1}{T}\sum _{t=0}^T\mathbb{E}[{\Delta }_t+\omega a_t{C}_e],$$

(5)

where $C_e$ is the energy consumption, and $\omega$ is the weighting factor.

3. Optimal Updating Policy

This section aims to investigate the optimal updating policy for the problem formulated in above section. In this section, our investigating problem is first formulated into an infinite horizon average cost MDP, and the existence of a stationary and deterministic policy that minimizes the average cost is proven. Then, the non-decreasing property of the value function is derived. Based on this property, we prove that the optimal update policy is of a threshold structure with respect to AoI, and the optimal threshold is a non-increasing function of the channel state. Aiming to reduce the computational complexity, a structure-aware policy iteration algorithm is proposed to find the optimal policy. Moreover, non-linear fractional programming is employed to directly compute the optimal thresholds in a special case where the channel is quantized into two states.

3.1. MDP Formulation

The Markov decision process (MDP) is typically applied to address the optimal decision problem when the investigation problem can be characterized by the evolution of the system state and the cost is per-stage. The optimization problem in (5) can be formulated as an infinite horizon average cost MDP, which is elaborated in the following.

• States: The state of the MDP in slot t is defined as ${\mathbf{x}}_t=({\Delta }_t,h_t)$, which takes values in ${\mathbb{Z}}^{+}\times \mathbb{H}$. Hence, the state space $\mathcal{S}$ is countable and infinite.
• Actions: The set of actions $a_t$ chosen in slot t is $\mathcal{A}=\{0,1\}$.
• Transition Probability: Let $Pr({\mathbf{x}}_{t+1}|{\mathbf{x}}_t,a_t)$ be the transition probability that the state ${\mathbf{x}}_t$ in slot t transits to ${\mathbf{x}}_{t+1}$ in slot $t+1$ after taking action $a_t$. According to the evolution of AoI in (4), the transition probability is given by

  $$\left\{\begin{array}{c}Pr({\mathbf{x}}_{t+1}=(\Delta +1,j)|{\mathbf{x}}_t=(\Delta ,i),a_t=0)=p_j,\hfill \\ Pr({\mathbf{x}}_{t+1}=(1,j)|{\mathbf{x}}_t=(\Delta ,i),a_t=1)=q_i{p}_j,\hfill \\ Pr({\mathbf{x}}_{t+1}=(\Delta +1,j)|{\mathbf{x}}_t=(\Delta ,i),a_t=1)=g_i{p}_j.\hfill \end{array}\right.$$

  (6)
• Cost: The instantaneous cost $C({\mathbf{x}}_t,a_t)$ at state ${\mathbf{x}}_t$ given action $a_t$ in slot t is

  $$C({\mathbf{x}}_t,a_t)={\Delta }_t+\omega a_t{C}_e.$$

  (7)

For an MDP with infinite states and unbounded cost, it is not guaranteed to have a stationary and deterministic policy that attains the minimum average cost in general. Fortunately, we can prove the existence of stationary and deterministic policy in next sub-section.

3.2. The Existence of Stationary and Deterministic Policy

For rigorous mathematical analysis, this section is purposed to prove the existence of a stationary and deterministic optimal policy. According to [16], we first analyze the associated discounted cost problem of the original MDP. The expectation of discount cost with respect to discounted factor $\gamma$ and initial state $\widehat{\mathbf{x}}$ under a policy $\pi$ is given by

$$V_{\pi ,\gamma }\left(\widehat{\mathbf{x}}\right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}C({\mathbf{x}}_t,a_t)|{\mathbf{x}}_0=\widehat{\mathbf{x}}\right],$$

(8)

where $a_t$ is the decision made in state $\widehat{\mathbf{x}}$ under policy $\pi$, and $\gamma \in (0,1)$ is the discounted factor. We first verify that $V_{\pi ,\gamma }\left(\widehat{\mathbf{x}}\right)$ is finite for any policy and all $\widehat{\mathbf{x}}\in \mathcal{S}$.

Lemma 1.

Given $\gamma \in (0,1)$, for any policy π and all $\widehat{\mathbf{x}}=(\widehat{\mathbf{x}},\widehat{h})\in \mathcal{S}$, we have

$$V_{\pi ,\gamma }\left(\widehat{\mathbf{x}}\right)={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}C({\mathbf{x}}_t,a_t)|{\mathbf{x}}_0=\widehat{\mathbf{x}}\right]<\infty .$$

(9)

Proof. 

By definition, the instantaneous cost in state ${\mathbf{x}}_t=({\Delta }_t,h_t)$ given action $a_t$ is

$$C({\mathbf{x}}_t,a_t)=\left\{\begin{array}{cc}{\Delta }_t,\hfill & \hfill \mathrm{if}{a}_t=0,\\ {\Delta }_t+\omega C_e,\hfill & \hfill \mathrm{if}{a}_t=1.\end{array}\right.$$

(10)

Therefore, $C({\mathbf{x}}_t,a_t)\le {\Delta }_t+\omega C_e$ holds. Combined with the fact that the AoI increases, at most, linearly at each slot for any policy, we have

$$\begin{array}{cc}\hfill & \sum _{t=0}^{\infty }{\gamma }^{t}C({\mathbf{x}}_t,a_t|{\mathbf{x}}_0=(\widehat{\Delta },\widehat{h}))\hfill \\ \hfill \le & \sum _{t=0}^{\infty }{\gamma }^t(\widehat{\Delta }+t+\omega C_e)\hfill \\ \hfill =& \frac{1}{1-\gamma }\left(\widehat{\Delta }+\frac{\gamma }{1-\gamma }+\omega \right)<\infty ,\hfill \end{array}$$

(11)

which completes the proof.    □

Let $V_{\gamma }\left(\widehat{\mathbf{x}}\right)={min}_{\pi }{V}_{\pi ,\gamma }\left(\widehat{\mathbf{x}}\right)$ denote the minimum expected discounted cost. By Lemma 1, $V_{\gamma }\left(\widehat{\mathbf{x}}\right)={min}_{\pi }{V}_{\pi ,\gamma }\left(\widehat{\mathbf{x}}\right)<\infty$ holds for every $\widehat{\mathbf{x}}$ and $\gamma \in (0,1)$.

According to [16] (Proposition 1), we have

$$V_{\gamma }\left(\widehat{\mathbf{x}}\right)=\underset{a\in \mathcal{A}}{min}\left\{C(\widehat{\mathbf{x}},a)+\gamma \sum _{{\mathbf{x}}^{\prime }\in \mathcal{S}}Pr({\mathbf{x}}^{\prime }|\widehat{\mathbf{x}},a)V_{\gamma }\left({\mathbf{x}}^{\prime }\right)\right\},$$

(12)

which implies that $V_{\gamma }\left(\widehat{\mathbf{x}}\right)$ satisfies the Bellman equation. $V_{\gamma }\left(\widehat{\mathbf{x}}\right)$ can be solved via a value iteration algorithm. In particular, we define $V_{\gamma ,0}\left(\widehat{\mathbf{x}}\right)=0$, and for all $n\ge 1$, we have

$$V_{\gamma ,n}\left(\widehat{\mathbf{x}}\right)=\underset{a\in \mathcal{A}}{min}{Q}_{\gamma ,n}(\widehat{\mathbf{x}},a),$$

(13)

where

$$Q_{\gamma ,n}(\widehat{\mathbf{x}},a)=\underset{a\in \mathcal{A}}{min}\left\{C(\widehat{\mathbf{x}},a)+\gamma \sum _{{\mathbf{x}}^{\prime }\in \mathcal{S}}Pr({\mathbf{x}}^{\prime }|\widehat{\mathbf{x}},a)V_{\gamma ,n-1}\left({\mathbf{x}}^{\prime }\right)\right\}$$

(14)

is related to the right-hand-side (RHS) of the discounted cost optimality equation. Then, ${lim}_{n\to \infty }{V}_{\gamma ,n}\left(\widehat{\mathbf{x}}\right)=V_{\gamma }\left(\widehat{\mathbf{x}}\right)$ for every $\widehat{\mathbf{x}}$ and $\gamma$.

Now, we can use the value iteration algorithm to establish the monotonic properties of $V_{\gamma }\left(\widehat{\mathbf{x}}\right)$

Lemma 2.

For all Δ and i, we have

$$V_{\gamma }(\Delta ,N-1)\le V_{\gamma }(\Delta ,i),$$

(15)

and for all ${\Delta }_1\le {\Delta }_2$ and i, we have

$$V_{\gamma }({\Delta }_1,i)\le V_{\gamma }({\Delta }_2,i).$$

(16)

Proof. 

See Appendix A.    □

Based on Lemmas 1 and 2, we are ready to show that the MDP has a stationary and deterministic optimal policy in the following theorem.

Theorem 1.

For the MDP in (5), there exists a stationary and deterministic optimal policy ${\pi }^{*}$ that minimizes the long-term average cost. Moreover, there exists a finite constant $\lambda ={lim}_{\gamma \to 1}(1-\gamma )V_{\gamma }\left(\mathbf{x}\right)$ for all states $\mathbf{x}$, where λ is independent of the initial state, and a value function $V\left(\mathbf{x}\right)$, such that

$$\lambda +V\left(\mathbf{x}\right)=\underset{a\in \mathcal{A}}{min}\left\{C(\mathbf{x},a)+\sum _{{\mathbf{x}}^{\prime }\in \mathcal{S}}Pr({\mathbf{x}}^{\prime }|\mathbf{x},a)V\left({\mathbf{x}}^{\prime }\right)\right\}$$

(17)

holds for all $\mathbf{x}$.

Proof. 

See Appendix B.    □

3.3. Structural Analysis

According to Theorem 1, the optimal policy for the average cost problem satisfies the following equation

$${\pi }^{*}\left(\mathbf{x}\right)=arg\underset{a\in \mathcal{A}}{min}Q(\mathbf{x},a),$$

(18)

where

$$Q(\mathbf{x},a)=C(\mathbf{x},a)+\sum _{{\mathbf{x}}^{\prime }\in \mathcal{S}}Pr({\mathbf{x}}^{\prime }|\mathbf{x},a)V\left({\mathbf{x}}^{\prime }\right).$$

(19)

Similar to Lemma 2, the monotonic property of the value function $V\left(\mathbf{x}\right)$ is given in the following lemma.

Lemma 3.

Given the channel state i, for any ${\Delta }_2\ge {\Delta }_1$, we have

$$V({\Delta }_2,i)\ge V({\Delta }_1,i).$$

(20)

Proof. 

This proof follows the same procedure of Lemma 2, with one exception being that the value iteration algorithm is based on Equation (17).    □

Moreover, based on Lemma 3, the property of the increment of the value function is established in following lemma.

Lemma 4.

Given the channel state i, for any ${\Delta }_2\ge {\Delta }_1$, we have

$$V({\Delta }_2,i)-V({\Delta }_1,i)\ge {\Delta }_2-{\Delta }_1.$$

(21)

Proof. 

We first examine the relation between the state-action value functions, i.e., $Q({\Delta }_2,i,a)$ and $Q({\Delta }_1,i,a)$. Specifically, based on Lemma 3, we have

$$\begin{array}{ll}& Q({\Delta }_2,i,0)-({\Delta }_2-{\Delta }_1)={\Delta }_1+{\displaystyle \sum _{j=0}^{N-1}}{p}_{j}V({\Delta }_2+1,j)\\ \ge & {\Delta }_1+{\displaystyle \sum _{j=0}^{N-1}}{p}_{j}V({\Delta }_1+1,j)=Q({\Delta }_1,i,0),\end{array}$$

(22)

and

$$\begin{array}{cc}\hfill & Q({\Delta }_2,i,1)-({\Delta }_2-{\Delta }_1)\hfill \\ \hfill =& {\Delta }_1+\omega C_e+q_i\sum _{j=0}^{N-1}{p}_{j}V(1,j)+g_i\sum _{j=0}^{N-1}{p}_{j}V({\Delta }_2+1,j)\hfill \\ \hfill \ge & {\Delta }_1+\omega C_e+q_i\sum _{j=0}^{N-1}{p}_{j}V(1,j)+g_i\sum _{j=0}^{N-1}{p}_{j}V({\Delta }_1+1,j)\hfill \\ \hfill =& Q({\Delta }_1,i,1).\hfill \end{array}$$

(23)

Since $V\left(\mathbf{x}\right)=\underset{a\in \mathcal{A}}{min}Q(\mathbf{x},a)$, we complete the proof.    □

Our main result is presented in the following theorem.

Theorem 2.

For any given channel state i, there exists a threshold ${\beta }_i$, such that when $\Delta \ge {\beta }_i$, the optimal action is to generate and transmit a new update, i.e., ${\pi }^{*}(\Delta ,i)=1$, and when $\Delta <{\beta }_i$, the optimal action is to remain idle, i.e., ${\pi }^{*}(\Delta ,i)=0$. Moreover, the optimal threshold ${\beta }_i$ is a non-increasing function of channel state i, i.e., ${\beta }_i\ge {\beta }_j$ holds for all $i,j\in \mathbb{H}$ and $i\le j$.

Proof. 

See Appendix C.    □

According to Theorem 2, the sensor will not update the status until the AoI exceeds the threshold. Moreover, if the channel condition is not good, i.e., channel state i is small, the sensor will wait for a longer time before it samples and transmits the status update packet so as to reduce the energy consumption because of a higher probability of transmission failure.

Based on the threshold structure, we can reduce the computational complexity of the policy iteration algorithm to find the optimal policy. The details of the algorithm are presented in Algorithm 1.

Algorithm 1 Policy iteration algorithm (PIA) based on the threshold structure.

1: Initialization: Set $k=0$, iteration threshold $ϵ$, and initialize the value function $V_0\left(\mathbf{x}\right)=0$ and policy $\pi \left(\mathbf{x}\right)=0$ for all state $\mathbf{x}\in \mathcal{S}$ 2: repeat 3: $\hspace{1em}k\leftarrow k+1$. 4:   Based on last iterative value function $V_{k-1}\left(\mathbf{x}\right)$, compute the current value function $V_k\left(\mathbf{x}\right)$ by calculating the following equations. 5: $\hspace{1em}{V}_k\left(\mathbf{x}\right)={\displaystyle \underset{a\in \mathcal{A}}{min}}\left\{C(\mathbf{x},a)+{\displaystyle \sum _{{\mathbf{x}}^{\prime }\in \mathcal{S}}}Pr\left({\mathbf{x}}^{\prime }\right|\mathbf{x},a)V_{k-1}\left({\mathbf{x}}^{\prime }\right)\right\}$ 6: until$|V_k\left(\mathbf{x}\right)-V_{k-1}\left(\mathbf{x}\right)|\le ϵ$ for all $\mathbf{x}\in \mathcal{S}$ 7: for$\mathbf{x}=(\Delta ,i)\in \mathcal{S}$do 8:   if ${\mathbf{x}}^{\prime }=(\Delta -1,i)\in \mathcal{S}$ and $\pi \left({\mathbf{x}}^{\prime }\right)=1$ then 9: $\hspace{1em}\hspace{1em}\pi \left(\mathbf{x}\right)\leftarrow 1$. 10:  else 11: $\hspace{1em} \pi \left(\mathbf{x}\right)\leftarrow arg{\displaystyle \underset{a\in \mathcal{A}}{min}}\left\{C(\mathbf{x},a)+\underset{{\mathbf{x}}^{\prime }\in \mathcal{S}}{Pr}\left({\mathbf{x}}^{\prime }\right|\mathbf{x},a)V\left({\mathbf{x}}^{\prime }\right)\right\}$ 12:  end if 13: end for 14: ${\pi }^{*}\leftarrow \pi$ 15: return the optimal policy ${\pi }^{*}$

3.4. Computing the Thresholds for a Special Case

In the above section, we have proven that the optimal policy has a threshold structure. Given the thresholds $({\beta }_0,{\beta }_1,\dots ,{\beta }_{N-1})$, a Markov chain can be induced by the threshold policy. A special Markov chain is depicted in Figure 3, where the channel has two states. By leveraging the Markov chain, we first derive the average cost of the special case, which is summarized in the following theorem.

Figure 3. An illustration of established Markov chain with two channel states.

Theorem 3.

Let $\phi \left(\mathbf{x}\right)$ be the steady state probability of state $\mathbf{x}$ of the corresponding Markov chain with two states and ${\beta }_0,{\beta }_1$ be the threshold with respect to the channel state, respectively. The steady state probability is given by

$$\phi (i,j)=\left\{\begin{array}{cc}{p}_j{\phi }_1,\hfill & \mathit{if}1\le i\le {\beta }_1,\hfill \\ p_j{s}_0^{i-{\beta }_1}{\phi }_1,\hfill & \mathit{if}{\beta }_1<i\le {\beta }_0,\hfill \\ p_j{s}_0^{{\beta }_0-{\beta }_1}{s}_1^{i-{\beta }_1}{\phi }_1,\hfill & \mathit{if}i>{\beta }_0,\hfill \end{array}\right.$$

(24)

where ${\phi }_1=\phi (1,0)+\phi (1,1)$, $s_0=1-p_1{q}_1$, $s_1=1-p_0{q}_0-p_1{q}_1$, and ${\phi }_1$ satisfies following equation:

$${\phi }_1=\frac{1}{{\beta }_1+\frac{s_0-s_0^{{\beta }_0-{\beta }_1}}{1-s_0}+s_0^{{\beta }_0-{\beta }_1}\frac{s_1}{1-s_1}}.$$

(25)

The average cost then is given by

$$\begin{array}{cc}\hfill & C_{mc}({\beta }_0,\dots ,{\beta }_{N-1})\hfill \\ \hfill =& {\phi }_1(\frac{{\beta }_1({\beta }_1+1)}{2}+A+B+\omega C_{e}E),\hfill \end{array}$$

(26)

where

$$A=\frac{s_0(({\beta }_1+1)-{\beta }_0{s}_0^{{\beta }_0-{\beta }_1})}{1-s_0}+\frac{s_0^3-s_0^{{\beta }_0-{\beta }_1+1}}{{(1-s_0)}^2},$$

(27)

$$B=\frac{({\beta }_0+1)s_1}{1-s_1}+\frac{s_1^3}{1-s_1},$$

(28)

and

$$\begin{array}{c}\hfill E=\left(s_0^{{\beta }_0-{\beta }_1}\frac{1}{1-s_1}+p_1\frac{1-s_0^{{\beta }_0-{\beta }_1}}{1-s_0}\right).\end{array}$$

(29)

Proof. 

See Appendix D.    □

Therefore, the closed form of the average cost is a function of thresholds. By linear search or gradient descent algorithm, the numerical solution of optimal thresholds can be obtained. However, computing its gradient directly requires a large amount of computation till convergence. Here, a nonlinear fractional programming (NLP) [17] based algorithm which can efficiently obtain the numerical solution is proposed.

Let $\mathbf{x}=({\beta }_0,{\beta }_1)$. We can rewrite the cost function as a fractional form, where the numerator is denoted as $N\left(\mathbf{x}\right)=-C_{mc}\left(\mathbf{x}\right)/{\phi }_1$, and the denominator term is $N\left(\mathbf{x}\right)=1/{\phi }_1$. The solution to an NLP problem with the form in the following

$$max\left\{\frac{N\left(\mathbf{x}\right)}{D\left(\mathbf{x}\right)}|\mathbf{x}\in A\right\}$$

(30)

is related to the optimization problem (31)

$$max\left\{N\left(\mathbf{x}\right)-qD\left(\mathbf{x}\right)|(\mathbf{x}\in A\right\},\mathrm{for}q\in \mathbb{R},$$

(31)

where the following assumption should also be satisfied:

$$D\left(\mathbf{x}\right)>0,\mathrm{for}\mathrm{all}\mathbf{x}\in A.$$

(32)

Define the function $F\left(q\right)$ with variable q as

$$F\left(q\right)=max\left\{N\left(\mathbf{x}\right)-qD\left(\mathbf{x}\right)|\mathbf{x}\in A\right\},\mathrm{for}q\in \mathbb{R}.$$

(33)

According to [17], $F\left(q\right)$ is a strictly monotonic decreasing function and is convex over $\mathbb{R}$. Furthermore, we have $q_0=N\left({\mathbf{x}}_0\right)/D\left({\mathbf{x}}_0\right)=max\{N\left(\mathbf{x}\right)-qD\left(\mathbf{x}\right)|\mathbf{x}\in A\}$ if, and only if,

$$F\left(q_0\right)=max\{N\left(\mathbf{x}\right)-q_{0}D\left(\mathbf{x}\right)|\mathbf{x}\in A\}=0.$$

(34)

Then, the algorithm can be described by two steps. The first step is to solve a convex optimization problem with a one dimensional parameter by a bisection method. The second step is to solve a high dimensional optimization problem by a gradient descent method.

According to [17], a bisection method can be used to solve the optimal $q_0$, under the assumption that the value of function $F\left(q\right)$ can be obtained exactly for given q. We will actually use the gradient descent algorithm to obtain the numerical solution of $F\left(q\right)$ since the global search method may not perform in polynomial time. As a trick, we alternate the optimization variables of thresholds $({\beta }_0,{\beta }_1)$ by the variables of the decrement of thresholds, i.e., $\mathbf{x}=({\beta }_0-{\beta }_1,{\beta }_1)$. To summarize, the numerical-based method for computing the optimal thresholds is given by Algorithm 2.

Algorithm 2 Numerical computation of the optimal thresholds.

Input:$\mathrm{Iteration}\mathrm{time}k,\mathrm{error}\mathrm{threshold}\delta$

Output:$\mathrm{Numerical}\mathrm{result}{\mathbf{x}}^{*}$

1: Let $N\left(\mathbf{x}\right)=-C_{mc}\left(\mathbf{x}\right)/{\phi }_1$, and $D\left(\mathbf{x}\right)=1/{\phi }_1$. Define $F\left(q\right)=max\left\{N\right(\mathbf{x})-qD(\mathbf{x}\left)\right|\mathbf{x}\ge \mathbf{0}\}$ 2: Let the iteration starts with $i=1$, search range $[a,b]$ of q. 3: while$i\le k$do 4: $\hspace{1em}m=\frac{a+b}{2}$; 5:   if$F\left(m\right)\ast F\left(a\right)<0$then 6: $\hspace{1em}\hspace{1em}b=m$; 7:   else 8: $\hspace{1em}\hspace{1em}a=m$; 9:   end if 10:  if$\frac{b-a}{2}<\delta$then 11: $\hspace{1em} {\mathbf{x}}^{*}=arg{min}_{\mathbf{x}}F\left(m\right)$ 12:    break; 13:  end if 14: $i=i+1$; 15: end while

4. Simulation Results and Discussions

In this section, the simulation results are presented to investigate the impacts of the system parameters. We also compare the optimal policy with the zero-wait policy and periodic policy, where the zero-wait policy immediately generates an update at each time slot and the periodic policy keeps a constant interval between two updates.

Figure 4 depicts the optimal policy for different AoI and channel states, where the number of channel states is 5. It can be seen that, for each channel state, the optimal policy has a threshold structure with respect to the AoI. In particular, when the AoI is small, it is not beneficial for the sensor to generate and transmit a new update because the energy consumption dominates the total cost. We can also see that the threshold is non-increasing with the channel state. In other words, if the channel condition is better, the threshold is smaller. This is because the success probability of packet transmission increases with the channel state.

Figure 4. Optimal policy for different AoI and channel states ($q_0=0.1, q_1=0.2, q_2=0.3, q_3=0.4, q_4=0.5, p_0=0.1, p_1=0.1, p_2=0.3, p_3=0.3, p_4=0.2, \omega =10, C_e=1$).

Figure 5 illustrates the thresholds for the MDP with two channel states with respect to the weighting factor $\omega$, in which the two dashed lines are obtained by PIA and the other two solid lines are obtained by the proposed numerical algorithm. Both of the thresholds grow with the increasing of $\omega$. Since the energy consumption has more weight, it is not efficient to update when the AoI is small. On the contrary, when $\omega$ decreases, the AoI dominates and the thresholds decline. In particular, both of the thresholds equal 1 when $\omega =0$. In this case, the optimal policy reduces to the zero-wait policy. We can also see that the value of the threshold for channel state 1 of the numerical algorithm is close to the optimal solution. In contrast, the value of the threshold for channel state 0 gradually deviates from the optimal value.

Figure 5. Optimal thresholds for two different channel states versus $\omega$ ($p_0=0.2, p_1=0.8, q_0=0.2, q_1=0.5, C_e=1$).

Figure 6 illustrates the performance comparison of four policies, i.e., the zero-wait policy, the periodic policy, the numerical-based policy, and the optimal policy, with respect to the weighting factor $\omega$. It is easy to see that the optimal policy has the lowest average cost. As we see in Figure 6, the zero-wait policy has the same performance with the optimal policy when $\omega =0$. As $\omega$ increases, the average cost of all three policies increases. However, the increment of the zero-wait policy is larger than the periodic policy and the optimal policy due to the frequent transmission in the zero-wait policy. Although the thresholds obtained by the PIA and the numerical algorithm are not exactly the same as shown in Figure 5, the performance of the numerical-based algorithm also coincides with the optimal policy. This is because the threshold for channel state 1 exists in the quadratic term of the cost function, while the threshold for channel state 0 exists in the negative exponential term of the cost function. As a result, the threshold for channel state 1 has a much more significant effect on the system performance.

Figure 6. Comparison of the zero-wait policy, the periodic policy with period being 5, the numerical-based policy, and the optimal policy with respect to the weighting factor $\omega$ ($p_0=0.2, p_1=0.8, q_0=0.2, q_1=0.5, C_e=1$).

Figure 7 compares the three policies with respect to the probability $p_1$ of the channel being in state 1. Since there is a higher probability that the channel has a good quality as $p_1$ increases, the average cost of all three policies decreases. We can see that, in the regime of $p_1$, the optimal policy has the lowest average cost, because it can achieve a good balance between the AoI and the energy consumption. We can also see that the cost of the periodic policy is greater than the zero-wait policy first, and smaller later. To further demonstrate these curves, we separate the energy consumption term and AoI term into different figures, i.e., Figure 8 and Figure 9. We see that the update cost of the zero-wait policy is smaller than that of the periodic policy, but the AoI of the zero-wait policy has a smaller decrease with respect to $p_1$ than the periodic policy.

Figure 7. Comparison of the zero-wait policy, the periodic policy with period being 5, and the optimal policy with respect to $p_1$ ($q_0=0.2, q_1=0.5, \omega =10, C_e=1$).

Figure 8. AoI comparison of the zero-wait policy, the periodic policy with period being 5, and the optimal policy with respect to $p_1$ ($q_0=0.2, q_1=0.5, \omega =10, C_e=1$).

Figure 9. Energy consumption comparison of the zero-wait policy, the periodic policy with period being 5, and the optimal policy with respect to $p_1$ ($q_0=0.2, q_1=0.5, \omega =10, C_e=1$).

5. Conclusions

In this paper, we have studied the optimal updating policy in an IoT system, where the channel gain is quantized into multiple states and the channel state is fed back to the sensor before the decision making. The status update problem has been formulated as an MDP to minimize the long-term average of the weighted sum of the AoI and the energy consumption. By investigating the properties of the value function, it is proven that the optimal policy has a threshold structure with respect to AoI for any given channel state. We have also proven that the threshold is a non-increasing function of the channel state. Simulation results show the impacts of system parameters on the optimal thresholds and the average cost. Through comparisons, we have also shown that our proposed policy outperforms the zero-wait policy and the periodic policy. In our future research, the time-varying channel model will be further involved for guiding the future design of realistic IoT systems.

Appendix A.1. Proof of Equation (16)

When $n=1$,

$$\begin{array}{cc}\hfill & Q_{\gamma ,1}({\Delta }_1,i,0)-Q_{\gamma ,1}({\Delta }_2,i,0)={\Delta }_1-{\Delta }_2\le 0,\hfill \end{array}$$

(A1)

and

$$\begin{array}{cc}\hfill & Q_{\gamma ,1}({\Delta }_1,i,1)-Q_{\gamma ,1}({\Delta }_2,i,1)={\Delta }_1+\omega C_e-({\Delta }_2+\omega C_e)\le 0,\hfill \end{array}$$

(A2)

hold due to ${\Delta }_1\le {\Delta }_2$, and we have $V_{\gamma ,1}({\Delta }_1,i)\le V_{\gamma ,1}({\Delta }_2,i)$.

Suppose that $V_{\gamma ,K}({\Delta }_1,i)\le V_{\gamma ,K}({\Delta }_2,i)$ holds for $k\le K$. Considering the case of $k=K+1$,

$$\begin{array}{cc}\hfill & Q_{\gamma ,K+1}({\Delta }_1,i,0)-Q_{\gamma ,K+1}({\Delta }_2,i,0)={\Delta }_1-{\Delta }_2\le 0,\hfill \end{array}$$

(A3)

and

$$\begin{array}{cc}\hfill & Q_{\gamma ,K+1}({\Delta }_1,i,1)-Q_{\gamma ,K+1}({\Delta }_2,i,1)\hfill \\ \hfill =& ({\Delta }_1-{\Delta }_2)+\gamma \sum _{j\in \mathbb{H}}{p}_j{g}_i\left(V_{\gamma ,K}({\Delta }_1+1,j)-V_{\gamma ,K}({\Delta }_2+1,j)\right)\hfill \\ \hfill \le & 0,\hfill \end{array}$$

(A4)

hold for all i according to ${\Delta }_1\le {\Delta }_2$. Therefore, we have $V_{\gamma ,K+1}({\Delta }_1,i)\le V_{\gamma ,K+1}({\Delta }_2,i)$. Since ${lim}_{n\to \infty }{V}_{\gamma ,n}=V_{\gamma }$, we have $V_{\gamma }({\Delta }_1,i)\le V_{\gamma }({\Delta }_2,i)$.

Appendix A.2. Proof of Equation (15)

By the definition of function $Q_{\gamma }(\mathbf{x},a)$, we have

$$Q_{\gamma }(\Delta ,i,0)=\Delta +\gamma \sum _{j\in \mathbb{H}}{p}_j{V}_{\gamma }(\Delta +1,j),$$

(A5)

and

$$\begin{array}{cc}\hfill & Q_{\gamma }(\Delta ,i,1)=\Delta +\omega C_e+\gamma \left(\sum _{j\in \mathbb{H}}{p}_j\left(g_i{V}_{\gamma }(\Delta +1,j)+q_i{V}_{\gamma }(1,j)\right)\right).\hfill \end{array}$$

(A6)

Therefore,

$$\begin{array}{c}\hfill Q_{\gamma }(\Delta ,N-1,0)-Q_{\gamma }(\Delta ,i,0)\le 0,\end{array}$$

(A7)

and

$$\begin{array}{cc}\hfill & Q_{\gamma }(\Delta ,N-1,1)-Q_{\gamma }(\Delta ,i,1)\hfill \\ \hfill \stackrel{\left(a\right)}{=}& \gamma \sum _{j\in \mathbb{H}}{p}_j(q_{N-1}-q_i)\left(V_{\gamma }(1,j)-V_{\gamma }(\Delta +1,j)\right)\le 0,\hfill \end{array}$$

(A8)

hold for all i, where step $\left(a\right)$ is due to Equation (16). Hence, we have $V_{\gamma }(\Delta ,N-1)\le V_{\gamma }(\Delta ,i)$. This completes the whole proof.