Scheduling to Minimize Age of Incorrect Information with Imperfect Channel State Information

Abstract

In this paper, we study a slotted-time system where a base station needs to update multiple users at the same time. Due to the limited resources, only part of the users can be updated in each time slot. We consider the problem of minimizing the Age of Incorrect Information (AoII) when imperfect Channel State Information (CSI) is available. Leveraging the notion of the Markov Decision Process (MDP), we obtain the structural properties of the optimal policy. By introducing a relaxed version of the original problem, we develop the Whittle’s index policy under a simple condition. However, indexability is required to ensure the existence of Whittle’s index. To avoid indexability, we develop Indexed priority policy based on the optimal policy for the relaxed problem. Finally, numerical results are laid out to showcase the application of the derived structural properties and highlight the performance of the developed scheduling policies.

1. Introduction

The Age of Incorrect Information (AoII) is introduced in [1] as a combination of age-based metrics (e.g., Age of Information (AoI)) and error-based metrics (e.g., Minimum Mean Square Error). In communication systems, AoII captures not only the information mismatch between the source and the destination but also the aging process of inconsistent information. Hence, two functions dominate AoII. The first is the time penalty function, which reflects how the inconsistency of information affects the system over time. In real-life applications, inconsistent information will affect different communication systems in different ways. For example, machine temperature monitoring is time-sensitive because the damage caused by overheating will accumulate quickly. However, reservoir water level monitoring is less sensitive to time. Therefore, by adopting different time penalty functions, AoII can capture different aging processes of the mismatch in different systems. The second is the information penalty function, which captures the information mismatch between the source and the destination. It allows us to measure mismatches in different ways, depending on how sensitive different systems are to information inconsistencies. For example, the navigation system requires precise information to give correct instructions, but the real-time delivery tracking system does not need very accurate location information. Since we can choose different penalty functions for different systems, AoII is adaptable to various communication goals, which is why it is regarded as a semantic metric [2].

Since the introduction of AoII, several studies have been performed to reveal its fundamental nature. The authors of [3] consider a system with random packet delivery times and compare AoII with AoI and real-time error via extensive numerical results. The authors of [4] study the problem of minimizing the AoII that takes the general time penalty function. Three real-life applications are considered to showcase the performance advantages of AoII over AoI and real-time error. In [5], the authors investigate the AoII that considers the quantified mismatch between the source and the destination. The optimization problem is studied when the system is resource-constrained. The authors of [6] studied the AoII minimization problem in the context of scheduling. It considers a system where the central scheduler needs to update multiple users at the same time. However, the central scheduler cannot know the states of the sources before receiving the updates. By introducing the belief value, Whittle’s index policy is developed and evaluated. In this paper, we also consider the problem of minimizing AoII in scheduling. Different from [6], we consider the generic time penalty function and study the minimization problem in the presence of imperfect Channel State Information (CSI). Due to the existence of CSI, Whittle’s index policy becomes infeasible in general. Hence, we introduce another scheduling policy that is more versatile and has comparable performance to Whittle’s index policy.

The problem of scheduling to minimize AoI is studied under various system settings in [7,8,9,10,11]. The problem studied in this paper is different and more complicated because AoII considers the aging process of inconsistent information rather than the aging process of updates. Meanwhile, none of them consider the case where CSI is available. The problem of optimizing information freshness in the presence of CSI is studied in [12,13]. However, they focus on the system with a single user and mainly discuss the case where CSI is perfect. The scheduling problems with the goal of minimizing an error-based performance measure are considered in [14,15,16]. Our problem is fundamentally different because AoII also considers the time effect. Moreover, we consider the system where a base station observes multiple sources simultaneously and needs to send updates to multiple destinations.

The main contributions of this work can be summarized as follows. (1) We study the problem of minimizing AoII in a multi-user system where imperfect CSI is available. Meanwhile, the time penalty function is generic. (2) We derive the structural properties of the optimal policy for the considered problem. (3) We establish the indexability of the considered problem under a simple condition and develop Whittle’s index policy. (4) We obtain the optimal policy for a relaxed version of the original problem. By exploring the characteristics of the relaxed problem, we provide an efficient algorithm to obtain the optimal policy. (5) Based on the optimal policy for the relaxed problem, we develop the Indexed priority policy that is free from indexability and has comparable performance to Whittle’s index policy.

The remainder of this paper is organized in the following way. In Section 2, we introduce the system model and formulate the primal problem. Section 3 explores the structural properties of the optimal policy for the primal problem. Under a simple condition, we develop Whittle’s index policy in Section 4. Section 5 presents the optimal policy for a relaxed version of the primal problem. On this basis, we develop the Indexed priority policy in Section 6. Finally, in Section 7, the numerical results are laid out.

2. System Overview

2.1. Communication Model

We consider a slotted-time system with N users and one base station. Each user is composed of a source process, a channel, and a receiver. We assume all the users share the same structure, but the parameters are different. The structure of the communication model is provided in Figure 1.

Figure 1. The structure of the communication model.

For user i, the source process is modeled by a two-state Markov chain where transitions happen between the two states with probability $p_i>0$ and self-transitions happen with probability $1-p_i$. At any time slot t, the state of the source process $X_{i,t}\in \{0,1\}$ will be reported to the base station as an update, and the base station will decide whether to transmit this update through the corresponding channel. The channel is unreliable, but the estimate of the Channel State Information (CSI) is available at the beginning of each time slot. Let $r_{i,t}\in \{0,1\}$ be the CSI at time t. We assume that $r_{i,t}$ is independent across time and user indices. $r_{i,t}=1$ if and only if the transmission attempt at time t will succeed and $r_{i,t}=0$ otherwise. Then, we denote by ${\widehat{r}}_{i,t}\in \{0,1\}$ the estimate of $r_{i,t}$. We assume that ${\widehat{r}}_{i,t}$ is an independent Bernoulli random variable with parameter ${\gamma }_i$, i.e., ${\widehat{r}}_{i,t}=1$ with probability ${\gamma }_i\in [0,1]$ and ${\widehat{r}}_{i,t}=0$ with probability $1-{\gamma }_i$. However, the estimate is imperfect. We assume that the error depends only on the user and its estimate. More precisely, we define the probability of error as $p_{e,i}^{{\widehat{r}}_i}\triangleq Pr[r_i\ne {\widehat{r}}_i\mid {\widehat{r}}_i]$. We assume $p_{e,i}^{{\widehat{r}}_i}<0.5$ because we can flip the estimate if $p_{e,i}^{{\widehat{r}}_i}>0.5$. We are not interested in the case of $p_{e,i}^{{\widehat{r}}_i}=0.5$ since ${\widehat{r}}_{i,t}$ is useless in this case. Although the channel is unreliable, each transmission attempt takes exactly one time slot regardless of the result, and the successfully transmitted update will not be corrupted. Every time an update is received, the receiver will use it as the new estimate ${\widehat{X}}_{i,t}$. The receiver will send an $ACK/NACK$ packet to inform the base station of its reception of the new update. Since an $ACK/NACK$ packet is generally very small and simple, we assume that it is transmitted reliably and received instantaneously. Then, if $ACK$ is received, the base station knows that the receiver’s estimate changed to the transmitted update. If $NACK$ is received, the base station knows that the receiver’s estimate did not change. Therefore, the base station always knows the estimate at the receiver side.

At the beginning of each time slot, the base station receives updates from each source and the estimates of CSI from each channel. The old updates and estimates are discarded upon the arrival of new ones. Then, the base station decides which updates to transmit, and the decision is independent of the transmission history. Due to the limited resources, at most $M<N$ updates are allowed per transmission attempt. We consider a base station that always transmits M updates.

2.2. Age of Incorrect Information

All the users adopt AoII as a performance metric, but the choices of penalty functions vary. Let $X_t$ and ${\widehat{X}}_t$ be the true state and the estimate of the source process, respectively. Then, in a slotted-time system, AoII can be expressed as follows

$${\Delta }_{AoII}(X_t,{\widehat{X}}_t,t)=\sum _{k=U_t+1}^t\left(g(X_k,{\widehat{X}}_k)\times F(k-U_t)\right),$$

(1)

where $U_t$ is the last time instance before time t (including t) that the receiver’s estimate is correct. $g(X_t,{\widehat{X}}_t)$ can be any information penalty function that captures the difference between $X_t$ and ${\widehat{X}}_t$. $F\left(t\right)\triangleq f\left(t\right)-f(t-1)$ where $f\left(t\right)$ can be any time penalty function that is non-decreasing in t. We consider the case where the users adopt the same information penalty function $g(X_t,{\widehat{X}}_t)=|X_t-{\widehat{X}}_t|$ but possibly different time penalty functions. To ease the analysis, we require $f\left(t\right)$ to be unbounded. Combined together, we require $f\left(t_1\right)\le f\left(t_2\right)$ if $t_1<t_2$ and ${lim}_{t\to +\infty }f\left(t\right)=+\infty$. Without a loss of generality, we assume $f\left(0\right)=0$, as the source is modeled by a two-state Markov chain, $g(X_t,{\widehat{X}}_t)\in \{0,1\}$. Hence, Equation (1) can be simplified to

$${\Delta }_{AoII}(X_t,{\widehat{X}}_t,t)=\sum _{k=U_t+1}^{t}F(k-U_t)=f\left(s_t\right),$$

where $s_t\triangleq t-U_t$. Therefore, the evolution of $s_t$ is sufficient to characterize the evolution of AoII. To this end, we distinguish between the following cases.

• When the receiver’s estimate is correct at time $t+1$, we have $U_{t+1}=t+1$. Then, by definition, $s_{t+1}=0$.
• When the receiver’s estimate is incorrect at time $t+1$, we have $U_{t+1}=U_t$. Then, by definition, $s_{t+1}=t+1-U_t=s_t+1$.

To sum up, we get

$$s_{t+1}={𝟙}_{\{U_{t+1}\ne t+1\}}\times (s_t+1).$$

(2)

A sample path of $s_t$ is shown in Figure 2. In the remainder of this paper, we use $f_i(·)$ to denote the time penalty function user i adopts.

Figure 2. A sample path of $s_t$.

Remark 1.

Under this particular choice of the penalty function, $s_t$ can be interpreted as the time elapsed since the last time the receiver’s estimate is correct. Please note that $s_t$ is different from the Age of Information (AoI) [17], which is defined as the time elapsed since the generation time of the last received update. We can see that AoI considers the aging process of the update, while AoII considers the aging process of the estimation error. At the same time, $s_t$ is also fundamentally different from the holding time, which, according to [18,19], is defined as the time elapsed since the last successful transmission. We notice that the receiver’s estimate can become correct even when no new update is successfully transmitted. Moreover, the information carried by the update may have become incorrect by the time it is received. We also notice that [18,19] consider the problem of minimizing the estimation error. However, by adopting AoII as the performance metric, we study the impact of estimation error on the system.

2.3. System Dynamic

In this section, we tackle the system dynamic. We notice that the status of user i can be captured by the pair $x_{i,t}\triangleq (s_{i,t},{\widehat{r}}_{i,t})$. In the following, we will use $x_{i,t}$ and $(s_{i,t},{\widehat{r}}_{i,t})$ interchangeably. Then, the system dynamic can be fully characterized by the dynamic of $x_t\triangleq (x_{1,t},\dots ,x_{N,t})$. Hence, it suffices to characterize the value of $x_{t+1}$ given $x_t$ and the base station’s action. To this end, we denote, by $a_t=(a_{1,t},\dots ,a_{N,t})$, the base station’s action at time t. $a_{i,t}=1$ if the base station transmits the update from user i at time t and $a_{i,t}=0$ otherwise. We notice that given action $a_t$, users are independent and the action taken on user i will only affect itself. Consequently

$$Pr(x_{t+1}\mid x_t,a_t)=\prod _{i=1}^{N}Pr(x_{i,t+1}\mid x_{i,t},a_t)=\prod _{i=1}^{N}Pr(x_{i,t+1}\mid x_{i,t},a_{i,t}).$$

Combined with the fact that all the users share the same structure, it is sufficient to study the dynamic of a single user. In the following discussions, we drop the user-dependent subscript i. We recall that ${\widehat{r}}_{t+1}$ is an independent Bernoulli random variable. Then, we have

$$Pr(x_{t+1}\mid x_t,a_t)=P\left({\widehat{r}}_{t+1}\right)\times Pr(s_{t+1}\mid x_t,a_t).$$

(3)

By definition, $P({\widehat{r}}_{t+1}=1)=\gamma$ and $P({\widehat{r}}_{t+1}=0)=1-\gamma$. Then, we only need to tackle the value of $Pr(s_{t+1}\mid x_t,a_t)$. To this end, we distinguish between the following cases

• When $x_t=(0,{\widehat{r}}_t)$, the estimate at time t is correct (i.e., ${\widehat{X}}_t=X_t$). Hence, for the receiver, $X_t$ carries no new information about the source process. In other words, ${\widehat{X}}_{t+1}={\widehat{X}}_t$ regardless of whether an update is transmitted at time t. We recall that $U_{t+1}=U_t$ if ${\widehat{X}}_{t+1}\ne X_{t+1}$ and $U_{t+1}=t+1$ otherwise. Since the source is binary, we obtain $U_{t+1}=U_t$ if $X_{t+1}\ne X_t$, which happens with probability p and $U_{t+1}=t+1$ otherwise. According to (2), we obtain

  $$Pr(1\mid (0,{\widehat{r}}_t),a_t)=p,$$

  $$Pr(0\mid (0,{\widehat{r}}_t),a_t)=1-p.$$
• When $a_t=0$ and $x_t=(s_t,{\widehat{r}}_t)$, where $s_t>0$, the channel will not be used and no new update will be received by the receiver, and so, ${\widehat{X}}_{t+1}={\widehat{X}}_t$. We recall that $U_{t+1}=U_t$ if ${\widehat{X}}_{t+1}\ne X_{t+1}$ and $U_{t+1}=t+1$ otherwise. Since $X_t\ne {\widehat{X}}_t$ and the source is binary, we have $U_{t+1}=U_t$ if $X_{t+1}=X_t$, which happens with probability $1-p$ and $U_{t+1}=t+1$ otherwise. According to (2), we obtain

  $$Pr(s_t+1\mid (s_t,{\widehat{r}}_t),a_t=0)=1-p,$$

  $$Pr(0\mid (s_t,{\widehat{r}}_t),a_t=0)=p.$$
• When $a_t=1$ and $x_t=(s_t,1)$ where $s_t>0$, the transmission attempt will succeed with probability $1-p_e^1$ and fail with probability $p_e^1$. We recall that $U_{t+1}=U_t$ if ${\widehat{X}}_{t+1}\ne X_{t+1}$ and $U_{t+1}=t+1$ otherwise. Then, when the transmission attempt succeeds (i.e., ${\widehat{X}}_{t+1}=X_t$), $U_{t+1}=U_t$ if $X_{t+1}\ne X_t$ and $U_{t+1}=t+1$ otherwise. When the transmission attempt fails (i.e., ${\widehat{X}}_{t+1}={\widehat{X}}_t\ne X_t$), we have $U_{t+1}=U_t$ if $X_{t+1}=X_t$ and $U_{t+1}=t+1$ otherwise. Combining (2) with the dynamic of the source process we obtain

  $$Pr(s_t+1\mid (s_t,1),a_t=1)=p_e^1(1-p)+(1-p_e^1)p\triangleq \alpha ,$$

  $$Pr(0\mid (s_t,1),a_t=1)=p_e^{1}p+(1-p_e^1)(1-p)=1-\alpha .$$
• When $a_t=1$ and $x_t=(s_t,0)$, where $s_t>0$, following the same line, we obtain

  $$Pr(s_t+1\mid (s_t,0),a_t=1)=p_e^{0}p+(1-p_e^0)(1-p)\triangleq \beta ,$$

  $$Pr(0\mid (s_t,0),a_t=1)=p_e^0(1-p)+(1-p_e^0)p=1-\beta .$$

Combines together, we obtain the value of $Pr(s_{t+1}\mid x_t,a_t)$ in all cases. As only M out of N updates are allowed per transmission attempt, we realize a necessity to require transmission attempts always help minimize AoII. It is equivalent to impose $Pr(s_{t+1}>s_t\mid (s_t,{\widehat{r}}_t),a_t=0)>Pr(s_{t+1}>s_t\mid (s_t,{\widehat{r}}_t),a_t=1)$ for any $(s_t,{\widehat{r}}_t)$. Leveraging the results above, it is sufficient to require $p<0.5$. As all the users share the same structure, we assume, for the rest of this paper, that $0<p_i<0.5$ for $1\le i\le N$.

2.4. Problem Formulation

The communication goal is to minimize the expected AoII. Therefore, the problem can be formulated as the following

$$\begin{array}{}\mathrm{(4a)}& \underset{\varphi \in \mathrm{\Phi }}{\arg \min }& \underset{T\to \infty }{\lim }\frac{1}{T}{\mathbb{E}}_{\varphi }(\sum _{t=0}^{T-1}\sum _{i=1}^N{f}_i(s_{i,t}))\mathrm{(4b)}& \mathrm{subject} \mathrm{to}& \sum _{i=1}^N{a}_{i,t} = M\hspace{1em}\forall t,\end{array}$$

where $\mathrm{\Phi }$ is the set of all causal policies. We refer to the constrained minimization problem reported in problem (4) as the Primal Problem (PP). We notice that the PP is a Restless Multi-Armed Bandit (RMAB) Problem. The optimal policy for this type of problem is far from reachable since it is PSPACE-hard in general [20]. However, we can still derive the structural properties of the optimal policy. These structural properties can be used as a guide for the development of scheduling policies and can indicate the good performance of the developed scheduling policies.

3. Structural Properties of the Optimal Policy

In this section, we investigate the structural properties of the optimal policy for PP. We first define an infinite horizon with an average cost Markov Decision Process (MDP) ${\mathcal{M}}_N(w,M)=({\mathcal{X}}_N,{\mathcal{A}}_N\left(M\right),{\mathcal{P}}_N,{\mathcal{C}}_N\left(w\right))$, where

• ${\mathcal{X}}_N$ denotes the state space. The state is $x=(x_1,\dots ,x_N)$ where $x_i=(s_i,{\widehat{r}}_i)$.
• ${\mathcal{A}}_N\left(M\right)$ denotes the action space. The feasible action is $\mathit{a}=(a_1,\dots ,a_N)$ where $a_i\in \{0,1\}$ and ${\sum }_{i=1}^N{a}_i=M$. Note that the feasible actions are independent of the state and the time.
• ${\mathcal{P}}_N$ denotes the state transition probabilities. We define $P_{\mathit{x},{\mathit{x}}^{\prime }}\left(\mathit{a}\right)$ as the probability that action $\mathit{a}$ at state $\mathit{x}$ will lead to state ${\mathit{x}}^{\prime }$. It is calculated by

  $$P_{\mathit{x},{\mathit{x}}^{\prime }}\left(\mathit{a}\right)=\prod _{i=1}^{N}P\left({\widehat{r}}_i^{\prime }\right)P_{s_i,s_i^{\prime }}(a_i,{\widehat{r}}_i),$$

  where $P_{s_i,s_i^{\prime }}(a_i,{\widehat{r}}_i)$ is the transition probability from $s_i$ to $s_i^{\prime }$ when the estimate of CSI is ${\widehat{r}}_i$ and action $a_i$ is taken. The values of $P_{s_i,s_i^{\prime }}(a_i,{\widehat{r}}_i)$ can be obtained easily from the results in Section 2.3.
• ${\mathcal{C}}_N\left(w\right)$ denotes the instant cost. When the system is at state $\mathit{x}$ and action $\mathit{a}$ is taken, the instant cost is $C(\mathit{x},\mathit{a})\triangleq {\sum }_{i=1}^{N}C(x_i,a_i)\triangleq {\sum }_{i=1}^N\left(f_i\left(s_i\right)+w{a}_i\right)$.

We notice that PP can be cast into ${\mathcal{M}}_N(0,M)$. Since $w=0$, the instant cost is independent of action $\mathit{a}$. Therefore, we abbreviate $C(\mathit{x},\mathit{a})$ as $C\left(\mathit{x}\right)$. To simplify the analysis, we consider the case of $M=1$. Equivalently, we investigate the structural properties of the optimal policy for ${\mathcal{M}}_N(0,1)$.

Remark 2.

For the case of $M>1$, we can apply the same methodology. However, as M increases, the action space will grow quickly, resulting in the need to consider more feasible actions in each step of the proof. Hence, to better demonstrate the methodology, we only consider the case of $M=1$ in this paper.

It is well known that the optimal policy for ${\mathcal{M}}_N(0,1)$ can be characterized by the value function. We denote the value function of state $\mathit{x}$ as $V\left(\mathit{x}\right)$. A canonical procedure to calculate $V\left(\mathit{x}\right)$ is applying the Value Iteration Algorithm (VIA). To this end, we define $V_{\nu }(·)$ as the estimated value function at iteration $\nu$ of VIA and initialize $V_0(·)=0$. Then, VIA updates the estimated value functions in the following way

$$\begin{array}{cc}\hfill V_{\nu +1}\left(\mathit{x}\right)& =C\left(\mathit{x}\right)-\theta +\underset{\mathit{a}\in {\mathcal{A}}_N\left(1\right)}{min}\left\{\sum _{{\mathit{x}}^{\prime }\in {\mathcal{X}}_N}{P}_{\mathit{x},{\mathit{x}}^{\prime }}\left(\mathit{a}\right)V_{\nu }\left({\mathit{x}}^{\prime }\right)\right\},\hfill \end{array}$$

(5)

where $\theta$ is the optimal value of ${\mathcal{M}}_N(0,1)$. VIA is guaranteed to converge to the value function [21]. More precisely, $V_{\nu }(·)=V(·)$ when $\nu \to +\infty$. However, the exact value function is impossible to get since we need infinite iterations and the state space is infinite. Instead, we provide two structural properties of the value function.

Lemma 1 (Monotonicity).

For ${\mathcal{M}}_N(0,1)$, $V\left(x\right)$ is non-decreasing in $s_i$ for $1\le i\le N$.

Proof. 

Leveraging the iterative nature of VIA, we use mathematical induction to prove the desired results. The complete proof can be found in Appendix A.    ☐

Before introducing the next structural property, we make the following definition.

Definition 1 (Statistically identical).

Two users are said to be statistically identical if the user-dependent parameters and the adopted time penalty functions are the same.

For the users that are statistically identical, we can prove the following

Lemma 2 (Equivalence).

For ${\mathcal{M}}_N(0,1)$, if users j and k are statistically identical, $V\left(\mathit{x}\right)=V$$\left(\mathcal{P}\right(\mathit{x}\left)\right)$ where $\mathcal{P}\left(\mathit{x}\right)$ is state $\mathit{x}$ with $x_j$ and $x_k$ exchanged.

Proof. 

Leveraging the iterative nature of VIA, we use mathematical induction to prove the desired results. At each iteration, we show that for each feasible action at state $\mathit{x}$, we can find an equivalent action at state $\mathcal{P}\left(\mathit{x}\right)$. Two actions are equivalent if they lead to the same value function. The complete proof can be found in Appendix B.    ☐

Equipped with the above lemmas, we proceed with characterizing the structural properties of the optimal policy. We recall that the optimal action at each state can be characterized by the value function. Hence, we denote, by $V^j\left(\mathit{x}\right)$, the value function resulting from choosing user j to update at state $\mathit{x}$. Then, $V^j\left(\mathit{x}\right)$ can be calculated by

$$V^j\left(\mathit{x}\right)=C\left(\mathit{x}\right)-\theta +\sum _{{\mathit{x}}^{\prime }-x_j^{\prime }}\left\{\left(\prod _{i\ne j}{P}_{x_i,x_i^{\prime }}\left(0\right)\right)\sum _{{\widehat{r}}_j^{\prime }}\left[P\left({\widehat{r}}_j^{\prime }\right)\left(\sum _{s_j^{\prime }}{P}_{s_j,s_j^{\prime }}(1,{\widehat{r}}_j)V\left({\mathit{x}}^{\prime }\right)\right)\right]\right\}.$$

If $V^j\left(\mathit{x}\right)<V^k\left(\mathit{x}\right)$ for all $k\ne j$, it is optimal to transmit the update from user j. When $V^j\left(\mathit{x}\right)=V^k\left(\mathit{x}\right)$, the two choices are equally desirable. In the following, we will characterize the properties of ${\delta }^{j,k}\left(\mathit{x}\right)\triangleq V^j\left(\mathit{x}\right)-V^k\left(\mathit{x}\right)$ for any j and k.

Theorem 1 (Structural properties).

For ${\mathcal{M}}_N(0,1)$, ${\delta }^{j,k}\left(x\right)$ has the following properties

• ${\delta }^{j,k}\left(\mathit{x}\right)\le 0$ if ${\widehat{r}}_k=p_{e,k}^0=0$. The equality holds when $s_j=0$ or ${\widehat{r}}_j=p_{e,j}^0=0$.
• ${\delta }^{j,k}\left(\mathit{x}\right)$ is non-increasing in ${\widehat{r}}_j$ and is non-decreasing in ${\widehat{r}}_k$ when $s_j,s_k>0$. At the same time, ${\delta }^{j,k}\left(\mathit{x}\right)$ is independent of ${\widehat{r}}_i$ for any $i\ne j,k$.
• ${\delta }^{j,k}\left(\mathit{x}\right)\le 0$ if $s_k=0$. The equality holds when $s_j=0$ or ${\widehat{r}}_j=p_{e,j}^0=0$.
• ${\delta }^{j,k}\left(\mathit{x}\right)$ is non-increasing in $s_j$ if ${\Gamma }_j^{{\widehat{r}}_j}\le {\Gamma }_k^{{\widehat{r}}_k}$ and is non-decreasing in $s_k$ if ${\Gamma }_j^{{\widehat{r}}_j}\ge {\Gamma }_k^{{\widehat{r}}_k}$ when $s_j,s_k>0$. We define ${\Gamma }_i^1\triangleq \frac{{\alpha }_i}{1-p_i}$ and ${\Gamma }_i^0\triangleq \frac{{\beta }_i}{1-p_i}$ for $1\le i\le N$.
• ${\delta }^{j,k}\left(\mathit{x}\right)\le 0$ if $s_j\ge s_k$, ${\widehat{r}}_j\ge {\widehat{r}}_k$, and users j and k are statistically identical.

Proof. 

The proof can be found in Appendix C.    ☐

We notice that ${\Gamma }_i^{{\widehat{r}}_i}$ can be written as

$${\Gamma }_i^{{\widehat{r}}_i}=\frac{Pr(s_i+1\mid (s_i,{\widehat{r}}_i),a_i=1)}{Pr(s_i+1\mid (s_i,{\widehat{r}}_i),a_i=0)}<1,$$

where $s_i$ can be any positive integer. Consequently, ${\Gamma }_i^{{\widehat{r}}_i}$ is independent of any $s_i>0$ and indicates the decrease in the probability of increasing $s_i$ caused by action $a_i=1$. When ${\Gamma }_i^{{\widehat{r}}_i}$ is large, action $a_i=1$ will achieve a small decrease in the probability of increasing $s_i$. In the following, we provide an intuitive interpretation of why the monotonicity in Property 4 of Theorem 1 depends on ${\Gamma }_i^{{\widehat{r}}_i}$. We take the case of ${\Gamma }_j^{{\widehat{r}}_j}\le {\Gamma }_k^{{\widehat{r}}_k}$ as an example and assume that there are only users j and k in the system. Then, according to Section 2.3, the dynamic of $s_j$ and $s_k$ can be divided into the following three cases

• Neither $s_j$ nor $s_k$ increases. In this case, both $s_j$ and $s_k$ become zero.
• Either $s_j$ or $s_k$ increases and the other becomes zero. We denote by $P_j^k$ the probability that only $s_k$ increases when $a_j=1$. The notation for other cases is defined analogously. The probabilities can be obtained easily using the results in Section 2.3.
• Both $s_j$ and $s_k$ increase. We denote by $P_j$ the probability that both $s_j$ and $s_k$ increase when $a_j=1$. $P_k$ is defined analogously. The probabilities can be obtained easily using the results in Section 2.3.

We notice that ${\delta }^{j,k}\left(\mathit{x}\right)$ implies the tendency of the base station to choose between the two users. The larger ${\delta }^{j,k}\left(\mathit{x}\right)$ is, the more the base station tends to choose user k. Thus, we investigate the base station’s propensity to choose user k when $s_k$ increases but $s_j$ stays the same. We ignore the case where the resulting $s_k$ is zero since it is independent of the increase in $s_k$. With this in mind, we first notice that $P_k^k\le P_j^k$. Meanwhile, we can easily verify that $\frac{P_j}{P_k}=\frac{{\Gamma }_j^{{\widehat{r}}_j}}{{\Gamma }_k^{{\widehat{r}}_k}}$. When ${\Gamma }_j^{{\widehat{r}}_j}\le {\Gamma }_k^{{\widehat{r}}_k}$, we have $P_j\le P_k$. Then, there exists a subtle trade-off. More precisely, choosing user k will result in $P_k^k\le P_j^k$, but at the cost of $P_k\ge P_j$. Hence, in this case, the propensity of the base station is hard to determine. Following the same line, we can show that choosing user j will lead to $P_j^j\le P_k^j$ and $P_j\le P_k$. Thus, there exists no such trade-off when we investigate the base station’s propensity to choose user j as $s_j$ increases but $s_k$ stays the same.

Leveraging Theorem 1, we can provide some specific structural properties of the optimal policy.

Corollary 1 (Application of Theorem 1).

When $M=1$, the optimal policy for PP must satisfy the following

• The user i with ${\widehat{r}}_i=p_{e,i}^0=0$ or $s_i=0$ will not be chosen unless it is to break the tie.
• When user j is chosen at state ${\mathit{x}}_1$, then for state ${\mathit{x}}_2$, such that ${\widehat{r}}_{1,j}\le {\widehat{r}}_{2,j}$ and $s_{1,i}=s_{2,i}$ for $1\le i\le N$, the optimal choice must be in the set $G=\left\{j\right\}\cup \{k:{\widehat{r}}_{1,k}<{\widehat{r}}_{2,k}\}$.
• When $N=2$, we consider two states, ${\mathit{x}}_1$ and ${\mathit{x}}_2$, which differ only in the value of $s_j$. Specifically, $s_{1,j}\le s_{2,j}$. If user j is chosen at state ${\mathit{x}}_1$ and ${\Gamma }_j^{{\widehat{r}}_{1,j}}\le {\Gamma }_k^{{\widehat{r}}_{1,k}}$, the optimal choice at state ${\mathit{x}}_2$ will also be user j.
• When $N=2$, we consider two states, ${\mathit{x}}_1$ and ${\mathit{x}}_2$, which differ only in the value of $s_k$. Specifically, $s_{1,k}\ge s_{2,k}$. If user j is chosen at state ${\mathit{x}}_1$ and ${\Gamma }_j^{{\widehat{r}}_{1,j}}\ge {\Gamma }_k^{{\widehat{r}}_{1,k}}$, the optimal choice at state ${\mathit{x}}_2$ will also be user j.
• When all users are statistically identical, the optimal choice at any time slot must be either the user with $x=(s_{max,1},1)$ where $s_{max,1}\triangleq {max}_{s_i}\left\{(s_i,1)\right\}$ or the user with $x=(s_{max,0},0)$ where $s_{max,0}\triangleq {max}_{s_i}\left\{(s_i,0)\right\}$. Moreover,
  • If $s_{max,1}\ge s_{max,0}$, it is optimal to choose the user with $x=(s_{max,1},1)$.
  • If $s_{max,1}<s_{max,0}$, the optimal choice will switch from the user with $x=(s_{max,0},0)$ to the user with $x=(s_{max,1},1)$ when $s_{max,1}$ increases from 0 to $s_{max,0}$ solely.

Proof. 

The first property follows directly from Property 1 and Property 3 of Theorem 1. For the second property, leveraging Property 2 of Theorem 1, we have ${\delta }^{j,k}\left(x_2\right)\le {\delta }^{j,k}\left(x_1\right)\le 0$ if ${\widehat{r}}_{1,j}\le {\widehat{r}}_{2,j}$, ${\widehat{r}}_{1,k}\ge {\widehat{r}}_{2,k}$, and $s_{1,i}=s_{2,i}$ for $1\le i\le N$. Thus, the optimal choice will not be user k in this case. Then, we can conclude that the optimal choice must be in the set $G=\left\{j\right\}\cup \{k:{\widehat{r}}_{1,k}<{\widehat{r}}_{2,k}\}$.

For the third property, we have proved in Property 4 of Theorem 1 that ${\delta }^{j,k}\left(\mathit{x}\right)$ is non-increasing in $s_j$ if ${\Gamma }_j^{{\widehat{r}}_j}\le {\Gamma }_k^{{\widehat{r}}_k}$. Hence, ${\delta }^{j,k}\left({\mathit{x}}_2\right)\le {\delta }^{j,k}\left({\mathit{x}}_1\right)\le 0$. As we consider the case of $N=2$, the optimal choice at state ${\mathit{x}}_2$ will also be user j. The fourth property can be shown in a similar way by noticing that ${\delta }^{j,k}\left(\mathit{x}\right)$ is non-decreasing in $s_k$ when ${\Gamma }_j^{{\widehat{r}}_j}\ge {\Gamma }_k^{{\widehat{r}}_k}$.

For the last property, we recall from Property 5 of Theorem 1 that it is always better to choose the user with a larger s if they are statistically identical and have the same $\widehat{r}$. Thus, we can conclude that the optimal choice must be either the user with $x=(s_{max,1},1)$ or the user with $x=(s_{max,0},0)$. Without a loss of generality, we assume $x_j=(s_{max,1},1)$ and $x_k=(s_{max,0},0)$. Now, we distinguish between the following cases

• According to Property 5 of Theorem 1, we can conclude that it is optimal to choose user j when $s_{max,1}\ge s_{max,0}$.
• To determine the optimal choice in the case of $s_{max,1}<s_{max,0}$, we recall that the optimal choice will be user k (i.e., ${\delta }^{j,k}\left(\mathit{x}\right)\ge 0$) if $s_j=0$ and will be user j (i.e., ${\delta }^{j,k}\left(\mathit{x}\right)\le 0$) if $s_j=s_k$. At the same time, Property 4 of Theorem 1 tells us that ${\delta }^{j,k}\left(\mathit{x}\right)$ is non-increasing in $s_j$ when users j and k are statistically identical. Therefore, we can conclude that the optimal choice will switch from user k to user j when $s_j$ increases from 0 to $s_k$ solely.

   ☐

4. Whittle’s Index Policy

Whittle’s index policy is a well-known low-complexity heuristic that shows a strong performance in many problems that belong to RMAB [22,23,24]. In this section, we develop Whittle’s index policy for PP. We first present the general procedures we adopt to obtain Whittle’s index.

• We first formulate a relaxed version of PP and apply the Lagrangian approach.
• Then, we decouple the problem of minimizing the Lagrangian function into N decoupled problems, each of which only considers a single user. By casting the decoupled problem into an MDP, we investigate the structural properties and performance of the optimal policy.
• Leveraging the results above and under a simple condition, we establish the indexability of the decoupled problem.
• Finally, we obtain the expression of Whittle’s index by solving the Bellman equation.

4.1. Relaxed Problem

The first step in obtaining Whittle’s index is to formulate the Relaxed Problem (RP). More precisely, instead of requiring the limit on the number of updates allowed per transmission attempt to be met in each time slot, we relax the constraint such that the limit is not violated in an average sense. Then, RP can be formulated as

$$\begin{array}{cc}\hfill \underset{\varphi \in \mathrm{\Phi }}{\arg \min }& \overline{\Delta } \triangleq \underset{T\to \infty }{\lim }\frac{1}{T}{\mathbb{E}}_{\varphi }(\sum _{t=0}^{T-1}\sum _{i=1}^N{f}_i(s_{i,t}))\hfill \end{array}$$

(6a)

$$\begin{array}{cc}\hfill \mathrm{subject} \mathrm{to}& {\overline{\rho }}_{\varphi } \triangleq \underset{T\to \infty }{\lim }\frac{1}{T}{\mathbb{E}}_{\varphi }\left(\sum _{t=0}^{T-1}\sum _{i=1}^N{a}_{i,t}\right) \le M.\hfill \end{array}$$

(6b)

As RP is specified, we apply the Lagrangian approach. First of all, we write RP into its Lagrangian form.

$$\mathcal{L}(\lambda ,\varphi )=\underset{T\to \infty }{lim}\frac{1}{T}{\mathbb{E}}_{\varphi }\left(\sum _{t=0}^{T-1}\sum _{i=1}^N(f_i\left(s_{i,t}\right)+\lambda a_{i,t})\right)-\lambda M,$$

where $\lambda \ge 0$ is the Lagrange multiplier. Then, we investigate the problem of minimizing the Lagrangian function. Since $\lambda M$ is independent of policies, we can ignore it. More precisely, we consider the following minimization problem

$$\begin{array}{cc}\hfill \underset{\varphi \in \mathrm{\Phi }}{\mathrm{minimize}}& \underset{T\to \infty }{\lim }\frac{1}{T}{\mathbb{E}}_{\varphi }(\sum _{t=0}^{T-1}\sum _{i=1}^N(f_i(s_{i,t}) + \lambda a_{i,t}).\hfill \end{array}$$

(7)

4.2. Decoupled Model

In this section, we formulate the decoupled problem and investigate its optimal policy. The decoupled model associated with each user follows the system model with $N=1$. Since all the users share the same structure, we drop the user-dependent subscript i for simplicity. Then, the decoupled problem can be formulated as

$$\begin{array}{cc}\hfill \underset{\varphi \in {\mathrm{\Phi }}^{\prime }}{\mathrm{minimize}}& \underset{T\to \infty }{\lim }\frac{1}{T}{\mathbb{E}}_{\varphi }(\sum _{t=0}^{T-1}(f(s_t) + \lambda a_t)),\hfill \end{array}$$

(8)

where ${\Phi }^{\prime }$ is the set of all causal policies when $N=1$. We notice that problem (8) can be cast into the MDP ${\mathcal{M}}_1(\lambda ,-1)$. We define $M=-1$ when there is no restriction on the number of updates allowed per transmission attempt.

We first investigate the structural properties of the optimal policy for ${\mathcal{M}}_1(\lambda ,-1)$ when $\lambda$ is a given non-negative constant. We start with characterizing the corresponding value function $V\left(x\right)$.

Corollary 2 (Extension of Lemma 1).

For ${\mathcal{M}}_1(\lambda ,-1)$, $V\left(x\right)$ is non-decreasing in s.

Proof. 

The proof follows the same steps as in the proof of Lemma 1. The complete proof can be found in Appendix D.    ☐

Equipped with the above corollary, we can characterize the structural properties of the optimal policy for (8).

Proposition 1 (Optimal policy for decoupled problem).

The optimal policy for the decoupled problem is a threshold policy with the following properties.

• The optimal policy can be fully captured by $\mathit{n}=(n_0,n_1)$. More precisely, when the system is at state $(s,\widehat{r})$, it is optimal to make a transmission attempt only when $s\ge n_{\widehat{r}}$.
• $n_0\ge n_1>0$.

Proof. 

We define $\Delta V\left(x\right)\triangleq V^1\left(x\right)-V^0\left(x\right)$, where $V^a\left(x\right)$ is the value function resulting from taking action a at state x. Then, the optimal action at state x is $a=1$ if $\Delta V\left(x\right)<0$, and $a=0$ is optimal otherwise. We use Corollary 2 to characterize the sign of $\Delta V\left(x\right)$. The complete proof can be found in Appendix E.    ☐

In the following, we evaluate the performance of the threshold policy detailed in Proposition 1. More precisely, we calculate the expected AoII ${\overline{\Delta }}_{\mathit{n}}$ and the expected transmission rate ${\overline{\rho }}_{\mathit{n}}$ resulting from the adoption of threshold policy $\mathit{n}$. We will see in the following that ${\overline{\Delta }}_{\mathit{n}}$ and ${\overline{\rho }}_{\mathit{n}}$ are essential for establishing the indexability and obtaining the expression of Whittle’s index.

Proposition 2 (Performance).

Under threshold policy $\mathit{n}=(n_0,n_1)$,

$${\overline{\Delta }}_{\mathit{n}}={\pi }_{0}p\left[\sum _{k=1}^{n_1-1}f\left(k\right){(1-p)}^{k-1}+{(1-p)}^{n_1-1}\left(\sum _{k=n_1}^{n_0-1}f\left(k\right)c_1^{k-n_1}+c_1^{n_0-n_1}\sum _{k=n_0}^{+\infty }f\left(k\right)c_2^{k-n_0}\right)\right],$$

$${\overline{\rho }}_{\mathit{n}}={\pi }_{0}p{(1-p)}^{n_1-1}\left[\frac{\gamma }{1-c_1}+c_1^{n_0-n_1}\left(\frac{1}{1-c_2}-\frac{\gamma }{1-c_1}\right)\right],$$

where

$${\pi }_0=\frac{{\displaystyle 1}}{{\displaystyle 2+p{(1-p)}^{n_1-1}\left[\frac{1}{1-c_1}-\frac{1}{p}+c_1^{n_0-n_1}\left(\frac{1}{1-c_2}-\frac{1}{1-c_1}\right)\right]}},$$

$c_1=(1-\gamma )(1-p)+\gamma \alpha$, and $c_2=(1-\gamma )\beta +\gamma \alpha$.

Proof. 

We notice that the dynamic of AoII under the threshold policy can be fully captured by a Discrete-Time Markov Chain (DTMC). Then, combined with the fact that $\widehat{r}$ is an independent Bernoulli random variable, we can obtain the desired results from the stationary distribution of the induced DTMC. The complete proof can be found in Appendix F.    ☐

As $f(·)$ can be any non-decreasing function, $\overline{\Delta }$ can grow indefinitely. Thus, it is necessary to require that there exists at least one threshold policy that causes a finite $\overline{\Delta }$. By noting that $1-p\ge c_1\ge c_2$, we have

$$\begin{array}{cc}\hfill \overline{\Delta }& \ge {\pi }_{0}p\left[\sum _{k=1}^{n_1-1}f\left(k\right)c_2^{k-1}+c_2^{n_1-1}\left(\sum _{k=n_1}^{n_0-1}f\left(k\right)c_2^{k-n_1}+c_2^{n_0-n_1}\sum _{k=n_0}^{+\infty }f\left(k\right)c_2^{k-n_0}\right)\right]\hfill \\ \hfill & ={\pi }_{0}p\left(\sum _{k=1}^{+\infty }f\left(k\right)c_2^{k-1}\right).\hfill \end{array}$$

The equality is achieved when $n_0=n_1=1$. Then, we can conclude that it is sufficient to require ${\sum }_{k=1}^{+\infty }f\left(k\right)c_2^{k-1}<+\infty$. This will be the underlying assumption throughout the rest of this paper.

4.3. Indexability

In this section, we establish the indexability of the decoupled problem, which ensures the existence of Whittle’s index. We start with the definition of indexability.

Definition 2

(Indexability). The decoupled problem is indexable if the set of states in which $a=0$ is the optimal action increases with λ, that is,

$${\lambda }^{\prime }<\lambda ⟹D\left({\lambda }^{\prime }\right)\subseteq D\left(\lambda \right),$$

where $D\left(\lambda \right)$ is the set of states in which $a=0$ is optimal when Lagrange multiplier λ is adopted.

The Lagrange multiplier $\lambda$ can be viewed as a cost associated with each transmission attempt. Intuitively, as $\lambda$ increases, the base station should stay idle (i.e., $a=0$) for a longer time until s becomes large enough to offset the cost. Although it is intuitively correct that the decoupled problem is indexable, the indexability is hard to establish as the optimal policy is characterized by two thresholds. Thus, Whittle’s index does not necessarily exist. However, the indexability can be established when the following condition is satisfied

$$p_{e,i}^0=0for1\le i\le N.$$

(9)

Remark 3.

Problem (9) only requires the estimate ${\widehat{r}}_i$ to be perfect when ${\widehat{r}}_i=0$. In the case of ${\widehat{r}}_i=1$, we still allow the estimate to be inaccurate.

When (9) is satisfied, Propositions 1 and 2 reduce to the following

Corollary 3 (Consequences of (9)).

When (9) is satisfied, the optimal policy for the decoupled problem (8) is the threshold policy $\mathit{n}=(+\infty ,n)$. The corresponding ${\overline{\Delta }}_{\mathit{n}}$ and ${\overline{\rho }}_{\mathit{n}}$ are

$${\overline{\Delta }}_{\mathit{n}}={\pi }_{0}p\left(\sum _{k=1}^{n-1}f\left(k\right){(1-p)}^{k-1}+{(1-p)}^{n-1}\sum _{k=n}^{+\infty }f\left(k\right)c_1^{k-n}\right),$$

$${\overline{\rho }}_{\mathit{n}}={\pi }_{0}p{(1-p)}^{n-1}\left(\frac{\gamma }{1-c_1}\right),$$

where

$${\pi }_0=\frac{{\displaystyle 1}}{{\displaystyle 2+p{(1-p)}^{n-1}\left(\frac{1}{1-c_1}-\frac{1}{p}\right)}}.$$

Proof. 

We continue with the same notations as in the proof of Propositions 1 and 2. It is sufficient to show that $n_0=+\infty$. To this end, we consider the state $x=(s,0)$. By following the same steps as in the proof of Proposition 1, we have

$$\Delta V(s,0)=\lambda \ge 0.$$

Therefore, it is optimal to stay idle (i.e., $a=0$) at state $x=(s,0)$ for any $s\ge 0$. Equivalently, $n_0=+\infty$. Then, the corresponding ${\overline{\Delta }}_{\mathit{n}}$ and ${\overline{\rho }}_{\mathit{n}}$ can be calculated as a special case of Proposition 2 where $n_0=+\infty$, $n_1=n$, and $p_e^0=0$.    ☐

Leveraging Corollary 3, we can establish the indexability of the decoupled problem.

Proposition 3 (Indexability of decoupled problem).

The decoupled problem is indexable when (9) is satisfied.

Proof. 

According to Proposition 2.2 of [25], we only need to verify that the expected transmission rate ${\overline{\rho }}_{\mathit{n}}$ is strictly decreasing in n. From Corollary 3, we have

$$\begin{array}{c}\hfill {\overline{\rho }}_{\mathit{n}}=\frac{{\displaystyle \gamma \left(\frac{p}{1-c_1}\right)}}{{\displaystyle \frac{2}{{(1-p)}^{n-1}}+\left(\frac{p}{1-c_1}-1\right)}}.\end{array}$$

As $\frac{1}{2}<1-p<1$, we can easily verify that ${\overline{\rho }}_{\mathit{n}}$ is strictly decreasing in n. Thus, the decoupled problem is indexable when (9) is satisfied.    ☐

4.4. Whittle’s Index Policy

In this section, we proceed with finding the expression of Whittle’s index and defining Whittle’s index policy. First of all, we give the definition of Whittle’s index.

Definition 3 (Whittle’s index).

When the decoupled problem is indexable, Whittle’s index at state x is defined as the infimum λ, such that both actions are equally desirable. Equivalently, Whittle’s index at state x is defined as the infimum λ such that $V^0\left(x\right)=V^1\left(x\right)$.

Let us denote by $W_x$ the Whittle’s index at state x. Then, the expression of Whittle’s index is given by the following Proposition.

Proposition 4 (Whittle’s index).

When (9) is satisfied, Whittle’s index is

$$W_x=\left\{\begin{array}{cc}0\hfill & whenx=(0,\widehat{r})orx=(s,0),\hfill \\ \frac{{\displaystyle \left(1-c_1\right)\sum _{k=s+1}^{+\infty }f\left(k\right)c_1^{k-s-1}-{\overline{\Delta }}_s}}{{\displaystyle \frac{(1-c_1)(1-p)-\gamma (1-p-\alpha )}{c_1(1-p-\alpha )}+{\overline{\rho }}_s}}\hfill & whenx=(s,1),\hfill \end{array}\right.$$

where $s>0$ and $c_1=(1-\gamma )(1-p)+\gamma \alpha$. ${\overline{\Delta }}_s$ and ${\overline{\rho }}_s$ are the expected AoII and the expected transmission rate when threshold policy $\mathit{n}=(+\infty ,s)$ is adopted, respectively. At the same time, $W_x$ is non-negative and is non-decreasing in s.

Proof. 

Whittle’s indexes at state $x=(0,\widehat{r})$ and $x=(s,0)$ are obtained easily from the proof of Proposition 1. For state $x=(s,1)$, we first use backward induction to calculate the expressions of some value functions. Then, the expression of Whittle’s index can be obtained from its definition. The complete proof can be found in Appendix G.    ☐

Definition 4

(Whittle’s index policy). At any state $\mathit{x}=(x_1,x_2,\dots ,x_N)$, the base station will transmit the updates from M users with the largest $W_{x_i}$. The ties are broken arbitrarily. $W_{x_i}$ is calculated using Proposition 4 with the parameters of user i.

Remark 4.

Whittle’s index policy possesses the structural properties detailed in Corollary 1.

• The first two properties can be verified by noting that $W_{x_i}\ge 0$ and the equality holds when ${\widehat{r}}_i=0$ or $s_i=0$. At the same time, $W_{x_i}$ is non-decreasing in ${\widehat{r}}_i$.
• The third and fourth properties can be verified by noting that $W_{x_i}$ is non-decreasing in $s_i$.
• For the last property, we first notice that $W_{x_j}=W_{x_k}$ when users j and k are statistically identical and $x_j=x_k$. Then, the property can be verified by noting that $W_{x_i}$ is non-decreasing in both $s_i$ and ${\widehat{r}}_i$.

5. Optimal Policy for Relaxed Problem

In this section, we provide an efficient algorithm to obtain the optimal policy for RP, based on which we will develop another scheduling policy for PP in the next section that is free from indexability. At the same time, the performance of the optimal policy for RP forms a universal lower bound because the following ordering holds

$${\overline{\Delta }}_{AoII}^{RP}\le {\overline{\Delta }}_{AoII}^{PP},$$

where ${\overline{\Delta }}_{AoII}^{RP}$ and ${\overline{\Delta }}_{AoII}^{PP}$ are the minimal expected AoII of RP and PP, respectively.

Remark 5.

Note that the optimal policy for RP may not necessarily be a valid policy for PP, as the transmitter may transmit more than M updates in one transmission attempt under RP-optimal policy.

To solve RP, we follow the discussion in Section 4.1. More precisely, we take the Lagrangian approach and consider the problem reported in (7). We will see in the following discussion that the optimal policy for RP can be characterized by the optimal policies for problem (7). Therefore, we first cast problem (7) into the MDP ${\mathcal{M}}_N(\lambda ,-1)$. However, the optimal policy for ${\mathcal{M}}_N(\lambda ,-1)$ is difficult to obtain because the state space is infinite. Even though we can make the state space finite by imposing an upper limit on the value of s, the state space and the action space grow exponentially with the number of users in the system. To overcome the difficulty, we investigate the optimal policy for ${\mathcal{M}}_1^i(\lambda ,-1)$ where $1\le i\le N$. The superscript i means that the only user in the system is user i. We will show later that the optimal policy for ${\mathcal{M}}_N(\lambda ,-1)$ can be fully characterized by the optimal policies for ${\mathcal{M}}_1^i(\lambda ,-1)$ where $1\le i\le N$.

5.1. Optimal Policy for Single User

In this section, we tackle the problem of finding the optimal policy for ${\mathcal{M}}_1^i(\lambda ,-1)$. Since the users share the same structure, we ignore the superscript i for simplicity. To find the optimal policy, we first use the Approximating Sequence Method (ASM) introduced in [26] to make the state space finite. More precisely, we impose $s\le m$ where m is a predetermined upper limit. The state transition probabilities $P_{s,s^{\prime }}^{\prime }(a,\widehat{r})$ are modified in the following way

$$P_{s,s^{\prime }}^{\prime }(a,\widehat{r})=\left\{\begin{array}{cc}{P}_{s,s^{\prime }}(a,\widehat{r})\hfill & if{s}^{\prime }<m,\hfill \\ P_{s,s^{\prime }}(a,\widehat{r})+\sum _{z>m}{P}_{s,z}(a,\widehat{r})\hfill & if{s}^{\prime }=m.\hfill \end{array}\right.$$

(10)

The action space and the instant cost remain unchanged. Then, we can apply Relative Value Iteration (RVI) with convergence criteria $ϵ$ to obtain the optimal policy. We notice that ${\mathcal{M}}_1(\lambda ,-1)$ coincides with the decoupled model studied in Section 4.2. Hence, we can utilize the threshold structure of the optimal policy to improve RVI. To this end, we class a state as active if the optimal action at this state is $a=1$. Then, the threshold structure detailed in Proposition 1 tells us the following. For any state x, if there exists an active state $x_1$ with $s_1\le s$ and ${\widehat{r}}_1\le \widehat{r}$, then x must also be active. Hence, we can determine the optimal action at state x immediately instead of comparing all feasible actions. In this way, we can reduce the running time of RVI. The pseudocode for the improved RVI can be found in Algorithm A1 of Appendix M. A similar technique is also presented in [5].

For ${\mathcal{M}}_1(\lambda ,-1)$, when problem (9) is satisfied, Whittle’s index exists and can be calculated efficiently using Proposition 4. Therefore, we can obtain the optimal policy using Whittle’s index and further reduce the computational complexity. To this end, we denote by ${\mathit{n}}_{\lambda }$ the optimal policy for ${\mathcal{M}}_1(\lambda ,-1)$ and present the following proposition

Proposition 5 (Optimal deterministic policy).

When (9) is satisfied, the optimal policy for ${\mathcal{M}}_1(\lambda ,-1)$ is ${\mathit{n}}_{\lambda }=(+\infty ,n)$ where n is given by

$$n=\left\{\begin{array}{cc}1\hfill & if\lambda =0,\hfill \\ max\{s\in {\mathbb{N}}_0:W_s\le \lambda \}+1\hfill & if\lambda >0.\hfill \end{array}\right.$$

$W_s$ is the Whittle’s index at state $(s,1)$.

Proof. 

We first notice that ${\mathcal{M}}_1(\lambda ,-1)$ coincides with the decoupled model studied in Section 4.2. Then, we show the optimal action for each state with $\widehat{r}=1$ using the definition of Whittle’s index and the fact that the decoupled problem is indexable when (9) is satisfied. The complete proof can be found in Appendix H.    ☐

In the following, we provide a randomized policy that is also optimal for ${\mathcal{M}}_1(\lambda ,-1)$. We will see later that the randomized policy is the key to obtaining the optimal policy for RP.

Theorem 2 (Optimal randomized policy).

There exist two deterministic policies ${\mathit{n}}_{{\lambda }_{+}}$ and ${\mathit{n}}_{{\lambda }_{-}}$, which are both optimal for ${\mathcal{M}}_1(\lambda ,-1)$. We consider the following randomized policy ${\mathit{n}}_{\lambda }$: every time the system reaches state $(0,0)$, the base station will make the choice between ${\mathit{n}}_{{\lambda }_{-}}$ with probability μ and ${\mathit{n}}_{{\lambda }_{+}}$ with probability $1-\mu$. The chosen policy will be followed until the next choice. Then, the randomized policy ${\mathit{n}}_{\lambda }$ is optimal for ${\mathcal{M}}_1(\lambda ,-1)$ under any $\mu \in [0,1]$.

Proof. 

We show that our system verifies the assumptions given in [27]. Then, leveraging the characteristics of our system, we can obtain the optimal randomized policy. The complete proof can be found in Appendix I.    ☐

In practice, we approximate ${\lambda }_{+}\approx \lambda +\xi$ and ${\lambda }_{-}\approx \lambda -\xi$ where $\xi$ is a small perturbation. Then, the deterministic policies ${\mathit{n}}_{{\lambda }_{+}}$ and ${\mathit{n}}_{{\lambda }_{-}}$ can be obtained by following the discussion at the beginning of this subsection. Note that, in most cases, ${\mathit{n}}_{{\lambda }_{+}}$ and ${\mathit{n}}_{{\lambda }_{-}}$ are the same.

5.2. Optimal Policy for RP

In this section, we characterize the optimal policy for RP. Let us denote by $V\left(x\right)$ and $V^i\left(x_i\right)$ the value functions of ${\mathcal{M}}_N(\lambda ,-1)$ and ${\mathcal{M}}_1^i(\lambda ,-1)$, respectively. Then, we can prove the following

Proposition 6 (Separability).

$V\left(x\right)={\sum }_{i=1}^N{V}^i\left(x_i\right)$ where $x=(x_1,\dots ,x_N)$. In other words, the policy, under which each user adopts its own optimal policy, is optimal for ${\mathcal{M}}_N(\lambda ,-1)$.

Proof. 

We show $V\left(x\right)={\sum }_{i=1}^N{V}^i\left(x_i\right)$ by comparing the Bellman equations they must satisfy. The complete proof can be found in Appendix J.    ☐

We denote the optimal policy for ${\mathcal{M}}_N(\lambda ,-1)$ as ${\varphi }_{\lambda }=[{\mathit{n}}_{\lambda ,1},\dots ,{\mathit{n}}_{\lambda ,N}]$ where ${\mathit{n}}_{\lambda ,i}$ is the optimal policy for ${\mathcal{M}}_1^i(\lambda ,-1)$. For simplicity, we define $\overline{\Delta }\left(\lambda \right)$ and $\overline{\rho }\left(\lambda \right)$ as the expected AoII and the expected transmission rate associated with ${\varphi }_{\lambda }$, respectively. ${\overline{\Delta }}^i\left(\lambda \right)$ and ${\overline{\rho }}^i\left(\lambda \right)$ are defined analogously for user i under policy ${\mathit{n}}_{\lambda ,i}$. We also define ${\lambda }^{*}\triangleq inf\{\lambda >0:$$\overline{\rho }\left(\lambda \right)\le M\}$. With Proposition 6 and the above definitions in mind, we proceed with constructing the optimal policy for RP.

Theorem 3 (Optimal policy for RP).

The optimal policy for RP can be characterized by two deterministic policies ${\varphi }_{{\lambda }_{+}^{*}}=[{\mathit{n}}_{{\lambda }_{+}^{*},1},\dots ,{\mathit{n}}_{{\lambda }_{+}^{*},N}]$ and ${\varphi }_{{\lambda }_{-}^{*}}=[{\mathit{n}}_{{\lambda }_{-}^{*},1},\dots ,{\mathit{n}}_{{\lambda }_{-}^{*},N}]$ where ${\mathit{n}}_{{\lambda }_{+}^{*},i}$ and ${\mathit{n}}_{{\lambda }_{-}^{*},i}$ are both the optimal deterministic policies for ${\mathcal{M}}_1^i({\lambda }^{*},-1)$. Then, we mix ${\varphi }_{{\lambda }_{+}^{*}}$ and ${\varphi }_{{\lambda }_{-}^{*}}$ in the following way: for each user i, every time the user reaches state $(0,0)$, the base station will make the choice between ${\mathit{n}}_{{\lambda }_{-}^{*},i}$ with probability ${\mu }_i$ and ${\mathit{n}}_{{\lambda }_{+}^{*},i}$ with probability $1-{\mu }_i$. The chosen policy will be followed by user i until the next choice. Where $1\le i\le N$, the ${\mu }_i$ is chosen in such a way as to satisfy

$$\sum _{i=1}^N{\overline{\rho }}^i\left({\lambda }^{*}\right)=\sum _{i=1}^N\left({\mu }_i{\overline{\rho }}^i\left({\lambda }_{-}^{*}\right)+(1-{\mu }_i){\overline{\rho }}^i\left({\lambda }_{+}^{*}\right)\right)=M.$$

(11)

Then, the mixed policy, denoted by ${\varphi }_{{\lambda }^{*}}$, is optimal for RP.

Proof. 

According to Lemma 3.10 of [27], a policy is optimal for RP if

• It is optimal for ${\mathcal{M}}_N({\lambda }^{*},-1)$;
• The resulting expected transmission rate is equal to M.

Then, we construct such a policy using Theorem 2 and Proposition 6. The complete proof can be found in Appendix K.    ☐

Since we approximate ${\lambda }_{+}^{*}\approx {\lambda }^{*}+\xi$ and ${\lambda }_{-}^{*}\approx {\lambda }^{*}-\xi$ in practice, ${\overline{\rho }}^i\left({\lambda }_{+}^{*}\right)\le {\overline{\rho }}^i\left({\lambda }_{-}^{*}\right)$ for all i according to the monotonicity given by Lemma 3.4 of [27]. Combining with the definition of ${\lambda }^{*}$, we must have $\overline{\rho }\left({\lambda }_{+}^{*}\right)\le M<\overline{\rho }\left({\lambda }_{-}^{*}\right)$. Therefore, we can always find ${\mu }_i$’s that realize (11). In this paper, we choose

$${\mu }_i=\mu =\frac{M-\overline{\rho }\left({\lambda }_{+}^{*}\right)}{\overline{\rho }\left({\lambda }_{-}^{*}\right)-\overline{\rho }\left({\lambda }_{+}^{*}\right)},for1\le i\le N.$$

(12)

Then, we describe the algorithm used to obtain the optimal policy for RP. As detailed in Theorem 3, it is essential to find ${\lambda }^{*}$. To this end, we recall that, for any user i under given $\lambda$, the optimal deterministic policy ${\mathit{n}}_{\lambda ,i}$ can be obtained using the results in Section 5.1 and the resulting expected transmission rate ${\overline{\rho }}^i\left(\lambda \right)$ is given by Proposition 2. Since ${\overline{\rho }}^i\left(\lambda \right)$ is non-increasing in $\lambda$ for all i according to Lemma 3.4 of [27], $\overline{\rho }\left(\lambda \right)={\sum }_{i=1}^N{\overline{\rho }}^i\left(\lambda \right)$ is also non-increasing in $\lambda$. Hence, we can regard $\overline{\rho }\left(\lambda \right)$ as a non-increasing function of $\lambda$. Then, according to the definition of ${\lambda }^{*}$, we can use the Bisection search to obtain ${\lambda }^{*}$ efficiently. The main steps can be summarized as follows.

• Initialize ${\lambda }_{-}=0$ and ${\lambda }_{+}=1$.
• Do ${\lambda }_{-}={\lambda }_{+}$ and ${\lambda }_{+}=2{\lambda }_{+}$ until $\overline{\rho }\left({\lambda }_{+}\right)<M$.
• Run Bisection search on the interval $[{\lambda }_{-},{\lambda }_{+}]$ until the tolerance $2\xi$ is met.

Then, ${\lambda }_{-}^{*}$ and ${\lambda }_{+}^{*}$ can simply be the boundaries of the final interval. The pseudocode for the Bisection search can be found in Algorithm A2 of Appendix M. After obtaining ${\lambda }_{-}^{*}$ and ${\lambda }_{+}^{*}$, the optimal policy ${\varphi }_{{\lambda }^{*}}$ is detailed in Theorem 3 and the mixing probabilities ${\mu }_i$’s are given by (12).

Remark 6.

We recall that the optimal deterministic policy for each user can be characterized by two positive thresholds (i.e., $n_0,n_1>0$). Consequently, under RP-optimal policy, the base station will never choose the user at state $(0,\widehat{r})$. Then, when M increases, the expected transmission rate achieved by RP-optimal policy will saturate before M reaches N. When the expected transmission rate saturates, the RP-optimal policy is ${\varphi }^{*}=[{\mathit{n}}_1,\dots ,{\mathit{n}}_N]$ where ${\mathit{n}}_i=(1,1)$ for $1\le i\le N$. The saturation happens when M is larger than or equal to the expected transmission rate achieved by ${\varphi }^{*}$.

6. Indexed Priority Policy

Although the performance of Whittle’s index policy is known to be good, it requires indexability, which is usually difficult to establish. In this section, based on the primal-dual heuristic introduced in [28], we develop a policy that does not require indexability and has comparable performance to Whittle’s index policy. We start with presenting the primal-dual heuristic.

6.1. Primal-Dual Heuristic

The heuristic is based on the optimal primal and dual solution pair to the linear program associated with RP. To introduce the linear program, we define ${\pi }_{x_i}^{a_i}\left(\varphi \right)\ge 0$ as the expected time that user i is at state $x_i$ and action $a_i$ is taken according to policy $\varphi$. Then, for any $\varphi$, ${\pi }_{x_i}^{a_i}\left(\varphi \right)$ must satisfy the following problems

$${\pi }_{x_i}^0\left(\varphi \right)+{\pi }_{x_i}^1\left(\varphi \right)=\sum _{x_i^{\prime }}\sum _{a_i^{\prime }}{P}_{x_i^{\prime },x_i}\left(a_i^{\prime }\right){\pi }_{x_i^{\prime }}^{a_i^{\prime }}\left(\varphi \right),\forall x_i,i.$$

$$\sum _{x_i}\sum _{a_i}{\pi }_{x_i}^{a_i}\left(\varphi \right)=1,\forall i.$$

The objective function of RP can be rewritten as

$$\begin{array}{cc}\hfill \underset{\varphi \in \mathrm{\Phi }}{\mathrm{minimize}}& \sum _{i=1}^N\sum _{x_i,a_i}C(x_i){\pi }_{x_i}^{a_i}(\varphi ),\hfill \end{array}$$

where $C\left(x_i\right)=f_i\left(s_i\right)$ is the instant cost at state $x_i$. The constraint on the expected transmission rate can be rewritten as

$$\sum _{i=1}^N\sum _{x_i}{\pi }_{x_i}^1\left(\varphi \right)\le M.$$

Thus, the linear program associated with RP can be formulated as the following

$$\begin{array}{}\mathrm{(13a)}& \underset{{\pi }_{x_i}^{a_i}}{\mathrm{minimize}}& \sum _{i=1}^N\sum _{x_i,a_i}C(x_i){\pi }_{x_i}^{a_i}\mathrm{(13b)}& \mathrm{subject} \mathrm{to}& {\pi }_{x_i}^0 + {\pi }_{x_i}^1 - \sum _{x_i^{\prime }}\sum _{a_i^{\prime }}{P}_{x_i^{\prime },x_i}(a_i^{\prime }){\pi }_{x_i^{\prime }}^{a_i^{\prime }} = 0\forall x_i,i,\mathrm{(13c)}& & \sum _{x_i}\sum _{a_i}{\pi }_{x_i}^{a_i} = 1\forall i,\mathrm{(13d)}& & \sum _{i=1}^N\sum _{x_i}{\pi }_{x_i}^1 \le M,\mathrm{(13e)}& & {\pi }_{x_i}^{a_i} \ge 0,\forall x_i,a_i,i.\end{array}$$

The corresponding dual problem is

$$\begin{array}{}\mathrm{(14a)}& \underset{\sigma ,{\sigma }_i,{\sigma }_{x_i}}{\mathrm{maximize}}& \sum _{i=1}^N{\sigma }_i - M_{\sigma }\mathrm{(14b)}& \mathrm{subject} \mathrm{to}& {\sigma }_{x_i} + {\sigma }_i - \sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}(0){\sigma }_{x_i^{\prime }} \le C(x_i),\forall x_i,i,\mathrm{(14c)}& & {\sigma }_{x_i} + {\sigma }_i - \sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}(1){\sigma }_{x_i^{\prime }} - \sigma \le C(x_i),\forall x_i,i,\mathrm{(14d)}& & \sigma \ge 0.\end{array}$$

Let $\left\{{\overline{\pi }}_{x_i}^{a_i}\right\}$ and $\{\overline{\sigma },{\overline{\sigma }}_i,{\overline{\sigma }}_{x_i}\}$ be the optimal primal and dual solution pair to the problems reported in (13) and (14). We define

$${\overline{\psi }}_{x_i}^0=\sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}\left(0\right){\overline{\sigma }}_{x_i^{\prime }}+C\left(x_i\right)-{\overline{\sigma }}_i-{\overline{\sigma }}_{x_i}\ge 0,$$

$${\overline{\psi }}_{x_i}^1=\sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}\left(1\right){\overline{\sigma }}_{x_i^{\prime }}+\overline{\sigma }+C\left(x_i\right)-{\overline{\sigma }}_i-{\overline{\sigma }}_{x_i}\ge 0.$$

For any state $x=(x_1,\dots ,x_N)$, let $h\left(x\right)={\sum }_{i=1}^N{𝟙}_{\{{\overline{\pi }}_{x_i}^1>0\}}$. Then, the heuristic operates in the following way

• If $h\left(\mathit{x}\right)\ge M$, the base station will choose the M users with the largest ${\overline{\psi }}_{x_i}^0$ among the $h\left(\mathit{x}\right)$ users.
• If $h\left(\mathit{x}\right)<M$, these $h\left(\mathit{x}\right)$ users are chosen by the base station. The base station will choose $M-h\left(\mathit{x}\right)$ additional users with the smallest ${\overline{\psi }}_{x_i}^1$.

However, Linear Programming (LP) is a very general technique and does not appear to take advantage of the special structure of the problem. Although there are algorithms for solving rational LP that take time polynomial in the number of variables and constraints, they run extremely slowly in practice [29]. For our problem, we notice that the users have separate activity areas that are linked through a common resource constraint. Therefore, the primal problem can be solved using Dantzig-Wolfe decomposition. Even so, the problem is still computationally demanding when the system scales up. We recall that we solved the exact problem efficiently using MDP-specific algorithms in Section 5. It is more efficient because of the following reasons

• According to Proposition 6, we can decompose the problem into N subproblems.
• For each subproblem, the threshold structure of the optimal policy is utilized to reduce the running time of RVI.
• As we will see later, the developed policy can be obtained directly from the result of RVI in practice.

In the following, we will translate the results in Section 5 into the optimal primal and dual solution pair and propose Indexed priority policy.

6.2. Indexed Priority Policy

We first define the Lagrangian function associated with (13).

$$\begin{array}{cc}\hfill \mathcal{L}({\pi }_{x_i}^{a_i},\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i})=& \left(\sum _{i=1}^N\sum _{x_i,a_i}C\left(x_i\right){\pi }_{x_i}^{a_i}\right)+\sum _{i,x_i}{\sigma }_{x_i}\left(\sum _{x_i^{\prime }}\sum _{a_i^{\prime }}{P}_{x_i^{\prime },x_i}\left(a_i^{\prime }\right){\pi }_{x_i^{\prime }}^{a_i^{\prime }}-{\pi }_{x_i}^0-{\pi }_{x_i}^1\right)+\hfill \\ & \sum _{i=1}^N{\sigma }_i\left(1-\sum _{x_i}\sum _{a_i}{\pi }_{x_i}^{a_i}\right)+\sigma \left(\sum _{i=1}^N\sum _{x_i}{\pi }_{x_i}^1-M\right)-\sum _{i,x_i,a_i}{\psi }_{x_i}^{a_i}{\pi }_{x_i}^{a_i}.\hfill \end{array}$$

Then, the corresponding Lagrangian dual function is

$$g(\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i})=\underset{{\pi }_{x_i}^{a_i}}{inf}\mathcal{L}({\pi }_{x_i}^{a_i},\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i}).$$

Let ${\pi }_{x_i}$ be the expected time that user i is at state $x_i$ caused by the adoption of ${\varphi }_{{\lambda }^{*}}$, where ${\varphi }_{{\lambda }^{*}}$ is the optimal policy detailed in Theorem 3. Then, we define $\left\{{\pi }_{x_i}^{a_i}\right\}$ as follows

• State $x_i$ is where randomization happens (randomization happens when the actions suggested by the two optimal deterministic policies are different), and it has a value of ${\pi }_{x_i}^0=a_{{\mathit{n}}_{{\lambda }_{-}^{*},i}}\left(x_i\right)(1-{\mu }_i){\pi }_{x_i}+a_{{\mathit{n}}_{{\lambda }_{+}^{*},i}}\left(x_i\right){\mu }_i{\pi }_{x_i}$ and ${\pi }_{x_i}^1={\pi }_{x_i}-{\pi }_{x_i}^0$ where ${\mu }_i$ is given by (12) and $a_{{\mathit{n}}_{\lambda ,i}}\left(x_i\right)$ is the action suggested by ${\mathit{n}}_{\lambda ,i}$ at state $x_i$.
• For other values of $x_i$, we have ${\pi }_{x_i}^0=(1-a_{{\mathit{n}}_{{\lambda }^{*},i}}\left(x_i\right)){\pi }_{x_i}$ and ${\pi }_{x_i}^1={\pi }_{x_i}-{\pi }_{x_i}^0$.

We also define $\sigma ={\lambda }^{*}$, ${\sigma }_i={\theta }_i$, and ${\sigma }_{x_i}=V^i\left(x_i\right)$ where ${\lambda }^{*}$ is specified in Section 5.2, ${\theta }_i$ is the optimal value of ${\mathcal{M}}_1^i({\lambda }^{*},-1)$, and $V^i\left(x_i\right)$ is the value function associated with ${\mathcal{M}}_1^i({\lambda }^{*},-1)$. Lastly, we define $\left\{{\psi }_{x_i}^{a_i}\right\}$ as follows

$${\psi }_{x_i}^0=\sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}\left(0\right){\sigma }_{x_i^{\prime }}+C\left(x_i\right)-{\sigma }_i-{\sigma }_{x_i},$$

$${\psi }_{x_i}^1=\sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}\left(1\right){\sigma }_{x_i^{\prime }}+\sigma +C\left(x_i\right)-{\sigma }_i-{\sigma }_{x_i}.$$

Then, we can prove the following proposition.

Proposition 7 (Optimal solution pair).

$\left\{{\pi }_{x_i}^{a_i}\right\}$ and $\{\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i}\}$ are primal and dual solutions to (13), respectively.

Proof. 

Since (13) is linear and strictly feasible, it is sufficient to show that $\left\{{\pi }_{x_i}^{a_i}\right\}$ and $\{\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i}\}$ verify the KKT conditions, which can be expressed as the following four conditions.

• Primal feasibility: the constraints in (13) are satisfied.
• Dual feasibility: $\sigma \ge 0$ and ${\psi }_{x_i}^{a_i}\ge 0$ for all $x_i$, $a_i$, and i.
• Complementary slackness: $\sigma \left({\sum }_{i=1}^N{\sum }_{x_i}{\pi }_{x_i}^1-M\right)=0$ and ${\psi }_{x_i}^{a_i}{\pi }_{x_i}^{a_i}=0$ for all $x_i$, $a_i$, and i.
• Stationarity: the gradient of $\mathcal{L}({\pi }_{x_i}^{a_i},\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i})$ with respect to $\left\{{\pi }_{x_i}^{a_i}\right\}$ vanishes.

Apparently, the first condition is satisfied by $\left\{{\pi }_{x_i}^{a_i}\right\}$. For the second condition, $\sigma \ge 0$ since $\sigma ={\lambda }^{*}\ge 0$ by definition. For ${\psi }_{x_i}^{a_i}$, we can verify that ${\psi }_{x_i}^{a_i}=V^{i,a_i}\left(x_i\right)-V^i\left(x_i\right)$ where $V^{i,a_i}\left(x_i\right)$ is the value function resulting from taking action $a_i$ at state $x_i$. Then, the non-negativity is guaranteed by the Bellman equation. For the third condition, the first term is zero because we choose the ${\mu }_i$’s given by (12). For the second term, we recall that ${\psi }_{x_i}^{a_i}=V^{i,a_i}\left(x_i\right)-V^i\left(x_i\right)$. According to the definition of ${\pi }_{x_i}^{a_i}$, we know $V^i\left(x_i\right)=V^{i,a_i}\left(x_i\right)$ if ${\pi }_{x_i}^{a_i}>0$. Combined together, we can conclude that ${\psi }_{x_i}^{a_i}=0$ when ${\pi }_{x_i}^{a_i}>0$. Thus, the third condition is satisfied. For the last condition, setting the gradient equal to zero yields a system of linear equations. More precisely, for each $x_i$ and $1\le i\le N$

$$\{\begin{array}{c}{\displaystyle \sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}\left(0\right){\sigma }_{x_i^{\prime }}+C\left(x_i\right)={\sigma }_{x_i}+{\sigma }_i+{\psi }_{x_i}^0.}\hfill \\ \hfill {\displaystyle \sum _{x_i^{\prime }}{P}_{x_i,x_i^{\prime }}\left(1\right){\sigma }_{x_i^{\prime }}+\sigma +C\left(x_i\right)={\sigma }_{x_i}+{\sigma }_i+{\psi }_{x_i}^1.}\end{array}$$

Then, $\{\sigma ,{\sigma }_i,{\sigma }_{x_i},{\psi }_{x_i}^{a_i}\}$ verifies the system of linear equations by definition. Since all four conditions are satisfied, we can conclude our proof.    ☐

According to Proposition 7, we know that $\left\{{\pi }_{x_i}^{a_i}\right\}$ and $\{\sigma ,{\sigma }_i,{\sigma }_{x_i}\}$ defined above are the optimal solutions to problems (13) and (14), respectively. As the optimal solutions are obtained, we can adopt the heuristic detailed in Section 6.1.

The heuristic can be expressed equivalently as an index policy. To this end, we define the index $I_{x_i}$ for state $x_i$ as

$$I_{x_i}\triangleq {\overline{\psi }}_{x_i}^0-{\overline{\psi }}_{x_i}^1.$$

According to the complementary slackness, $I_{x_i}$ can be reduced to the following.

• For state $x_i$ such that ${\overline{\pi }}_{x_i}^1>0$ and ${\overline{\pi }}_{x_i}^0=0$, we have ${\overline{\psi }}_{x_i}^1=0$. Therefore, $I_{x_i}={\overline{\psi }}_{x_i}^0\ge 0$.
• For state $x_i$ such that ${\overline{\pi }}_{x_i}^1>0$ and ${\overline{\pi }}_{x_i}^0>0$, we have ${\overline{\psi }}_{x_i}^1={\overline{\psi }}_{x_i}^0=0$. Therefore, $I_{x_i}=0$.
• For state $x_i$ such that ${\overline{\pi }}_{x_i}^1=0$ and ${\overline{\pi }}_{x_i}^0>0$, we have ${\overline{\psi }}_{x_i}^0=0$. Therefore, $I_{x_i}=-{\overline{\psi }}_{x_i}^1\le 0$.

We can show that $I_{x_i}$ possesses the following properties.

Proposition 8 (Properties of $I_{x_i}$).

For $1\le i\le N$, $I_{x_i}\ge -{\lambda }^{*}$ for any $x_i$. The equality holds when ${\widehat{r}}_i=p_{e,i}^0=0$ or $s_i=0$. At the same time, $I_{x_i}$ is non-decreasing in both $s_i$ and ${\widehat{r}}_i$.

Proof. 

We notice that $I_{x_i}$ can be expressed as a function of $V^i\left(x_i\right)$ and ${\lambda }^{*}$. Meanwhile, ${\mathcal{M}}_1^i({\lambda }^{*},-1)$ coincides with the decoupled model studied in Section 4.2. Then, we can verify the properties of $I_{x_i}$ using the results in Section 4.2. The complete proof can be found in Appendix L.    ☐

Comparing with the heuristic detailed in Section 6.1, we can define the Indexed priority policy.

Definition 5 (Indexed priority policy).

At any state $\mathit{x}=(x_1,x_2,\dots ,x_N)$, the base station will transmit the updates from M users with the largest $I_{x_i}$. The ties are broken arbitrarily.

Remark 7.

Indexed priority policy belongs to the class of priority policies introduced in [30]. These priority policies are asymptotically optimal when certain conditions are satisfied.

Remark 8.

Indexed priority policy possesses the structural properties detailed in Corollary 1.

• The first two properties can be verified by noting that $I_{x_i}\ge -{\lambda }^{*}$ and the equality holds when ${\widehat{r}}_i=p_{e,i}^0=0$ or $s_i=0$. At the same time, $I_{x_i}$ is non-decreasing in ${\widehat{r}}_i$.
• The third and fourth properties can be verified by noting that $I_{x_i}$ is non-decreasing in $s_i$.
• For the last property, we first notice that $I_{x_j}=I_{x_k}$ when users j and k are statistically identical and $x_j=x_k$. Then, the property can be verified by noting that $I_{x_i}$ is non-decreasing in both $s_i$ and ${\widehat{r}}_i$.

We notice that ${\theta }_i$’s and $C\left(x_i\right)$’s are canceled out by the definition of $I_{x_i}$. Therefore, $I_{x_i}$ can be calculated using ${\lambda }^{*}$ and the value function of ${\mathcal{M}}_1^i({\lambda }^{*},-1)$. In practice, we can use either ${\lambda }_{-}^{*}$ or ${\lambda }_{+}^{*}$ to approximate ${\lambda }^{*}$, and the value function can be approximated by the result of the RVI detailed in Section 5.1. Since the state space is infinite, we only calculate a finite number of $V^i\left(x_i\right)$, the number of which depends on the truncation parameter m of ASM. Meanwhile, the probabilities $P_{x_i,x_i^{\prime }}\left(a_i\right)$ in $I_{x_i}$ are modified according to (10).

7. Numerical Results

In this section, we provide numerical results to showcase the performance of the developed scheduling policies. To eliminate the effect of N, we plot the expected average AoII. In particular, we provide the expected average AoII achieved by the Indexed priority policy and Whittle’s index policy when $M=1$. The policies are calculated using the results detailed in Section 4, Section 5 and Section 6. When obtaining the Indexed priority policy, we set the tolerance in the Bisection search to $\xi =0.005$. Meanwhile, we choose the truncation parameter in ASM $m=800$ and the convergence criteria in RVI $ϵ=0.01$. We notice that the calculation of Whittle’s index involves an infinite sum. In practice, we approximate the result by replacing $+\infty$ with a large enough number $k_{max}$. Here, we choose $k_{max}=800$. For both scheduling policies, the resulting expected average AoII is obtained via simulations. Each data point is the average of 15 runs with 15,000 time slots considered in each run.

We also compare the developed policies with the optimal policy for RP, which can be calculated by following the discussion in Section 5.2. We adopt the same choices of parameters as we used to obtain the developed policies. The corresponding performance is calculated using Proposition 2. Like before, the infinite sum is approximated by replacing $+\infty$ with $k_{max}=800$. We also provide the expected average AoII achieved by the Greedy policy to show the performance advantages of the developed policies. When the Greedy policy is adopted, the base station always chooses the user with the largest AoII. The resulting expected average AoII is obtained via the same simulations as applied to the developed policies.

Figure 3 and Figure 4 illustrate the performance when the source processes have different dynamics and when each user’s communication goal is different, respectively. Figure 3a provides the performance when $p_i=0.05+\frac{0.4(i-1)}{N-1}$ for $1\le i\le N$. For other parameters, the users make the same choices. More precisely, $f_i\left(s\right)=s$, ${\gamma }_i=0.6$, and $p_{e,i}^0=p_{e,i}^1=0.1$ for $1\le i\le N$. Figure 4a provides the performance when $f_i\left(s\right)=s^{0.5+\frac{i-1}{N-1}}$ for $1\le i\le N$. Same as before, the users make the same choices for other parameters. More precisely, $p_i=0.3$, ${\gamma }_i=0.6$, and $p_{e,i}^0=p_{e,i}^1=0.1$ for $1\le i\le N$. In Figure 3b and Figure 4b, we force $p_{e,i}^0=0$ for all users to ensure the existence of Whittle’s index. Other choices remain the same as in Figure 3a and Figure 4a. According to Corollary 1, the optimal policy will never choose the user with $\widehat{r}=p_e^0=0$ unless it is to break the tie. Therefore, in Figure 3b and Figure 4b, we also consider the Greedy+ policy where the base station always chooses the user with the largest AoII among the users with $\widehat{r}=1$. The resulting expected average AoII is obtained via the same simulations as applied to the Greedy policy.

Figure 3. Performance when the source processes vary. We choose $p_i=0.05+\frac{0.4(i-1)}{N-1}$, $f_i\left(s\right)=s$, ${\gamma }_i=0.6$, $p_{e,i}^0=p_e^0$, and $p_{e,i}^1=0.1$ for $1\le i\le N$.

Figure 4. Performance when the communication goals vary. We choose $f_i\left(s\right)=s^{0.5+\frac{i-1}{N-1}}$, $p_i=0.3$, ${\gamma }_i=0.6$, $p_{e,i}^0=p_e^0$, and $p_{e,i}^1=0.1$ for $1\le i\le N$.

Figure 5 shows the performance in systems where the parameters for each user are generated uniformly and randomly within their ranges. In Figure 5a, we consider $N=5$, $\gamma \in [0,1]$, $p\in [0.05,0.45]$, $p_e^{\widehat{r}}\in [0,0.45]$, and $f\left(s\right)=s^{\tau }$, where $\tau \in [0.5,1.5]$. There are a total of 300 different choices and the results are sorted by the performance of RP-optimal policy in ascending order. Figure 5b adopts the same system settings except that we impose $p_{e,i}^0=0$ for $1\le i\le N$ to ensure the feasibility of Whittle’s index policy. Meanwhile, we ignore the Greedy policy since the Greedy+ policy achieves a better performance, as indicated by Figure 3b and Figure 4b.

Figure 5. Performance in systems with random parameters when $N=5$. The parameters for each user are chosen randomly within the following intervals: $\gamma \in [0,1]$, $p\in [0.05,0.45]$, $p_e^0\in I$, $p_e^1\in [0,0.45]$, and $f\left(s\right)=s^{\tau }$ where $\tau \in [0.5,1.5]$.

We can make the following observations from the figures.

• The Greedy+ policy yields a smaller expected average AoII than that achieved by the Greedy policy. Recall that we obtained the Greedy+ policy by applying the structural properties detailed in Corollary 1. Therefore, simple applications of the structural properties of the optimal policy can improve the performance of scheduling policies.
• The Indexed priority policy has comparable performance to Whittle’s index policy in all the system settings considered. The two policies have their own advantages. The Indexed priority policy has a broader scope of application, while Whittle’s index policy has a lower computational complexity.
• The performance of the Indexed priority policy and Whittle’s index policy is better than that of the Greedy/Greedy+ policies and is not far from the performance of the RP-optimal policy. Recall that the performance of the RP-optimal policy forms a universal lower bound on the performance of all admissible policies for PP. Hence, we can conclude that both the Indexed priority policy and Whittle’s index policy achieve good performances.

8. Conclusions

In this paper, we studied the problem of minimizing the Age of Incorrect Information in a slotted-time system where a base station needs to schedule M users among N available users. Meanwhile, the base station has access to imperfect channel state information in each time slot. The problem is a restless multi-armed bandit problem which is SPACE-hard. However, by casting the problem into a Markov decision process, we obtain the structural properties of the optimal policy. Then, we introduce a relaxed version of the original problem and investigate the decoupled model. Under a simple condition, we establish the indexability of the decoupled problem and obtain the expression of Whittle’s index. On this basis, we developed Whittle’s index policy. To get rid of the requirement for indexability, we developed the Indexed priority policy based on the optimal policy for the relaxed problem. The characteristics of the relaxed problem are explored to make the calculation of its optimal policy more efficient. Finally, through numerical results, we show that simple applications of the structural properties can improve the performance of scheduling policies. Moreover, Whittle’s index policy and the Indexed priority policy achieve good and comparable performances.