A UoI-Optimal Policy for Timely Status Updates with Resource Constraint

Abstract

Timely status updates are critical in remote control systems such as autonomous driving and the industrial Internet of Things, where timeliness requirements are usually context dependent. Accordingly, the Urgency of Information (UoI) has been proposed beyond the well-known Age of Information (AoI) by further including context-aware weights which indicate whether the monitored process is in an emergency. However, the optimal updating and scheduling strategies in terms of UoI remain open. In this paper, we propose a UoI-optimal updating policy for timely status information with resource constraint. We first formulate the problem in a constrained Markov decision process and prove that the UoI-optimal policy has a threshold structure. When the context-aware weights are known, we propose a numerical method based on linear programming. When the weights are unknown, we further design a reinforcement learning (RL)-based scheduling policy. The simulation reveals that the threshold of the UoI-optimal policy increases as the resource constraint tightens. In addition, the UoI-optimal policy outperforms the AoI-optimal policy in terms of average squared estimation error, and the proposed RL-based updating policy achieves a near-optimal performance without the advanced knowledge of the system model.

1. Introduction

With the development of 5G and the Internet of Things (IoT), requirements for wireless communication have shifted from merely providing communication channels to covering the entire process of various IoT applications, e.g., autonomous vehicle [1] and virtual reality (VR) [2], where sensing, communication, computation, and control form a closed loop. Therefore, in addition to the communication delay, it is necessary to consider the information delay counted from the generation of the state information to the execution, namely the timeliness of information. For this purpose, Age of Information (AoI) has been proposed, which is defined as the time elapsed since the generation time of the latest received packets [3]. Due to its concise definition and clear physical meaning, AoI has been widely used for the design of scheduling and updating policies in remote estimation [4,5,6] and wireless communication networks [7,8,9,10,11,12]. Most existing works focus on optimizing average AoI or peak age. In [13], the authors claim that minimizing average age cannot satisfy the requirements for ultra-reliable low-latency communication (URLLC) and study the tail distribution of AoI. The violation probability for peak age is derived in [14] and the stationary distribution of AoI is studied in [15].

Nevertheless, the AoI still has some limitations. First, it fails to measure the nonlinear performance degradation caused by information staleness. In [16,17,18,19], nonlinear age penalty functions were introduced to solve this problem. Meanwhile, the Age of Synchronization (AoS) [20] and Age of Incorrect Information (AoII) [21] are defined to associate information freshness with the content of information. AoS is the time elapsed since the information at the receiver becomes desynchronized with the actual status of the monitored process. AoII is defined as the product of an increasing time penalty function and a penalty function of the estimation error. In addition, the status of heterogeneous data sources may change at different rates. A fast-changing process may require information with a lower age. However, age is independent of the changing rate and thus is not proper in the cases when heterogeneous data sources are jointly considered. To solve this problem, weighted age was introduced in [22,23] to distinguish important monitored processes. In [24], the metric based on information theory is proposed as a replacement of the time-based metric, AoI, to characterize the changing rate. In [5], the authors claim that minimizing age is not equivalent to minimizing the estimation error in a remote estimation problem and propose an effective age to solve this problem [25].

Practical systems (e.g., V2X-communication systems) may have different requirements for information freshness with different contexts. The context refers to all environmental factors that affect the requirement for information freshness. Therefore, resources should be reserved for frequent status updates in emergency to ensure safety.

However, the timeliness metrics mentioned above pay no attention to the significance of context information. To solve this problem, Urgency of Information (UoI) has been proposed in [26,27,28] to measure the influence of inaccurate information on performance under different contexts. To be specific, UoI uses a time-variant context-aware weight $\omega \left(t\right)$ to distinguish different contexts. A higher $\omega \left(t\right)$ indicates that the system is in more urgent situations (e.g., when a vehicle is approaching an intersection or overtaking) and therefore requires frequent updates. For example, when a vehicle passes through an intersection, the context-aware weight increases as the distance between the vehicle and the center of the intersection decreases. Meanwhile, the estimation error $Q\left(t\right)$ is introduced to measure the information inaccuracy, which is defined as the difference between the actual status and the estimated status at the receiver. The larger the absolute value of $Q\left(t\right)$ is, the less accurate the estimated status is. Therefore, UoI is defined as the product of context-aware weight and a cost function of the estimation error $Q\left(t\right)$:

$$F\left(t\right)=\omega \left(t\right)\delta \left(Q\right(t\left)\right).$$

(1)

In discrete-time systems, the estimation error $Q\left(t\right)$ is:

$$Q\left(t\right)=\sum _{\tau =g\left(t\right)}^{t-1}A\left(\tau \right),$$

(2)

where $g\left(t\right)$ is the generation time of the latest status update at the receiver and $A\left(t\right)$ is the increment in estimation error in time slot t. Specifically, if the context-aware weight is time-invariant (i.e., $\omega \left(t\right)=1$), and $A\left(t\right)=1$ as well as $\delta \left(Q\right(t\left)\right)=Q\left(t\right)$, UoI is the same as AoI. If the context-aware weight is process-dependent, UoI can represent weighted age. If the cost function $\delta \left(Q\right(t\left)\right)$ is nonlinear, UoI can represent the nonlinear age penalty function. For example, when the outdated information is worthless, e.g., the information is about sales that expire after some time [29], then the shifted unit step cost function $\delta \left(Q\right(t\left)\right)=u\left(Q\right(t)-\tau )$, $\tau >0$ is recommended. For the unit step function, $u\left(x\right)=1$ when $x\ge 0$ and otherwise $u\left(x\right)=0$.

In this work, we considered a single-user remote monitoring system, and the objective was to find an updating policy minimizing the average UoI over time under the constraint on average update frequency. To solve this problem, Refs. [27,30] proposed update-index-based adaptive schemes with Lyapunov optimization but did not conduct a theoretical analysis of their optimality. In addition, the constrained Markov decision process (CMDP) formulation was only used in the simulation for a numerically solved benchmark. Based on the existing works, in this paper, we theoretically analyzed the structure of the UoI-optimal policy and focused on how to derive an updating policy in an unknown environment.

The main contributions of this paper are summarized as follows.

• In contrast to [27,30], we assumed that the context-aware weight is a first-order irreducible positive recurrent Markov process or independent and identically distributed (i.i.d.) over time. We formulated the updating problem as a CMDP problem and proved the single threshold structure of the UoI-optimal policy. We then derived the policy through LP with the threshold structure and discussed the conditions that the monitored process needs to satisfy for the threshold structure.
• When the distributions of the context-aware weight and the increment in estimation error were unknown, we used model-based RL method to learn the state transitions of the whole system and derive a near-optimal RL-based updating policy.
• Simulations were conducted to verify the theoretical analysis of the threshold structure and show the near-optimal performance of the RL-based updating policy. The results indicate that: (i) the update thresholds decrease when the maximum average update frequency becomes large; (ii) the update threshold for emergency can actually be larger than that for ordinary states when the probability of transferring from emergency to ordinary states tends to 1.

The rest of this paper is organized as follows. The system model and the problem formulation are described in Section 2. In Section 3, we obtain the CMDP formulation of the problem with the given distribution of context-aware weight and prove the threshold structure of the UoI-optimal policy. The proposed model-based RL updating policy is obtained in Section 4. In Section 5, the simulation results are shown and discussed while the conclusions are drawn in Section 6.

2. System Model and Problem Formulation

In this paper, we considered a remote monitoring system, in which a fusion center collects the status information (e.g., current location, velocity, information of surrounding) from a vehicle of interest via a wireless channel with limited resources, as shown in Figure 1. The whole system is considered as a discrete-time system and the status can be generated at will. Due to the limitations on the wireless resources and energy supply, there is a constraint on the average update frequency of the vehicle. The update decision in time slot t is denoted by $U\left(t\right)\in \{0,1\}$, where $U\left(t\right)=1$ means that the vehicle decides to transmit the current status to the center, and $U\left(t\right)=0$ denotes that the vehicle decides to stay idle.

Figure 1. Remote control and monitoring model. The vehicle of interest is shown in red.

The wireless channel is assumed as a block fading channel with successful transmission probability $p_s$. Let $S\left(t\right)\in \{0,1\}$ be the state of the channel. $S\left(t\right)=0$ represents that the channel is in deep fading, and no packet can be successfully transmitted. $S\left(t\right)=1$ means the packets can be successfully transmitted to the center through the channel. If the center receives an update, then $U\left(t\right)S\left(t\right)=1$ and an ACK will be sent to the vehicle.

Let $x\left(t\right)$ and $\widehat{x}\left(t\right)$ denote the current status of the monitored vehicle and the estimated status of the vehicle at the center, and $Q\left(t\right)=x\left(t\right)-\widehat{x}\left(t\right)$ denotes the estimation error. Similar to [26], we further assume that the time period of a packet transmission is less than a time slot and the estimation at the center equals the latest status information received by the center. This estimation scheme is easy to implement, theoretically tractable and has been proven to be an optimal policy that can minimize the average squared error of status estimation in a remote estimation system under energy constraints when the monitored process is a Wiener process [31]. Then, the recurrence relation of the estimation error $Q\left(t\right)$ is:

$$Q(t+1)=(1-U(t\left)S\right(t\left)\right)Q\left(t\right)+A\left(t\right).$$

(3)

Equation (3) indicates that the estimation error will be the amount of variation of the monitored process from the generation time of the latest received status to the current time. The increment $A\left(t\right)$ represents the variation of the monitored process. For example, when $A\left(t\right)$ follows a Gaussian distribution with a mean of zero and variance of ${\sigma }^2$, represented by $N(0,{\sigma }^2)$, the monitored status follows a Wiener process. When $A\left(t\right)$ takes values from $\{0,1,-1\}$ with a probability of $\{1-2p_{rw},p_{rw},p_{rw}\}$, where $0<p_{rw}<\frac{1}{2}$, then the status of the monitored source will be a one-dimensional random walk. In this paper, we assumed that the monitored status of the vehicle is a Wiener process and $A\left(t\right)$ is i.i.d. over time. However, the increment in estimation error during a single slot cannot be infinite in practical systems. Therefore, in contrast to [27,30], we assumed that increment $A\left(t\right)$ obeys a truncated Gaussian distribution, i.e., the probability density function (PDF) of $A\left(t\right)$ is:

$$\begin{array}{c}\hfill f_{A\left(t\right)}\left(a\right)=\frac{\frac{1}{\sigma }\varphi \left(\frac{a-\mu }{\sigma }\right)}{\mathrm{\Phi }\left(\frac{A_{max}-\mu }{\sigma }\right)-\mathrm{\Phi }\left(\frac{-A_{min}-\mu }{\sigma }\right)},\end{array}$$

(4)

where $\mu$ and $\sigma$ are the expectation and standard deviation of increment $A\left(t\right)$. $\varphi$ and $\mathrm{\Phi }$ are the PDF and the cumulative distribution function (CDF) of standard normal distribution. We also assumed $A\left(t\right)\in [-A_{min},A_{max}],A_{max}=A_{min}>0$ and $\mu =0$.

Meanwhile, the scheduling policy of information updates should also be related to the situation and environment of the system. For example, when the system is in an emergency, it should be very sensitive to the accuracy and the delay of the status information, thus the status should be updated more frequently. Therefore, our objective is to find a policy telling the vehicle whether to transmit status information or not in each slot for a minimum average UoI over time under the constraint:

$$\begin{array}{cc}\hfill \underset{U\left(t\right)}{min}& \underset{T\to \infty }{lim\; sup}\frac{1}{T}E\left[\sum _{t=0}^{T-1}w\left(t\right)Q{\left(t\right)}^2\right]\hfill \\ \hfill s.t.& \underset{T\to \infty }{lim\; sup}\frac{1}{T}\sum _{t=0}^{T-1}E\left[U\left(t\right)\right]\le \rho ,\hfill \end{array}$$

(5)

where $\omega \left(t\right)>0$ is the context-aware weight, which is independent with $Q\left(t\right)$. $\rho \in (0,1]$ is the maximum average update frequency. The cost function of the estimation error used here is $\delta \left(Q\left(t\right)\right)={\left(Q\left(t\right)\right)}^2$, which is inspired by the squared error of status estimation.

3. Scheduling with CMDP-Based Approach

In this section, we start by formulating problem (5) into a constrained Markov decision process (CMDP) with assumptions on the distribution of the context-aware weight. We will prove the threshold structure of the UoI-optimal updating policy and derive the optimal policy through a linear programming (LP) formulation.

3.1. Constrained Markov Decision Process Formulation

In the remote monitoring system, the context may be related to the distance between adjacent vehicles/mobile devices, the unexpected maneuver of the neighboring vehicles, etc. In [32], the authors prove that whether the distance between two mobile wireless devices with Ornstein–Uhlenbeck mobility is less than a certain threshold follows a first-order Markov process. When the two devices are closer, they are more interested in each other’s status information, communication and computing resources to facilitate cooperation, share resources, and avoid collisions. At this time, the transmission of status information is more urgent than when the two devices are far apart. As for the unexpected maneuver of the neighboring vehicles, it is very challenging to find a proper formulation. Instead, we assumed that such emergencies occur independently in each slot according to a certain probability. Therefore, in contrast to [27,30], we assumed that the context-aware weight $\omega \left(t\right)$ is i.i.d. over time or a first-order irreducible positive recurrent Markov process and formulated the problem (5) as a CMDP problem. The irreducible positive recurrent Markov formulation guarantees the existence of the UoI-optimal policy (see Appendix A). In this section, we will first focus on the situation where $\omega \left(t\right)$ is a first-order Markov process:

• State space: The state of the vehicle in slot t, denoted by $s\left(t\right)=\left(Q\right(t),\omega (t\left)\right)$, includes the current estimation error and the context-aware weight. Then, we discretize $Q\left(t\right)$ with the step size ${\Delta }_Q>0$, i.e., the estimation error $Q\left(t\right)\in \mathbb{Q}=\{0,\pm {\Delta }_Q,\pm 2{\Delta }_Q,\cdots ,\pm n{\Delta }_Q,\cdots \}$. For example, when $Q\left(t\right)\in [n{\Delta }_Q-\frac{1}{2}{\Delta }_Q,n{\Delta }_Q+\frac{1}{2}{\Delta }_Q)$, its value will be taken as $n{\Delta }_Q$. The smaller the step size ${\Delta }_Q$, the smaller the performance degradation caused by discretization. In addition, the value set of the context-aware weight is denoted by $\mathbb{W}$. Then, the state space $\mathbb{S}=\{\mathbb{Q}\times \mathbb{W}\}$ is thus countable but infinite.
• Action space: At each slot, the vehicle can take two actions, namely $U\left(t\right)\in \mathbb{U}=\{0,1\}$, where $U\left(t\right)=1$ denotes the vehicle deciding to transmit updates in slot t and $U\left(t\right)=0$ denotes the vehicle deciding to wait.
• Probability transfer function: After taking action U at state $s=(Q,\omega )$, the next state is denoted by $s^{\prime }=(Q^{\prime },{\omega }^{\prime })$. When the vehicle decides not to transmit or the transmission fails, the probability of the estimation error transferring from Q to $Q^{\prime }$ is written as $\mathrm{Pr}\{Q^{\prime }-Q=a\}=p_a$. Due to the discretization of the estimation error, the increment $a\in \mathbb{A}=\{0,\pm {\Delta }_Q,\pm 2{\Delta }_Q,\cdots ,\pm A_m\}$, where $A_m=\lfloor \frac{A_{max}}{{\Delta }_Q}\rfloor {\Delta }_Q>0$. In addition, $p_a=F_A(a+\frac{1}{2}{\Delta }_Q)-F_A(a-\frac{1}{2}{\Delta }_Q)$, where $F_A\left(a\right)$ is the CDF of increment $A\left(t\right)$. In addition, the probability of the context-aware weight transferring from $\omega$ to ${\omega }^{\prime }$ is written as $\mathrm{Pr}\{\omega \to {\omega }^{\prime }\}=p_{\omega {\omega }^{\prime }}$. Based on the assumption that the context-aware weight $\omega \left(t\right)$ is independent with the estimation error $Q\left(t\right)$, then the probability of the state transferring from $s=(Q,\omega )$ to $s^{\prime }=(Q^{\prime },{\omega }^{\prime })$ given action U is:

  $$\begin{array}{cc}\hfill & \mathrm{Pr}\{s\to s^{\prime }|U\}=\mathrm{Pr}\{(Q,\omega )\to (Q^{\prime },{\omega }^{\prime })|U\}\hfill \\ \hfill & =\left\{\begin{array}{cc}{p}_{\omega {\omega }^{\prime }}{p}_{Q^{\prime }-Q}\hfill & ,U=0,\hfill \\ p_{\omega {\omega }^{\prime }}((1-p_s)p_{Q^{\prime }-Q}+p_s{p}_{Q^{\prime }-0})\hfill & ,U=1.\hfill \end{array}\right.\hfill \end{array}$$

  (6)
• One-step cost: The cost caused by taking action U in state $(Q,\omega )$ is:

  $$\begin{array}{c}\hfill C(Q,\omega ,U)=\omega Q^2,\end{array}$$

  (7)

  while the one-step updating penalty only depends on the chosen action:

  $$\begin{array}{c}\hfill D(Q,\omega ,U)=U.\end{array}$$

  (8)

The average cost caused under a certain policy $\pi$ is the average UoI, which is defined as ${\overline{C}}^{\pi }$ and the average updating penalty under $\pi$ is defined as ${\overline{D}}^{\pi }$. We aimed to find the UoI-optimal policy which minimizes the average cost under the resources constraint. Therefore, problem (5) can be formulated into the following CMDP problem:

$$\begin{array}{cc}\hfill \underset{\pi }{min}& {\overline{C}}^{\pi }=\underset{T\to \infty }{lim}\frac{1}{T}{\mathbb{E}}_{\pi }\left[\sum _{t=1}^{T}C(Q\left(t\right),\omega \left(t\right),U\left(t\right))\right]\hfill \\ \hfill s.t.& {\overline{D}}^{\pi }=\underset{T\to \infty }{lim}\frac{1}{T}{\mathbb{E}}_{\pi }\left[\sum _{t=1}^{T}D(Q\left(t\right),\omega \left(t\right),U\left(t\right))\right]\le \rho .\hfill \end{array}$$

(9)

3.2. Threshold Structure of the Optimal Policy

We start from some basic definitions in [33] and show the properties of problem (9).

Definition 1.

A stationary deterministic policy is a policy that takes the same action whenever in a given state $s=(Q,\omega )$, while a stationary randomized policy chooses to update or not in state s with a certain probability.

Theorem 1.

There exists an optimal stationary randomized policy for problem (9). The optimal policy is a probabilistic combination of two stationary deterministic policies. The two deterministic policies only differ on at most one state and each policy minimizes the unconstrained cost in (10) with a different Lagrange multiplier λ:

$$\begin{array}{c}\hfill L_{\lambda }^{\pi }=\underset{T\to \infty }{lim}\frac{1}{T}{\mathbb{E}}_{\pi }\left[\sum _{t=1}^T\left[C\left(Q\right(t),\omega (t),U(t\left)\right)+\lambda D\left(Q\right(t),\omega (t),U(t\left)\right)\right]\right].\end{array}$$

(10)

Proof of Theorem 1.

The proof is shown in Appendix A. □

We denote the optimal policy that minimizes the unconstrained cost in (10) with a given $\lambda$ by ${\pi }^{★}$ and the cost obtained under policy ${\pi }^{★}$ by $L_{\lambda }^{{\pi }^{★}}$, namely $L_{\lambda }^{{\pi }^{★}}={min}_{\pi }{L}_{\lambda }^{\pi }$. Then, there exists a differential cost function $V(Q,\omega )$ that satisfies the Bellman Equation [34]:

$$\begin{array}{cc}\hfill & V(Q,\omega )+L_{\lambda }^{{\pi }^{★}}=min\{C(Q,\omega ,1)+\lambda D(Q,\omega ,1)\hfill \\ & +(1-p_s)\sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_{a}V\left(Q+a,{\omega }^{\prime }\right)+p_s\sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_{a}V\left(a,{\omega }^{\prime }\right),\hfill \\ & C(Q,\omega ,0)+\sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_{a}V\left(Q+a,{\omega }^{\prime }\right)\}.\hfill \end{array}$$

(11)

To solve problem (5), we first prove that with a given $\lambda$, the optimal stationary deterministic policy ${\pi }^{★}$ has a threshold structure. We then introduce a discounted problem with a discount factor $\alpha$ and the discounted cost starting from state $(Q,\omega )$ under a certain policy $\pi$ is:

$$\begin{array}{cc}\hfill J_{\alpha ,\pi }(Q,\omega )=& \underset{T\to \infty }{lim}{\mathbb{E}}_{\pi }[\sum _{t=0}^T{\alpha }^t\left[C\left(Q\right(t),\omega (t),U(t\left)\right)\right.\hfill \\ \hfill & \left.+\lambda D\left(Q\right(t),\omega (t),U(t\left)\right)\right]\mid (Q\left(0\right)=Q,\omega \left(0\right)=\omega )].\hfill \end{array}$$

(12)

Denote the minimum cost starting from state $(Q,\omega )$ by $V_{\alpha }(Q,\omega )={min}_{\pi }{J}_{\alpha ,\pi }(Q,\omega )$. Then, we have:

$$\begin{array}{cc}\hfill V_{\alpha }(Q,\omega )=& min\{C(Q,\omega ,1)+\lambda D(Q,\omega ,1)+(1-p_s)\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(Q+a,{\omega }^{\prime }\right)\hfill \\ \hfill +& p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(a,{\omega }^{\prime }\right),C(Q,\omega ,0)\hfill \\ \hfill +& \alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(Q+a,{\omega }^{\prime }\right)\}.\hfill \end{array}$$

(13)

Define $\Delta (Q,\omega )$ as the difference between the value functions by taking the two different actions $U=0,1$, meaning that:

$$\begin{array}{cc}\hfill & \Delta (Q,\omega )=C(Q,\omega ,0)+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(Q+a,{\omega }^{\prime }\right)\hfill \\ \hfill -& C(Q,\omega ,1)-\lambda D(Q,\omega ,1)-p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(a,{\omega }^{\prime }\right)\hfill \\ \hfill -& (1-p_s)\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(Q+a,{\omega }^{\prime }\right)\hfill \\ \hfill =& p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a\left\{V_{\alpha }\left(Q+a,{\omega }^{\prime }\right)-V_{\alpha }\left(a,{\omega }^{\prime }\right)\right\}-\lambda .\hfill \end{array}$$

(14)

Define ${\sum }_{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(Q+a,\omega \right)$ as a function $f_{\alpha }(Q,\omega )$. Then we will prove that for $\forall |Q_1|<|Q_2|$, we have $f_{\alpha }(Q_1,\omega )<f_{\alpha }(Q_2,\omega )$. To this end, we first prove the following Lemma 1.

Lemma 1.

For a given discount factor α and a fixed context-aware weight ω, the value function for Q equals the value function for $-Q$, namely:

$$V_{\alpha }(Q,\omega )=V_{\alpha }(-Q,\omega ).$$

Proof of Lemma 1.

The Lemma is proven by induction. Define $V_{\alpha }^{\left(k\right)}(Q,\omega )$ as the value function obtained after the $k^{\mathrm{th}}$ iteration. Assume that for $\forall Q$, we have: $V_{\alpha }^{\left(k\right)}(Q,\omega )=V_{\alpha }^{\left(k\right)}(-Q,\omega )$. If action U is taken in the $k^{\mathrm{th}}$ iteration, then the expected discounted cost is defined as $J_{\alpha ,U}^{\left(k\right)}(Q,\omega )$. Therefore, $V_{\alpha }^{(k+1)}\left(Q,\omega \right)={min}_U{J}_{\alpha ,U}^{\left(k\right)}(Q,\omega )$. We have:

$$\begin{array}{cc}\hfill J_{\alpha ,0}^{\left(k\right)}\left(Q,\omega \right)& =C\left(Q,\omega ,0\right)+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }^{\left(k\right)}\left(Q+a,{\omega }^{\prime }\right)\hfill \\ \hfill & =\omega {(-Q)}^2+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }^{\left(k\right)}\left(-Q-a,{\omega }^{\prime }\right)\hfill \\ \hfill & =C_X\left(-Q,\omega ,0\right)+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }^{\left(k\right)}\left(-Q+a,{\omega }^{\prime }\right)=J_{\alpha ,0}^{\left(k\right)}\left(-Q,\omega \right).\hfill \end{array}$$

(15)

Similarly, we can further prove that $J_{\alpha ,1}^{\left(k\right)}\left(Q,\omega \right)=J_{\alpha ,1}^{\left(k\right)}\left(-Q,\omega \right)$. Notice that the value function obtained in ${(k+1)}^{\mathrm{th}}$ iteration is obtained by: $V_{\alpha }^{(k+1)}\left(Q,\omega \right)={min}_U{J}_{\alpha ,U}^{\left(k\right)}(Q,\omega ),$ and for any action U, $J_{\alpha ,U}^{\left(k\right)}\left(Q,\omega \right)=J_{\alpha ,U}^{\left(k\right)}\left(-Q,\omega \right)$. Thus, $V_{\alpha }^{(k+1)}(Q,\omega )=V_{\alpha }^{(k+1)}(-Q,\omega )$. By letting $k\to \infty$, $V_{\alpha }^{\left(k\right)}(Q,\omega )\to V_{\alpha }(Q,\omega )$. Hence, $V_{\alpha }(Q,\omega )=V_{\alpha }(-Q,\omega )$. □

Lemma 2.

For a given discount factor α and a fixed context-aware weight ω, function $f_{\alpha }(Q,\omega )$ for Q increases monotonically with the absolute value of Q, namely: for $\forall |Q_1|<|Q_2|,f_{\alpha }(Q_1,\omega )<f_{\alpha }(Q_2,\omega )$.

Proof of Lemma 2.

Using the induction method, we first assume that for $\forall |Q_1|<|Q_2|$, we have $f_{\alpha }^{\left(k\right)}(Q_1,\omega )<f_{\alpha }^{\left(k\right)}(Q_2,\omega )$. Therefore:

$$\begin{array}{cc}\hfill J_{\alpha ,0}^{\left(k\right)}\left(Q_1,\omega \right)& =C\left(Q_1,\omega ,0\right)+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }^{\left(k\right)}\left(Q_1+a,{\omega }^{\prime }\right)\hfill \\ \hfill & =\omega Q_1^2+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}{f}_{\alpha }^{\left(k\right)}(Q_1,{\omega }^{\prime })\hfill \\ \hfill & <C\left(Q_2,\omega ,0\right)+\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}{f}_{\alpha }^{\left(k\right)}(Q_2,{\omega }^{\prime })\hfill \\ & =J_{\alpha ,0}^{\left(k\right)}\left(Q_2,\omega \right).\hfill \end{array}$$

(16)

Similarly, we can obtain $J_{\alpha ,1}^{\left(k\right)}\left(Q_1,\omega \right)<J_{\alpha ,1}^{\left(k\right)}\left(Q_2,\omega \right)$. Meanwhile, $V_{\alpha }^{(k+1)}\left(Q,\omega \right)={min}_U{J}_{\alpha ,U}^{\left(k\right)}(Q,\omega )$, then we have $V_{\alpha }^{(k+1)}\left(Q_1,\omega \right)<V_{\alpha }^{(k+1)}\left(Q_2,\omega \right)$, for $\forall |Q_1|<|Q_2|$. Obviously, if we want to use induction to complete the proof of Lemma 2, we have to prove that: $f_{\alpha }^{(k+1)}(Q_1,\omega )<f_{\alpha }^{(k+1)}(Q_2,\omega )$, for $\forall |Q_1|<|Q_2|$. To simplify the proof, it is assumed that $Q_2>Q_1>0$. The discussion will be divided into the following three situations.

• When $A_m\le \left|Q_1\right|$, then $|Q_1+a|<|Q_2+a|$, for $\forall a\in [-A_m,A_m]$, we can derive that:

  $$\begin{array}{cc}\hfill f_{\alpha }^{(k+1)}(Q_1,\omega )=& \sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)\hfill \\ \hfill <& \sum _{a=-A_m}^{A_m}{p}_a{V}_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)=f_{\alpha }^{(k+1)}(Q_2,\omega ).\hfill \end{array}$$

  (17)
• When $A_m>\left|Q_2\right|$, there exists an increment $a^{\prime }\in {\mathcal{A}}^{\prime }=\left\{a\right|a\in [-A_m,-\frac{1}{2}(Q_1+Q_2)\}$, such that $|Q_1+a^{\prime }|>|Q_2+a^{\prime }|$, and $V_{\alpha }^{(k+1)}\left(Q_1+a^{\prime },{\omega }^{\prime }\right)>V_{\alpha }^{(k+1)}\left(Q_2+a^{\prime },{\omega }^{\prime }\right)$. Notice that $-Q_1-a^{\prime }\in (\frac{1}{2}(Q_2-Q_1),A_m-Q_1]$ and $Q_2+a\in [-A_m+Q_2,A_m+Q_2]$, then $p_{-Q_1-a^{\prime }-Q_2}{V}_{\alpha }^{(k+1)}\left(-Q_1-a^{\prime },\omega \right)$ is a term in the summation $f_{\alpha }^{(k+1)}(Q_2,\omega )$, namely ${\sum }_{a=-A_m}^{A_m}{p}_a{V}_{\alpha }\left(Q+a,\omega \right)$. Similarly, $p_{-Q_2-a^{\prime }-Q_1}{V}_{\alpha }^{(k+1)}\left(-Q_2-a^{\prime },\omega \right)$ is a term in the summation $f_{\alpha }^{(k+1)}(Q_1,\omega )$. We further define ${\mathcal{A}}^{″}=\left\{a\right|a=-Q_1-Q_2-a^{\prime }\}$, since $-Q_1-Q_2-a^{\prime }\in \left(-\frac{1}{2}(Q_1+Q_2),A_m-Q_1-Q_2\right]$, then $A^{\prime }\cap A^{″}=⌀$.
  Furthermore, the probability of the estimation error transferring from $Q_1$ to $-Q_2-a^{\prime }$, i.e., $p_{-Q_2-a^{\prime }-Q_1}$ equals $p_{-Q_1-a^{\prime }-Q_2}$, the probability of the estimation error transferring from $Q_2$ to $-Q_1-a^{\prime }$. Since $-a^{\prime }\in (\frac{1}{2}(Q_1+Q_2),A_m]$, then $|a^{\prime }|>|-Q_1-Q_2-a^{\prime }|$. According to our assumption of the increment, we can prove that for any $a^{\prime }\in {\mathcal{A}}^{\prime }$, $p_{a^{\prime }}<p_{-Q_1-Q_2-a^{\prime }}$. Then, we can derive:

  $$\begin{array}{cc}\hfill & f_{\alpha }^{(k+1)}(Q_1,\omega )-f_{\alpha }^{(k+1)}(Q_2,\omega )\hfill \\ \hfill =& \sum _{a\in {\mathcal{A}}^{\prime }}{p}_a{V}_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)+\sum _{a\in {\mathcal{A}}^{″}}{p}_a{V}_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)\hfill \\ \hfill -& \sum _{a\in {\mathcal{A}}^{\prime }}{p}_a{V}_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)-\sum _{a\in {\mathcal{A}}^{″}}{p}_a{V}_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)+M(Q_1,Q_2)\hfill \\ \hfill =& \sum _{a\in {\mathcal{A}}^{\prime }}{p}_a\{V_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)-V_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)\}\hfill \\ \hfill +& \sum _{a\in {\mathcal{A}}^{\prime }}{p}_{-Q_1-Q_2-a}\{V_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)-V_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)\}+M(Q_1,Q_2)\hfill \\ \hfill =& \sum _{a\in {\mathcal{A}}^{\prime }}\left(p_a-p_{-Q_1-Q_2-a}\right)\{V_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)-V_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)\}+M(Q_1,Q_2)<0,\hfill \end{array}$$

  (18)

  where $M(Q_1,Q_2)={\sum }_{a\notin {\mathcal{A}}^{\prime }\cup {\mathcal{A}}^{″}}{p}_a\left(V_{\alpha }^{(k+1)}\left(Q_1+a,\omega \right)-V_{\alpha }^{(k+1)}\left(Q_2+a,\omega \right)\right)<0$.
• When $|Q_2|>A_m>\left|Q_1\right|$, since $a^{\prime }\in [-A_m,-\frac{1}{2}(Q_1+Q_2))$, we only need to consider the case when $A_m>\frac{1}{2}(Q_1+Q_2)$, in this case $-Q_1-a^{\prime }>\frac{1}{2}(Q_2-Q_1)>Q_2-A_m$. Therefore, $p_{-Q_1-a^{\prime }-Q_2}{V}_{\alpha }^{(k+1)}\left(-Q_1-a^{\prime },\omega \right)$ is a term in the summation $f_{\alpha }^{(k+1)}(Q_2,\omega )$. Similarly, we can also prove that $f_{\alpha }^{(k+1)}(Q_1,\omega )<f_{\alpha }^{(k+1)}(Q_2,\omega )$ when $|Q_2|>A_m>\left|Q_1\right|$.

According to Lemma 1, the conclusions above can be easily generalized to the cases without the condition $Q_2>Q_1>0$. Finally, by letting $k\to \infty$, $V_{\alpha }^{(k+1)}(Q,\omega )\to V_{\alpha }(Q,\omega )$, therefore: $f_{\alpha }^{(k+1)}(Q,\omega )\to f_{\alpha }(Q,\omega )$. Hence: $f_{\alpha }(Q_1,\omega )<f_{\alpha }(Q_2,\omega )$. □

Remark 1.

Lemma 2 holds when $f_A\left(a\right)$, i.e., the PDF of increment $A\left(t\right)$ satisfies the following conditions:

• $f_A\left(a\right)=f_A(-a)$, $\mu =0$;
• $f_A\left(a_2\right)\le f_A\left(a_1\right),\forall a_2\ge a_1\ge 0$.

Then, with Lemmas 1 and 2, we can prove the threshold structure of the optimal stationary deterministic policy which minimizes $L_{\lambda }^{\pi }$ in (10).

Theorem 2.

For a given λ, the optimal stationary deterministic policy which minimizes $L_{\lambda }^{\pi }$ in (10) has a threshold structure when the context-aware weight is a first-order irreducible positive recurrent Markov process.

Proof of Theorem 2.

Let $s_{\alpha }^{*}(Q,\omega )$ denote the optimal action which minimizes the discounted cost $V_{\alpha }(Q,\omega )$ at state $(Q,\omega )$. If the optimal action $s_{\alpha }^{*}(Q,\omega )=1$, then the vehicle will transmit its status update to the center at state $(Q,\omega )$ and $\Delta (Q,\omega )\ge 0$. Thus, we have:

$$\begin{array}{c}\hfill \Delta (Q,\omega )=p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a\left\{V_{\alpha }\left(Q+a,{\omega }^{\prime }\right)-V_{\alpha }\left(a,{\omega }^{\prime }\right)\right\}-\lambda \ge 0.\end{array}$$

(19)

According to Lemma 2, for any $|Q^{\prime }|>|Q|$, $\Delta (Q^{\prime },\omega )$ can be lower bounded by

$$\begin{array}{cc}\hfill \Delta (Q^{\prime },\omega )& =p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a\left\{V_{\alpha }\left(Q^{\prime }+a,{\omega }^{\prime }\right)-V_{\alpha }\left(a,{\omega }^{\prime }\right)\right\}-\lambda \hfill \\ \hfill & \ge p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{\omega {\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a\left\{V_{\alpha }\left(Q+a,{\omega }^{\prime }\right)-V_{\alpha }\left(a,{\omega }^{\prime }\right)\right\}-\lambda \ge 0.\hfill \end{array}$$

(20)

If $\Delta (Q,\omega )>0$, then for any states with $|Q^{\prime }|>|Q|$, the optimal policy is to transmit the status to the center. If $\Delta (Q,\omega )<0$, then for any states with $|Q^{\prime }|<|Q|$, the optimal action is not to transmit. In addition, the optimal policy will not be choosing to wait in all the slots. Therefore, for each context-aware weight $\omega$, there must be a threshold ${\tau }_{\omega }\ge 0$. For any state $(Q,\omega )$ with $\left|Q\right|>{\tau }_{\omega }$, the optimal choice is to transmit the status update. We can then conclude that for a given weight $\omega$, the optimal policy with a discount factor $\alpha$ has a threshold structure.

Let $\{{\alpha }_1,{\alpha }_2,\cdots ,{\alpha }_k\}$ denote a sequence of discount factors and ${\alpha }_k$ converges to 1 when $k\to \infty$. Then, the optimal deterministic policy for $\alpha =1$ will also converge to the optimal policy with a discount factor which is less than 1 [35]. Similar derivation is also applied in [12]. Therefore, we can prove the threshold structure of the optimal stationary deterministic policy which minimizes $L_{\lambda }^{\pi }$. □

Similarly, when the context-aware weight is i.i.d. over time, we can obtain the following theorem:

Theorem 3.

For a given λ, the optimal stationary deterministic policy which minimizes $L_{\lambda }^{\pi }$ in (10) has a threshold structure when the context-aware weight is i.i.d. over time. The thresholds are the same for each state of the context-aware weight.

Proof of Theorem 3.

If the context-aware weight is i.i.d. over time, then we have:

$$\begin{array}{c}\hfill \Delta (Q,\omega )=p_s\alpha \sum _{{\omega }^{\prime }\in \mathbb{W}}{p}_{{\omega }^{\prime }}\sum _{a=-A_m}^{A_m}{p}_a\left\{V_{\alpha }\left(Q+a,{\omega }^{\prime }\right)-V_{\alpha }\left(a,{\omega }^{\prime }\right)\right\}-\lambda =\Delta \left(Q\right),\end{array}$$

(21)

where $p_{\omega }$ is the probability of the value of the context-aware weight being in state $\omega$. Therefore, in this case, the state will be reduced to one dimension and the thresholds will be the same for all the states of the context-aware weight. □

According to Theorems 2 and 3, we proved the threshold structure of the two stationary deterministic policies that compose the UoI-optimal policy. Since the UoI-optimal policy for problem (9) is a probabilistic combination of two deterministic policies with threshold structures, we can finally draw the conclusion that the UoI-optimal policy also has a threshold structure.

3.3. Numerical Solution of Optimal Strategy

Based on Theorem 2, we only need to consider the policy that chooses to update with a probability of 1 in state $(Q,\omega )$, for $\forall \left|Q\right|\ge Q_{max}={max}_{\omega }{\tau }_{\omega }$. Let ${\mu }_{Q,\omega }$ denote the probability that the state of the vehicle is $(Q,\omega )$. $y_{Q,\omega }$ denotes the probability that the state is $(Q,\omega )$ and the vehicle chooses to transmit an update. Therefore, we have:

Theorem 4.

When the context-aware weight is a first-order irreducible positive recurrent Markov process, the UoI-optimal policy can be derived by solving the following LP problem:

$$\begin{array}{cc}\hfill \{{\mu }_{Q,\omega }^{*},y_{Q,\omega }^{*}\}=& arg\underset{\{{\mu }_{Q,\omega },y_{Q,\omega }\}}{min}\sum _{\omega \in \mathbb{W}}\sum _{Q=-Q_{max}}^{Q_{max}}\omega Q^2{\mu }_{Q,\omega },\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill \mathrm{s}.\mathrm{t}.& \sum _{\omega \in \mathbb{W}}\sum _{Q=-Q_{max}}^{Q_{max}}{\mu }_{Q,\omega }=1,\hfill \end{array}$$

(22b)

$$\begin{array}{cc}\hfill & \sum _{\omega \in \mathbb{W}}\sum _{Q=-Q_{max}}^{Q_{max}}{y}_{Q,\omega }\le \rho ,\hfill \end{array}$$

(22c)

$$\begin{array}{cc}\hfill & y_{Q,\omega }\le {\mu }_{Q,\omega },\forall Q,\omega ,\hfill \end{array}$$

(22d)

$$\begin{array}{cc}\hfill & 0\le y_{Q,\omega }\le 1,0\le {\mu }_{Q,\omega }\le 1,\forall Q,\omega ,\hfill \\ \hfill & {\mu }_{Q,\omega }=\sum _{{\omega }^{\prime }\in \mathbb{W}}\sum _{Q^{\prime }=-Q_{max}}^{Q_{max}}{y}_{Q^{\prime },{\omega }^{\prime }}{p}_s{p}_Q{p}_{\omega {\omega }^{\prime }}\hfill \end{array}$$

(22e)

$$\begin{array}{c}\hfill +\sum _{{\omega }^{\prime }\in \mathbb{W}}\sum _{Q^{\prime }=-Q_{max}}^{Q_{max}}({\mu }_{Q^{\prime },{\omega }^{\prime }}-y_{Q^{\prime },{\omega }^{\prime }}{p}_s)p_{Q^{\prime }-Q}{p}_{\omega {\omega }^{\prime }}.\end{array}$$

(22f)

Proof of Theorem 4.

We first derive the average UoI ${\overline{C}}^{\pi }$ as a function of ${\mu }_{Q,\omega }$ and $y_{Q,\omega }$. The vehicle is in state $(Q,\omega )$ and produces a cost of $C(Q,\omega ,u)=\omega Q^2$ with a probability of ${\mu }_{Q,\omega }$. Therefore, the average UoI is:

$$\begin{array}{c}\hfill \sum _{\omega \in \mathbb{W}}\sum _{Q=-Q_{max}}^{Q_{max}}\omega Q^2{\mu }_{Q,\omega }.\end{array}$$

(23)

As for the constraints, (22b) means that the sum of the probabilities of all the states should be 1. To explain (22c), we note that $y_{Q,\omega }$ is the probability of the vehicle being in state $(Q,\omega )$ and choosing to transmit the update, then the expectation of a one-step updating penalty for state $(Q,\omega )$ in (8) is ${\mu }_{Q,\omega }$. Therefore, the constraint on average update frequency ${\overline{D}}^{\pi }$ can be illustrated by

$$\begin{array}{c}\hfill \sum _{\omega \in \mathbb{W}}\sum _{Q=-Q_{max}}^{Q_{max}}{\mu }_{Q,\omega }\le \rho .\end{array}$$

(24)

Then, we introduce ${\xi }_{Q,\omega }\in [0,1]$ to represent that the probability of the vehicle choosing to transmit updates in state $(Q,\omega )$ and (22d) can be obtained by the fact that $y_{Q,\omega }={\mu }_{Q,\omega }{\xi }_{Q,\omega }$, while (22e) is derived by the nature of probability.

The right-hand side of (22f) can be viewed as two terms. The first term is the sum of transition probability from all the states to state $(Q,\omega )$ when the vehicle chooses to update and the transmission of status is successful. The second term is the sum of transition probability from all the states to state $(Q,\omega )$ when the transmission is failed or the vehicle chooses to wait. Therefore, we can prove that the optimal solution of problem (5) equals the solution of the LP problem. □

When $\omega \left(t\right)$ is i.i.d. over time, we can also obtain the UoI-optimal policy through the LP problem proposed in Theorem 4 and only need to use $p_{{\omega }^{\prime }}$ as a replacement of $p_{\omega {\omega }^{\prime }}$.

4. Scheduling in Unknown Contexts

To make decisions, the UoI-optimal updating policy obtained in Section 3 still needs the distributions of the context-aware weight $\omega \left(t\right)$, the increment $A\left(t\right)$ and the successful transmission probability, which may not be available in advance or may change over time in most practical systems. To solve this problem, we will assume that the distribution of the context-aware weight is not pre-determined and the vehicle has to learn it. In this section, we use the reinforcement learning (RL) algorithm to learn the dynamic of the context and the characteristic of the wireless channel.

To solve this problem, we turn to the model-based RL framework proposed in [36]. We only consider the cases when the UoI-optimal policy has a threshold structure. This assumption makes the optimal policy based on the truncated state space equal the optimal policy of the original problem.

We use the 3-tuple $(s,s^{\prime },U)$ to formulate the proposed RL-based updating policy. The states in the current slot and next slot are denoted by s and $s^{\prime }$, respectively. U denotes the action chosen in the current slot. The settings of the discretized state space and the action space are the same as the settings proposed in Section 3.1. The smaller the step size used in the discretization is, the closer our results are to those in continuous state space. In addition, the selection of the step size only affects the accuracy of the update threshold. Therefore, the performance loss caused by discretization can be reduced by choosing a smaller step size.

We display details about the proposed RL-based updating policy in Algorithm 1. At the beginning of episode k, we randomly decide whether to explore or exploit. $l\in [0,1]$ represents the trade-off between exploration and exploitation during the following episode. A larger l means a higher frequency of exploration and vice versa. If the algorithm chooses to explore during this episode, a random policy ${\pi }_{rand}\left(s\right)$ will be used, i.e., we randomly choose to update or not in each state to find more valuable actions. If the algorithm chooses to exploit, then we have to obtain the probability transfer functions ${\tilde{p}}_k\left(s^{\prime }\right|s,U)$ for each state transmission pair. In Algorithm 1, $N(s,U)$ and $N(s,U,s^{\prime })$ represent the number of occurrences of state–action pair $s,U$ and state transition from s to $s^{\prime }$ given action U, respectively. Based on the assumption that the optimal policy has a threshold structure, the policy $\pi \left(k\right)$ which can minimize the average UoI with the estimated probability transfer functions, can be directly solved through the LP problem proposed in Theorem 4. Then, the vehicle will use policy ${\pi }_k$ to derive state–action pairs and the state transitions in the following $\lceil L_k\rceil$ slots. Here, $L>0$ is defined to control the number of state transitions observed in each episode. At the end of each episode, the model will be updated according to the state–action pairs and the state transitions observed during the episode. Finally, after K episodes, the algorithm will output the RL-based updating policy ${\pi }^{★}\left(s\right)$, which is derived based on ${\tilde{p}}_K\left(s^{\prime }\right|s,U)$.

Algorithm 1 RL-based Updating Policy

Input:  $l\in [0,1],L>0,K>0$ 1: for episodes $k=1,2,\cdots ,K$ do 2:  Set $L_k=L\sqrt{k},{ϵ}_k=l/\sqrt{k}$, uniformly draw $\alpha \in [0,1]$. 3:  if $\alpha <{ϵ}_k$ then 4:   Set ${\pi }_k\left(s\right)={\pi }_{rand}\left(s\right),$ 5:  else 6:   for each state $s,s^{\prime }\in \mathbb{S}$ and $U\in \mathbb{U}$ do 7:    if $N(s,U)>0$ then 8:     Let ${\tilde{p}}_k\left(s^{\prime }\right|s,U)=N(s,U,s^{\prime })/N(s,u),$ 9:    else 10:     ${\tilde{p}}_k\left(s^{\prime }\right|s,U)=1/\left|\mathbb{S}\right|.$ 11:    end if 12:   end for 13:   obtain policy ${\pi }_k\left(s\right)$ by solving the estimated CMDP 14:  end if 15:  Randomly choose an initial state $s\left(1\right)$. 16:  for slots $t=1,2,\cdots ,\lceil L_k\rceil -1$ do 17:   Choose action $U\left(t\right)$ as ${\pi }_k\left(s\left(t\right)\right)$. 18:   Observe the next state $s(t+1)$. 19:   $N\left(s\right(t),U(t),s(t+1\left)\right)=N\left(s\right(t),U(t),s(t+1\left)\right)+1$. 20:   $N\left(s\right(t),U(t\left)\right)=N\left(s\right(t),U(t\left)\right)+1$. 21:   $s\left(t\right)\leftarrow s(t+1)$. 22:  end for 23: end for 24: obtain policy ${\pi }^{★}\left(s\right)$ by solving the estimated CMDP based on ${\tilde{p}}_k\left(s^{\prime }\right|s,U)$, $s,s^{\prime }\in \mathbb{S},U\in \mathbb{U}$. Output: output the RL-based updating policy ${\pi }^{★}\left(s\right)$

5. Simulation Results and Discussion

5.1. Simulation Setup

To facilitate the simulation, we consider the case where the context-aware weight of the vehicle only has two different states: the ’normal’ state and ’urgent’ state. The ’normal’ state means that the vehicle is in ordinary situations and the significance of accuracy of status information is relatively low. We set $\omega \left(t\right)$ as 1 in ’normal’ state while $\omega \left(t\right)$ is set as a constant much larger than 1, ${\omega }_e$, in ’urgent’ state to show that the vehicle is in emergencies. Two different distributions of the context-aware weight are taken into consideration to conform to the assumptions about $\omega \left(t\right)$ used in Section 3.1:

• The context-aware weight $\omega \left(t\right)$ has the first-order Markov property. The state transition diagram of $\omega \left(t\right)$ is shown in Figure 2 and $\omega \left(t\right)$ is irreducible and positive recurrent. $p_1$ is the probability of the context-aware weight transferring from the normal state to the urgent state, while $p_2$ is the probability of the weight transferring from the urgent state to the normal state;
  

  Figure 2. The state transition diagram of $\omega \left(t\right)$.
• The context-aware weight $\omega \left(t\right)$ is i.i.d. over time. The probability of the weight being in the urgent state and the normal state are denoted by $p_h$ and $p_l$, respectively.

As for the increment $A\left(t\right)$, $A_{max}$ is set to a large enough positive number to simplify the simulations.

5.2. Numerical Results

Figure 3 shows the structure of the UoI-optimal updating policy. For the discretization of the estimation error, the step size used is 1. It can be seen that under the two different distributions of the context-aware weight mentioned above, the optimal updating policies all have threshold structures. Especially when the context-aware weight is i.i.d. over time, Figure 3b shows that thresholds for all the states of the context-aware weight are the same, which matches well with theoretical analysis. From Figure 3c, we can find that the UoI-optimal policy also has threshold structure when increment $A\left(t\right)$ obeys a uniform distribution $Unif(-3,3)$, which verifies Remark 1. We then simulate the UoI-optimal policy under the contexts with more states to show the policy is generic. We consider a three-state context-aware weight which takes value from ${\omega }_1=1,{\omega }_2=50,{\omega }_3=100$. The state transition matrix $P_3$ of the three-state context-aware weight is:

$$P_3=\left[\begin{array}{ccc}0.997& 0.002& 0.001\\ 0.02& 0.97& 0.01\\ 0.2& 0.1& 0.7\end{array}\right],$$

(25)

where the j-th element on the i-th row indicates the probability that the context transfers from state ${\omega }_i$ to state ${\omega }_j$. The numerical results (Figure 3d) show that when the context-aware weight has more states, the UoI-optimal policy still has a threshold structure, which verifies our theoretical results.

Figure 3. Threshold structure of the UoI-optimal updating policy when: (a) the context-aware weight is a first-order Markov process, $\rho =0.05,p_1=0.001,p_2=0.01,p_s=0.9,{\sigma }^2=1,{\omega }_e=100$; (b) the context-aware weight is i.i.d. over time, $\rho =0.05,p_l=0.999,p_h=0.001,p_s=0.9,{\sigma }^2=1,{\omega }_e=100$; (c) the context-aware weight is a first order Markov process, $\rho =0.05,p_1=0.001,p_2=0.01,p_s=0.9,{\omega }_e=100$, increment in the estimation error during one slot, i.e., $A\left(t\right)\sim Unif(-3,3)$, for $\forall t$; and (d) the context-aware weight is a three-state first-order Markov process, which takes value from ${\omega }_1=1,{\omega }_2=50,{\omega }_3=100$ and evolves according to the state transition matrix $P_3$, $\rho =0.05,p_s=0.9,{\sigma }^2=1$.

Then, we will focus on the results obtained when the context-aware weight is a first-order irreducible positive recurrent Markov process, as shown in Figure 2. Figure 4 shows the average UoI of the UoI-optimal policy, the AoI-optimal policy derived by CMDP, the RL-based updating policy, and the update-index-based adaptive scheme [27]. In the RL-based updating policy, $L=8000$, $l=1$ and $K=50$. All the numerical results of the RL-based policy are averaged over 100 runs.

Figure 4. Average UoI of the UoI-optimal updating policy, the RL-based updating policy, the update-index-based adaptive scheme [27], and the AoI-optimal updating policy when $p_1=0.001,p_2=0.01,p_s=0.9,{\sigma }^2=1,{\omega }_e=100$.

First of all, the UoI-optimal policy can only be obtained based on advanced information about the system dynamics. However, the RL-based policy achieves near-optimal without knowing the system dynamics, which indicates that Algorithm 1 learns relatively accurate probability transfer functions from the observed state–action pairs and state transitions during the training.

Secondly, according to Figure 4, the AoI-optimal policy yields a much higher UoI than the three UoI-based policies, namely the UoI-optimal policy, the RL-based updating policy, and the update-index-based adaptive scheme. On the one hand, AoI is one special case of UoI. When the context-aware weight $\omega \left(t\right)=1$, the increment $A\left(t\right)=1$, and the cost function $\delta \left(Q\right(t\left)\right)=Q\left(t\right)$, then UoI equals AoI. Therefore, the AoI-optimal policy ignores the fact that different contexts have different requirements for information freshness. In the proposed UoI-based updating policies, different contexts have different policies and update thresholds, while the AoI-optimal updating policies for different contexts are the same. On the other hand, Figure 5 reveals that the AoI-optimal policy leads to a much higher estimation error, which results in worse performance in terms of UoI. The AoI-optimal policy is an oblivious policy, which is independent of the monitored process. Since AoI increases linearly with time, the AoI-optimal policy can only minimize the linear performance degradation in terms of time. However, the UoI-based policies (the cost function $\delta \left(Q\left(t\right)\right)={\left(Q\left(t\right)\right)}^2$) considered in this paper are process-dependent, which are called non-oblivious policies, and can benefit from both age and process realization [37]. These policies can directly minimize the nonlinear impact exerted by information staleness and the gap between the actual status and the estimated status.

Figure 5. Average squared estimation error of the UoI-optimal updating policy and the AoI-optimal updating policy when $p_1=0.001,p_2=0.01,{\sigma }^2=1,{\omega }_e=100$.

Thirdly, our updating policies outperform the update-index-based adaptive scheme [27] in terms of UoI. Under the adaptive scheme, the vehicle will derive an update index as a function of the current estimation error and the context-aware weight for the next slot. If the index is larger than the adaptive update threshold, then the vehicle is supposed to transmit its status information to the center. If the vehicle transmits an update in slot t, then the adaptive threshold will increase in the next slot; otherwise, the adaptive threshold will decrease. The adaptive scheme will cause an overuse of the resource in `urgent’ states and lead to the fact that the vehicles cannot receive resources in `normal’ states. However, the UoI-optimal policy and the trained RL-updating policy are fixed schemes, which can avoid the extremely unbalanced resource allocation between the two contexts and achieve better performance.

Figure 6 shows the influence of the maximum average update frequency $\rho$ and the context weight for emergency, ${\omega }_e$, on update threshold of UoI-optimal policy. In order to obtain more accurate results, the step size used here is 0.25. The solid curves show update thresholds for the normal state while the dashed curves show update thresholds for the urgent state. When the constraint on update resources is strict, the update thresholds fall faster. Furthermore, a larger ${\omega }_e$ results in a lower update threshold for the urgent state and a higher threshold for the normal state. This phenomenon indicates that the value of ${\omega }_e$ means the tolerance of estimation error in the emergency. When $\rho <0.1$, the influence of ${\omega }_e$ on the update threshold for the normal state is larger than the urgent state. For the cases where the maximum average update frequency is relatively large, ${\omega }_e$ has little effect on update thresholds for both normal state and urgent state.

Figure 6. Update thresholds of the UoI-optimal updating policy with different values of ${\omega }_e$ when $p_1=0.001,p_2=0.01,p_s=0.9,{\sigma }^2=1$.

Figure 7 shows that the update thresholds also depend on the dynamic of context-aware weight when the weight has first-order Markov property. When $p_2$ is approaching $1-p_1$, the gap between update thresholds for the urgent state and the normal state becomes smaller for the context-aware weight which tends to be i.i.d. over time. When $p_2=1$, the update threshold for the urgent state exceeds the threshold for the normal state. Therefore, the update threshold for the urgent state is not necessarily lower than the update threshold for the normal state.

Figure 7. Update thresholds of the UoI-optimal updating policy with different values of $p_2$ when $p_1=0.01,p_s=0.9,{\sigma }^2=1,{\omega }_e=100$.

Figure 8 shows the performance of the RL-based updating policy with different values of L. According to Algorithm 1, the number of state transitions observed in episode k is $\lceil L\sqrt{k}\rceil$. Therefore, L denotes the number of state transitions observed during the whole learning process. Generally speaking, a larger L reduces the randomness of the performance and achieves a better UoI. The performance of the RL-based updating policy depends on the accuracy of the model obtained through training, namely whether the estimated probability transfer function of the system is accurate. A larger L means that the algorithm can collect more data or state transitions and obtain a more accurate model.

Figure 8. Average UoI of the RL-based updating policy with different values of L when $p_1=0.001,p_2=0.01,{\sigma }^2=1,{\omega }_e=100$.

Figure 9 shows the influence of the number of episodes, i.e., K, on the performance of the RL-based updating policy. A larger K leads to a lower average UoI and smaller randomness over 100 runs. On the one hand, the more episodes and the more data the algorithm observes, the more accurate the model obtained will be and the better the performance of the updating policy will be. On the other hand, the value of K is the number of iterations for the policy obtained through the estimated CMDP. The policy ${\pi }_k\left(s\right)$ used in episode k is derived based on the state–action pairs and the state transitions observed in the previous $k-1$ episodes. Therefore, more frequent iterations of the updating policy can obtain more valuable state–action pairs and better performance.

Figure 9. Average UoI of the RL-based updating policy with different values of K when $p_1=0.001,p_2=0.01,{\sigma }^2=1,{\omega }_e=100$.

6. Conclusions

In this work, we studied how to minimize the performance degradation caused by outdated information in terms of UoI, which is a new metric jointly considering context and information freshness. We proved that the UoI-optimal updating policy for the considered single-user remote monitoring system has a single threshold structure. Then, the policy was obtained through linear programming by assuming that the state transition probability of the system is known in advance. In unknown contexts, we further used a reinforcement learning algorithm to learn the dynamics of the system. Simulations verified the threshold structure of the UoI-optimal policies and showed that the update thresholds decrease as the maximum average update frequency increases. In addition, a larger context-aware weight in emergencies resulted in a lower update threshold for urgent states. However, since the state transition probability also influenced the update thresholds, the update threshold for emergencies was not necessarily higher than the update threshold for normal states, especially when the probability of transferring from urgent states to normal states tended towards 1. Furthermore, the numerical results showed that the proposed RL-based updating policy achieved a near-optimal performance without advanced knowledge of the system model.

In fact, determining the context-aware weight in practical systems, where the models of the context are often very complicated and difficult to obtain in advance, remains open. As for future work, we plan to use deep RL algorithms to learn the models of the context variation. We believe that UoI can provide a new performance metric for information timeliness measurement in the future V2X scenario. In addition, we believe the proposed UoI metric and the context-aware scheduling policy can shed some light on low-latency and ultra-reliable wireless communication in the future 5G/6G systems.