Optimal Information Update for Energy Harvesting Sensor with Reliable Backup Energy

Abstract

The timely delivery of status information collected from sensors is critical in many real-time applications, e.g., monitoring and control. In this paper, we consider a scenario where a wireless sensor sends updates to the destination over an erasure channel with the supply of harvested energy and reliable backup energy. We adopt the metric age of information (AoI) to measure the timeliness of the received updates at the destination. We aim to find the optimal information updating policy that minimizes the time-average weighted sum of the AoI and the reliable backup energy cost. First, when all the environmental statistics are assumed to be known, the optimal information updating policy exists and is proved to have a threshold structure. Based on this special structure, an algorithm for efficiently computing the optimal policy is proposed. Then, for the unknown environment, a learning-based algorithm is employed to find a near-optimal policy. The simulation results verify the correctness of the theoretical derivation and the effectiveness of the proposed method.

1. Introduction

Timely information updates from wireless sensors to destinations are essential for real-time monitoring and control systems. To describe the timeliness of information updates from the receivers’ perspective, a new metric called age of information (AoI) is proposed [1,2,3]. Unlike general performance metrics, such as delay and throughput, AoI refers to the time elapsed since the generation of the latest received information. A lower AoI generally reflects more timely information at the destination. Therefore, the AoI-minimal status updating policies in sensor networks have been widely studied [4,5,6,7].

The destinations always desire information updates in as timely a manner as possible, which is typically constrained by sensors’ energy. Generally, energy sources include the grid and sensors’ own non-rechargeable batteries. We call these sources reliable energy since they enable sensors to reliably operate until the power grid is cut off or sensors’ batteries are exhausted [8]. Specifically, if sensors consume energy from the grid, they need to pay the electricity bill; if sensors only use the power of their own batteries, the price of sensing and transmitting updates will be the cost of frequent battery replacement. There is clearly a price to pay for using reliable energy to update. Energy harvesting (EH) is a promising technology that can help reduce the consumption of reliable energy for information update [9,10]. It can continuously extract energy from solar power, ambient RF, and thermal energy and store the harvested energy in sensors’ rechargeable batteries. The stored energy is renewable and can be used for free. Hence, in this case, the reliable energy can serve as backup energy. The design of the coexistence of reliable backup energy and harvested energy has been researched and promoted in academia and industry [8,11,12,13,14]. The mixed energy supply mode can enhance the reliability of the system.

However, the irregular arrivals of harvested energy and the limited capacity of rechargeable batteries still motivate us to schedule the energy usage properly to reduce the cost of using reliable backup energy while maintaining the timeliness of information updates (i.e., the average AoI). Intuitively, the average AoI and the cost of using reliable energy cannot be minimized simultaneously. On the one hand, a lower average AoI means that the sensor senses and transmits updates more frequently, which will increase the consumption of reliable backup energy since the harvested energy is limited. On the other hand, to reduce the cost of reliable backup energy, the sensor will only exploit the harvested energy. Due to the uncertainty of the energy harvesting behavior, the average AoI of the system will inevitably increase. Therefore, in this paper, we focus on achieving the best trade-off between the average AoI and the cost of reliable backup energy.

We consider a sensor-based information update system, where an energy harvesting sensor with reliable backup energy sends timely updates to the destination through an erasure channel. Based on our settings, we will minimize the long-term average weighted sum of the AoI and the paid reliable energy cost to find the optimal information updating policy by Markov decision process (MDP) theory [15]. First, we assume that the sensor knows the relevant statistics in advance, such as the success probability of each transmission and the probability of energy arrival, so that the sensor can make the optimal decision at any time. Then we consider a more realistic scenario where the sensor has no knowledge of the environment. In such an unknown environment, learning-based approaches should be adopted to obtain the updating policy.

1.1. Related Work

There have been a series of related works studying AoI minimization in EH communication systems [16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34]. In these systems, each update consumes harvested energy and is constrained by the energy causality.

Refs. [16,17,18,19,20,21,22,23] focus on how to optimize AoI under general energy causality constraints, where different battery model settings are considered. Constrained by the average power available in the infinite-sized battery, ref. [16] shows that a lazy policy which leaves a certain idle period between updates outperforms the greedy policy under random service times. With the same assumption of an infinite-sized battery, ref. [17] focuses on both offline and online policies under energy replenishment constraints with zero service time. While considering fixed service times, the offline results in [17] are extended to a two-hop scenario in [18], and online policy is provided in [19]. In the case of the delay being controlled by transmission energy, ref. [20] also investigated the optimal offline policy. For the error-free and delay-free channel, the optimal updating policies were investigated for different battery settings [21,22]. Ref. [21] derived the asymptotically optimal policies for the infinite-sized, finite-sized, and unit-sized battery by renewal theory. It turned out to be a threshold policy for the unit-sized battery case. More general battery models were considered in [22]. The optimal policy was also proved to be multi-threshold and the energy-dependent thresholds were characterized explicitly. When the battery is finite sized and there is no feedback from the destination, it was shown that the optimal updating policy is of a threshold structure and the threshold is non-increasing with the battery level [23].

Refs. [24,25,26,27,28,29,30] studied how to properly utilize the harvested energy to transmit updates over imperfect channels. For the noisy channel, ref. [24] considered an infinite-sized battery model and derived the different optimal policies for updating with and without feedback. Ref. [25] further derived a closed-form expression for the threshold of the unit-sized battery model and extended the threshold-based policies to multiple sources case. To combat the noisy channel, some channel coding schemes for EH communication were investigated in [26,27]. In [28], the HARQ protocol was applied for a single EH sensor to send updates to the destination. The optimal policies were obtained by employing reinforcement learning in both known and unknown environments, but no clear intuition on the policy structure was provided. Considering energy harvesting wireless sensor networks (EH-WSNs), ref. [29] suggested to estimate the channel state of a Rayleigh fading channel before transmitting to improve the AoI, update interval and packet loss performance, despite the associated time and energy costs. Ref. [30] aimed to minimize the average AoI of an EH-aided secondary user(SU) in a cognitive ratio network. The SU has to make sensing and updating decisions subject to random energy arrivals and the available spectrum. The sequential decision problem is formulated as a partially observable Markov decision process (POMDP).

Refs. [31,32,33,34] paid attention to other AoI-related metrics in EH communication and even the distributional properties of AoI, not just the average AoI. Different freshness metrics were considered, such as nonlinear AoI [31], urgency-aware AoI (U-AoI) [32], and peak AoI [33] in EH sensor network. To better understand the distributional properties of AoI, ref. [34] further derived closed-form expressions of the moment generating function (MGF) of AoI in an EH-powered queuing system using the stochastic hybrid systems (SHS) framework.

The above works focus on optimizing information freshness under the EH supply. Different from them, energy sources in this paper include both harvested energy and reliable backup energy, and our goal is to achieve the best trade-off between age and reliable energy consumption, instead of merely optimizing AoI. Among the above works, refs. [23,25] are the most related to our paper. The following

Table 1. Comparative summary of the most related works in contrast to our paper.

	Ref	[23]	[25]	Our
Feature
Energy supply		EH	EH	EH + reliable energy
Battery capacity		Finite-sized	Unit-sized	Finite-sized
Wireless channel		Error-free	Error-prone	Error-prone
Optimization objective		AoI	AoI	AoI-reliable energy trade-off

summarizes the detailed differences. It is worth noting that by letting the reliable energy consumption be small enough, our results can be compared with some prior results in [23,25].

The age–energy trade-off has been widely studied in [35,36,37,38,39]. The age–energy trade-off in the erasure channel was studied in [35], and the fading channel case was investigated in [36]. Ref. [37] adopted a truncated automatic repeat request (TARQ) scheme and characterized the age–energy trade-off for the IoT monitoring system. Optimum energy efficiency and AoI trade-off was considered in a multicast system in [38]. In [39], the authors investigated the optimal age–energy trade-off, where status sensing and data transmission can be carried out separately. By the MDP analysis similar to [6,15], the optimal policy exists and is proved to have two thresholds. The energy sources are all reliable in these works, which means that the energy cost of the update is easy to track. However, the uncertainty of the energy arrival and mixed energy supplies bring more challenges to the MDP analysis in this paper. To the best of our knowledge, this paper is the first to consider the timeliness of the system under mixed energy supplies. The preliminary results of this paper are presented in [40].

1.2. Main Contributions

The main contributions of this paper are as follows:

• We consider an information update system where the harvested energy and reliable energy coexist. The goal is to find the optimal policy that achieves the best trade-off between age and reliable energy consumption. Compared to the existing works [23,25], our problem is more practical and general, which will provide some insights for future green and durable update system designs.
• For the case that all the statistics such as channel erasure probability and EH probability are known a priori, we formulate an unconstrained infinite space Markov decision process (MDP) problem, and prove the existence of the optimal policy. By revealing the monotonicity and proportional differential propertyof the value function, we find that the optimal policy is of the threshold-type. Based on this special structure, we propose an efficient algorithm to compute the optimal policy.
• In an unknown environment, we propose an average cost Q-learning algorithm to obtain the updating policy.
• Simulation results show that the optimal policy outperforms other baseline policies when the environmental statistics are known. At the same time, the performance of the policy learned in the unknown environment is very close to the theoretical optimal policy. We also compare the age-reliable energy trade-off curves of the optimal updating policies under different energy supply conditions, which reflects the rationality of mixed energy supplies. The optimal policy can also be particularized to a special case, where the sensor can only utilize the harvested energy and the battery is unit-sized, and its performance coincides with the existing results in [23,25].

1.3. Organization

The rest of this paper is organized as follows. In Section 2, we introduce the model of the information update system and formulate the problem. In Section 3, we analyze the optimal policy when all the statistics are known. In Section 4, we aim to minimize the average cost of updating in an unknown environment. In Section 5, we present the simulation results. Finally, in Section 6, we conclude the paper.

2. System Model and Problem Formulation

2.1. System Model

In this paper, we consider a point-to-point information update system, where a wireless sensor and a destination are connected by an erasure channel, as shown in Figure 1. The channel is assumed to be noisy and time invariant, and each update is corrupted with probability p during transmission (Note $p\in (0,1)$). Both the free harvested energy stored in the rechargeable battery and the reliable backup energy that needs to be paid can be used for real-time environmental status updates.

Without loss of generality, time is slotted with equal length and indexed by $t=0,1,2\cdots$. At the beginning of each time slot, the sensor decides whether to generate and transmit an update to the destination or stay idle. The decision action at slot t, denoted by $a\left[t\right]$, takes value from action set $\mathcal{A}=\left\{0,1\right\}$, where $a\left[t\right]=1$ means that the sensor decides to generate and transmit an update to the destination while $a\left[t\right]=0$ means the sensor is idle. The destination will feed back an instantaneous ACK to the sensor through an error-free channel when it has successfully received an update and a NACK otherwise. We assume the above processes can be completed in one time slot. The destination keeps track of the environment status through the received updates. We apply the metric age of information to measure the freshness of the status information available at the destination.

2.1.1. Age of Information

Age of Information (AoI) is defined as the elapsed time since the generation of the latest successfully received update [1,2,3]. Denote $\Delta \left[t\right]$ as the AoI of destination in time slot t. Then, we have

$$\Delta \left[t\right]=t-U\left[t\right].$$

(1)

where $U\left[t\right]$ denotes the time slot when the most recently received update was generated before time slot t. In particular, the AoI will decrease to one if a new update is successfully received. Otherwise, it will increase by one. The evolution of AoI can be expressed as follows:

$$\Delta [t+1]=\left\{\begin{array}{cc}1,\hfill & \mathrm{successful}\mathrm{transmission},\hfill \\ \Delta \left[t\right]+1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(2)

A sample path of AoI is depicted in Figure 2.

2.1.2. Description of Energy Supply

We assume that only the sensor’s measurement and transmission process will consume energy and ignore other energy consumption. The energy unit is normalized, so the generation and transmission for each update will consume one energy unit. As previously described, the energy sources of the sensor include energy harvested from nature and reliable backup energy.

For the harvested energy, the sensor can store it in a rechargeable battery for later use. The maximum capacity of the rechargeable battery is B units ($B>1$). Considering the scarcity of energy in nature, the total energy harvested in one time slot may sometimes not reach an energy unit. So we consider using the Bernoulli process with the parameter $\lambda$ to approximately capture the arrival process of harvested energy, which was also adopted in [41,42,43]. Let $b\left[t\right]$ be the accumulated harvested energy in time slot t. That is, we have $Pr\left\{b\left[t\right]=1\right\}=\lambda$ and $Pr\left\{b\left[t\right]=0\right\}=1-\lambda$ in each time slot t (note $\lambda \in (0,1)$). Here, we assume that the energy arrival at each slot is independently and identically distributed. Time-correlated energy arrival processes, such as Markov process, will be considered in future work.

For reliable backup energy, we assume that it contains much more energy units than the rechargeable battery, so the energy it contains can be viewed as infinite. However, it needs to be used for a fee compared to the free renewable energy stored in the rechargeable battery. Therefore, when the stored renewable energy is not zero, the sensor will prioritize using it for status updates; otherwise, it will automatically switch to the reliable backup energy until the sensor has harvested energy. Defining the power of the rechargeable battery at the beginning of time slot t as the battery state $q\left[t\right]$, then the evolution of battery state from time slot t to $t+1$ can be summarized as follows:

$$q[t+1]=min\left\{q\right[t]+b[t]-a[t\left]u\right(q\left[t\right]),B\},$$

(3)

where $u(·)$ is unit step function, which is defined as

$$u\left(x\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}x>0,\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

Suppose that under reliable energy supply, the cost of generating and transmitting an update is a non-negative value $C_r$. Defining $E\left[t\right]$ as the paid reliable energy cost at the time slot t, then we have

$$E\left[t\right]=C_{r}a\left[t\right](1-u\left(q\left[t\right]\right)).$$

(5)

2.2. Problem Formulation

Let $\mathsf{\Pi }$ denote the set of non-anticipative policies in which scheduling decision $a\left[t\right]$ are made based on the action history ${\left\{a\left[k\right]\right\}}_{k=0}^{t-1}$, the evolution of AoI ${\left\{\Delta \left[k\right]\right\}}_{k=0}^t$, the evolution of battery state ${\left\{q\left[k\right]\right\}}_{k=0}^t$ as well as the system parameters (e.g., p and $\lambda$). In order to keep the information freshness at the destination, the sensor needs to send updates. However, due to the randomness of energy arrivals, the battery energy may sometimes be insufficient to support updates, and the sensor has to take energy from reliable backup energy. To balance the information freshness and the paid reliable backup energy cost, we aim to find the optimal information updating policy $\pi \in \mathsf{\Pi }$ that achieves the minimum of the time-average weighted sum of the AoI and the paid reliable backup energy cost. The problem is formulated as follows:

$$\begin{array}{cc}\hfill & \underset{\pi \in \mathsf{\Pi }}{min}\underset{T\to \infty }{limsup}\frac{1}{T}{\mathbb{E}}_{\pi }\left\{\sum _{t=0}^{T-1}[\Delta \left[t\right]+\omega E\left[t\right]]\right\},\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.\left(2\right),\left(3\right),\left(5\right),\hfill \end{array}$$

(6)

where $\omega$ is the pre-defined non-negative weighting factor. If $\omega$ = 0, the optimal policy is to update in each time slot, i.e., zero-wait policy [4]. Since the effect of energy can be ignored, if the rechargeable battery is not empty, the sensor uses the renewable energy; otherwise, the sensor will use the reliable energy directly. When $\omega >0$, the optimal policy is non-trivial. So we will focus on the optimal policy for $\omega >0$ in the rest of the paper. The smaller $\omega$ is, the more we attach importance to the system AoI; otherwise, the more emphasis is placed on the cost of reliable energy.

Remark 1.

The optimal trade-off between age and reliable energy consumption can also be formulated as a constrained problem, where the reliable energy consumption serves as a constraint (not exceeding $E_m$) but not a penalty, and the goal is to minimize the long-term average age. By the Lagrangian method, it can be converted into an unconstrained weighted sum problem, where the Lagrangian multiplier is exactly the weight factor ω. So the solution proposed in this paper can be used. If there exists an ω such that the average reliable energy consumption in the minimum weighted sum is $E_m$, the optimal policy of the weighted sum problem also minimizes the long-term average age with the $E_m$ constraint. Otherwise, a randomized optimal policy for the constrained problem needs to be considered; see details in [44].

3. Optimal Policy Analysis in A Known Environment

In this section, we aim to solve the problem (6) in a known environment and obtain the optimal policy. It is difficult to solve the original problem directly due to the random erasures and the temporal dependency in both AoI and battery state evolution. However, since the statistics such as channel erasure probability and EH probability are known, we can reformulate the original problem as a time-average cost MDP with infinite state space and analyze the structure of the optimal policy.

3.1. Markov Decision Process Formulation

According to the system description mentioned in the previous section, the MDP is formulated as follows:

• State space. The sensor’s state $\mathbf{x}\left[t\right]$ in slot t is a couple of the current destination AoI and the battery state, i.e., $(\Delta [t],q[t\left]\right)$. Define $\mathcal{B}=\left\{0,1,\dots ,B\right\}$. The state space $\mathcal{S}={\mathbb{Z}}^{+}\times \mathcal{B}$ is thus infinite countable.
• Action space. The sensor’s action $a\left[t\right]$ in time slot t takes value from the action set $\mathcal{A}=\left\{0,1\right\}$.
• Transition probabilities. Denote $Pr\left(\mathbf{x}\right[t+1\left]\right|\mathbf{x}\left[t\right],a\left[t\right])$ as the transition probability that current state $\mathbf{x}\left[t\right]$ transits to next state $\mathbf{x}[t+1]$ after taking action $a\left[t\right]$. Suppose the current state $\mathbf{x}\left[t\right]=(\Delta ,q)$ and action $a\left[t\right]=a$, then the transition probability is divided into two following cases conditioned on different values of action.
  Case 1. $a=0$,

  $$\left\{\begin{array}{cc}Pr\left\{\right(\Delta +1,q+1\left)\right|(\Delta ,q),0\}=\lambda ,\hfill & \mathrm{if}q<B,\hfill \\ Pr\left\{\right(\Delta +1,B\left)\right|(\Delta ,B),0\}=1,\hfill & \mathrm{if}q=B,\hfill \\ Pr\left\{\right(\Delta +1,q\left)\right|(\Delta ,q),0\}=1-\lambda ,\hfill & \mathrm{if}q<B.\hfill \end{array}\right.$$

  (7)
  Case 2. $a=1$,

  $$\left\{\begin{array}{cc}Pr\left\{\right(\Delta +1,q\left)\right|(\Delta ,q),1\}=p\lambda ,\hfill & \mathrm{if}q>0,\hfill \\ Pr\left\{\right(1,q\left)\right|(\Delta ,q),1\}=(1-p)\lambda ,\hfill & \mathrm{if}q>0,\hfill \\ Pr\{\Delta +1,q-1)\left|\right(\Delta ,q),1\}=p(1-\lambda ),\hfill & \mathrm{if}q>0,\hfill \\ Pr\left\{\right(1,q-1\left)\right|(\Delta ,q),1\}=(1-p\left)\right(1-\lambda ),\hfill & \mathrm{if}q>0,\hfill \\ Pr\left\{\right(\Delta +1,1\left)\right|(\Delta ,0),1\}=p\lambda ,\hfill & \mathrm{if}q=0,\hfill \\ Pr\left\{\right(1,1\left)\right|(\Delta ,0),1\}=(1-p)\lambda ,\hfill & \mathrm{if}q=0,\hfill \\ Pr\left\{\right(\Delta +1,0\left)\right|(\Delta ,0),0\}=p(1-\lambda ),\hfill & \mathrm{if}q=0,\hfill \\ Pr\left\{\right(1,0\left)\right|(\Delta ,0),0\}=(1-p\left)\right(1-\lambda ),\hfill & \mathrm{if}q=0.\hfill \end{array}\right.$$

  (8)
  In both cases, the evolution of AoI still follows Equation (2) and the evolution of battery state follows Equation (3).
• One-step cost. For the current state $\mathbf{x}=(\Delta ,q)$, the one-step cost $C(\mathbf{x},a)$ of taking action a is expressed by

  $$C(\mathbf{x},a)=\Delta +\omega C_{r}a(1-u\left(q\right)).$$

  (9)

After the above modeling, the original problem (6) is transformed into obtaining the optimal policy for the MDP to minimize the average cost in an infinite horizon:

$$\underset{\pi \in \mathsf{\Pi }}{min}\underset{T\to \infty }{limsup}\frac{1}{T}{\mathbb{E}}_{\pi }\left\{\sum _{t=0}^{T-1}C(\mathbf{x}\left[t\right],a\left[t\right])\right\}.$$

(10)

Denote ${\mathsf{\Pi }}_{SD}$ as the set of stationary deterministic policies. Given observation $(\Delta [t],q[t\left]\right)=(\Delta ,q)$, the policy $\pi \in {\mathsf{\Pi }}_{SD}$ selects action $a\left[t\right]=\pi (\Delta ,q)$, where $\pi (·):(\Delta ,q)\to \left\{0,1\right\}$ is a deterministic function from state space $\mathcal{S}$ to action space $\mathcal{A}$. In the next section, we prove that there is an optimal stationary deterministic policy for the above unconstrained MDP with infinite countable state and action space.

3.2. The Existence of the Optimal Stationary Deterministic Policy

According to [15], we need to first address a discounted cost MDP, then relate it to the original average cost problem. Given an initial state $\mathbf{x}\left[0\right]=\widehat{\mathbf{x}}$, the total expected discounted cost under a policy $\pi$ is given by

$$V_{\gamma }^{\pi }\left(\widehat{\mathbf{x}}\right)=\underset{T\to \infty }{limsup}{\mathbb{E}}_{\pi }\left\{\sum _{t=0}^{T-1}{\gamma }^{t}C(\mathbf{x}\left[t\right],a\left[t\right])|\mathbf{x}\left[0\right]=\widehat{\mathbf{x}}\right\},$$

(11)

where the discounted factor is $\gamma \in (0,1)$. Therefore, the problem of minimizing the expected discounted cost can be formulated as

$$V_{\gamma }\left(\widehat{\mathbf{x}}\right)\triangleq \underset{\pi \in \mathsf{\Pi }}{min}{V}_{\gamma }^{\pi }\left(\widehat{\mathbf{x}}\right),$$

(12)

where value function $V_{\gamma }\left(\widehat{\mathbf{x}}\right)$ denotes the minimum expected discounted cost. The policy is $\gamma$-optimal if it minimizes the above discounted cost. The optimality equation of $V_{\gamma }\left(\widehat{\mathbf{x}}\right)$ is introduced in Proposition 1.

Proposition 1.

 (a) 

The optimal expected discounted cost $V_{\gamma }\left(\widehat{\mathit{x}}\right)$ satisfies the Bellman equation as follows:

$$V_{\gamma }\left(\widehat{\mathit{x}}\right)=\underset{a\in \mathcal{A}}{min}{Q}_{\gamma }(\widehat{\mathit{x}},a),$$

(13)

where the state–action value function $Q_{\gamma }(\widehat{\mathit{x}},a)$ is defined as

$$Q_{\gamma }(\widehat{\mathit{x}},a)=C(\widehat{\mathit{x}},a)+\gamma \sum _{{\mathit{x}}^{\prime }\in \mathcal{S}}Pr\left({\mathit{x}}^{\prime }\right|\widehat{\mathit{x}},a)V_{\gamma }\left({\mathit{x}}^{\prime }\right).$$

(14)

 (b) 

The policy π determined by the right hand side of  (13) is γ-optimal, and $\pi \in {\mathsf{\Pi }}_{SD}$.

 (c) 

$V_{\gamma }\left(\widehat{\mathit{x}}\right)$ can be solved by value iteration algorithm. Specifically, let $V_{\gamma ,n}\left(\widehat{\mathit{x}}\right)$ be the cost-to-go function and $V_{\gamma ,0}\left(\widehat{\mathit{x}}\right)=0$ for all state $\widehat{\mathit{x}}\in \mathcal{S}$. For all $n\ge 1$, we have:

$$V_{\gamma ,n}\left(\widehat{\mathit{x}}\right)=\underset{a\in \mathcal{A}}{min}{Q}_{\gamma ,n}(\widehat{\mathit{x}},a),$$

(15)

where $Q_{\gamma ,n}(\widehat{\mathit{x}},a)$ is obtained as follows:

$$Q_{\gamma ,n}(\widehat{\mathit{x}},a)=C(\widehat{\mathit{x}},a)+\gamma \sum _{{\mathit{x}}^{\prime }\in \mathcal{S}}Pr\left({\mathit{x}}^{\prime }\right|\widehat{\mathit{x}},a)V_{\gamma ,n-1}\left({\mathit{x}}^{\prime }\right).$$

(16)

Then the equation $\underset{n\to \infty }{lim}{V}_{\gamma ,n}\left(\widehat{\mathit{x}}\right)=V_{\gamma }\left(\widehat{\mathit{x}}\right)$ holds for every state $\widehat{\mathit{x}}$ and γ.

Proof. 

See Appendix A.    □

Now, we can show the monotonic properties of $V_{\gamma }\left(\widehat{\mathbf{x}}\right)$ in the following lemma by using (c) in Proposition 1.

Lemma 1.

Given fixed channel erasure probability p and EH probability λ, then

 (a) 

value function $V_{\gamma }(\Delta ,q)$ is non-decreasing in Δ, i.e., for any $1\le {\Delta }_1\le {\Delta }_2$ and any battery state $q\in \mathcal{B}$, we have

$$V_{\gamma }({\Delta }_1,q)\le V_{\gamma }({\Delta }_2,q),$$

(17)

and

$$V_{\gamma }({\Delta }_2,q)-V_{\gamma }({\Delta }_1,q)\ge {\Delta }_2-{\Delta }_1.$$

(18)

 (b) 

value function $V_{\gamma }(\Delta ,q)$ is non-increasing in q, i.e., for AoI $\Delta \ge 1$ and any battery state $q\in \left\{0,1,\cdots ,B-1\right\}$, we have

$$V_{\gamma }(\Delta ,q)\ge V_{\gamma }(\Delta ,q+1),$$

(19)

Proof. 

See Appendix B.    □

Based on the Proposition 1 and Lemma 1, we will verify the existence of the optimal stationary deterministic policy for the average cost problem (10) in the following theorem.

Theorem 1.

There exists an optimal policy ${\pi }^{☆}\in {\mathsf{\Pi }}_{SD}$ for the average cost MDP in (10). Moreover, for every state $\mathit{x}$, there exists a value function $V(·):\mathcal{S}\to \mathbb{R}$ and a unique constant $g^{☆}\in \mathbb{R}$ such that:

$$g^{☆}+V\left(\mathit{x}\right)=\underset{a\in \mathcal{A}}{min}\left\{C(\mathit{x},a)+\sum _{{\mathit{x}}^{\prime }\in \mathcal{S}}Pr\left({\mathit{x}}^{\prime }\right|\mathit{x},a)V\left({\mathit{x}}^{\prime }\right)\right\},$$

(20)

where $g^{☆}$ is the optimal average cost of problem (10) and satisfies $g^{☆}=\underset{\gamma \to 1}{lim}(1-\gamma )V_{\gamma }\left(\mathit{x}\right)$ for every state $\mathbf{x}$, and the value function $V\left(\mathit{x}\right)$ satisfies

$$V\left(\mathit{x}\right)=\underset{\gamma \to 1}{lim}\gamma V_{\gamma }\left(\mathit{x}\right)=\underset{\gamma \to 1}{lim}{V}_{\gamma }\left(\mathit{x}\right)-g^{☆}=\underset{T\to \infty }{limsup}\frac{1}{T}{\mathbb{E}}_{\pi }\left\{\sum _{t=0}^{T-1}[C(\mathit{x}\left[t\right],a\left[t\right])-g^{☆}]\right\}.$$

(21)

Proof. 

See Appendix C.    □

Based on Theorem 1, we have the following corollary:

Corollary 1.

The state–action value function $Q(\mathit{x},a)$ for the average cost is given as follows:

$$Q(\mathit{x},a)=C(\mathit{x},a)+\sum _{{\mathit{x}}^{\prime }\in \mathcal{S}}Pr\left({\mathit{x}}^{\prime }\right|\mathit{x},a)V\left({\mathit{x}}^{\prime }\right),$$

(22)

which is similar to $Q_{\gamma }(\mathit{x},a)$ in (14) by letting $\gamma \to 1$. Then the optimal policy ${\pi }^{☆}\in {\mathsf{\Pi }}_{SD}$ for the average cost MDP in (10) can be expressed as follows:

$${\pi }^{☆}\left(\mathit{x}\right)=arg\underset{a\in \mathcal{A}}{min}Q(\mathbf{x},a),\forall \mathit{x}\in \mathcal{S}.$$

(23)

3.3. Structure Analysis of Optimal Policy

Before analyzing the structure of the optimal policy ${\pi }^{☆}$, we first prove some monotonic properties of the value function $V\left(\mathbf{x}\right)$ on different dimensions, which is summarized in the following lemma.

Lemma 2.

Given fixed channel erasure probability p and EH probability λ, then

 (a) 

value function $V(\Delta ,q)$ isnon-decreasingin Δ, i.e., for any $1\le {\Delta }_1\le {\Delta }_2$ and any battery state $q\in \mathcal{B}$, we have

$$V({\Delta }_1,q)\le V({\Delta }_2,q),$$

(24)

and

$$V({\Delta }_2,q)-V({\Delta }_1,q)\ge {\Delta }_2-{\Delta }_1.$$

(25)

 (b) 

value function $V(\Delta ,q)$ is non-increasing in q, i.e., for AoI $\Delta \ge 1$ and any battery state $q\in \left\{0,1,\cdots ,B-1\right\}$, we have

$$V(\Delta ,q)\ge V(\Delta ,q+1),$$

(26)

Proof. 

According to the (21), $V\left(x\right)=\underset{\gamma \to 1}{lim}{V}_{\gamma }\left(\mathbf{x}\right)-g$. Therefore, the monotonic properties of $V_{\gamma }\left(\mathbf{x}\right)$ in Lemma 1 are also valid for $V\left(x\right)$, which completes the proof.    □

Based on Lemma 2, we will derive the proportional differential property of the value function in Lemma 3.

Lemma 3.

Given fixed channel erasure probability p and EH probability λ, then value function $V(\Delta ,q)$ has the proportional differential property, i.e., the inequality

$$\frac{V(\Delta +1,q+1)-V(\Delta ,q+1)}{V(\Delta +1,q)-V(\Delta ,q)}\ge p$$

(27)

holds for AoI $\Delta \ge 1$ and any battery state $q\in \left\{0,1,\dots ,B-1\right\}$.

Proof. 

See Appendix D.    □

With Corollary 1, Lemmas 2 and 3, we directly provide our main result in the following theorem.

Theorem 2.

Assuming that the channel erasure probability p and EH probability λ are both fixed, there exists a threshold ${\Delta }_q\in {\mathbb{Z}}^{+}$ for given battery state q, such that when $\Delta <{\Delta }_q$, the optimal action ${\pi }^{☆}(\Delta ,q)=0$, i.e., the sensor keeps idle; when $\Delta \ge {\Delta }_q$, the optimal action ${\pi }^{☆}(\Delta ,q)=1$, i.e., the sensor chooses to generate and transmit a new update.

Proof. 

See Appendix E.    □

Theorem 2 reveals the threshold structure of the optimal policy: if the optimal action in a certain state is to generate and transmit an update, then in the state with the same battery state and larger AoI, the optimal action must be the same. Note that the threshold ${\Delta }_q$ is actually determined by the channel erasure probability p, EH probability $\lambda$ and pre-defined weighting factor $\omega$. The closed-form expression of the threshold is difficult to be derived due to the complex transition probabilities. In the next section, we will show how to compute the optimal policy numerically.

3.4. Modified Relative Value Iteration Algorithm Design

In this section, we will propose a computationally efficient algorithm to find the optimal stationary deterministic policy based on the threshold structure.

Since the state space $\mathcal{S}$ is infinite, we will use a truncated space ${\mathcal{S}}^N$ for approximation in practice, where ${\mathcal{S}}^N=\left\{(\Delta ,q)|\Delta \le N,\Delta \in {\mathbb{Z}}^{+},q\in \mathcal{B}\right\}$. It can be proved that when N is large enough, the optimal policy of the approximated MDP will be identical to that of the original problem [6].

However, the value iteration algorithm in Proposition 1 for the discounted cost problem cannot be applied to the average cost problem by letting $\gamma =1$. It does not converge because the value function $V(·)$ in (20) is not unique. One can check if $V(·)$ satisfies (20), a new function $V^{\prime }(·)=V(·)+c$ also satisfies (20), where $c\in \mathbb{R}$. Therefore, we introduce a relative value iterative (RVI) algorithm to obtain the optimal policy of the approximate average cost MDP [45]. We choose a reference state $\widehat{x}\in {\mathcal{S}}^N$ and set $V_0\left(\mathbf{x}\right)=0$ for all states $\mathbf{x}\in {\mathcal{S}}^N$ Then for all $n\ge 0$, we have

$$V_{n+1}\left(\mathbf{x}\right)=\underset{a\in \mathcal{A}}{min}{Q}_{n+1}(\mathbf{x},a),$$

(28)

and $Q_{n+1}(\mathbf{x},a)$ is obtained as follows:

$$Q_{n+1}(\mathbf{x},a)=C(\mathbf{x},a)+\sum _{{\mathbf{x}}^{\prime }\in {\mathcal{S}}^N}Pr\left({\mathbf{x}}^{\prime }\right|\mathbf{x},a)h_n\left({\mathbf{x}}^{\prime }\right),$$

(29)

where the differential value function is $h_n\left(\mathbf{x}\right)=V_n\left(\mathbf{x}\right)-V_n\left(\widehat{\mathbf{x}}\right)$. The equation $\underset{n\to \infty }{lim}{Q}_n(\mathbf{x},a)=Q(\mathbf{x},a)$ holds for every state $\mathbf{x}\in {\mathcal{S}}^N$ and action $a\in \mathcal{A}$. Finally, we compute the optimal policy by

$${\pi }^{☆}\left(\mathbf{x}\right)=arg\underset{a\in \mathcal{A}}{min}Q(\mathbf{x},a).$$

(30)

Note that the optimal policy is still of a threshold structure. The corresponding proof is similar to that of Theorem 2.

Moreover, based on the RVI algorithm, we can exploit this threshold structure to reduce the computational complexity. When the optimal policy of a state ${\mathbf{x}}^{\prime }=({\Delta }^{\prime },q^{\prime })$ is 1, the optimal policy of state ${\mathbf{x}}^{\prime }\in \left\{(\Delta ,q)|\Delta >{\Delta }^{\prime },\Delta \le N,q=q^{\prime }\right\}$ will also be 1 without the need to calculate (30). Therefore, we propose a modified RVI algorithm, and the details are given in Algorithm 1.

Algorithm 1 Modified relative value iteration algorithm.

Input:

Iteration number K,

Iteration threshold $ϵ$,

Maximum of AoI N,

Maximum of battery state B,

Reference state $\widehat{\mathbf{x}}$.

Output:

Optimal policy ${\pi }^{☆}\left(\mathbf{x}\right)$ for all state $\mathbf{x}$.

1: Initialization: $h_0\left(\mathbf{x}\right)=0$, for all $\mathbf{x}\in {\mathcal{S}}^N$

2: for episodes $n=0,1,2,\cdots ,K$ do

3:  for state $\mathbf{x}\in {\mathcal{S}}^N$ do

4:   for action $a\in \mathcal{A}$ do

5:    $Q_n(\mathbf{x},a)\leftarrow C(\mathbf{x},a)+{\displaystyle \sum _{{\mathbf{x}}^{\prime }\in {\mathcal{S}}^N}}Pr\left({\mathbf{x}}^{\prime }\right|\mathbf{x},a)h_n\left({\mathbf{x}}^{\prime }\right)$// Update the state-action value function.

6:    end for

7:    $V_{n+1}\left(\mathbf{x}\right)\leftarrow \underset{a\in \mathcal{A}}{min}{Q}_n(\mathbf{x},a)$// Update the value function.

8:    $h_{n+1}\left(\mathbf{x}\right)\leftarrow V_{n+1}\left(\mathbf{x}\right)-V_{n+1}\left(\widehat{\mathbf{x}}\right)$// Update the differential value function.

9:   end for

10:   if $\parallel h_{n+1}\left(\mathbf{x}\right)-h_n\left(\mathbf{x}\right)\parallel \le ϵ,\forall \mathbf{x}\in {\mathcal{S}}^N$ then

11:    for $\mathbf{x}=(\Delta ,q)\in {\mathcal{S}}^N$ do

12:     if ${\pi }^{☆}(\Delta -1,q)=1$ then

13:     ${\pi }^{☆}\left(\mathbf{x}\right)\leftarrow 1$, // Leverage the threshold structure of the optimal policy.

14:     else

15:      ${\pi }^{☆}\left(\mathbf{x}\right)\leftarrow arg\underset{a\in \mathcal{A}}{min}{Q}_n(\mathbf{x},a)$

16:     end if

17:    end for

18:    break

19:   end if

20:  end for

4. Minimize Average Cost in an Unknown Environment

In the previous sections, we assumed that the channel erasure probability p and EH probability $\lambda$ are known in advance. Thus, the model-based RVI method can be employed to obtain the optimal updating policy. However, statistics such as p and $\lambda$ may be unknown and even time variant in many practical scenarios, which makes it impossible to apply modified RVI algorithm because the transition probabilities are not explicit and Equation (29) cannot be applied to estimate the state-action value function $Q(\mathbf{x},a)$. In the field of reinforcement learning, alternatively, model-free methods can solve MDP problems with unknown transition probabilities. An example of a model-free algorithm is Q-learning [46]. Q-learning finds an optimal policy in the sense of maximizing the expected value of the total reward over any and all successive steps. However, it is only designed for discounted MDP. For the average cost problem in (10), we employ an average cost Q-learning algorithm. The basic idea of this algorithm comes from the SMART algorithm in [47], which is a model-free reinforcement learning algorithm proposed for semi-Markov decision problems (SMDP) under the average-reward criterion. We modify it to fit the average cost MDP problem.

The state–action value function $Q(\mathbf{x},a)$ is essential for solving the optimal policy. When the model is unknown, as long as $Q(\mathbf{x},a)$ can be estimated accurately, the optimal policy can also be obtained immediately by (30). So the key question is how to estimate the $Q(\mathbf{x},a)$ function, or equivalently, the value of all state–action pairs. Similar to Q-learning, the average cost Q-learning algorithm uses the minimum value of the next state–action pairs to update the value of the current state–action pair. Moreover, it needs to estimate the shift value g by averaging all immediate cost.

Specifically, the average cost Q-learning algorithm learns $Q(\mathbf{x},a)$ by episodes. Each episode contains several iterations, and each iteration corresponds to one time slot. Then in the nth time slot of an episode, the algorithm first observes the current state $\mathbf{x}\left[n\right]=(\Delta [n],q[n\left]\right)$, selects an action $a\left[n\right]$ according to the $ϵ$-greedy policy:

$$a\left[n\right]=\left\{\begin{array}{cc}arg\underset{a\in \mathcal{A}}{min}Q(\mathbf{x}\left[n\right],a),\hfill & \mathrm{with}\mathrm{probability}1-ϵ,\hfill \\ \mathrm{random}\mathrm{action},\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(31)

By (9), the immediate cost $C\left[n\right]=\Delta \left[n\right]+\omega C_{r}a\left[n\right](1-u\left(q\left[n\right]\right))$ occurs, and the system will transit to the next state $\mathbf{x}[n+1]$. The value of $Q\left(\mathbf{x}\right[n],a[n\left]\right)$ is updated as follows:

$$Q(\mathbf{x}\left[n\right],a\left[n\right])=(1-\alpha \left[n\right])Q(\mathbf{x}\left[n\right],a\left[n\right])+\alpha \left[n\right](C\left[n\right]-g+\underset{a\in \mathcal{A}}{min}Q(\mathbf{x}[n+1],a)),$$

(32)

where $\alpha \left[n\right]$ is the learning rate. The shift value g is updated as follows:

$$g=(1-\beta [n\left]\right)g+\beta \left[n\right]C\left[n\right]$$

(33)

where $\beta \left[n\right]=\frac{1}{n}$. The details are given in Algorithm 2. We leverage the parameter $ϵ$ to balance exploration and exploitation. As the number of epochs increases, the learned $Q(\mathbf{x},a)$ value will approach its true value, so we can gradually decrease $ϵ$ to 0 to reduce invalid exploration. At the same time, the shift value g will also be close to the optimal average cost $g^{☆}$ in (20). Note that in [47], the shift value g is updated only in a non-exploratory time slot. Here we update it by simply averaging all cost, similar to [48]. The performance comparison of the average cost Q-learing algorithm and modified RVI algorithm is shown in the next section.

Algorithm 2 Average cost Q-learning algorithm.

Input:

Maximum number of episodes K,

Maximum iteration number of an episode $N_e$,

Maximum of AoI N,

Maximum of battery state B,

Initial value of $Q^{N\times B\times 2}\leftarrow \mathbf{0}$,

Initial value of $ϵ\leftarrow 0$,

Initial value of the shift value $g\leftarrow 0$.

Output:

Learned policy $\pi \left(\mathbf{x}\right)$ for all state $\mathbf{x}$,

Average cost $g^{☆}$ by following the policy $\pi$.

1: for episodes $k=0,1,2,\cdots ,K$ do

2:  $g\leftarrow 0$ // Initialize the shift value at the beginning of every episode.

3:  for $n=1,2,\cdots ,N_e$ do

4:   Observe the current state $\mathbf{x}\left[n\right]$

5:   Select an action $a\left[n\right]$ according to $ϵ$-greedy policy in (31)

6:   Calculate immediate cost $C\left[n\right]\leftarrow \Delta \left[n\right]+\omega C_{r}a\left[n\right](1-u\left(q\left[n\right]\right))$

7:   Observe the next state $\mathbf{x}[n+1]$

8:   $\alpha \left[n\right]\leftarrow \frac{1}{\sqrt{n}}$

9:   $Q(\mathbf{x}\left[n\right],a\left[n\right])\leftarrow (1-\alpha \left[n\right])Q(\mathbf{x}\left[n\right],a\left[n\right])+\alpha \left[n\right](C\left[n\right]-g+\underset{a\in \mathcal{A}}{min}Q(\mathbf{x}[n+1],a))$ // Update the state-action value function.

10:   $\beta \left[n\right]\leftarrow \frac{1}{n}$

11:   $g\leftarrow (1-\beta [n\left]\right)g+\beta \left[n\right]C\left[n\right]$ // Update the shift value.

12:  end for

13:  Decrease $ϵ$

14: end for

15: for$\mathbf{x}=(\Delta ,q)\in {\mathcal{S}}^N$do

16:  $\pi \left(\mathbf{x}\right)\leftarrow arg\underset{a\in \mathcal{A}}{min}Q(\mathbf{x},a)$// Calculate the learned policy $\pi$.

17: end for

5. Numerical Results

In this section, we first show the threshold structure of the optimal policy by the simulation results. Then we compare the performance of the optimal policy with the following representative policies under different system parameters:

• Zero-wait policy [4]. The sensor generates and transmits an update in every time slot.
• Periodic policy. The sensor periodically generates and sends updates to the destination.
• Randomized policy. The sensor chooses to send an update or stay idle in each time slot with the same probability.
• Energy first policy. The sensor only uses the harvested energy, that is, as long as the battery state is not zero, it will choose to sense and send updates, otherwise it will remain idle.

Moreover, we will show the average cost Q-learning algorithm performs very close to the modified RVI with known statistics. We will also compare age and reliable energy cost trade-off curves of the optimal updating policies under EH supply, reliable energy supply and mixed energy supplies. Finally, we compare the performance of the optimal policy under the only EH supply and unit-sized battery setting with the prior results in [23,25].

5.1. Simulation Setup

In our simulations, we set the maximum of AoI $N=500$, and the maximum of battery state $B=20$. So the finite state space ${\mathcal{S}}^N=\left\{(\Delta ,q)|\Delta \le 500,\Delta \in {\mathbb{Z}}^{+},q\in \mathcal{B}\right\}$. The cost of reliable energy $C_r$ for one update is equal to 2. For the modified RVI algorithm, we set the iteration number $K=1000$, iteration threshold $ϵ={10}^{-5}$ and reference state $\widehat{\mathbf{x}}=(1,B)$. The optimal policy and other baseline policies are run for $T=$ 10,000 time slots to compute the average cost. For the average cost Q-learning algorithm, we set the total number of episodes $K=1000$, and the maximum iteration number in an episode $N_e=$ 100,000.

5.2. Results

Figure 3 shows the optimal policy under different system parameters. All the subfigures in Figure 3 exhibit the threshold structure described in Theorem 2. Intuitively, when $\omega$ is very small, the optimal action for every state should be 1, and when $\omega$ is very large, the optimal action for battery state $q=0$ should be 0. It can be observed from Figure 3a,b that when $\omega$ is small (i.e., $\omega =0.1$), the optimal policy is to update for every state, which is exactly the zero-wait policy. Figure 3 also shows that when $\omega$ is relatively large (e.g., $\omega =10$), and the AoI is small, even if the battery state is not zero, the optimal action in the corresponding state is to keep idle. When the AoI is large or the battery state is large, the optimal action is to measure and send updates. Moreover, in all the subfigures, the threshold ${\Delta }_q$ keeps monotonically non-increasing with the battery state q. However, this conclusion has not been rigorously proven.

Figure 4 shows the time average cost with respect to $\omega$ under different policies. Here, we set the period of the periodic policy to 5 and 10 for comparison without loss of generality. It can be found that under different weighting factor $\omega$, the optimal policy proposed in this paper can obtain the minimum long-term average cost compared with the other policies, which indicates the best trade-off between the average AoI and the cost of reliable energy. When $\omega$ tends to 0, the zero-wait policy tends to be optimal. Since there is no need to consider the update cost brought by paid reliable backup energy, the optimal policy should maximize the utilization of the updating opportunities.

It can also be observed from Figure 4 that the growth of the optimal policy curve slows down as $\omega$ increases. This is because the optimal policy in the case of large $\omega$ does not tend to use the reliable energy when battery state $q=0$, but prefers to wait for harvested energy, as shown in Figure 3. Since the EH probability is constant, the average AoI does not change much, resulting in no significant increase in the total average cost. Comparing Figure 4a,b, it is found that the larger the $\lambda$, the smaller the average cost variation with $\omega$. This is because there is not much opportunity for the sensor to use reliable energy in the case of sufficient harvested energy.

Figure 5 reveals the impact of EH probabilities $\lambda$. In Figure 5a,b, we set $p=0.2$, $\omega =10$ and $p=0.2,\omega =1$, respectively.

It can also be found from both Figure 5a,b that the proposed optimal update policy outperforms all other policies under different EH probability. The interesting point is that when the EH probability tends to 1, i.e., energy arrives in each time slot, the performance of the zero-wait policy and the energy first policy is equal to the optimal policy, while there is still a performance gap between the optimal policy and the other two polices. This is intuitive because when the free harvested energy is sufficient, the optimal policy must be to generate and transmit updates in every time slot. However, the periodic policy and the randomized policy still keep idle in many time slots, which will lead to a higher average AoI and thus increase the average cost. Results show that the performance of zero-wait policy approaches the optimal policy for large $\lambda$, which is consistent with our findings in Figure 4.

In Figure 6, we compare the five policies under different channel erasure probability p.

It can be found that when the erasure probability increases from 0 to 0.9, the proposed optimal update policy always performs better than the other baseline policies. As p tends to 1, the average cost under all policies theoretically tends to infinity because all updates will be erased by the noisy channel and cannot be received by the destination. The simulation results confirmed this conjecture. Comparing Figure 6a,b, we can observe that when $\lambda$ is large, the energy-first strategy will be close to the optimal strategy, which is also illustrated in Figure 5.

Figure 7 shows the performance of the average cost Q-learning algorithm. In every episode, the shift value g of the last inner iteration is recorded as the average cost. It can be found from Figure 7a that the average cost achieved by Algorithm 2 converges to that obtained by the modified RVI algorithm under different EH probability $\lambda$ and channel erasure probability p. The age–energy trade-off is shown in Figure 7b. By fixing $\lambda$ and p and changing $\omega$ from 0 to 1000, we run the modified RVI algorithm and average cost Q-learning algorithm to obtain the corresponding trade-off curve. It can be found that the curve obtained by the average cost Q-learning algorithm is very close to the optimal trade-off curve under the same condition, which further verifies the near-optimal performance of the average cost Q-learning algorithm in an unknown environment.

Figure 8 shows the optimal age and reliable energy cost trade-off curves for different energy supplies. By fixing EH probability $\lambda$ and channel erasure probability p and changing $\omega$ from 0 to 10,000, we run the modified RVI algorithm to get the optimal trade-off curve for mixed energy supplies. By letting EH probability $\lambda =0$ and following the same steps, we can obtain the optimal trade-off curve for reliable energy supply. By letting weighting factor $\omega$ go to infinity, we can theoretically obtain the optimal trade-off “curve” corresponding to the EH supply. The “curve” contains only one point because the reliable energy consumption can only be 0 for the EH supply case. It should be noted that $\omega$ cannot be infinite in a simulation. Instead, we can set $\omega$ to a relatively large number (e.g., 10,000). To facilitate comparison, the channel erasure probability is set as $p=0.2$, and the EH probability $\lambda$ is set as $0.1$, $0.3$ and $0.7$. It can be observed that the curves for the mixed energy supplies are always at the lower left of the curve for relying solely on reliable energy, which indicates that under the same average AoI, the reliable energy required by the system under the mixed energy supplies is smaller, and under the same reliable energy consumption, the AoI of the system under the mixed energy supplies is lower. The mixed energy design also achieves lower AoI than that with only EH, at the cost of paying for reliable energy. The optimal updating policy proposed in this paper makes full use of the harvested energy.

Figure 9 compares the performance of the optimal policy with the prior results in [23,25] for a special case where the sensor only uses the harvested energy and the battery capacity $B=1$. Both [23,25] considered a continuous-time model, i.e., the energy arrival process is a Poisson process with an arrival rate of $\mathsf{\Lambda }$ energy units per time unit (TU), and proved that the optimal policies are threshold structure, in which a new update is generated and transmitted only if the time until the next energy arrival since the latest successful transmission exceeds a certain threshold. Specifically, [23] (Theorem 4, Equation (13)) provided the average AoI and threshold in closed-form under the optimal update policy for any energy arrival rate $\mathsf{\Lambda }$ in the error-free channel case. It is interesting that the optimal average AoI and the corresponding threshold are equal. Ref. [25] (Theorem 4, Equation (14)) extended the results of [23] to an error-prone channel case, while the energy arrival rate $\mathsf{\Lambda }$ is assumed to be 1. So we first show the results of [23,25] vs. different channel erasure probability p in Figure 9, where the energy arrival rate $\mathsf{\Lambda }=1$. It should be emphasized that the unit of the average AoI and threshold is TU. According to Theorems 1 and 2 in this paper, the optimal update policy exists and admits a threshold structure for any EH probability $\lambda$, channel erasure probability p, weighting factor $\omega$ and battery capacity B. This conclusion is based on the discrete-time model, i.e., the energy arrives as a Bernoulli process with parameter $\lambda$, which is different from the continuous-time model in [23,25], and the reliable backup energy is also considered. However, by the choice of some parameters (large $\omega$, small $\lambda$), our results can be a good approximation of the results in [23,25]. First, by choosing a large $\omega$, the reliable energy will almost never be used, and equivalently, only the EH supply exists. Secondly, by choosing a small $\lambda$, the Poisson process can be approximated as a Bernoulli process. This is because for a Poisson process with parameter $\mathsf{\Lambda }$, we can discretize a TU into n small time slots of equal length, then according to probability theory, when n is large enough, the energy arrival process within a time slot can be approximated as a Bernoulli process with parameter $\lambda =\mathsf{\Lambda }/n$, which is relatively small. In our simulation, we set the battery capacity $B=1$, and take $\lambda =0.1$ (i.e., $n=10$) and $\omega =$ 10,000. By changing the channel erasure probability p, we can run the modified RVI algorithm to compute the minimum average AoI and the optimal threshold. It needs to be mentioned that the unit of them is a time slot. For comparison, we need to divide their values by n to obtain the average AoI and threshold in TU. The final results are shown by the dashed lines in Figure 9. It can be observed that the results of this paper are extremely close to the explicit results in [23,25], which verifies the correctness of the analysis and also reflects the generality of our system model.

6. Conclusions

In this paper, we studied the optimal updating policy for an information update system, where a wireless sensor sends updates over an erasure channel using both harvested energy and reliable backup energy. Theoretical analysis indicates the threshold structure of the optimal policy and simulation results verify its performance. For the practical case where the statistics, such as the EH probability and channel erasure probability, are unknown in advance, a learning-based algorithm is proposed to compute the updating policy. Simulation results show its performance is close to that of the optimal policy. With the optimal policy, the design of mixed energy supplies can make full use of harvested energy and achieve the best age–energy trade-off. In future work, we will focus on the timeliness of the multi-sensor system under mixed energy supplies.