Power Control and Link Selection for Wireless Relay Networks with Hybrid Energy Sources

Abstract

The Hybrid energy supply (HES) wireless relay system is a new green network technology, where the source node is powered by the grid and relay is powered by harvested renewable energy. However, the network’s performance may degrade due to the intermittent nature of renewable energy. In this paper, our purpose is to minimize grid energy consumption and maximize throughput. However, improving the throughput requires increasing the transmission power of the source node, which will lead to a higher grid energy consumption. Linear weighted summation method is used to turn the two conflicting objectives into a single objective. Link assignment and a power control strategy are adopted to maximize the total reward of the network. The problem is formulated as a discrete Markov decision model. In addition, a backwards induction method based on state deletion is proposed to reduce the computational complexity. Simulation results show that the proposed algorithm can effectively alleviate performance degradation caused by the lack of renewable energy, and present the trade-off between energy consumption and throughput.

1. Introduction

In order to expand the coverage of wireless networks and improve the communication quality of edge users, relay has been widely used in wireless networks such as long term evolution (LTE) and 5G [1,2]. However, dense deployment of relays can lead to problems such as high energy consumption and greenhouse gas emissions [3,4]. Energy harvesting technology is the most promising technology to solve the economic and environmental problems caused by the dense deployment of relays, which can collect and utilize renewable energy such as solar and wind energy [5,6,7,8,9]. In addition, energy harvesting technology can reduce the dependence of wireless networks on grid energy. Deploying green relays in areas where grid energy is scarce can effectively expand the coverage of wireless communication networks. However, the intermittent and random nature of the renewable energy may cause a decline in network performance. Therefore, it is critical to establish a renewable energy allocation and link selection mechanism to ensure the performance of wireless relay networks.

Optimal power control strategies to alleviate network performance degradation caused by the lack of renewable energy have been proposed in [10,11,12,13]. In [10], both the source and relay nodes are powered by renewable energy, and an off-line power control strategy is made to minimize data transmission time with throughput constraints. It was also proved that the power control strategy has water injection structure. Differing from [10], literature [11] considered how the source node is powered by the grid while the relay node is powered by renewable energy. An off-line power control strategy is proposed to maximize the grid energy efficiency of the source node. In [12], the research was extended to a Gauss fading channel, with both off-line and on-line power control schemes are proposed. In addition to the traditional relay, the literature [13] considers the relay with cache function. Off-line and on-line power control strategies are developed to maximize network throughput. All of the above studies have developed off-line or on-line power control strategies under the constraints of user’s quality of service (QoS).

Besides power control schemes, link selection strategies are also critical. In [14], the source node is powered by grid energy while the relay is powered by renewable energy. With causal side information, a link selection strategy is made to maximize the average throughput of all time slots. In [15], the relay is powered by radio frequency (RF) energy with a finite battery capacity, and only one data packet needs to be transmitted in each slot. A link selection scheme based on battery level is proposed to minimize the outage probability of the relay. Further, literature [16] jointly optimizes power and link selection to reduce outage probability of the relay. In [17], the research was extended to multi-relay with data caching, Under the constraints of battery capacity, on-line and off-line power allocation and link selection mechanisms were developed to minimize data transmission time.

Most previous studies on hybrid energy supply (HES) wireless relay systems assume that the amount of data transmitted by the source node are fixed in each slot, and then optimize the outage probability or transmission time. However, in practical applications, the source node needs to serve a lot of different users and deliver unfixed data bits in each slot. So, data bits should be transmitted as much as possible to reduce the network congestion with grid energy consumption constraints. Consequently, in this paper, we consider that the source node has available bits to be transmitted all the time. Our goal is to maximize network throughput while minimizing grid energy consumption.

The rest of this paper is organized as follows. Section 2 describes the system model. In Section 3, the Markov decision process (MDP) problem is formulated, for which a low-complexity algorithm is proposed. Simulations are shown in Section 4. Finally, Section 5 highlights the conclusion.

2. System Model

2.1. Network Model

We consider a green wireless relay network as shown in Figure 1, which consists of a source node (S), a relay node (R), and a destination node (D). The source node is powered by the electric grid, and its maximum transmit power is denoted as $p_S^{\max }$. The relay is powered purely by the renewable energy which is harvested from the nature and stored in a battery, with the maximum transmit power as $p_R^{\max }$. The distances between source node and relay, relay and destination node, source node and destination node are $d_{SR}$, $d_{RD}$ and $d_{SD}$, respectively. We consider the performance changes of the network in N slots with length T. The slot index set is denoted as $\mathbb{N}=\left\{1,2\cdots \cdots ,N\right\}$.

We assume that the source node has available bits to be transmitted all the time. The relay operates in a half-duplex manner that it receives data from the base station (BS) in the first half of the time slot and forwards it in the second half [11]. In each slot, there are two links for S to transmit bits: the relay link and direct link. The link assignment indicator in the $i$-th slot is denoted as $I_j^i\in \left\{0,1\right\}$, with $j\in \left\{SRD,SD\right\}$. ${\mathrm{I}}_{SRD}^i=1$ indicates that the relay link is assigned to deliver the bits, and ${\mathrm{I}}_{SD}^i=1$ represents the event that the direct link is selected. In each slot, only one of these links can be selected, so

$${\mathrm{I}}_{SD}^i+{\mathrm{I}}_{SRD}^i=1,\forall i\in \mathbb{N}.$$

(1)

In addition, transmission power of source node S and relay R in slot $i$ is $p_S^i$ and $p_R^i$, respectively. Static power consumption is neglected in this work.

We consider that the channel is time-varying with small-scale fading; denoted ${{\displaystyle h}}_i^k$, with $k\in \left\{SR,RD,SD\right\}$ as the channel fading factor between two nodes. In addition, all channels share the bandwidth (B) together. Therefore, once the direct link is selected in the $i$-th slot, the number of bits transmitted by the source node are calculated as

$$R_{SD}^i=BT\log (1+h_{SD}^i{g}_0{d}_{SD}^{-\alpha }{\sigma }^{-2}{P}_{SD}^i),$$

(2)

according to the Shannon theorem, where ${\sigma }^{-2}$ is the noise variance of node D, $g_0$ is the channel fading constant, $p_{SD}^i$ is the transmit power of node S and $\alpha$ is the path loss exponent. While the relay link is selected, the total bits delivered by the source node and the relay are given as

$$R_{SR}^i=B\frac{T}{2}\log (1+h_{SR}^i{g}_0{d}_{SR}^{-\alpha }{\sigma }^{-2}{p}_{SR}^i)$$

(3)

and

$$R_{RD}^i=B\frac{T}{2}\log (1+h_{RD}^i{g}_0{d}_{RD}^{-\alpha }{\sigma }^{-2}{p}_R^i),$$

(4)

where $p_{SR}^i$ is the transmit power of node S when selecting the relay link.

2.2. Energy Model

In order to simplify the energy harvesting model, the harvesting process is thought to be accomplished at the beginning of each time slot [18]. Then, discrete energy model is adopted to describe the process of energy harvesting with $E_H^i$ energy packets arriving at each time slot [19], which obeys the Poisson distribution with mean λ [20]. And λ represents the intensity of energy harvesting. In addition, each packet contains the energy of $E_e$. In the initial time slot, the energy stored in the battery is the original energy plus the harvested energy. While in the slot of $i>1$, the energy consumed by the relay should be subtracted. Therefore, battery level in the $i$-th slot is expressed as:

$$E(i)=\{\begin{array}{ll}{E}_0+E_H^i{E}_e& i=1\\ E(i)-C_R^{i-1}+E_H^i{E}_e& i>1\end{array},$$

(5)

where $E_0$ is the initial energy, and $C_R^{i-1}$ is energy consumed by the relay in the last slot, which is given by:

$${\mathrm{C}}_R^{i-1}={\mathrm{I}}_{SD}^{i-1}T{\mathrm{p}}_{SD}^{i-1}+{\mathrm{I}}_{SRD}^{i-1}\frac{T}{2}{\mathrm{p}}_{SRD}^{i-1}.$$

(6)

Since the battery is of the limited size, the following energy constraint should be satisfied:

$$E(i)\le E_{\max },$$

(7)

where $E_{\max }$ is the maximum capacity of the battery. The energy consumed by the relay should be no more than the energy of the battery, that is $C_R^i\le E(i)$.

2.3. Optimizing Objective

Most previous studies on two-hop green wireless relay networks concentrated on minimizing the outage probability or transmission time with fixed bits at each slot [10,21,22]. However, when the network is busy and needs to provide services for multiple users, there will be a continuous stream of data bits to be transmitted. In this case, throughput can show the carrying capacity of the network, which is attractive to us. In addition, the source node powered by the grid energy can also be applied to some wireless networks. For example, macro base stations powered by the grid energy can maintain basic coverage in heterogeneous wireless networks. To take above aspects into consideration, we set grid energy consumption and throughput as optimization objectives. However, according to Equations (2) and (3), improving the throughput requires increasing the transmission power of the source node, which will lead to higher grid energy consumption. Therefore, increasing throughput and reducing grid energy consumption are conflicting objectives, which can be formulated as:

$$obj=\{\begin{array}{l}\max {\displaystyle \sum _{i=1}^N{I}_{SD}^i{R}_{SD}^i+I_{SRD}^i{R}_{RD}^i}\\ \min {\displaystyle \sum _{i=1}^N{I}_{SD}^{i}T{p}_{SD}^i+I_{SRD}^i\frac{T}{2}{p}_{SR}^i}\end{array}.$$

(8)

There are many ways to solve multi-objective problems, and linear weighted summation is an effective method of them [23]. We use the weighted summation method to transform the conflicting multi-objectives into a single objective. Thus, the conflict objectives can be denoted as:

$$TR={\displaystyle \sum _{i=1}^N[{\omega }_t(I_{SD}^i{R}_{SD}^i+I_{SRD}^i{R}_{RD}^i)}-{\omega }_g(I_{SD}^{i}T{p}_{SD}^i+I_{SRD}^i\frac{T}{2}{p}_{SR}^i)],$$

(9)

where ${\omega }_t$ and ${\omega }_g$ are weight coefficients of throughput and grid energy consumption, respectively. ${\omega }_t(I_{SD}^i{R}_{SD}^i+I_{SRD}^i{R}_{RD}^i)$ and $-{\omega }_g(I_{SD}^{i}T{p}_{SD}^i+I_{SRD}^i\frac{T}{2}{p}_{SR}^i)$ are the weighted throughput reward and grid energy reward in the $i$-th slot.

3. Optimal Control for Expected Total Rewards

In this section, we assume $h_{SR}^i$, $h_{SD}^i$ and $h_{RD}^i$ are causally known. Consequently, we aim to adapt the transmission power and link selection to maximize the expected total rewards. Thus, the problem can be formulated as:

$${\mathrm{P}}_1:\underset{I_j^i,p_{SR}^i,p_{SD}^i,p_R^i}{\max }E\left\{{\displaystyle \sum _{i=1}^N[{\omega }_t(I_{SD}^i{R}_{SD}^i+I_{SRD}^i{R}_{RD}^i)-{\omega }_g(I_{SD}^{i}T{p}_{SD}^i+I_{SRD}^i\frac{T}{2}{p}_{SR}^i)]}\right\}$$

(10)

s.t. Equations (1) and (7)

(11)

$$C_R^i\le E(i)$$

(12)

$$R_{SR}^i=R_{RD}^i$$

(13)

$${{\displaystyle p}}_j^i\le {{\displaystyle p}}_j^{\max }\forall i\in \mathbb{N},j\in \left\{S,R\right\}$$

(14)

$$I_j^i\in \left\{0,1\right\},\forall i\in N,j\in \left\{SD,SRD\right\}.$$

(15)

In ${\mathrm{P}}_1$, the objective is the expected total rewards over N time slots. Equation (12) is the energy constraint that the energy consumed by relay should not exceed that of the battery. Equation (13) is the throughput constraint that the bits delivered in the two stages of relay link should be Equations (14) and (15) are the power constraint and the link selection constraint.

3.1. Problem Simplification

From Equation (10), we can see that the optimized variables of ${\mathrm{P}}_1$ include the 0–1 variable $I_j^i$ and the continuous variable $p_{SR}^i$, $p_{SD}^i$ and $p_R^i$. Thus, it is very difficult to optimize so many different types of variables at the same time. Therefore, we do some simplification of ${\mathrm{P}}_1$ to reduce the optimized variables. And, the problem of ${\mathrm{P}}_1$ can be expressed as:

$$P_2:\underset{p_R^i}{\max }E\left\{{\displaystyle \sum _{i=1}^N[{\beta }_i{D}_i-(1-{\beta }_i)H_i]}\right\}$$

(16)

$$\mathrm{s}.\mathrm{t}.C_R^i\le E\left(i\right)$$

(17)

$$p_R^i\le p_R^{\max }\forall i\in \mathbb{N}.$$

(18)

And ${\beta }_i$ is the 0–1 variable, which can be calculated as:

$${\beta }_i=\{\begin{array}{c}1p_R^i\ne 0\\ 0p_R^i=0\end{array}\forall i\in \mathbb{N}.$$

(19)

That is, when the transmission power of relay is non-zero, ${\beta }_i=1$ and relay link is chosen to transmit data. While the power of relay is zero, ${{\displaystyle \beta }}_i=0$ and direct link is selected to deliver bits. In addition, ${{\displaystyle D}}_i$ is the network total reward when relay link is chosen, which is given by:

$${\mathrm{D}}_i={\omega }_t{R}_{RD}^i-{\omega }_g{G}_{SR}^i,$$

(20)

where $R_{RD}^i$ is the throughput, and $G_{SR}^i$ is the grid energy consumed by the source node. We assume that the amount of data bits transmitted in the two stages of relay link should be equal. So, there is

$$R_{SR}^i=R_{RD}^i=B\frac{T}{2}\log (1+h_{SR}^i{g}_0{d}_{SR}^{-\alpha }{\sigma }^{-2}{p}_{SR}^i)=B\frac{T}{2}\log (1+h_{RD}^i{g}_0{d}_{RD}^{-\alpha }{\sigma }^{-2}{p}_R^i).$$

(21)

From equation (21), we can know that

$$h_{SR}^i{d}_{SR}^{-\alpha }{p}_{SR}^i=h_{RD}^i{d}_{RD}^{-\alpha }{p}_R^i.$$

(22)

According to Equation (20) to (22), $D_i$ can be further given as:

$${{\displaystyle D}}_i={{\displaystyle w}}_{t}B\frac{T}{2}\log (1+{{\displaystyle h}}_{RD}^i{{\displaystyle g}}_0{{\displaystyle d}}_{RD}^{-\alpha }{{\displaystyle \sigma }}^{-2}{{\displaystyle p}}_R^i)-{{\displaystyle w}}_g\frac{T}{2}\frac{{{\displaystyle h}}_{RD}^i{{\displaystyle d}}_{RD}^{-\alpha }{{\displaystyle p}}_R^i}{{{\displaystyle h}}_{SR}^i{{\displaystyle d}}_{SR}^{-\alpha }}.$$

(23)

$H_i$ is the reward when direct link is selected, and the problem of solving ${{\displaystyle H}}_i$ can be expressed as:

$$P_H:\underset{p_{SD}^i}{\max }{H}_i=\underset{p_{SD}^i}{\max }[{\omega }_t{R}_{SD}^i-{\omega }_g{G}_{SD}^i]={\omega }_{t}TB\log (1+h_{SD}^i{g}_0^{\prime }{d}_{SD}^{-\alpha }{\sigma }^{-2}{p}_{SD}^i)-{\omega }_{g}T{p}_{SD}^i$$

(24)

$$\mathrm{s}.\mathrm{t}.p_{SD}^i\le p_S^{\max }\forall i\in \mathbb{N}.$$

(25)

Proposition 1.

The reward of the direct link $H_i$ is convex in transmission power $p_S^i$.

Proof: 

The second derivative of formula (24) is

$$f{(p_{SD}^i)}^{″}=-\frac{w_{t}TB{(h_{SD}^i{g}_0{d}_{SD}^{-\alpha }{\sigma }^{-2})}^2}{{(1+h_{SD}^i{g}_0{d}_{SD}^{-\alpha }{\sigma }^{-2}{p}_{SD}^i)}^2\ln 2}<0,$$

(26)

which means that $f(p_{SD}^i)$ is a convex function. □

According to the properties of convex functions, we can easily know that $H_i$ gets the maximum value at $p_{SD}^i=\frac{{\omega }_t}{{\omega }_g}-\frac{1}{h_{SD}^i{g}_0{d}_{SD}^{-\alpha }{\sigma }^{-2}}$. Therefore, the optimal $p_{SD}^i$ and $H_i$ are known once $h_{SD}^i$ is given. However, $p_{SD}^i$ has practical significance only on [0, $p_S^{\max }$] in this work. Consequently, when $\frac{{\omega }_t}{{\omega }_g}-\frac{1}{h_{SD}^i{g}_0{d}_{SD}^{-\alpha }{\sigma }^{-2}}<0$, $P_H$ decreases monotonously on [0, $p_S^{\max }$] with maximum value $p_H(0)$. While $\frac{{\omega }_t}{{\omega }_g}-\frac{1}{h_{SD}^i{g}_0{d}_{SD}^{-\alpha }{\sigma }^{-2}}>p_S^{\max }$, $P_H$ increases monotonously on [0, $p_S^{\max }$] with maximum value $p_H(\max )$.

It can be seen from Equation (16) to (26), that the values of ${\beta }_i$ and $D_i$ are determined by $p_R^i$, and the maximum value of $H_i^{\ast }$ is a fixed value in each time slot which is only related to $h_{SD}^i$. Therefore, the optimal variable of problem $P_2$ is only $p_R^i$.

3.2. MDP Model for Expected Total Rewards

Our goal is to maximize the total rewards over N slots through a relay power control scheme. However, due to the limited battery capacity, the relay power selection results in each slot will affect the initial battery capacity at the next moment. So, power decisions on different time slots are mutually influential. The MDP is a useful model to handle such decision problems, and backward induction is an effective algorithm to solve this problem [24].

Therefore, we formulate $P_2$ as a Markov decision process (MDP) problem, which can also be expressed as:

$${\mathrm{P}}_2:\underset{\pi \in \prod }{\max }{{\displaystyle E}}^{\pi }{\displaystyle \sum _{i=1}^{N}R(i)},$$

(27)

where $\pi$ is a feasible relay power policy, $\prod$ denotes the set of all feasible policies. $R(i)$ is the reward of slot i, which is given by $[{{\displaystyle \beta }}_i{{\displaystyle D}}_i-(1-{{\displaystyle \beta }}_i){{\displaystyle H}}_i]$.

3.2.1. MDP Basics

A sequential decision-making method is the selection of one of several action strategies in each time slot during the operation of the system [25]. In the sequential decision process, if the transfer of the system state obeys the known probability law and is independent of the previous history, then this sequential problem is called an MDP problem [26]. An MDP model consists of a reward function, system states, actions, state transition probability and objective, each of which will be described in detail later.

3.2.2. Reward Function

In an MDP model, the reward function is defined as $r(j,a_i)$. It indicates that the system gets the reward with action $a_i$ at state j [27]. This is denoted $o_i\in O,i\in \mathbb{N}$ as the rule for selecting relay power in slot i. Thus, the rules over N slots can be expressed as $\pi {=(\mathrm{o}}_1{,\mathrm{o}}_2{,\mathrm{o}}_3,\cdots {\mathrm{o}}_N)$, and the set of all possible rules is denoted as $\prod$. Given the initial state k and strategy $\pi \in {\prod }_m^O$, the expected total rewards can be also written as:

$${{\displaystyle V}}_N(\pi ,{{\displaystyle s}}_1)={\displaystyle \sum _{i=1}^N{\displaystyle \sum _{{{\displaystyle a}}_i\in {{\displaystyle A}}_i,j\in S}{{\displaystyle P}}_{\pi }}}\left\{{{\displaystyle s}}_i=j,{{\displaystyle \Delta }}_i={{\displaystyle a}}_i|{{\displaystyle s}}_1=k\right\}r(j,{{\displaystyle a}}_i),$$

(28)

where ${{\displaystyle s}}_i$ and ${{\displaystyle \Delta }}_i$ are the states of the relay system and the selected action in the slot i, respectively. ${{\displaystyle P}}_{\pi }\left\{{{\displaystyle s}}_i=j,{{\displaystyle \Delta }}_i={{\displaystyle a}}_i|{{\displaystyle s}}_1=k\right\}$ is the conditional probability of using strategy $\pi \in {\prod }_m^O$, starting from state k, selecting action $a_i$, and moving to state j at slot $i$. Our aim is to find the optimal action selection scheme as ${\pi }^{\ast }{=(\mathrm{o}}_1^{\ast }{,\mathrm{o}}_2^{\ast },\cdots ,o_N^{\ast })$, which makes ${{\displaystyle V}}_N({{\displaystyle \pi }}^{\ast },{{\displaystyle s}}_1)$ the maximum value.

3.2.3. Discretization of System States and Actions

The optional values of the system states and actions should be finite in MDP model. However, the system states include links and the battery states, and the relay power actions are continuous values in the wireless relay network. Therefore, it is necessary to discretize the system states and relay power actions. The relay system states consist of channel fading values and battery levels, which can be given as $s_i\triangleq \langle h_{SR}^i,h_{SD}^i,h_{RD}^i,\epsilon (i)\rangle$. We discretize the channel states by reference to the method in literature [28]. Denoting $H=\left\{H_1,H_2,\dots H_3\right\}$ as the set of channel fading values, which is an equal-difference sequence. The probability when the channel fading is $H_k$ with $k\in \left\{1,2\cdots K\right\}$ can be calculated as:

$$p\left\{h_j^i=H_k\right\}=\frac{1}{K}\forall i\in \mathbb{N},j\in \left\{SR,SD,SRD\right\},k\in \left\{1,2\cdots K\right\}.$$

(29)

We divide the battery into M + 1 energy level. And the battery states set is taken as $\epsilon (i)\in \epsilon =[0,1,\cdots ,m,\cdots M]$. The real-time energy level of the battery can be calculated by

$$\epsilon (i)=m=\lfloor \frac{E_{i}M}{E_{\max }}\rfloor .$$

(30)

Denote $A_i^{\prime }=[0,\frac{p_R^{\max }}{L},\dots ,p_R^{\max }]$ as the action set, which is also an equal-difference sequence. Actually, the value of the relay transmit power is constrained by the battery level. Thus, the action set is given as $A_i=[0,\frac{p_R^{\max }}{L},\dots ,\min (p_R^{\max },p_x^i)]$, where $p_x^i$ is calculated by:

$$p_x^i=\lfloor \frac{2\epsilon (i)E_{\max }L}{MT{p}_R^{\max }}\rfloor \ast \frac{p_R^{\max }}{L}.$$

(31)

$\lfloor x\rfloor$ is the function that rounds the variable x down.

3.2.4. State Transition Probability

After action $a_i$ is selected, the system states will migrate from $s_i$ to $s_{i+1}$, which can be expressed as $s_i\triangleq \langle h_{SR}^i,h_{SD}^i,h_{RD}^i,\epsilon (i)\rangle \to s_{i+1}\triangleq \langle h_{SR}^{i+1},h_{SD}^{i+1},h_{RD}^{i+1},\epsilon (i+1)\rangle$. Since the value of channel fading is equal probability, the state transition probability is:

$$p\left\{s_{i+1}|s_i,a_i\right\}=\frac{1}{K^3}p\left\{{\epsilon }_{i+1}|{\epsilon }_i,a_i\right\}.$$

(32)

We assume that $\epsilon (i)$ and $\epsilon (i+1)$ are in the $M_1$ and $M_2$ levels of the battery, which should satisfy the Equation: $\frac{M_2{E}_{\max }}{M}\le \frac{M_1{E}_{\max }}{M}-a_i\ast \frac{T}{2}+E_H^i{E}_e<\frac{(M_2+1)E_{\max }}{M}$. As mentioned in Section 2.2, the energy harvesting process obeys the Poisson distribution with mean λ. Therefore, Equation (32) can be further given as:

$$p\left\{s_{i+1}|s_i,a_i\right\}=\frac{1}{K^3}{\displaystyle {\sum }_{n=n_1}^{n_2}\frac{{\lambda }^n}{n!}}{e}^{-\lambda },$$

(33)

where $n_1$ and $n_2$ is given as $n_1=\lceil \frac{\frac{M_2{E}_{\max }}{M}+a_i\ast \frac{T}{2}-\frac{M_1{E}_{\max }}{M}}{E_e}\rceil$ and $n_2=\lceil \frac{\frac{(M_2+1)E_{\max }}{M}+a_i-\frac{M_1{E}_{\max }}{M}}{E_e}\rceil -1$, respectively. $\lceil x\rceil$ is the function that rounds the variable x up.

3.3. The Backward Induction Algorithm for MDP Problem

The backward induction algorithm is an effective solution to the optimal strategy and value function in the finite-stage Markov decision programming problem [26]. A new function, ${{\displaystyle V}}_{\ast }^n(i)$, was proposed based on the backward induction algorithm, which is formulated as:

$$\begin{array}{ll}{{\displaystyle V}}_{\ast }^n(i)& =\underset{{{\displaystyle a}}_i\in {{\displaystyle A}}_i}{\max }[r(k,{{\displaystyle a}}_i)+{\displaystyle \sum _{j\in s}p(j|k,{{\displaystyle a}}_i)}{{\displaystyle V}}_{\ast }^{i+1}(j)]\\ & =r(k,{{\displaystyle f}}_{\ast }^i(k))+{\displaystyle \sum _{j\in s}p(j|k,{{\displaystyle a}}_i)}{{\displaystyle V}}_{\ast }^{i+1}(j)\\ & (k\in s,i=\left\{N,N-1,N-2,\cdots ,0\right\})\end{array},$$

(34)

where $V_{\ast }^{N+1}(k)=0,\forall k\in s$ [26]. According to Equation (34), the optimal value function of the expected total rewards can be calculated as $V_{\ast }^1=(V_{\ast }^1(1),V_{\ast }^1(2),\cdots ,V_{\ast }^1(q))$. Meanwhile, the decision sequence ${\pi }^{\ast }=(o_{\ast }^1,o_{\ast }^2,\cdots ,o_{\ast }^N)$ obtained is the optimal strategy.

With the backward induction algorithm, the number of states required to traverse is $M\times {{\displaystyle L}}^3$. The state space may be very large if some of the elements are of large size and may encounter the curse of dimensionality [29]. An effective method to reduce the computational complexity in MDP model is proposed in literature [28]. In this case, we also eliminate some states that do not need to be searched according to the wireless relay network properties in our model by reference [28].

Proposition 2.

When $h_{SR}^i$, $h_{RD}^i$, $\epsilon (i)$ are fixed value, and the optimal action is $a_i=0$ at state $s_i^{-}\triangleq \langle h_{SR}^i,h_{SD}^{i^{-}},h_{RD}^i,\epsilon (i)\rangle$, the optimal action is $a_i=0$ for any state $s_i^{+}$ of $h_{SD}^{i^{+}}>h_{SD}^{i^{-}}$.

Proof: 

If the optimal action for the state $s_i^{-}$ is $a_i=0$, according to Equation (34), we know that:

$$r(s_i^{-},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{-},0)}{V}_{i+1}^{\ast }(j)>\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{-},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{-},a_i)}{V}_{i+1}^{\ast }(j)\right].$$

(35)

From Equation (21), we know that

$$r(s_i,0)=H_i^{\ast }={\omega }_{t}TB\log (1+h_{SD}^i{g}_0^{\prime }{d}_{SD}^{-\alpha }{\sigma }^{-2}{p_{SD}^i}^{\ast })-{\omega }_{g}T{p_{SD}^i}^{\ast },$$

(36)

Which becomes larger as $h_{SD}^i$ grows. Therefore, for any state $s_i^{+}$ with $h_i^{S{D}^{+}}>h_i^{S{D}^{-}}$, $r(s_i^{+},0)>r(s_i^{-},0)$. Since $h_i^{SR}$, $h_i^{RD}$ and ${\epsilon }_i$ are fixed value, we can get

$${\displaystyle \sum _{j\in s}p(j|s_i^{+},0)}{V}_{i+1}^{\ast }(j)={\displaystyle \sum _{j\in s}p(j|s_i^{-},0)}{V}_{i+1}^{\ast }(j),$$

(37)

and

$$\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{+},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},a_i)}{V}_{i+1}^{\ast }(j)\right]=\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{-},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{-},a_i)}{V}_{i+1}^{\ast }(j)\right].$$

(38)

Finally,

$$r(s_i^{+},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},0)V_{i+1}^{\ast }(j)}>\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{+},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},a_i)}{V}_{i+1}^{\ast }(j)\right],$$

(39)

which proves that the optimal action is $a_i=0$ in the state $s_i^{+}$. □

Proposition 3.

When $h_{SD}^i$, $h_{RD}^i$, $\epsilon (i)$ are fixed value, and the optimal action is $a_i=0$ at state $s_i^{+}\triangleq \langle h_{SR}^{i^{+}},h_{SD}^i,h_{RD}^i,\epsilon (i)\rangle$, the optimal action is $a_i=0$ for any state $s_i^{-}$ of $h_{SR}^{i^{-}}<h_{SR}^{i^{+}}$.

Proof: 

If the optimal action for state $s_i^{+}$ is $a_i=0$, according to Equation (34), we can get:

$$r(s_i^{+},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},0)}{V}_{i+1}^{\ast }(j)>\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{+},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},a_i)}{V}_{i+1}^{\ast }(j)\right].$$

(40)

While $p_R^i\ne 0$, $r(s_i,p_R^i)$ is given as

$$r(s_i,p_R^i)={{\displaystyle D}}_i={{\displaystyle w}}_{t}B\frac{T}{2}\log (1+{{\displaystyle h}}_{RD}^i{{\displaystyle g}}_0{{\displaystyle d}}_{RD}^{-\alpha }{{\displaystyle \sigma }}^{-2}{{\displaystyle p}}_R^i)-{{\displaystyle w}}_g\frac{T}{2}\frac{{{\displaystyle h}}_{RD}^i{{\displaystyle d}}_{RD}^{-\alpha }{{\displaystyle p}}_R^i}{{{\displaystyle h}}_{SR}^i{{\displaystyle d}}_{SR}^{-\alpha }},$$

(41)

which becomes smaller as $h_{SR}^i$ grows. Thus, For the stat $s_i^{-}$ with $h_i^{S{R}^{-}}<h_i^{S{R}^{+}}$, $r(S_i^{-},a_i)<(S_i^{+},a_i)a_i\in A_i and a_i\ne 0$. Since $h_i^{SD}$, $h_i^{RD}$ and ${\epsilon }_i$ are fixed value, there are:

$$\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{-},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{-},a_i)}{V}_{i+1}^{\ast }(j)\right]<\underset{a_i\in A_{i}and{a}_i\ne 0}{\max }\left[r(s_i^{+},a_i)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},a_i)}{V}_{i+1}^{\ast }(j)\right],$$

(42)

and

$$r(s_i^{-},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{-},0)}{V}_{i+1}^{\ast }(j)=r(s_i^{+},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},0)}{V}_{i+1}^{\ast }(j).$$

(43)

Finally,

$$r(s_i^{-},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{-},0)}{V}_{i+1}^{\ast }(j)=r(s_i^{+},0)+{\displaystyle \sum _{j\in s}p(j|s_i^{+},0)}{V}_{i+1}^{\ast }(j),$$

(44)

which indicates that the optimal action is $a_i=0$ in the state $s_i^{-}$.

Algorithm 1. Backward Induction Algorithm Based on States Elimination

Input: $p_R^{\max }$, $p_S^{\max }$, $d_{SD}$, $d_{RD}$, $d_{SR}$, $T$, $B$, $N$, $K$, $L$, $\lambda$, $E_e$, ${\omega }_t$, ${\omega }_g$, $\epsilon$

Output: ${\pi }^{\ast }$

1: Initialize ${\pi }^{\ast }=zeros(N,K^3\times M)$, $V_{\ast }^{N+1}=0$

2: While N $\ne 1$

3:  ${\mathrm{p}}_x^N=\lfloor \frac{2\epsilon (N)L}{T{p}_R^{\max }}\rfloor \ast \frac{p_R^{\max }}{L}$, ${\mathrm{A}}_N=[0,\frac{p_R^{\max }}{L},\dots ,\min (p_R^{\max },p_x^N)]$;

4:     For m = 1 to M, $k_{SR}$ = 1 to K

5:       $k_{RD}$ = K, $k_{SD}$ = 1;

6:          While $k_{SD}\ne K+1$, $k_{RD}\ne 0$

7:          $s=<H_{k_{SR}},H_{k_{SD}},H_{k_{RD}},\epsilon (m)>$;

8:              For j = 1: length ($A_N$)

9:      Calculate ${\pi }^{\ast }(N,s)=\arg \underset{A_N(j)}{\max }{V}^N(s)=\arg \underset{A_N(j)}{\max }(r(s),A_N(j))+{\displaystyle \sum _{l\in s}p(l|s,A_N(j))}{V}_{\ast }^{N+1}(l))$;

10:               End For

11:                 If ${\pi }^{\ast }(N,s_{k_{SR},k_{SD},k_{RD},m})=0$

12:                   ${\pi }^{\ast }(N,s_{k_{SR},k_{SD}^{+},k_{RD},m})=0\forall k_{SD}^{+}>k_{SD}$, ${\pi }^{\ast }(N,s_{k_{SR},k_{SD},k_{RD}^{-},m})=0\forall k_{RD}^{-}<k_{RD}$;

13:                   $k_{RD}$ = $k_{RD}$ − 1, $k_{SD}$ = $k_{SD}$ + 1;

14:                 Else

15:                   if $k_{SD}$ = K

16:                    $k_{RD}$ = $k_{RD}$ − 1, $k_{SD}$ = 1;

17:                    Else

18:                     $k_{SD}$ = $k_{SD}$ + 1;

19:                    End If

20:                 End If

21:       End For

23:       N = N − 1;

24: End While

4. Numerical Simulations

In this section, we run some numerical simulations to analyze the total reward, grid energy consumption and throughput in two-hop wireless relay networks. In the simulations, we set B = 10 MHz, T = 1 ms, $p_S^{\max }$ = 2 W, $p_R^{\max }$ = 0.5 W, ${\sigma }^2$ = −97.5 dBm, $g_0$ = −40 dB, $\alpha$ = 4 [28]. And $E_e$ = 0.01 mJ, $E_{\max }$ = 1.6 mJ, K = 10, L = 20, $d_{SD}$ = 80 m, ${\omega }_g=1$. The detailed numerical results are shown as follows.

4.1. Baseline Schemes

Joint Power Control and Link Selection Algorithm (JPLA): The JPCALSA only considers the current system state, and calculates the maximum rewards of relay and direct link, respectively. Then, the optimal access link is selected by comparing the rewards.

Power Control Algorithm (PCA): The PCA is to maximize the reward in single slot by adjusting the power of the relay, and link selection scheme is not taken into account [11].

When the energy of relay is sufficient, the system will be in the ideal state. In order to compare the ideal results with our results in different situations, we propose JPLA-F and BIABoSE-F, which are JPLA and BIABoSE with enough renewable energy.

4.2. Parameter Analysis

Figure 2 demonstrates the total rewards with different number of battery levels at different time slots. In any slot, the total rewards increase with the number of battery levels rises. Actually, the energy between two adjacent levels is expressed by the lower level, and the interval of two adjacent levels is smaller as the number of levels becomes larger. At this point, the error between the true value and the expressed value will be smaller, which makes a more accurate result. When the number of battery levels reach 80 and 160, their rewards are close and maximal. Consequently, for reducing the computational complexity, M = 80 is used for simulation analysis in the follow-up.

We assume that $d_{SD}=d_{SR}+d_{RD}=80m$, and the total rewards vary with $d_{SD}$ is shown in Figure 3 and Figure 4. As $d_{SD}$ increases, the rewards of all algorithms become larger first and then decrease. When $d_{RD}$ is small, $d_{SR}=80-d_{RD}$ is large, and the path loss between the source node and relay is high. In this case, the source node delivers a few bits to relay with high grid energy consumption, which leads to low total rewards. As $d_{RD}$ increases, the path loss between the source node and relay decreases, and the total rewards rise. Once $d_{RD}$ is larger than a certain threshold, the path loss between the relay and destination is high, the number of bits that can be transmitted by relay is lower than that by source node. In this case, the total reward is gradually reduced as the throughput of relay tapers off.

In addition, the JPLA-F and BIBAoSE-F achieve the maximum value near $d_{RD}^{\ast }$ = 40 m in both Figures. Meanwhile, the PCA, JPLA and BIBAoSE obtain the maximum value at different $d_{RD}^{\ast }$ in two Figures. Unlike the JPLA-F and BIBAoSE-F, the other three algorithms are affected by the energy harvesting intensity. The energy that the relay needed for data transmission grows larger as $d_{RD}$ increases. Therefore, when the energy is more sufficient, the total rewards will be closer to optimal result. The total rewards reach the maximum value at $d_{RD}$ = 40. Thus, we choose $d_{RD}$ = 40 for subsequent simulations to better observe the improvement of system performance in the absence of energy.

4.3. Total Reward Maximization

Figure 5 shows the total rewards changes with the time slots. Compared with the PCA, the JPLA adds a link selection mechanism. Therefore, the JPLA can transmit data through the direct link when the battery is very low, which can increase the total rewards. The BIBAoSE takes the future system states into account, which makes a more efficient green energy allocation over N slots than the JPLA. However, all the algorithms can only alleviate the system performance degradation caused by insufficient energy and cannot replace the green energy supply. Therefore, the JPLA-F and the BIBAoSE-F always have the highest total rewards.

Figure 6 displays the total rewards vary as energy harvesting intensity increases. The system is in a green energy-deficient state, when the energy intensity is low. In this case, the relay can deliver more bits as the intensity increases, which leads to a higher reward. However, the rewards will be constant once the energy intensity reaches a certain threshold, because the battery capacity is limited. It should be noted that the rewards of the BIABoSE are lower than the other algorithms when the green energy is enough due to the discretization of states. However, the BIABoSE achieves better performance in our main application scenario, which is a lack of green energy.

4.4. Grid Energy Consumption and Throughput Trade-Off

Figure 7 shows the grid energy consumption and throughput when ${\omega }_t$ takes different values. When ${\omega }_t$ is very small, the energy consumption and throughput of all schemes are similar. In this case, the system has a high demand for grid energy consumption, which will impose strict limits on energy consumption. When ${\omega }_t$ increases, the throughput plays an increasingly important role in the reward. Although our schemes consume a little more energy than the JPLA and PCA, it greatly improves the throughput. When the value of ${\omega }_t$ is large, all schemes pursue maximum throughput regardless of energy consumption costs. Therefore, all throughput gains are very close. However, the BIBAoSE consumes the least energy and is closest to the JPLA-F and BIBAoSE-F. In addition, once the throughput constraints are given, we can find the value of ${\omega }_t$ and get the minimum grid energy consumption.

The energy consumption and throughput are shown in Figure 8. As can be seen from the graph, the BIBAoSE consumes less grid energy than the JPLA when achieves the same throughput. And the BIBAoSE can transmit more bits than the JPLA with the same grid energy supply. In short, the BIBAoSE has a better trade-off between energy consumption and throughput, which is closer to the ideal situation such as the JPLA-F and BIBAoSE-F.

5. Conclusions

In this paper, we proposed an online power allocation and link selection strategy to maximize the total rewards of two-hop relay wireless networks where the source node and relay are powered by grid and green energy, respectively. Simulation results show that the total reward of this scheme is optimal under different settings compared with some conventional schemes. Next, we will continue to study energy harvesting technology in multifunctional relay nodes. Then, the research results will be applied to practical scenarios such as 5G heterogeneous networks, the Internet of Things and other networks.