# Designing Transmission Strategies for Enhancing Communications in Medical IoT Using Markov Decision Process

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

- An MDP formulation to develop the transmission strategy for multi-hop communication within WBAN which not only focuses on obtaining optimal transmission power subject to the input conditions but also reflects the necessity of multi-hop data transmission as well.
- A routing algorithm is designed based on the effective transmission strategies obtained by solving MDP formulation prior to deployment. These transmission strategies correspond to the system states that the network may undergo after deployment in terms of energy level, event occurrence, packet transmission rate and link quality. The nodes route data following a simple but effective routing algorithm and make a decision to transmit via multi-hop or single-hop based on suitable transmission power.
- The effectiveness of the designed solutions is verified with extensive simulations, and performance is evaluated with respect to the existing multi-hop protocol for WBAN.

## 2. Related Work

## 3. Markov Decision Process

## 4. Our Work

#### 4.1. System Model

#### 4.2. Phase I (Pre-Deployment Phase)

#### Markov Decision Process Formulation

#### 4.3. Phase II (Post-Deployment Phase)

Algorithm 1: EstimateTransmissionPower (). |

## 5. Simulation Results

#### 5.1. Experimental Results of Phase I

#### 5.2. Experimental Results of Phase II

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Xu, B.; Xu, L.D.; Cai, H.; Xie, C.; Hu, J.; Bu, F. Ubiquitous data accessing method in IoT-based information system for emergency medical services. IEEE Trans. Ind. Inf.
**2014**, 10, 1578–1586. [Google Scholar] - Movassaghi, S.; Abolhasan, M.; Lipman, J.; Smith, D.; Jamalipour, A. Wireless body area networks: A survey. IEEE Commun. Surv. Tutor.
**2014**, 16, 1658–1686. [Google Scholar] [CrossRef] - Cavallari, R.; Martelli, F.; Rosini, R.; Buratti, C.; Verdone, R. A survey on wireless body area networks: Technologies and design challenges. IEEE Commun. Surv. Tutor.
**2014**, 16, 1635–1657. [Google Scholar] [CrossRef] - Zhang, Y.; Zhang, B.; Zhang, S. A lifetime maximization relay selection scheme in wireless body area networks. Sensors
**2017**, 17, 1267. [Google Scholar] [CrossRef] - Sodhro, A.H.; Li, Y. Battery-friendly packet transmission strategies for wireless capsule endoscopy. In The International Conference on Health Informatics; Springer: Berlin/Heidelberg, Germany, 2014; pp. 236–239. [Google Scholar]
- Hughes, L.; Wang, X.; Chen, T. A review of protocol implementations and energy efficient cross-layer design for wireless body area networks. Sensors
**2012**, 12, 14730–14773. [Google Scholar] [CrossRef] [PubMed] - Chen, X.; Xu, Y.; Liu, A. Cross layer design for optimizing transmission reliability, energy efficiency, and lifetime in body sensor networks. Sensors
**2017**, 17, 900. [Google Scholar] [CrossRef] [PubMed] - Chen, M.; Gonzalez, S.; Vasilakos, A.; Cao, H.; Leung, V.C. Body Area Networks: A Survey. Mob. Netw. Appl.
**2011**, 16, 171–193. [Google Scholar] [CrossRef] - Waheed, M.; Ahmad, R.; Ahmed, W.; Drieberg, M.; Alam, M.M. Towards Efficient Wireless Body Area Network Using Two-Way Relay Cooperation. Sensors
**2018**, 18, 565. [Google Scholar] [CrossRef] - Roy, M.; Chowdhury, C.; Aslam, N. Designing an energy efficient WBAN routing protocol. In Proceedings of the International Conference on Communication Systems and Networks (COMSNETS), Bangalore, India, 4–8 January 2017. [Google Scholar]
- Lei, J.; Yates, R.; Greenstein, L. A generic model for optimizing single-hop transmission policy of replenishable sensors. IEEE Trans. Wirel. Commun.
**2009**, 8, 547–551. [Google Scholar] [CrossRef] - Seyedi, A.; Sikdar, B. Energy efficient transmission strategies for body sensor networks with energy harvesting. IEEE Trans. Commun.
**2010**, 58, 2116–2126. [Google Scholar] [CrossRef] - Michelusi, N.; Stamatiou, K.; Zorzi, M. On optimal transmission policies for energy harvesting devices. In Proceedings of the Information Theory and Applications Workshop (ITA), San Diego, CA, USA, 5–10 February 2012; pp. 249–254. [Google Scholar]
- Michelusi, N.; Stamatiou, K.; Zorzi, M. Transmission policies for energy harvesting sensors with time-correlated energy supply. IEEE Trans. Commun.
**2013**, 61, 2988–3001. [Google Scholar] [CrossRef] - Puterman, M.L. Markov dEcision Processes: Discrete Stochastic Dynamic Programming; John Wiley & Sons: Hoboken, NJ, USA, 2014. [Google Scholar]
- Monowar, M.M.; Bajaber, F. On designing thermal-aware localized QoS routing protocol for in-vivo sensor nodes in wireless body area networks. Sensors
**2015**, 15, 14016–14044. [Google Scholar] [CrossRef] [PubMed] - Chen, Y.M.; Peng, Y. Energy Efficient Fuzzy Routing Protocol in Wireless Body Area Networks. Int. J. Eng.
**2013**, 4, 59–63. [Google Scholar] - Liang, L.; Ge, Y.; Feng, G.; Ni, W.; Wai, A.A.P. Experimental study on adaptive power control based routing in multi-hop wireless body area networks. In Proceedings of the Global Communications Conference (GLOBECOM), Anaheim, CA, USA, 3–7 December 2012; pp. 572–577. [Google Scholar]
- Heinzelman, W.R.; Chandrakasan, A.; Balakrishnan, H. Energy-efficient communication protocol for wireless microsensor networks. In Proceedings of the 33rd Annual Hawaii International Conference On System Sciences, Maui, HI, USA, 7 January 2000. [Google Scholar]
- Younis, O.; Fahmy, S. HEED: A hybrid, energy-efficient, distributed clustering approach for ad hoc sensor networks. IEEE Trans. Mob. Comput.
**2004**, 3, 366–379. [Google Scholar] [CrossRef] - Liu, X. A survey on clustering routing protocols in wireless sensor networks. Sensors
**2012**, 12, 11113–11153. [Google Scholar] [CrossRef] [PubMed] - Arroyo-Valles, R.; Marques, A.G.; Cid-Sueiro, J. Optimal selective forwarding for energy saving in wireless sensor networks. IEEE Trans. Wirel. Commun.
**2011**, 10, 164–175. [Google Scholar] [CrossRef] - Li, M.; Reeves, J.; Moreno, C. Multi-level sample importance ranking based progressive transmission strategy for time series body sensor data. Comput. Netw.
**2018**, 136, 119–127. [Google Scholar] [CrossRef] - Biason, A.; Zorzi, M. Transmission policies for an energy harvesting device with a data queue. In Proceedings of the International Conference on Computing, Networking and Communications, Garden Grove, CA, USA, 16–19 February 2015; pp. 189–195. [Google Scholar]
- Sodhro, A.H.; Li, Y.; Shah, M.A. Energy-efficient adaptive transmission power control for wireless body area networks. IET Commun.
**2016**, 10, 81–90. [Google Scholar] [CrossRef] - Marcano, N.J.H.; Heide, J.; Lucani, D.E.; Fitzek, F.H. On Transmission Policies in Multihop Device-to-Device Communications with Network Coded Cooperation. In Proceedings of the European Wireless 2016, 22th European Wireless Conference, Oulu, Finland, 8–20 May 2016; pp. 1–5. [Google Scholar]
- Yi, C.; Wang, L.; Li, Y. Energy efficient transmission approach for WBAN based on threshold distance. IEEE Sens. J.
**2015**, 15, 5133–5141. [Google Scholar] [CrossRef] - Zhao, Z.; Huang, S.; Cai, J. Energy Efficient Packet Transmission Strategies for Wireless Body Area Networks with Rechargeable Sensors. In Proceedings of the 2017 IEEE 86th Vehicular Technology Conference (VTC-Fall), Toronto, ON, Canada, 24–27 September 2017; pp. 1–5. [Google Scholar]
- Orhan, O.; Erkip, E. Optimal transmission policies for energy harvesting two-hop networks. In Proceedings of the 46th Annual Conference on Information Sciences and Systems, Princeton, NJ, USA, 21–23 March 2012; pp. 1–6. [Google Scholar]
- Shaikh, F.K.; Zeadally, S. Energy harvesting in wireless sensor networks: A comprehensive review. Renew. Sustain. Energy Rev.
**2016**, 55, 1041–1054. [Google Scholar] [CrossRef] - Tutuncuoglu, K.; Yener, A. Optimum transmission policies for battery limited energy harvesting nodes. IEEE Trans. Wirel. Commun.
**2012**, 11, 1180–1189. [Google Scholar] [CrossRef] - Gupta, S.K.; Lalwani, S.; Prakash, Y.; Elsharawy, E.; Schwiebert, L. Towards a propagation model for wireless biomedical applications. IEEE Int. Conf. Commun.
**2003**, 3, 1993–1997. [Google Scholar] - Javaid, N.; Ahmad, A.; Nadeem, Q.; Imran, M.; Haider, N. iM-SIMPLE: iMproved stable increased-throughput multi-hop link efficient routing protocol for Wireless Body Area Networks. Comput. Hum. Behav.
**2015**, 51, 1003–1011. [Google Scholar] [CrossRef] - The R Project for Statistical Computing. Available online: https://www.r-project.org/ (accessed on 20 May 2017).
- Castalia. Available online: https://castalia.forge.nicta.com.au/index.php/en/ (accessed on 20 May 2017).
- Bettstetter, C. Smooth is better than sharp: A random mobility model for simulation of wireless networks. In Proceedings of the 4th ACM International Workshop on Modeling, Analysis and Simulation of Wireless and Mobile Systems, Rome, Italy, 15–20 July 2001; pp. 19–27. [Google Scholar]

**Figure 3.**Structure of State transition matrix (P) and corresponding reward matrix (R) for m system states subject to an action (a

_{t}).

**Figure 5.**Performance of MDP with varying discount factor for different combinations of probability values related to event generation P (p

_{on}, p

_{off}), PR (p

_{on}, p

_{off}) and link quality LQ (lq

_{on}, lq

_{off}).

**Figure 6.**Variation of resultant discounted utility with system state for different combinations of probability values related to remaining battery power ${L}_{t}$, event generation P (${p}_{on},{p}_{off}$), packet transmission rate ($p{r}_{on},p{r}_{off}$) and link quality ($l{q}_{on},l{q}_{off}$).

**Figure 8.**Mapping of MDP formulation into routing strategy. (

**a**) Variation of utility values with varying ${p}_{on},{p}_{off},p{r}_{on},p{r}_{off},l{q}_{on},l{q}_{off}$. (

**b**) Variation of data packets received by sink with varying ${p}_{on}$, ${p}_{off}$, $p{r}_{on}$, $p{r}_{off}$, $l{q}_{on}$, $l{q}_{off}$.

**Figure 9.**Obtaining threshold values for packet transmission rate ($P{R}_{th}$) and link quality ($L{Q}_{th}$).

**Figure 10.**Performance evaluation of the proposed routing strategy with respect to time. (

**a**) Variation of data packets received by sink with time following Line Mobility Model (LMM). (

**b**) Variation of energy consumption with varying time following LMM. (

**c**) Variation of data packets received by sink with time following Smooth Random Mobility Model (SRMM). (

**d**) Variation of energy consumption with varying time following SRMM.

**Figure 12.**Reliability assessment of the proposed routing strategy in terms of Packet Delivery Ratio (PDR) with respect to growing network size. (

**a**) Variation of PDR with growing network size following LMM. (

**b**) Variation of PDR with growing network size following SRMM.

Year | Existing Work | Network | Topology Used | Issues Handled | Mathematical Model Used | Input Conditions | Performance Metric |
---|---|---|---|---|---|---|---|

2009 | Generic model [11] | WSN | Single-hop | Energy replenishment | Markov model | Energy status | Battery capacity, reward rate |

2010 | Transmission strategies [12] | WBAN | Single-hop | Energy harvesting, energy efficiency, reliability | Markov model | Current energy level, state of data generation process, battery recharge, packet error probability | Quality of coverage |

2011 | Selective forwarding [22] | WSN | Multi-hop | Energy effiviency | Markov Decision Process | available battery of the node, the energy cost of retransmitting a message or the importance of messages | Suboptimal scheme and reduced computational cost |

2012 | Transmission policies [13] | WSN | Single-hop | Energy harvesting | Markov model | Current energy level of sensors, data importance | Transmission probability, energy level |

Routing protocol [18] | WBAN | Multi-hop | Energy efficiency, power control, transmission reliability, low overhead | CTP | Changing link quality, end to end delay, packet loss | Packet reception ratio, delay, energy consumption, energy balancing | |

2013 | Transmission policies [14] | WSN | Single-hop | Energy harvesting | Markov model | battery capacity, data transmission with a given energy cost | Asymptotic average reward as a function of SNR, transmission probability |

Routing protocol [17] | WBAN | Multi-hop | Energy efficiency, power control, lifetime | - | Distance of the receiver | Remaining energy | |

2015 | Transmission policies [24] | WSN | Energy harvesting | Markov model | Energy level, data queue | Buffer size and battery, large data buffer case, low complexity policy | |

Transmission approach [27] | WBAN | Single-hop | Energy efficiency | Circuit energy, transmission energy on distance | Energy consumption, recovery energy, transmission time, duty cycle | ||

2016 | Transmission policies [26] | WSN | Multi-hop | Energy efficiency | Network coding | No. of relay, recoding scheme and field size, Source and relay transmission | Mean transmission, medium access probability |

2017 | Transmission strategies [28] | WBAN | Energy efficiency | Discrete Markov Arrival Process | Channel state, battery state, no. of buffered packet in the system | ||

Optimizing transmission [7] | WBAN | Multi-hop | Transmission reliability, energy efficiency, lifetime, body movement | Signal to noise ratio, bit error rate | Transmission success rate, packet size, sensed data percent, burden packets per sec, transmission reliability, energy efficiency, energy consuming speed, energy balance degree, lifetime | ||

2018 | This study | WBAN | Multi-hop | Transmission power, energy efficiency, body movement, heat generation | Markov Decision Process | Energy level, event generation, packet transmission rate, link quality | Packet received by sink, consumed energy, packet delivery ratio, heating ratio |

Terms | Description |
---|---|

${X}_{t}$ | A finite set of states |

${A}_{t}$ | A finite set of actions $({a}_{t})$ to be taken |

P | Transition probability matrix, where the state transitions are given by $p({x}_{(t+1)}\mid {x}_{t},{a}_{t})=p({x}_{(t+1)}|{x}_{0}\dots {x}_{t},{a}_{0}\dots {a}_{t})$. This matrix plays the key role in finding the next state ${x}_{(t+1)}$ which is considered to be a possible consequence of performing an action $({a}_{t})$ in a state $({x}_{t})$. Hence it is depicted as a set of square matrices one for each action having both dimensions indexed by states. |

R | Reward matrix where each entry gives the immediate reward (or expected immediate reward) $r({x}_{t},{a}_{t})$ received for state transition from ${x}_{t}$ to ${x}_{(t+1)}$ performing action ${a}_{t}$. |

${\gamma}^{\prime}$ | [0,1] Discount factor denoting the importance of future reward in present reward. |

$\Pi (x)$ | A policy $\Pi $ gives an action for each state x, ${\Pi}^{*}(x)$ is optimal policy, i.e., the sequence of actions which maximizes expected utility if followed |

${U}_{dis}(x)$ | Expected discounted resultant utility value at each state obtained using value iteration process |

${P}_{tx}$ | Transmission power |

${p}_{on}$ | [0.5,1]Probability of occurrence of an event in next slot when there is an event in present slot |

${p}_{off}$ | [0.5,1]Probability of occurrence of no event in next slot when there is no event in present slot |

$p{r}_{on}$ | [0.5,1]Probability of exceeding maximum limit of packet transmission rate, $P{R}_{th}$ in next slot when $P{R}_{th}$ exceeded in current slot |

$p{r}_{off}$ | [0.5,1]Probability of not exceeding $P{R}_{th}$ in next slot when $P{R}_{th}$ not exceeded in current slot |

$l{q}_{on}$ | [0.5,1]Probability that indicates stable channel condition in next slot when link quality is above threshold ($LQ\phantom{\rule{4pt}{0ex}}>\phantom{\rule{4pt}{0ex}}L{Q}_{th}$) in present slot |

$l{q}_{off}$ | [0.5,1]Probability that indicates unstable channel condition in next slot when link quality is below threshold ($LQ\phantom{\rule{4pt}{0ex}}<\phantom{\rule{4pt}{0ex}}L{Q}_{th}$) in present slot |

${E}_{rem}$ | Remaining energy of a node |

Simulation Parameter | Default Value |
---|---|

Simulation area | $10\phantom{\rule{3.33333pt}{0ex}}\mathrm{m}\times 10\phantom{\rule{3.33333pt}{0ex}}\mathrm{m}$ |

Simulation time | 10,000 s |

Network size | 13 |

Mobility model | LineMobility model [35] |

MAC protocol | IEEE 802.15.4 |

Data generation rate | 14 packets/s |

$SA{R}_{th}$ | 0.3357 Watt/Kg [3] |

Tunable Parameters | Tunable Values |
---|---|

Phase I: | |

${\gamma}^{\prime}$ | 0.9 |

number of iterations | Up to 19 |

Phase II: | |

PR | 10 kbps to 125 kbps |

$P{R}_{th}$ | 50 |

$L{Q}_{th}$ | 100 |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Roy, M.; Chowdhury, C.; Aslam, N.
Designing Transmission Strategies for Enhancing Communications in Medical IoT Using Markov Decision Process. *Sensors* **2018**, *18*, 4450.
https://doi.org/10.3390/s18124450

**AMA Style**

Roy M, Chowdhury C, Aslam N.
Designing Transmission Strategies for Enhancing Communications in Medical IoT Using Markov Decision Process. *Sensors*. 2018; 18(12):4450.
https://doi.org/10.3390/s18124450

**Chicago/Turabian Style**

Roy, Moumita, Chandreyee Chowdhury, and Nauman Aslam.
2018. "Designing Transmission Strategies for Enhancing Communications in Medical IoT Using Markov Decision Process" *Sensors* 18, no. 12: 4450.
https://doi.org/10.3390/s18124450