# Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

^{7}

^{8}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

#### 1.1. Related Work

#### 1.2. Paper Contributions

- We aim to relax/reformulate the HBS multi-objective optimization problem and obtain acceptable solutions (i.e., sub-optimal HBS decisions) in real time without prior channel knowledge using online learning techniques that easily handle blockage and energy consumption during the selection process.
- We reformulate the HBS optimization problem into a cost subsidy multi-armed bandit (CS-MAB) that accounts for the cost during selection in both exploitation and exploration terms.
- We propose CS—upper confidence bound (CSUCB-HBS) and CS—Thompson sampling (CSTS-HBS) algorithms and evaluate their performance compared with the ordinary MAB techniques (UCB and TS), and traditional HBS (i.e., conventional and random choice).
- Simulation results indicate the superior performance of our proposed CS-MAB techniques over classical MAB methods, especially CSTS-HBS, which exhibits better performance than CSUCB-HBS and others.

## 2. System Model

## 3. Problem Formulation

## 4. Envisioned CS-HBS Methods

#### 4.1. CSUCB-HBS Algorithm

#### 4.2. CSTS-HBS Algorithm

Algorithm 1: CSUCB/CSTS-HBS Algorithms |

## 5. Results

## 6. Conclusions and Outlook

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Acknowledgments

## Conflicts of Interest

## References

- Abuella, H.; Elamassie, M.; Uysal, M.; Xu, Z.; Serpedin, E.; Qaraqe, K.A.; Ekin, S. Hybrid RF/VLC Systems: A Comprehensive Survey on Network Topologies, Performance Analyses, Applications, and Future Directions. IEEE Access
**2021**, 9, 160402–160436. [Google Scholar] [CrossRef] - Chen, Y.; Ai, B.; Niu, Y.; He, R.; Zhong, Z.; Han, Z. Resource Allocation for Device-to-Device Communications in Multi-Cell Multi-Band Heterogeneous Cellular Networks. IEEE Trans. Veh. Technol.
**2019**, 68, 4760–4773. [Google Scholar] [CrossRef] [Green Version] - Mughal, B.; Fadlullah, Z.M.; Fouda, M.M.; Ikki, S. Allocation Schemes for Relay Communications: A Multiband Multichannel Approach Using Game Theory. IEEE Sens. Lett.
**2022**, 6, 7500104. [Google Scholar] [CrossRef] - Najla, M.; Mach, P.; Becvar, Z. Deep Learning for Selection Between RF and VLC Bands in Device-to-Device Communication. IEEE Wirel. Commun. Lett.
**2020**, 9, 1763–1767. [Google Scholar] [CrossRef] - Sakib, S.; Tazrin, T.; Fouda, M.M.; Fadlullah, Z.M.; Nasser, N. A Deep Learning Method for Predictive Channel Assignment in Beyond 5G Networks. IEEE Netw.
**2021**, 35, 266–272. [Google Scholar] [CrossRef] - Shrivastava, S.; Chen, B.; Chen, C.; Wang, H.; Dai, M. Deep Q-Network Learning Based Downlink Resource Allocation for Hybrid RF/VLC Systems. IEEE Access
**2020**, 8, 149412–149434. [Google Scholar] [CrossRef] - Sakib, S.; Tazrin, T.; Fouda, M.M.; Fadlullah, Z.M.; Nasser, N. An Efficient and Light-weight Predictive Channel Assignment Scheme for Multi-Band B5G Enabled Massive IoT: A Deep Learning Approach. IEEE Internet Things J.
**2021**, 8, 5285–5297. [Google Scholar] [CrossRef] - Bakri, S.; Brik, B.; Ksentini, A. On using reinforcement learning for network slice admission control in 5G: Offline vs. online. Int. J. Commun. Syst.
**2021**, 34, 1987–2007. [Google Scholar] [CrossRef] - Nasir, Y.S.; Guo, D. Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks. IEEE J. Sel. Areas Commun.
**2019**, 37, 2239–2250. [Google Scholar] [CrossRef] [Green Version] - Wang, S.; Lv, T. Dynamic Multichannel Access for 5G and Beyond with Fast Time-Varying Channel. In Proceedings of the ICC 2020—2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 7–11 June 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Wang, Z.; Zhang, T.; Liu, Y.; Xu, W. Caching Placement and Resource Allocation for AR Application in UAV NOMA Networks. In Proceedings of the GLOBECOM 2020—2020 IEEE Global Communications Conference, Taipei, Taiwan, 7–11 December 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Hashima, S.; Fadlullah, Z.M.; Fouda, M.M.; Mohamed, E.M.; Hatano, K.; ElHalawany, B.M.; Guizani, M. On Softwarization of Intelligence in 6G Networks for Ultra-Fast Optimal Policy Selection: Challenges and Opportunities. IEEE Netw.
**2022**, 1–9. [Google Scholar] [CrossRef] - Yin, R.; Wu, Z.; Liu, S.; Wu, C.; Yuan, J.; Chen, X. Decentralized Radio Resource Adaptation in D2D-U Networks. IEEE Internet Things J.
**2021**, 8, 6720–6732. [Google Scholar] [CrossRef] - Du, Z.; Wang, C.; Sun, Y.; Wu, G. Context-Aware Indoor VLC/RF Heterogeneous Network Selection: Reinforcement Learning With Knowledge Transfer. IEEE Access
**2018**, 6, 33275–33284. [Google Scholar] [CrossRef] - Wang, C.; Wu, G.; Du, Z.; Jiang, B. Reinforcement learning based network selection for hybrid VLC and RF systems. In MATEC Web of Conferences; EDP Sciences: Les Ulis, France, 2018; Volume 173, p. 03014. [Google Scholar] [CrossRef] [Green Version]
- Lattimore, T. Bandit Algorithms; Cambridge University Press: Cambridge, UK, 2020. [Google Scholar]
- Hashima, S.; ElHalawany, B.M.; Hatano, K.; Wu, K.; Mohamed, E.M. Leveraging Machine-Learning for D2D Communications in 5G/Beyond 5G Networks. Electronics
**2021**, 10, 169. [Google Scholar] [CrossRef] - Hashima, S.; Hatano, K.; Takimoto, E.; Mohamed, E.M. Neighbor Discovery and Selection in Millimeter Wave D2D Networks Using Stochastic MAB. IEEE Commun. Lett.
**2020**, 24, 1840–1844. [Google Scholar] [CrossRef] - Hashima, S.; Hatano, K.; Kasban, H.; Mohamed, E.M. Wi-Fi Assisted Contextual Multi-Armed Bandit for Neighbor Discovery and Selection in Millimeter Wave Device to Device Communications. Sensors
**2021**, 21, 2835. [Google Scholar] [CrossRef] - Hashima, S.; Hatano, K.; Takimoto, E.; Mohamed, E.M. Minimax Optimal Stochastic Strategy (MOSS) For Neighbor Discovery and Selection In Millimeter Wave D2D Networks. In Proceedings of the 2020 23rd International Symposium on Wireless Personal Multimedia Communications (WPMC), Okayama, Japan, 19–26 October 2020; pp. 1–6. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Hatano, K.; Aldossari, S.A.; Zareei, M.; Rihan, M. Two-Hop Relay Probing in WiGig Device-to-Device Networks Using Sleeping Contextual Bandits. IEEE Wirel. Commun. Lett.
**2021**, 10, 1581–1585. [Google Scholar] [CrossRef] - Mohamed, E.M.; Hashima, S.; Aldosary, A.; Hatano, K.; Abdelghany, M.A. Gateway Selection in Millimeter Wave UAV Wireless Networks Using Multi-Player Multi-Armed Bandit. Sensors
**2020**, 20, 3947. [Google Scholar] [CrossRef] - Hashima, S.; Mohamed, E.M.; Hatano, K.; Takimoto, E. WiGig Wireless Sensor Selection Using Sophisticated Multi Armed Bandit Schemes. In Proceedings of the 2021 Thirteenth International Conference on Mobile Computing and Ubiquitous Network (ICMU), Tokyo, Japan, 17–19 November 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Barrachina-Muñoz, S.; Chiumento, A.; Bellalta, B. Multi-Armed Bandits for Spectrum Allocation in Multi-Agent Channel Bonding WLANs. IEEE Access
**2021**, 9, 133472–133490. [Google Scholar] [CrossRef] - Zuo, J.; Joe-Wong, C. Combinatorial Multi-armed Bandits for Resource Allocation. In Proceedings of the 2021 55th Annual Conference on Information Sciences and Systems (CISS), Baltimore, MD, USA, 24–26 March 2021. [Google Scholar] [CrossRef]
- Mohamed, E.M.; Hashima, S.; Hatano, K. Energy Aware Multi-Armed Bandit for Millimeter Wave Based UAV Mounted RIS Networks. IEEE Wirel. Commun. Lett.
**2022**. [Google Scholar] [CrossRef] - Mohamed, E.M.; Hashima, S.; Hatano, K.; Aldossari, S.A. Two-Stage Multiarmed Bandit for Reconfigurable Intelligent Surface Aided Millimeter Wave Communications. Sensors
**2022**, 22, 2179. [Google Scholar] [CrossRef] - Mohamed, E.M.; Hashima, S.; Hatano, K.; Kasban, H.; Rihan, M. Millimeter-Wave Concurrent Beamforming: A Multi-Player Multi-Armed Bandit Approach. Comput. Mater. Contin.
**2020**, 65, 1987–2007. [Google Scholar] [CrossRef] - ElHalawany, B.M.; Hashima, S.; Hatano, K.; Wu, K.; Mohamed, E.M. Leveraging Machine Learning for Millimeter Wave Beamforming in Beyond 5G Networks. IEEE Syst. J.
**2021**, 1–12. [Google Scholar] [CrossRef] - Hashima, S.; Hatano, K.; Kasban, H.; Rihan, M.; Mohamed, E.M. Multiagent Multi-Armed Bandit Techniques for Millimeter Wave Concurrent Beamforming. In Proceedings of the 2020 8th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC), Virtual, 14–15 December 2020; pp. 56–59. [Google Scholar] [CrossRef]
- Fouda, M.; Hashima, S.; Sakib, S.; Fadlullah, Z.; Hatano, K.; Shen, X. Optimal Channel Selection in Hybrid RF/VLC Networks: A Multi-Armed Bandit Approach. IEEE Trans. Veh. Technol.
**2022**. [Google Scholar] [CrossRef] - Hashima, S.; Fouda, M.M.; Sakib, S.; Fadlullah, Z.M.; Hatano, K.; Mohamed, E.M.; Shen, X. Energy-Aware Hybrid RF-VLC Multi-Band Selection in D2D Communication: A Stochastic Multi-Armed Bandit Approach. IEEE Internet Things J.
**2022**. [Google Scholar] [CrossRef] - Hashima, S.; Fouda, M.M.; Fadlullah, Z.M.; Mohamed, E.M.; Hatano, K. Improved UCB-based Energy-Efficient Channel Selection in Hybrid-Band Wireless Communication. In Proceedings of the 2021 IEEE Global Communications Conference (GLOBECOM), Madrid, Spain, 7–11 December 2021; pp. 1–6. [Google Scholar] [CrossRef]
- Wei, N.; Lin, X.; Zhang, Z. Optimal Relay Probing in Millimeter-Wave Cellular Systems with Device-to-Device Relaying. IEEE Trans. Veh. Technol.
**2016**, 65, 10218–10222. [Google Scholar] [CrossRef] [Green Version] - Boban, M.; Dupleich, D.; Iqbal, N.; Luo, J.; Schneider, C.; Müller, R.; Yu, Z.; Steer, D.; Jämsä, T.; Li, J.; et al. Multi-Band Vehicle-to-Vehicle Channel Characterization in the Presence of Vehicle Blockage. IEEE Access
**2019**, 7, 9724–9735. [Google Scholar] [CrossRef]

**Figure 1.**Hybrid band system model: How to self optimize hybrid channels in fluctuating channel conditions (distance, energy level, and blocking?).

**Figure 3.**Average throughput comparison of CSTS/CSUCB-HBS approaches vs. separation distances at distinct blocking layouts. (

**a**) No blockage. (

**b**) Small blockage. (

**c**) Large blockage.

**Figure 6.**Energy efficiency performance of CSTS/CSUCB-HBS approaches vs. separation distances at distinct blocking layouts. (

**a**) No blockage. (

**b**) Small blockage. (

**c**) Large blockage.

Simulation Parameters | Value | |
---|---|---|

Number of channels | 4 (WiFi 2.4 GHz, 5.25 GHz, WiGig 38 GHz, VLC ${10}^{5}$ GHz) | |

T, ${L}_{D}$, ${E}_{th}$ | 1000, 1 TB, 1% | |

Operating frequencies of each channel | 5.25, 2.4, 38, ${10}^{5}$ GHz | |

$BW$ | 40, 20, 40, 20 MHz | |

r | {10–100} m | |

Blocking model [35] | Small blocker: {length, width, height} | {5.07, 1.69, 1.93} m |

Large blocker: {length, width, height} | {7.01, 2.04, 2.63} m |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Hashima, S.; Hatano, K.; Fouda, M.M.; Fadlullah, Z.M.; Mohamed, E.M.
Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks. *Electronics* **2022**, *11*, 1782.
https://doi.org/10.3390/electronics11111782

**AMA Style**

Hashima S, Hatano K, Fouda MM, Fadlullah ZM, Mohamed EM.
Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks. *Electronics*. 2022; 11(11):1782.
https://doi.org/10.3390/electronics11111782

**Chicago/Turabian Style**

Hashima, Sherief, Kohei Hatano, Mostafa M. Fouda, Zubair M. Fadlullah, and Ehab Mahmoud Mohamed.
2022. "Cost-Aware Bandits for Efficient Channel Selection in Hybrid Band Networks" *Electronics* 11, no. 11: 1782.
https://doi.org/10.3390/electronics11111782