Access Control in NB-IoT Networks: A Deep Reinforcement Learning Strategy
Abstract
:1. Introduction
- We propose a detailed and up-to-date state of the art.
- We describe a fluid model of IoT access in NB-IoT networks.
- We formulated the problem of IoT access as a Markov Decision Process (MDP).
- We design a specific reward function in order to guide the agent to improve the quality of the solutions.
- We provide an in-depth analysis of the problem.
2. State of the Art
2.1. Random Access Fundamentals
- The terminal transmits the selected preamble at the first RAO and sets a timer to receive the Random Access Response (RAR);
- If the preamble is well detected by the base station, the base station sends a RAR response carrying the synchronization advance and the allocated resource;
- The base station executes the contention resolution and sends the identity of the winning terminal in the contention resolution message. If the message doesn’t arrive at the terminal side, the terminal continues waiting until the timer expires;
- The terminal then sends a connection request, using the resource allocated to it, and re-arms a contention resolution timer. This request, named msg3, carries the identity of the terminal.
2.2. Related Work
3. A Fluid Model for IoT Device Access
- –
- is the number of backlogged devices at time t;
- –
- is the number of blocked devices waiting for a re-attempt at time t, after having failed an ACB check;
- –
- is the total number of devices from the different classes that pass the ACB check and wait to start Random Access (RA) attempt at time t;
- –
- is the number of blocked devices at time t after a failed RA attempt and waiting to try again;
- –
- is the arrival rate of devices. Different traffic patterns could be considered, depending on the type of IoT applications;
- –
- is the rate of ACB re-attempts;
- –
- is the rate of RA failure, which is equal to when is equal to 0 (see last item);
- –
- is the rate of RA re-attempts;
- –
- is the rate at which the devices abort the transmission after reaching the maximum number of RA attempts; in a correctly dimensioned system, we should have ;
- –
- p is the ACB factor.
- –
- , , and should be non negative,
- –
- , , , , and ,
- –
- .
4. Reinforcement Learning-Based Access Controller for IoT Devices
4.1. Problem Formulation
- State: Given the non-availability of the number of devices attempting the access at a given time k, the state we consider is based on the collected estimated values. Since a single measurement of this number is necessarily very noisy, we consider a series of several measurements, which we believe allow us to better reveal the current state of the network. The state is, thus, defined as the vector where H represents the measurement horizon.In our problem, k progresses according to the preambles’ arrival, whose frequency is constant.
- Action: At each step, the agent has to select the blocking factor p that will be considered by the IoT objects. This value is continuous and deterministic, in the problem we are considering, i.e., the same state will always give the same action .
- Revenue: This is a feedback signal received by the agent from the environment after the completion of an action. Thus, at step k, the agent obtains a revenue as a consequence of the action that was performed in the state . This revenue will allow the agent to be informed of the quality of the executed action. The objective of the agent is to maximize this revenue.Note that the maximum number of successes is equal to the number of available preambles N, so this metric could be used as a parameter for the calculation of the revenue, which is given by the following equation:
4.2. Arrival Regulation System
5. Performance Evaluation
6. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- Lin, X.; Adhikary, A.; Eric Wang, Y. Random Access Preamble Design and Detection for 3GPP Narrowband IoT Systems. IEEE Wirel. Commun. Lett. 2016, 5, 640–643. [Google Scholar] [CrossRef] [Green Version]
- Bouzouita, M.; Hadjadj-Aoult, Y.; Zangar, N.; Tabbane, S. On the risk of congestion collapse in heavily congested M2M networks. In Proceedings of the 2016 International Symposium on Networks, Computers and Communications (ISNCC), Yasmine Hammamet, Tunisia, 11–13 May 2016; pp. 1–5. [Google Scholar]
- 3GPP. RAN Improvements for Machine-Type Communications; Technical Report (TR) 37.868, 3rd Generation Partnership Project (3GPP), Version 11.0.0.; 2011. Available online: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2630 (accessed on 20 November 2020).
- Bouzouita, M.; Hadjadj-Aoul, Y.; Zangar, N.; Rubino, G. Estimating the number of contending IoT devices in 5G networks: Revealing the invisible. Trans. Emerg. Telecommun. Technol. 2019, 30, e3513. [Google Scholar] [CrossRef]
- Fujimoto, S.; van Hoof, H.; Meger, D. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018; pp. 1582–1591. [Google Scholar]
- 3GPP. Cellular System Support for Ultra-Low Complexity and Low Throughput Internet of Things (CIoT); Technical Report (TR) 45.820, 3rd Generation Partnership Project (3GPP), Version 13.1.0.; 2015. Available online: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2719 (accessed on 20 November 2020).
- ETSI. Evolved Universal Terrestrial Radio Access (E-UTRA); Physical channels and modulation; Protocol Specification (3GPP TS 36.211 version 16.3.0 Release 16), 2020. Available online: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspxspecificationId=2425 (accessed on 23 October 2020).
- ETSI. Evolved Universal Terrestrial Radio Access (E-UTRA); Medium Access Control (MAC) protocol Specification; Protocol specification (3GPP TS 36.321 version 16.2.0 Release 16), 2020. Available online: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2437 (accessed on 23 October 2020).
- ETSI. LTE; Evolved Universal Terrestrial Radio Access (E-UTRA); Radio Resource Control (RRC); Protocol Specification (3GPP TS 36.331 version 14.12.0 Release 14), 2019. Available online: https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2440 (accessed on 22 October 2020).
- Harwahyu, R.; Cheng, R.; Tsai, W.; Hwang, J.; Bianchi, G. Repetitions Versus Retransmissions: Tradeoff in Configuring NB-IoT Random Access Channels. IEEE Internet Things J. 2019, 6, 3796–3805. [Google Scholar] [CrossRef]
- Baracat, G.H.; Brito, J.M.C. NB-IoT Random Access Procedure Analysis. In Proceedings of the 2018 IEEE 10th Latin-American Conference on Communications (LATINCOM), Guadalajara, Mexico, 14–16 November 2018; pp. 1–6. [Google Scholar] [CrossRef]
- Jiang, N.; Deng, Y.; Condoluci, M.; Guo, W.; Nallanathan, A.; Dohler, M. RACH Preamble Repetition in NB-IoT Network. IEEE Commun. Lett. 2018, 22, 1244–1247. [Google Scholar] [CrossRef]
- Harwahyu, R.; Cheng, R.; Wei, C.; Sari, R.F. Optimization of Random Access Channel in NB-IoT. IEEE Internet Things J. 2018, 5, 391–402. [Google Scholar] [CrossRef]
- Sun, Y.; Tong, F.; Zhang, Z.; He, S. Throughput Modeling and Analysis of Random Access in Narrowband Internet of Things. IEEE Internet Things J. 2018, 5, 1485–1493. [Google Scholar] [CrossRef]
- Jeon, W.S.; Seo, S.B.; Jeong, D.G. Effective Frequency Hopping Pattern for ToA Estimation in NB-IoT Random Access. IEEE Trans. Veh. Technol. 2018, 67, 10150–10154. [Google Scholar] [CrossRef]
- Hwang, J.; Li, C.; Ma, C. Efficient Detection and Synchronization of Superimposed NB-IoT NPRACH Preambles. IEEE Internet Things J. 2019, 6, 1173–1182. [Google Scholar] [CrossRef]
- Zhang, J.; Xie, D.; Wang, X. TARA: An Efficient Random Access Mechanism for NB-IoT by Exploiting TA Value Difference in Collided Preambles. IEEE Trans. Mob. Comput. 2020, 1. [Google Scholar] [CrossRef]
- Ali, M.S.; Hossain, E.; Kim, D.I. LTE/LTE-A Random Access for Massive Machine-Type Communications in Smart Cities. IEEE Commun. Mag. 2017, 55, 76–83. [Google Scholar] [CrossRef] [Green Version]
- Toor, W.T.; Jin, H. Comparative study of access class barring and extended access barring for machine type communications. In Proceedings of the 2017 International Conference on Information and Communication Technology Convergence (ICTC), Jeju, Korea, 18–20 October 2017; pp. 604–609. [Google Scholar] [CrossRef]
- Park, J.; Lim, Y. Adaptive Access Class Barring Method for Machine Generated Communications. Mob. Inf. Syst. 2016, 2016. [Google Scholar] [CrossRef] [Green Version]
- Liu, J.; Agiwal, M.; Qu, M.; Jin, H. Online Control of Preamble Groups with Priority in Massive IoT Networks. IEEE J. Sel. Areas Commun. 2020, 1. [Google Scholar] [CrossRef]
- Jin, H.; Toor, W.T.; Jung, B.C.; Seo, J. Recursive Pseudo-Bayesian Access Class Barring for M2M Communications in LTE Systems. IEEE Trans. Veh. Technol. 2017, 66, 8595–8599. [Google Scholar] [CrossRef]
- Cheng, R.; Chen, J.; Chen, D.; Wei, C. Modeling and Analysis of an Extended Access Barring Algorithm for Machine-Type Communications in LTE-A Networks. IEEE Trans. Wirel. Commun. 2015, 14, 2956–2968. [Google Scholar] [CrossRef]
- Agiwal, M.; Maheshwari, M.K.; Jin, H. Power Efficient Random Access for Massive NB-IoT Connectivity. Sensors 2019, 19, 4944. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Bouzouita, M.; Hadjadj-Aoul, Y.; Zangar, N.; Rubino, G.; Tabbane, S. Multiple Access Class Barring Factors Algorithm for M2M Communications in LTE-Advanced Networks. In Proceedings of the 18th ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems (MSWiM ’15), Cancun, Mexico, 2–6 November 2015; Association for Computing Machinery: New York, NY, USA, 2015; pp. 195–199. [Google Scholar] [CrossRef]
- Rolnick, D.; Veit, A.; Belongie, S.J.; Shavit, N. Deep Learning is Robust to Massive Label Noise. arXiv 2017, arXiv:1705.10694. [Google Scholar]
- Fujimoto, S.; van Hoof, H.; Meger, D. Addressing Function Approximation Error in Actor-Critic Methods. arXiv 2018, arXiv:1802.09477. [Google Scholar]
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction, 2nd ed.; A Bradford Book: Cambridge, MA, USA, 2018. [Google Scholar]
- Thrun, S.; Schwartz, A. Issues in Using Function Approximation for Reinforcement Learning, 1993. Available online: https://www.ri.cmu.edu/pub_files/pub1/thrun_sebastian_1993_1/thrun_sebastian_1993_1.pdf (accessed on 20 November 2020).
- Bouzouita, M.; Hadjadj-Aoul, Y.; Zangar, N.; Tabbane, S.; Viho, C. Chapter 20—A random access model for M2M communications in LTE-advanced mobile networks. In Modeling and Simulation of Computer Networks and Systems; Obaidat, M.S., Nicopolitidis, P., Zarai, F., Eds.; Morgan Kaufmann: Boston, MA, USA, 2015; pp. 577–599. [Google Scholar] [CrossRef]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Hadjadj-Aoul, Y.; Ait-Chellouche, S. Access Control in NB-IoT Networks: A Deep Reinforcement Learning Strategy. Information 2020, 11, 541. https://doi.org/10.3390/info11110541
Hadjadj-Aoul Y, Ait-Chellouche S. Access Control in NB-IoT Networks: A Deep Reinforcement Learning Strategy. Information. 2020; 11(11):541. https://doi.org/10.3390/info11110541
Chicago/Turabian StyleHadjadj-Aoul, Yassine, and Soraya Ait-Chellouche. 2020. "Access Control in NB-IoT Networks: A Deep Reinforcement Learning Strategy" Information 11, no. 11: 541. https://doi.org/10.3390/info11110541