Hybrid Services Collaborative Resource Scheduling Strategy towards Artificial Intelligence of Things

Abstract

The increasing number of Artificial Intelligence of Things (AIoT) devices at the edge layer brings serious challenges to the traditional access network architecture, which results in a decrease in data transmission due to different QoS requirements. To improve the QoS of the URLLC service and mMTC service in the AIoT, a Hybrid Services Collaborative Resource Scheduling Strategy (HSCRS) is proposed. First, a multi-layer collaborative resource scheduling framework for the AIoT hybrid services is designed based on the F-RAN. Then, a throughput weighting model for hybrid services is constructed to analyze the throughput characteristics of mMTC service and URLLC service. Furthermore, a sub-channel allocation and power control method is designed to solve the resource scheduling strategy of hybrid services. Experimental results show that the proposed method can largely improve the network throughput performance.

1. Introduction

With the development of the Artificial Intelligence of Things (AIoT), massive devices are deployed at the edge of the network to provide support for various applications [1,2]. However, the increasing number of AIoT devices at the edge layer brings serious challenges to the traditional access network architecture. To meet the requirements of communications in diverse AIoT scenarios, a Fog Radio Access Network (F-RAN) is proposed, which focuses on localized services. In the F-RAN, the Fog Access Point (F-AP) is equipped with communication, computing, and storage capabilities, which can realize key data storage, end-to-end low-latency communication, and lightweight data processing. Therefore, the F-RAN has advantages in AIoT application scenarios [3,4]. However, AIoT services can become more diversified with the gradual increase in AIoT application scenarios at the network edge, and data interaction between devices is more frequent. Therefore, it can increase the possibility of different types of AIoT services being transmitted at the same time. However, different AIoT services have different requirements for communication, which makes network resource scheduling more difficult. Therefore, we focus on the resource scheduling problem for AIoT hybrid services in the F-RAN.

In the 5G era, the AIoT application scenarios are divided into three categories. Massive machine-type communications (mMTCs) and ultra-reliable and ultra-reliable low latency communications (URLLCs) are more relevant to AIoT services [5]. In our paper, the mMTC service mainly refers to the communication between AIoT devices and access nodes, which is oriented to latency-insensitive services. The URLLC service aims at the transmission of some security-related information and delay-sensitive services. Therefore, in the F-RAN, distributed F-APs can be flexibly deployed to meet the access requirements of different AIoT services within its coverage area. However, the URLLC service has high requirements for delay and reliability. When an access node needs to meet the co-transmission of the URLLC and mMTC services, the URLLC service will increase the transmission delay due to the reduction in the bandwidth resources obtained. The Quality of Service (QoS) requirements cannot be satisfied. Therefore, in the coexistence scenario of URLLC and mMTC services, it is important to schedule resources according to the QoS of different services for improving the network throughput and realizing the coordinated transmission of mixed services.

Therefore, to improve the URLLC service and mMTC service in the network, we propose a Hybrid Services Collaborative Resource Scheduling Strategy (HSCRS). Firstly, the throughput-weighted modeling is carried out based on the data characteristics of the URLLC service. Secondly, to achieve the goal of improving the network throughput, a multi-agent model for the coexistence scenario of URLLC and mMTC mixed services is constructed. The contributions are as follows:

• A multi-layer collaborative resource scheduling framework for the AIoT hybrid services is designed based on the F-RAN, and resource scheduling is performed based on the QoS requirements of different IoT service types.
• A throughput weighting model for hybrid services is constructed to analyze the throughput characteristics of the mMTC service and URLLC service in the AIoT.
• A sub-channel allocation and power control method is designed to solve the better resource scheduling strategy of AIoT hybrid services in complex environments. At the same time, a multi-agent model is constructed to improve the network throughput in a mixed service scenario.

The remainder of this paper is organized as follows. Section 2 introduces the related work. Section 3 designs the hybrid business system model. Section 4 proposes the throughput optimization strategy of the hybrid service. Section 5 is the experiment analysis. Section 6 concludes the paper.

2. Related Works

The research on the AIoT network throughput mainly focuses on two aspects: technical characteristics and scene characteristics. For the technical characteristics, the throughput performance under the current conditions is solved by setting specific constraints. For example, Hu et al. [6] studied the use of finite-length coding in QoS-constrained AIoT systems to support low-latency communication. For the scene characteristics, the network throughput is mainly improved by utilizing the advantageous resources in the scene. For example, Li et al. [7] studied the throughput problem of the AIoT in an ultra-dense network. Because the network capacity will cause huge energy consumption when deploying a large number of small base stations (BSs) in an ultra-dense network, we therefore use energy harvesting and energy cooperation technology to maximize the throughput of the system. Huang et al. [8] studied the throughput performance of the license-free multiple access mode in the AIoT. To address the sequence information conflict problem in the unauthorized multiple access mode, we propose a sequence selection scheme based on deep reinforcement learning to reduce the collision risk. Mankar et al. [9] studied the throughput performance of Device-to-Device (D2D) communication in cellular-based AIoT networks. In addition, AIoT devices often send latency-sensitive information to their nearest BS. Based on the above assumptions, the location of the AIoT device is modeled as a bipolar Poisson point process, and the BS location is modeled as another independent Poisson point process. When an AIoT device sends a status update to the BS, it is considered to control the transmission power based on the distance. Therefore, based on the available maximum transmission power, the AIoT devices covered by the BS can be determined and associated. Then, by deriving the average success probability of the D2D link, the average throughput of the AIoT network can be obtained. Xu et al. [10] studied the throughput performance of the network after introducing blockchain and edge computing in AIoT applications. In large-scale AIoT scenarios, blockchain technology was used for encryption to ensure the security of transmission. However, due to the limited throughput caused by blockchain technology, AIoT applications with frequent interactions cannot be supported. To meet AIoT application requirements and enhance the network throughput, we propose a comprehensive framework that integrates blockchain and edge computing based on spatially structured ledgers. The framework takes into account the collaboration between AIoT devices and edge servers and optimizes bandwidth and computing allocation strategies to improve the network throughput.

The F-RAN is divided into two types in terms of optimizing the network throughput performance in AIoT scenarios. The first type combines traditional F-RAN architectures with the characteristics of AIoT applications to improve the system throughput performance [11]. The second type combines fog computing with existing open-source frameworks on the infrastructure and leverages the programmable features of the framework to enable the software-based processing of multiple functions. Different types of AIoT services make the new fog architecture better suited for various types of AIoT services.

2.1. Throughput Optimization Strategy Based on Traditional Fog Architecture

In traditional fog architectures, a multitude of F-APs were dispersed at the network edge, and their service coverage in the area can fluctuate over time. To efficiently allocate resources among different F-APs and improve the network throughput, a distributed online optimization method was required. Ren et al. [12] proposed a fully distributed online learning method based on the non-uniform connection between AIoT devices and F-APs. This method constructed a graph for each time slot and business and used the stochastic gradient descent to decouple the optimal decision between each time slot, which can improve the network throughput.

However, the deployment of dense F-APs at the edge of the network exacerbated the degree of interference between adjacent F-APs as the network throughput increases. To solve the above problems, Kharel et al. [13] analyzed the change in the throughput performance when utilizing distributed F-APs for joint transmission in an F-RAN with limited interference. Experiment results showed that the F-AP joint transmission scheme significantly improves the transmission reliability and reduces transmission delays. However, the algorithm was disabled when dealing with strong interference sources, without analyzing the root cause of the interference.

To fully utilize the resources of each F-AP, some researchers proposed a multi-layer operation scheduling algorithm based on the fog architecture, which decomposed the AIoT service scheduling problem into two different operations: access control and resource scheduling [14]. Then, the Lyapunov optimization technique was used to analyze the trade-off relationship between the average network throughput and service delay in dynamic wireless networks based on fog architecture. Experimental results indicated that the multi-layer operation scheduling algorithm outperformed traditional algorithms in terms of the network throughput and service delay across various network scales and traffic load scenarios. The study [15] pointed out that the large quantity of data transmitted by the IoT devices resulted in the overhead of the bandwidth and an increased delay and large amounts of data transmission generated resource management issues and decreased the system’s throughput; thus, an optimized task scheduling and preemption (OSCAR) model was designed to overcome the limitations and improve the QoS.

2.2. Throughput Optimization Strategy Based on New Fog Architecture

Due to the diverse nature of AIoT application scenarios, optimizing methods based solely on the traditional fog architecture must redesign their optimization strategies to cater to the unique characteristics of each scenario. This can pose difficulties for researchers and hinder technological evolution. Therefore, the applicability of the fog architecture was improved by introducing a computing framework that promoted the further development of subsequent research. Three main computing frameworks were combined with fog architecture to optimize system throughput: Fog Flow, Kafka-ML, and Elastic Fog.

Fog Flow [16] is a cloud and edge computing platform that has been designed and implemented to meet the needs of AIoT service developers working in smart city scenarios. It enables developers to make elastic settings both in the cloud and at the edge to meet the requirements of different businesses. Additionally, Fog Flow supports standard interfaces that allow for the sharing and reuse of contextual data information between services. In contrast, Kafka-ML is an open-source framework that specializes in data streams for application scenarios. It can manage machine learning algorithms and AI pipelines deployed in scenarios and facilitate the deployment of DNNs in real-world applications. However, the traditional Kafka-ML cannot account for the deployment of hierarchical DNN models in a continuous structure from the cloud to the network edge. To address this problem, Carnero et al. [17] introduced fog computing into the Kafka-ML framework to support the deployment of distributed DNNs in a continuous structure from the cloud to the network edge. Experiment results demonstrated that the integration of fog computing with Kafka-ML can significantly enhance the system throughput and reduce the response time when compared to deploying DNNs solely in the cloud.

To further optimize the deployment of fog computing in different AIoT application scenarios, a container-based solution is an effective option for massive AIoT services. Currently, the Kubernetes platform provides powerful and flexible features and has been leveraged to manage containerized applications in fog computing. An implementation of the Elastic Fog framework [18], based on the Kubernetes platform, collects real-time network traffic information and combines it with the Kubernetes scheduler to provide an elastic resource configuration of applications for each fog node. Furthermore, Kubernetes supports various efficient allocation strategies to make resource allocation decisions for applications. Experimental results demonstrate that, in comparison to the default mechanism in Kubernetes, the Elastic Fog framework significantly optimizes the system throughput and latency.

Due to the fact that the joint optimization of virtual network function (VNF) embedding and flow scheduling faces several challenges of a differentiated QoS guarantee, the coupling and externality of VNF embedding, and route selection conflicts, in [19], a multi-timescale VNF embedding and flow scheduling algorithm named NEWS was proposed to maximize the throughput while reducing the VNF embedding cost and energy consumption. In [20], the issue that the lack of interconnection and collaboration between devices leaded to poor timeliness and security in IIoT resource scheduling was found, and an intelligent blockchain-enabled adaptive collaborative resource scheduling method was proposed. It introduced blockchain and AI technology to support dynamic resource scheduling in untrustworthy environments. In order to better guarantee diversified time-sensitive services for B5G/6G connected automated vehicles, literature [21] proposed a spectrum resource scheduling model for time-sensitive services and designed a two-tier joint resource scheduling method based on the age of information and bandwidth requirements under the macro base station and Road Side Unit. The Lagrange relaxation algorithm was utilized to obtain the optimal solution of spectrum resource allocation for enhancing the delay, throughput, and packet loss ratio performances.

3. Hybrid Business System Model

This paper focuses on the resource scheduling problem for URLLC and mMTC services in the AIoT. Given the presence of a security-oriented URLLC short data packet transmission service in the access scenario of massive AIoT devices, we employ Orthogonal Frequency Division Multiplexing (OFDM) to facilitate the collaborative transmission of URLLC and mMTC services. Figure 1 shows the architecture used, in which the F-AP collects data information from the previous cycle and estimates the channel state in the next cycle, before performing bandwidth resource scheduling based on the number of tasks transmitted by the mMTC service and URLLC service in the downlink. The coordinated transmission of mixed services is implemented to ensure that the QoS requirements of different services are met.

In our paper, the F-AP is responsible for transmitting H URLLC services, while there are also J mMTC services within the coverage area of the F-AP, facilitating communication between AIoT devices. Among these services, the URLLC service is provided with a higher transmission rate by connecting to the F-AP, whereas the mMTC service is primarily designed for the mutual transmission of information between specific AIoT devices. Since AIoT devices typically use a single antenna, we assume that all devices employ a single antenna in this scenario. Then, the URLLC service set in the scenario can be expressed as $H=\{0,\cdots ,h\}$, and the mMTC service set can be expressed as $J=\{0,\cdots ,j\}$.

3.1. Analysis for System SNR

OFDM can convert a wireless channel into a flat channel across different sub-carriers, with several consecutive sub-carriers forming a spectral sub-bandwidth. We assume that the fading of the channel is roughly the same within a sub-band and that different sub-bands are independent of each other. If, during a slot interval, the URLLC service shares the transmission sub-channel with the mMTC service, the URLLC service will experience interference from the AIoT device transmitter, which can interfere with the URLLC service. Based on this analysis, when the URLLC link h and the mMTC link j are on the same sub-channel, the Signal Interference Plus Noise Ratio (SINR) of the URLLC service is calculated as follows:

$$SIN{R}_{h,n}=\frac{P_n^h{g}_{h,n}}{{\displaystyle \sum _j^J}{\rho }_{j,n}{P}_n^j{g}_{j,n}+{\sigma }^2},$$

(1)

where $P_n^h$ is the power required to transmit the h-th URLLC service in the n-th bandwidth, $g_h$ denotes the channel coefficient, and $g_{j,n}$ is the interference coefficient caused by the transmission of the j-th mMTC service in n-th sub bandwidth. ${\rho }_{j,n}\in \{0,1\}$ represents the sub-bandwidth allocation control coefficient: when ${\rho }_{j,n}=1$, it denotes that the j-th mMTC service can use the n-th sub bandwidth for transmission, and ${\rho }_{j,n}=0$ represents that the n-th sub bandwidth cannot be used. ${\sigma }^2$ is the noise power spectral density. Similarly, the noise of the mMTC service mainly comes from the URLLC service in the same bandwidth. Then, when the j-th mMTC service uses the n-th bandwidth for transmission, the SINR of the mMTC service is calculated as follows:

$$SIN{R}_{j,n}=\frac{P_n^j{g}_{j,n}}{P_n^h{g}_{h,n}+{\displaystyle \sum _{j^{{}^{\prime }}\ne j}^J}{\rho }_{j^{{}^{\prime }},n}{P}_n^{j^{{}^{\prime }}}{g}_{j^{{}^{\prime }},n}+{\sigma }^2},$$

(2)

where $P_n^j$ is the power required to transmit the j-th mMTC service in the n-th bandwidth. ${\displaystyle \sum _{j^{{}^{\prime }}\ne j}^J}{\rho }_{j^{{}^{\prime }},n}{P}_n^{j^{{}^{\prime }}}{g}_{j^{{}^{\prime }},n}$ represents the noise impact of the remaining mMTC services on the current mMTC service in the n-th bandwidth. ${\rho }_{j^{{}^{\prime }},n}$ is the bandwidth allocation control coefficient for the remaining mMTC services. If ${\rho }_{j^{{}^{\prime }},n}=1$, it means that more mMTC services can be added to the current sub-bandwidth for transmission.

3.2. Analysis for Throughput in Mixed Services

The traditional channel capacity is calculated by Shannon’s theorem based on infinite code and the bit error rate $\epsilon \approx 0$. However, the bit error rate is non-zero and the code length of the channel transmission is limited in the actual environment. A longer code length can effectively improve the transmission rate [22]. However, there are many small data packets that need to be transmitted for AIoT services, and adding a significant amount of redundant information to each AIoT service to ensure high-reliability transmission would cause a considerable amount of useless information to be transmitted in the network. Therefore, the channel rate (bits/s/Hz) is obtained as follows:

$$\frac{1}{n}{log}_2{M}^{*}(n,\epsilon )\approx C-\sqrt{\frac{V}{n}}{Q}^{-1}\left(\epsilon \right),$$

(3)

where n is the length of the code. $C={log}_2(1+\gamma )$ is the unit channel capacity in the infinite code. $V=1-{(1+\gamma )}^{-2}$ is the channel dispersion, which is used to measure the random variability of the channel relative to the deterministic channel with the same capacity. $Q^{-1}(·)$ is the inverse of the function $Q\left(x\right)=\underset{x}{\overset{\infty }{\int }}\frac{1}{\sqrt{2\pi }}{e}^{-\frac{t^2}{2}}dt$.

In the OFDM system, $n_x$ is the length of the code, and the number of occupied symbols is positively correlated with the duration of the signal. When the total occupied bandwidth is B, the given channel code length for a transmission task can be seen as $N_x=BTx\in \{h,j\}$. T is the duration of one symbol during the service transmission. The system bandwidth is divided into multiple bandwidths with a size of $B_0$. In the system of URLLC service and mMTC service, when the number of symbols occupied by the URLLC service is $n_{x_1}$, the total traffic transmitted by the URLLC service can be approximately calculated as follows:

$$C_{URLLC}\approx n_{x_1}{B}_{0}TC-\sqrt{n_{x_1}{B}_{0}TV}{Q}^{-1}\left(\epsilon \right),$$

(4)

where mMTC service does not have high requirements for the delay; the transmission traffic can be increased by increasing the length of the code. When the number of symbols occupied by the mMTC service is $n_{x_2}$, the mMTC traffic transmitted during the signal duration can be calculated as follows:

$$C_{mMTC}=n_{x_2}{B}_{{}_0}TC,$$

(5)

3.3. Throughput Modeling

URLLC services are typically characterized by an end-to-end transmission delay of less than 1 ms and a bit error rate $\epsilon \le {10}^{-5}$. However, the transmission time interval in LTE-based systems is fixed at 1 ms, which falls short of meeting the ultra-low latency transmission requirements of URLLC services. To address this problem, the 5G system leverages flexible carrier spacing to reduce time slots and accommodate ultra-low latency services. The objective of this chapter is to increase the amount of data transmitted by the network URLLC and mMTC services in a hybrid service system. That is correct. By increasing the total system capacity to meet the low-latency and high-reliability requirements of the URLLC link, more mMTC service information can also be transmitted, thus increasing the overall system throughput. Currently, mMTC services are mainly latency-insensitive services. Therefore, a mMTC business admission control model is built as follows:

$$Pr\left\{{\displaystyle \sum _{n=1}^N}{\displaystyle \sum _{h=1}^H}{\rho }_{j,h}{C}_{j,t}\ge C_{mMTC,t}\right\},$$

(6)

where $C_{j,t}$ is the link capacity of the j-th mMTC service in the time slot t. $C_{mMTC,t}$ denotes the traffic volume that needs to be transmitted by the mMTC service in the time slot t. ${\rho }_{j,h}$ is the h-th URLLC service bandwidth allocation coefficient and j-th mMTC service when they are in the same sub-bandwidth.

In summary, the resource scheduling problem can be viewed as optimizing the bandwidth allocation strategy of the mMTC service to enhance the traffic volume of both URLLC and mMTC in the communication link. Therefore, the optimal bandwidth model in finite code is calculated as follows:

$$\begin{array}{c}\underset{h,j}{Max}{C}_{sum}=C_{URLLC}+C_{mMTC}\hfill \\ s.t.{\rho }_{j,h},{\rho }_{j,n},{\rho }_{j^{{}^{\prime }},n}\in \{0,1\}\forall j,j^{{}^{\prime }},n;\hfill \\ {\displaystyle \sum _h^H}{\displaystyle \sum _j^J}{n}_{x_1}+n_{x_2}\le n_{max};\hfill \\ {\displaystyle \sum _n}{P}_{h,n}\le P_{h,max},\forall h\in H;\hfill \\ {\displaystyle \sum _n}{P}_{j,n}\le P_{j,max},\forall j\in J;\hfill \end{array}$$

(7)

where $n_{max}=14$ is the maximum number of symbols that can be used in a time slot, according to the definition in the 5G system. $P_{h,n}$ is the transmission power of the h-th URLLC service in the n-th sub-bandwidth; $P_{h,max}$ is the maximum transmission power of all URLLC services in the total bandwidth. $P_{j,n}$ is the transmission power of the j-th mMTC service within the n-th bandwidth. $P_{j,max}$ is the maximum transmission power of all mMTC services within the total bandwidth.

4. Throughput Optimization Strategy of Hybrid Service

4.1. Multi-Agent Modeling in Hybrid Service

In a hybrid service coexistence system, both mMTC and URLLC services can be transmitted within the same bandwidth. However, due to the different transmission power and delay requirements of these services, the resource scheduling problem for different services can be modeled as a multi-agent reinforcement learning problem [23]. Reinforcement learning can be used to iteratively find better resource scheduling strategies in different environments and meet the transmission requirements of more services as much as possible. Each communication link between the URLLC service and the mMTC service is considered an agent. By collecting experience from different decisions made in various environments, an appropriate scheduling strategy can be selected for execution in the current environment state. When multiple services need to be transmitted simultaneously, multiple communication links form a multi-agent cluster to jointly explore the sub-bandwidth scheduling strategy and power control strategy in the current environment.

In the multi-agent reinforcement learning model, as shown in Figure 2, the decision-making process of each agent can be defined as $(S,A,P,r,d)$, where S is the state space set. A denotes the action space set. P represents the state transition probability, which means that in the current state $s_t$, if the action $a_t$ is taken, the probability of obtaining a new state is $s_{t+1}$. r is the reward value, because it is necessary to connect as many mMTCs as possible while trying to complete the URLLC service. d is the loss factor. It can obtain the corresponding reward value after each agent takes an action, then update the reinforcement learning model and gradually find better decisions. When the training of the model is completed, each agent selects an action based on the historical experience obtained from the trained model that can bring greater benefits. The details of the multi-agent model in the hybrid system of URLLC and mMTC are shown in Figure 2.

Agent: Each communication link in the network.

State: The network performance parameters mainly considered in this scenario include the status of sub-channels, remaining capacity of communication links, service tolerance for delay, and channel conditions in the sub-bandwidth. Therefore, the network state can be defined as $s=\{s_{state},s_{capacity},s_{time},s_{SNR}\}\in S$. $s_{state}$ is the sub-bandwidth occupation state. $s_{capacity}$ denotes the remaining capacity of the bandwidth. $s_{time}$ represents the service delay tolerance. $s_{SNR}$ is the channel signal-to-noise ratio.

Action: For the resource scheduling problem in the mixed service system, each agent can decide to utilize any sub-bandwidth for transmission. For the downlink transmission power of F-AP, we define the transmission power control of the mMTC service as $P_{mMTC}=\{50,\begin{array}{c}\end{array}100,\begin{array}{c}\end{array}150,\begin{array}{c}\end{array}200\}$ mW. Therefore, the dimension of the action space is $4\times H$, and each action corresponds to a sub-bandwidth and transmission power. When the number of communication links is large, the overall action space will become larger.

Reward: To optimize the resource scheduling in the network, a reward function must be established to obtain the optimal strategy. Each agent can adjust its scheduling strategy by receiving reward values from each decision, allowing it to approach the optimal decision. The goal of this chapter is to enhance the network throughput performance and the transmission success rate of mMTC services in mixed service scenarios within a given time interval. To achieve this, network performance parameters, such as the sub-channel status, remaining capacity of communication links, service tolerance delay, and channel conditions in the sub-bandwidth are taken into consideration.

Therefore, the reward function for the URLLC link in each iteration can be calculated by ${\displaystyle \sum _{h\in H}}{C}_{URLLC}$. To access as many mMTC services as possible, we set the reward function for the mMTC communication link agent as follows:

$$M_j=\left\{\begin{array}{c}{C}_{max}-{\mu }_1{C}_{URLLC}-{\mu }_2{C}_{mMTC},when C_{re-mMTC}\ge 0\hfill \\ \lambda ,otherwise\hfill \end{array}\right.$$

(8)

where $C_{max}$ is the total capacity of Shannon in the current bandwidth. ${\mu }_1\in \{0,1\}$ is the agent factor of the URLLC link, indicating whether there is any URLLC service that needs to be transmitted in the current sub-bandwidth. ${\mu }_2\in \{0,1\}$ is the agent factor of the mMTC link: when ${\mu }_1=1$, it indicates that there is URLLC service transmission within the bandwidth. Similarly, ${\mu }_2$ represents the transmission status of mMTC services. $C_{re-mMTC}$ is the remaining mMTC service volume. $\lambda$ denotes a hyperparameter used to adjust the willingness of the network to access mMTC services. Therefore, the reward function in each step t is calculated as follows:

$$r_{t+1}={\omega }_{URLLC}{\displaystyle \sum _{h\in H}}{C}_{URLLC,t}+{\omega }_{mMTC}{\displaystyle \sum _j^J}{M}_{j,t}$$

(9)

where ${\omega }_{URLLC}$ and ${\omega }_{mMTC}$ are weight coefficients used to adjust the transmission traffic volume of these two services.

4.2. Throughput Optimization Algorithm for Hybrid Service

In reinforcement learning, each agent chooses a strategy $\pi$ that maximizes the cumulative reward. Among them, the strategy $\pi$ refers to the probability distribution of the agent mapped to the action a when the agent is in the current state s. The cumulative discount function is usually used to represent the expected return in the strategy $\pi$, calculated as follows:

$$V_{\pi }(s,a)={\displaystyle \sum _t}{\xi }_t{r}_t(s_t,a_t)$$

(10)

where ${\xi }_t$ is the impairment rate. When ${\xi }_t$ tends to 1, it means that the agents in the network will obtain greater cumulative returns, and the transmission rate of the mMTC link will be improved. Therefore, when the remaining capacity in the communication link is not zero, the value is increased during the training process, encouraging each agent to access as many mMTC services as possible. Therefore, our model aims to find an optimization strategy, and its definition is as follows:

$$V_{\pi }^{*}(s,a)=\underset{\pi }{max}{V}_{\pi }(s,a)$$

(11)

When the model finds $V_{\pi }^{*}(s,a)$, it means that the optimization strategy in the current state can be obtained as follows:

$${\pi }^{*}\left(s_t\right)=arg\underset{a_t\in A}{max}{\displaystyle \sum _t}{r}_t(s_t,a_t)+{\displaystyle \sum _{s_{t+1}}}P\left(s_{t+1}\right|s_t,a_t)V_{\pi }^{*}(s_{t+1},a_{t+1})$$

(12)

To find the optimal strategy $V_{\pi }^{*}(s,a)$, an iterative algorithm can be used. However, in the actual environment, it is difficult to obtain the state transition probability BB due to the lack of prior knowledge $P\left(s_{t+1}\right|s_t,a_t)$. Therefore, we utilize the Deep Q-Network to deal with the lack of experience in unknown environments. Specifically, each agent has one DQN and takes the state space S as input. Then, it outputs the value function corresponding to all actions A. The DQN is trained through multiple iterations, and, in each iteration step, all agents adopt some soft policy to select the action with the largest estimate in the state–action space with probability $1-\epsilon$ and random action with probability $\epsilon$ [23]. The channel state and environment change when an agent takes an action. Each agent will collect and store the current state–action space, reward value, and state space in the experience pool L for the next step. In each iteration, part of the information will be extracted from the experience pool to update the parameter $\theta$ in the stochastic gradient descent method. Then, a fixed set of parameters will be obtained to reduce the error value. When the impairment factor is $\gamma$, the error function is calculated as follows:

$$\psi \left({\theta }_t\right)={\left[r_{t+1}(s_t,a_t)+\gamma \underset{a\in A}{max}{Q}_t(s_{t+1},a_{t+1},{\theta }_t^{{}^{\prime }})-Q_t(s_t,a_t,{\theta }_t)\right]}^2$$

(13)

In summary, the proposed HSCRS algorithm involves building each AIoT communication link into an agent, exploring the reward value of different strategies in unknown environments during iterative training, and learning to make up for any lack of experience. This is illustrated in Algorithm 1.

Algorithm 1 HSCRS.

Input:  DQN Architecture, Environment Model, Parameters of Agents. Output:  The trained DQN model. 1: for each iteration do 2:    Initialize all agent parameters, $Q(s,a)$, $\pi (s,a)$, $\theta$, L 3:    for each step t in each iteration do 4:      Each agent observes its state $s_t$ 5:      Select a random action $a_t$ with probability $\epsilon$ 6:      Execute the action and obtain the reward value according to (9), then get the new state $s_{t+1}$ 7:      Save information to experience pool L 8:    end for 9:    for each agent do 10:      Pull part of the information from the experience pool 11:      Update the error function according to (13) 12:    end for 13: end for

5. Experiment

5.1. Settings

We utilize PyCharm Professional 2020.3.2 as the simulation software for our experiments, running on the Ubuntu 18.04.6 operating system. In the multi-agent model, each agent’s DQN model consisted of three fully connected hidden layers, with RMSProp serving as the optimizer. The learning rate was set to 0.001, and the Q network of each agent was trained for 3000 iterations. To gradually decrease the exploration rate from 1 to 0.01, we used linear annealing, after which we kept it constant. Three baselines were used in our paper: Single Agent Reinforcement Learning (SARL), Random Access Control (RAC), and mMTC Priority (mMTCP). The SARL algorithm uses only one communication link as an agent and updates the state only through its reward value during the training process, ignoring the state of other links in the environment. The RAC algorithm randomly selects the bandwidth and transmission power for communication links. The mMTCP algorithm prioritizes the transmission of the mMTC service without considering the impact on the URLLC link. The mMTC traffic grows by $2^n\times 1024,\begin{array}{c}\end{array}n\in \{0,\begin{array}{c}\end{array}1,\cdots \}$ bytes. To ensure the validity of the URLLC link, we set the transmission power to its maximum value. The parameter settings used in the experiment are summarized in

Table 1. Parameter Settings.

Parameter	Value
Number of URLLC links	4
Number of mMTC links	4
Carrier frequency	2 GHz
Bandwidth	4 MHz
F-AP antenna gain	8 dBi
F-AP receiver noise coefficient	3 dB
AIoT devices antenna gain	1 dBi
AIoT devices receiver noise coefficient	5 dB
mMTC link power	[50, 100, 150, 200] mW
URLLC link power	200 mW
Noise power	−114 dBm
mMTC business volume	$2^n \times$ 1024 bytes
mMTC limitation of service transmission delay	100 ms
URLLC limitation of service transmission delay	1 ms [b]

.

5.2. Main Results

Figure 3 shows the URLLC link throughput under HSCRS, SARL, RAC, and mMTCP as the mMTC traffic volume increases. As can be observed from the figure, the throughput of the URLLC link decreases with increases in mMTC traffic. As the mMTC traffic increases, the required transmission delay also increases. To meet the delay limit, the transmission power of the mMTC link needs to be increased, causing greater interference to the URLLC link and resulting in a degraded throughput performance of the URLLC link. When the mMTC traffic volume is small, the performance of mMTCP is better than that of HSCRS. This is because the transmission delay with small mMTC traffic is also small and mMTC prioritizes mMTC traffic transmission, enabling it to complete the transmission task earlier. However, the transmission delay increases as the mMTC traffic increases, which leads to more interference to the URLLC link, reducing the throughput performance of the URLLC link. When the mMTC traffic is 256 Kbytes, the mMTCP performance is 8 percent lower than that of the HSCRS proposed in our paper.

Figure 4 illustrates the access success rates of HSCRS, SARL, RAC, and mMTCP algorithms for varying mMTC traffic levels. As shown in Figure 4, HSCRS, RAC, and mMTCP demonstrate a superior access performance when the mMTC traffic volume is low. At the same time, the access probability of SARL is low, because it refuses to access the mMTC service with strong interference to ensure the performance optimization of a single communication link. As the mMTC traffic increases, all algorithms experience a rapid deterioration in access performance. This is because the transmission delay increases, which makes it impossible to complete the transmission task within the delay limit.

Figure 5 compares the changes in the traffic volume of each communication link over time steps, for the HSCRS and RAC algorithms, when the traffic volume of each mMTC link is fixed at 4 KBytes. As can be seen from Figure 5a, the proposed HSCRS algorithm is able to rapidly transmit all mMTC services, in contrast to the RAC algorithm shown in Figure 5b, where only three mMTC links complete the transmission task. This is because all mMTC links compete for resources simultaneously in the RAC algorithm, resulting in fewer resources obtained by each communication link, which increases the transmission delay and slows down the initial traffic volume transmitted in the stage. As one of the mMTC communication links completes its transmission task, the remaining communication links can obtain more resources, resulting in a rapid increase in the traffic volume transmitted. However, due to the suboptimal resource scheduling in the early stage, excessive time is consumed, and one communication link is unable to complete the transmission of all traffic.

Figure 6 shows the transmission rate of each communication link. In the HSCRS, each mMTC link operates in a cooperative manner. When a communication link performs a transmission task, the other links transmit at a lower rate to reduce the transmission delay of that link. Moreover, in Figure 6b, each mMTC communication link competes to obtain resources, and while each link can obtain a certain amount of resources, the cooperative resource allocation scheme in HSCRS results in a better network performance.

6. Conclusions

This paper investigates the optimization of the throughput for hybrid services in the AIoT and proposes an HSCRS strategy based on multi-agent reinforcement learning. Firstly, the channel model in the coexistence scenario of URLLC and mMTC service is analyzed. Secondly, the network throughput of URLLC services is described under a limited code length, and a throughput weighting model is constructed. The proposed method models the throughput optimization problem as a multi-agent reinforcement learning model and iteratively finds the optimal policy in a complex environment. Experiment results show that HSCRS can effectively improve the network throughput performance for mixed services in the AIoT. However, this paper focuses on the throughput performance index of URLLC and mMTC mixed services and does not consider the low latency and high reliability of URLLC services. The latency and reliability of URLLC services are also important indicators. In the future, an AIoT multi-objective optimization resource scheduling strategy will be designed. Moreover, the current resource scheduling strategy assumes that the channel quality presents a certain regularity and does not consider the resource optimization scheduling problem of mixed services when the channel quality suddenly changes. In the future, we also plan to introduce a backup mechanism to improve the robustness performance of the resource scheduling strategy.