Subchannel Allocation in Massive Multiple-Input Multiple-Output Orthogonal Frequency-Division Multiple Access and Hybrid Beamforming Systems with Deep Reinforcement Learning ^†

Abstract

In this study, we emphasize that the maximum sum rate can be achieved through AI-based subchannel allocation, while taking into account all users’ quality of service (QoS) requirements in data rates for hybrid beamforming systems. We assume a limited number of radio frequency (RF) chains in practical hybrid beamforming architectures. This constraint makes subchannel allocation a critical aspect of hybrid beamforming in massive multiple-input multiple-output (MIMO) systems with orthogonal frequency division multiple access (MIMO-OFDMA), as it enables the system to serve more users within a single time slot. Unlike conventional subcarrier allocation methods, we employ a deep reinforcement learning (DRL)-based algorithm to address real-time decision-making challenges. Specifically, we propose a dueling double deep Q-network (Dueling-DDQN) to implement dynamic subchannel allocation. Simulation results demonstrate that the performance of the proposed algorithm gradually approaches that of the greedy method. Furthermore, both the average sum rate and the average spectral efficiency per user improve with a reasonable variation in outage probability.

1. Introduction

To meet users’ requirements, millimeter-wave (mmWave) technology, operating in the 30 to 300 GHz frequency band, has emerged as a promising candidate and has been widely discussed in 5G communication systems [1,2]. However, due to its high carrier frequency, mmWave signals suffer from severe path loss. Fortunately, the short wavelength of mmWave allows for dense antenna array configurations within a limited physical space. This enables significant beamforming gain through large antenna arrays, which can compensate for the high path loss [3,4,5], making massive multiple-input multiple-output (MIMO) an attractive solution for mmWave systems.

Traditional MIMO systems typically employ a fully digital architecture at the baseband for beamforming. While effective, this approach is power-hungry and costly, particaularly for massive MIMO and mmWave systems, as it requires a dedicated radio frequency (RF) chain for each antenna element [6]. To address this issue, hybrid beamforming architectures have been proposed [7,8].

As previously mentioned, the limited bandwidth must be shared among multiple users. To enhance data rates in wireless communication, orthogonal frequency division multiple access (OFDMA) combined with space division multiple access (SDMA) has been proposed [9]. OFDMA with hybrid beamforming systems can select the optimal set of users to serve within the same resource block (RB) and adjust beamforming weight vectors to minimize interference among users sharing the same subcarrier.

Undoubtedly, optimizing resource allocation across frequency, space, and time is a highly complex problem. To tackle this, various AI-based approaches have been explored. However, implementing real-time subchannel allocation in complex communication systems remains an open challenge. Motivated by the capabilities of AI, we propose an AI-based approach to solve the dynamic subchannel allocation problem. Specifically, we employ a deep reinforcement learning (DRL) algorithm to handle subchannel scheduling in a massive MIMO-OFDMA system with hybrid beamforming, aiming to maximize the overall data rate while also satisfying each user’s quality of service (QoS) requirements. Nevertheless, designing an appropriate network architecture and tuning hyperparameters to suit the specific scenario remains a significant challenge.

2. System Model

In the downlink of an mmWave and large-scale MIMO-OFDMA system with a fully connected hybrid beamforming design for a frequency-selective channel, we consider that base station (BS) equips $N_{\mathrm{t}}$ antennas and $N_{\mathrm{t}\mathrm{R}\mathrm{F}}$ RF chains. BS transmits $N_{\mathrm{t}\mathrm{s}}$ data streams simultaneously at each subcarrier [10,11]. The total number of subcarriers is denoted by $K$. Moreover, a total $N$ of users are served in each time slot, with each user equipped with a single antenna. The number of RF chains cannot exceed the number of transceiver antennas, i.e., $N_{\mathrm{t}\mathrm{s}}\le N_{\mathrm{t}\mathrm{R}\mathrm{F}}\le N_{\mathrm{t}}$, and $N_{\mathrm{r}\mathrm{s}}\le N_{\mathrm{r}\mathrm{R}\mathrm{F}}\le N_{\mathrm{r}}$. We assume that the number of scheduled users $U$ per resource block (RB) is equal to $N_{\mathrm{t}\mathrm{R}\mathrm{F}}$, and these users are selected from the total user set $N$, i.e., $U=N_{\mathrm{t}\mathrm{R}\mathrm{F}}<N$. Let $\left\{{\mathcal{F}}_b\right\}$ denote the set of users scheduled in the $b^{\mathrm{t}\mathrm{h}}$ RB, where $b=1,\dots ,B$, and $\left|{\mathcal{F}}_b\right|$ is $U$. Furthermore, all N users must be served in at least one RB per time slot. Therefore, in our simulation setup, we assume that $U=N_{\mathrm{t}\mathrm{s}}= N_{\mathrm{t}\mathrm{R}\mathrm{F}}\ll N_{\mathrm{t}}$, with the hybrid beamforming structure employed at the transmitter and single-antenna multi-user terminals at the receiver.

Although subchannel allocation is implemented in units of RBs indexed by k, the digital precoder is designed for each subcarrier. At each subcarrier k, where $k=1,\dots ,K$, $N_{\mathrm{t}\mathrm{s}}$ data symbols $\mathbf{s}\left[k\right]\in {\mathbb{C}}^{N_{\mathrm{t}\mathrm{s}}\times 1}$ are first processed by a low-dimensional digital precoder ${\mathbf{F}}_{\mathrm{B}\mathrm{B}}\left[k\right]\in {\mathbb{C}}^{N_{\mathrm{t}\mathrm{R}\mathrm{F}}\times N_{\mathrm{t}\mathrm{s}}}$. The digitally precoded signals are then transformed into the time domain using $N_{\mathrm{t}\mathrm{R}\mathrm{F}}$ $K$-point inverse fast Fourier transforms (IFFTs). After this, cyclic prefixes (CPs) are added to each stream to mitigate inter-symbol interference (ISI). Following the CP addition, the signal is passed through a high-dimensional analog precoder ${\mathbf{F}}_{\mathrm{R}\mathrm{F}}\in {\mathbb{C}}^{N_{\mathrm{t}}\times N_{\mathrm{t}\mathrm{R}\mathrm{F}}}$. In a fully connected structure, each RF chain is connected to all antennas via phase shifters. Since the analog precoder operates in the time domain, it remains identical across all subcarriers. Finally, the transmitted signal at subcarrier k is expressed as:

$$\mathbf{x}\left[k\right]={\mathbf{F}}_{\mathrm{R}\mathrm{F}}{\mathbf{F}}_{\mathrm{B}\mathrm{B}}\left[k\right]\mathbf{s}\left[k\right]=\sum _{u=1}^U{\mathbf{F}}_{\mathrm{R}\mathrm{F}}{\mathbf{f}}_{{\mathrm{B}\mathrm{B}}_u}\left[k\right]{\mathrm{s}}_u\left[k\right]$$

(1)

where $\mathbf{s}\left[k\right]={\left[{\mathrm{s}}_1\left[k\right],{\mathrm{s}}_2\left[k\right]\dots , {\mathrm{s}}_U\left[k\right]\right]}^T$ represent the combined data symbols for $U$ served users at the $k$ subcarrier/RB. We assume that the total transmit power $P$ is equally divided among each symbol as $\mathbb{E}\left[\mathbf{s}\left[k\right]{\mathbf{s}}^H\left[k\right]\right]={\displaystyle \frac{P}{UK}}{\mathbf{I}}_{N_{\mathrm{s}}}$; ${\mathrm{s}}_u\left[k\right]\in \mathbb{C}$ is the data symbol of user $u$ in $\left\{{\mathcal{F}}_b\right\}$ at the subcarrier/RB $k$. Next, ${\mathbf{F}}_{\mathrm{B}\mathrm{B}}\left[k\right]=\left[{\mathbf{f}}_{{\mathrm{B}\mathrm{B}}_1}\left[k\right],{\mathbf{f}}_{{\mathrm{B}\mathrm{B}}_2}\left[k\right],\dots ,{\mathbf{f}}_{{\mathrm{B}\mathrm{B}}_U}\left[k\right]\right]$ is the digital precoder matrix at the $k$ subcarrier/RB for $U$ served users, and ${\mathbf{f}}_{{\mathrm{B}\mathrm{B}}_u}\left[k\right]\in {\mathbb{C}}^{N_{\mathrm{t}\mathrm{R}\mathrm{F}}\times 1}$ is the digital precoder for the user $u$. The received signal for the $n^{\mathrm{t}\mathrm{h}}$ user at the $k$ subcarrier/RB is given by

$${\mathrm{y}}_n\left[k\right]={\mathbf{h}}_n^H\left[k\right]\mathbf{x}\left[k\right]+{\mathrm{z}}_n\left[k\right]$$

(2)

where ${\mathrm{z}}_n\left[k\right]~\mathcal{C}\mathcal{N}\left(0,{\sigma }^2\right)$ and ${\mathbf{h}}_n^H\left[k\right]\in {\mathbb{C}}^{1\times N_{\mathrm{t}}}$ are the additive white Gaussian noise (AWGN) and the channel vector at the $k$ subcarrier for user $n$, respectively.

The main goal of this paper is to maximize the sum rate while considering the Quality of Service (QoS) for each user’s data rate through dynamic subcarrier/RB allocation. The problem formulation can be expressed as:

$$\underset{{\rho }_{n,k},{ \mathbf{F}}_{\mathrm{RF}},{ \mathbf{F}}_{\mathrm{BB}}\left[k\right]}{\mathbf{max}}\sum _{k=1}^K\sum _{n=1}^N{R}_n\left[k\right]$$

(3a)

$${\mathbf{a}}_r^H{\mathbf{a}}_r=1, \forall r=1,\dots ,N_{\mathrm{t}\mathrm{R}\mathrm{F}};$$

(3b)

$${‖{\mathbf{F}}_{\mathrm{R}\mathrm{F}}{\mathbf{f}}_{{\mathrm{B}\mathrm{B}}_n}\left[k\right]‖}^2=1, \forall k=1,\dots ,K, \forall n=1,\dots ,N;$$

(3c)

$$\mathrm{\Omega }\le \sum _{k=1}^K{R}_n\left[k\right],\forall n=1,\dots , N;$$

(3d)

where (3b) represents the hardware constraint ensuring that each column of the analog precoder has unit norm, and ${\mathbf{F}}_{\mathrm{R}\mathrm{F}}=\left[{\mathbf{a}}_1,{\mathbf{a}}_2,\dots ,{\mathbf{a}}_{N_{\mathrm{t}\mathrm{R}\mathrm{F}}}\right]$. Equation (3c) is the transmit power budget constraint for each subcarrier and user, and $\mathrm{\Omega }$ is the data rate constraint for each user in (3d). In (3a), it is important to note that ${ \mathbf{F}}_{\mathrm{R}\mathrm{F}}$ and ${ \mathbf{F}}_{\mathrm{B}\mathrm{B}}\left[k\right]$ are designed as in [12]. Furthermore, $R_n\left[k\right]$ denotes the data rate for user $n$ at subcarrier $k$, and the rate can be expressed as

$$R_n\left[k\right]=B_{\mathrm{S}\mathrm{C}\mathrm{S}}{\log }_2\left(1+{\mathrm{S}\mathrm{I}\mathrm{N}\mathrm{R}}_n\left[k\right]\right)$$

(4)

where $B_{\mathrm{S}\mathrm{C}\mathrm{S}}$ is the subcarrier spacing (SCS). In the subchannel allocation strategy for maximizing the sum rate under deep fading, we define the outage probability as follows.

$$p_{\mathrm{o}\mathrm{u}\mathrm{t}}(t)=\mathrm{P}\mathrm{r}\mathrm{o}\mathrm{b}\{N_{\mathrm{o}\mathrm{u}\mathrm{t}}(t)\ge 1\}$$

(5)

where $N_{\mathrm{o}\mathrm{u}\mathrm{t}}(t)$ is the number of users whose achievable data rates fall below the target constraint $\mathrm{\Omega }$ at the slot $t$.

3. Dynamic Subchannel Allocation with DRL-Based Method

We define the states, actions, rewards, and next states for our proposed method. The downlink transmissions are organized into frames of 10 ms, and each frame contains 10 subframes [13].

• States ($s_t$): In the proposed algorithm, $s_t$ is a set that combines different meaningful vectors or scalars, with time-variant feature data as the input to the network. We define $s_t=\left\{\mathit{p}\mathit{l}\left(t\right),\mathit{g}\left(t\right),\mathit{d}\mathit{r}\left(t\right),\mathit{d}\left(t\right),o_t,a_t,{sr}_t\right\}$, where $\mathit{p}\mathit{l}\left(t\right)=\left[1/\sqrt{{pl}_1(t)},\dots ,1/\sqrt{{pl}_n(t)},\dots ,1/\sqrt{{pl}_N(t)}\right]$, and ${pl}_n\left(t\right)$ represents the path loss of the $n^{\mathrm{t}\mathrm{h}}$ user at the $t^{\mathrm{t}\mathrm{h}}$ time slot. Here, we assume that the path loss does not change at all time slots at an episode for each user, and each user’s path loss is only updated at the beginning of the next episode. Next, $\mathit{g}\left(t\right)=\left[{\mathit{g}}_1^T\left(t\right),\dots ,{\mathit{g}}_n^T\left(t\right),\dots ,{\mathit{g}}_N^T\left(t\right)\right]\in {\mathbb{C}}^{1\times KN}$ is the vector representing each user’s channel state, where ${\mathit{g}}_n^T\left(t\right)=\left[g_{n,1}\left(t\right),\dots ,g_{n,k}\left(t\right),\dots ,g_{n,K}\left(t\right)\right]\in {\mathbb{C}}^{1\times K}$. We preprocess channel gain for all subcarriers of each user before inputting them into the neural network. $g_{n,k}\left(t\right)$ is computed by the following equation

$$g_{n,k}\left(t\right)={10\mathrm{l}\mathrm{o}\mathrm{g}}_{10}\left({‖{\mathbf{h}}_n^H\left[k\right]‖}_2\right)-{avg\_p}_n$$

(6)

where ${avg\_p}_n$ is the average power across all subcarriers for the $n^{\mathrm{t}\mathrm{h}}$ user expressed as $\left(\sum _{k=1}^K{10\mathrm{l}\mathrm{o}\mathrm{g}}_{10}\left({‖{\mathbf{h}}_n^H\left[k\right]‖}_2\right)\right)/K$. Thus, $g_{n,k}\left(t\right)$ represents the power difference between subcarrier $k$ for the $n^{\mathrm{t}\mathrm{h}}$ user and the average power of all subcarriers. $\mathit{d}\mathit{r}\left(t\right)$ is the data rate of each user after the agent takes the action at the $t^{\mathrm{t}\mathrm{h}}$ time slot, and is denoted as $\mathit{d}\mathit{r}\left(t\right)=\left[{dr}_1\left(t\right),\dots ,d{r}_n\left(t\right),\dots ,{dr}_N\left(t\right)\right]$. At the beginning of each episode, all elements in $\mathit{d}\mathit{r}\left(t\right)$ are initialized to 0, as no action has been taken yet. $\mathit{d}\left(t\right)$ is a detection/indicator vector that determines whether the user satisfies the QoS requirement. The vector is denoted by $\mathit{d}\left(t\right)=\left[d_1\left(t\right),\dots ,d_n\left(t\right),\dots ,d_N\left(t\right)\right]$. If $d_n\left(t\right)$ is 1, it means that the constraint of the data rate for the $n^{\mathrm{t}\mathrm{h}}$ user is satisfied after taking the current subchannel allocation. Otherwise, $d_n\left(t\right)$ = 0. All the elements $\mathit{d}\left(t\right)$ are set to −1 at the beginning of the episode. $o_t$ is the outage probability of an episode. An outage is counted when one of the served users cannot meet the data rate constraint at the $t^{\mathrm{t}\mathrm{h}}$ time slot. $a_t$ and ${sr}_t$ are the current action chosen by the agent and the sum rate of all users after taking the $a_t$, respectively. The action $a_t$ is a number representing a particular subchannel allocation, and as we mentioned earlier, $a_t$ = −1 corresponds to a non-allocation state. At the start of the episode, ${sr}_t$ = 0 because the agent has not taken any action.

• Actions ($a_t$): We treat each possible subchannel assignment as an action and implement the allocation at each time slot. For partial allocation, we randomly generate some allocation cases as an action set, without considering all possible subchannel allocations. For full allocation, the action set includes all possible subchannel allocation solutions. According to [13], RB consists of 12 consecutive subcarriers in the frequency domain. For simplicity in this simulation, we assume that each RB group (RBG) contains only one RB, rather than multiple RBs, as defined in the specification. We assume that each user in the set $\left\{{\mathcal{F}}_b\right\}$ is served at least one RB in each time slot. Therefore, we eliminate the options where a user is not allocated any RBs. In summary, we represent $a_t$ as a number corresponding to a bitmap that indicates the subchannel allocation, and the agent selects $a_t$ based on the current $s_t$ at each time slot $t$.
• Rewards ($r_t$): We define a reward function to guide the agent, rewarding correct actions and penalizing irregular ones. The agent’s goal is to maximize the long-term expected cumulative reward. The reward function is expressed as:

$$r_t=\left\{\begin{array}{c}{\displaystyle \sum _{n=1}^N}{f_{\mathrm{p}\mathrm{r}}(dr}_n\left(t\right)-\mathrm{\Omega }), \mathrm{if} {\displaystyle \sum _{n=1}^N}{d}_n\left(t\right)=N\\ f_{\mathrm{n}\mathrm{r}}\left(o_t\right) , \mathrm{if} {\displaystyle \sum _{n=1}^N}{d}_n\left(t\right)<N\end{array}\right.$$

(7)

where $f_{\mathrm{p}\mathrm{r}}(·$) and $f_{\mathrm{n}\mathrm{r}}(·)$ are the positive reward and negative reward functions, respectively. In (7), if the data rate of all users achieves $\mathrm{\Omega }$ after the agent takes $a_t$, the function $f_{\mathrm{p}\mathrm{r}}(·)$ generates a positive reward. Each user’s positive reward increases as their data rate increases, and then we sum the output of the $f_{\mathrm{p}\mathrm{r}}(·)$ for each user as the current reward $r_t$. If a user’s data rate does not meet the constraint after taking action $a_t$, the function $f_{\mathrm{n}\mathrm{r}}(·)$ generates a negative reward. The negative reward depends on the outage probability $o_t$, and the reward becomes more negative as combines these two streams using the following equation:the outage probability increases.

Dueling-DDQN Algorithm and Network Structure

In the proposed Dueling-DDQN structure, the weights and biases of the neural network are updated via backpropagation (BP) to minimize the loss value in the loss function. The specific structure of the proposed Dueling-DDQN is shown in Figure 1, where “Act” is an activation function. The activation function used is the rectified linear unit (ReLU). The number of outputs corresponds to the size of the action set, and each output represents the expected cumulative reward for that action.

The output layer of the Dueling-DDQN has two streams: one for estimating the state value (a scalar) and another for estimating the advantages of each action. The blue line in Figure 1 combines these two streams using the following equation:

$$\begin{gathered} \mathrm{Q}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}};\mathrm{\theta }\right)=\mathrm{V}\left({\mathrm{s}}_{\mathrm{t}};\mathrm{\theta }\right)+ \\ \left(\mathrm{A}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}};\mathrm{\theta }\right)-{\displaystyle \frac{1}{\left|\mathcal{A}\right|}}\sum _{{\mathrm{a}}_{\mathrm{t}}\in \mathcal{A}}\mathrm{A}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}};\mathrm{\theta }\right)\right) \end{gathered}$$

(8)

where $\mathrm{V}\left({\mathrm{s}}_{\mathrm{t}};\mathrm{\theta }\right)$ estimates the state-value and its output is a scalar, and $\mathrm{A}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}};\mathrm{\theta }\right)$ estimates the advantages for each action, and its output is a vector whose size is the same as the action set. Further, $\mathrm{\theta }$ is the parameter of the Q-network, and $\left|\mathcal{A}\right|$ is the total number of actions. The specific process of the complete model is shown in Figure 2. Due to limited space, we omit the detailed discussions.

For the hybrid beamforming design, we apply the signal-to-leakage-plus-noise ratio (SLNR) approach, which is commonly used in MU-MIMO beamformer design to mitigate the challenges of non-convex design problems [12]. We use the SLNR method to design an analog precoder and the zero-forcing (ZF) method to design a digital precoder. Due to space constraints, we also omit the details of this part, which will be discussed in the full version of the journal paper.

4. Simulation Results

The bandwidth part (BWP) size refers to the number of Resource Blocks (RBs) for a given bandwidth range, while the RBG size represents the number of RBs in a group. We assume a subcarrier spacing (SCS) of 60 kHz, as this value is suitable for both frequency range 1 (FR1) and frequency range 2 (FR2) [14]. Additionally, subchannel allocation is performed at each time slot, with a total of 40 slots in one radio frame. The environment is modeled with two clusters and ten rays, where each cluster has an angular spread of 10 degrees, and the cluster angle range is [0, 2π). The path loss ${pl}_n$ for the $n$ user is generated using the Non-Line of Sight (NLOS) urban microcell (UMi)-Street Canyon scenario described in [15]. In the numerical simulation, unless otherwise mentioned in the article, the following experiments are used $N_{\mathrm{t}}=16$, $N_{\mathrm{t}\mathrm{R}\mathrm{F}}=U=2$, $N=3$, and $\mathrm{B}\mathrm{W}\mathrm{P}=4$.

Figure 3 illustrates the average data rate based on the agent’s policy in a fixed path loss environment. The average data rate for each user is computed for one episode, i.e., one radio frame. In this experiment, a random episode is selected from the testing data, and this selection is fixed for each test, as previously mentioned. The testing results are recorded after 5, 50, and 100 training epochs. As shown in Figure 3, the sum rate increases as the training steps progress for the observed episode. However, this allocation may slightly increase the outage probability, as the spectrum resources available to disadvantaged users may not be sufficient to meet their data rate constraints. Even when the policy allocates most of the subchannels to these users, they may still fail to achieve the required data rate in some cases. As a result, the agent prioritizes increasing spectral efficiency to raise the sum rate, rather than allocating most resources to disadvantaged users.

The outage probability during testing is shown in Figure 4, where each circular marker on the curve represents the outage probability for a specific testing time. The average curve is computed by averaging the outage probabilities from every 10 testing times. It should be noted that the experiments shown in Figure 3 focus on a single radio frame’s state. To demonstrate the stable performance of the proposed algorithm, we test the well-trained network on identical testing data.

The performance of the proposed algorithm and the compared methods is shown in Figure 5. The greedy line and the random line are computed over the testing set. The greedy method tests all subchannel allocations and selects the action that avoids outage. It then chooses the action that results in the highest sum rate from the non-outaging actions. In contrast, the random method randomly selects an action from all possible actions. If the channel state is suboptimal and none of the allocations can meet each user’s QoS in a radio frame, the greedy-allocation method selects an action from the ineligible options, while the random-allocation method picks an action at random. Similarly, for both methods, the channel parameters vary with testing steps, though the path loss values remain fixed. As observed, the performance of the partial allocation method is comparable to that of the full allocation method, with both curves closely matching the greedy line after training on the testing set. This suggests that the algorithm is capable of handling large subchannel allocations in real-world scenarios and demonstrates good generalization.

5. Conclusions

In this study, we consider the hybrid beamforming and the subchannel allocation with the AI-based algorithm for the mmWave massive MIMO-OFDMA systems. The simulation results verify that our proposed algorithm can gradually approach the performance of the greedy allocation in testing for both architectures. Moreover, stability and generalization are also shown in the results. Additionally, we try both cases of full assignment and partial assignment for subchannel allocation, which shows that the performance of the proposed algorithm could achieve the same effect as long as we consider some parts of all the assignments as actions. Finally, compared to the different scenes, we refer that the subchannel allocation with the Dueling-DDQN algorithm could be applied to the practical mmWave and massive MIMO-OFDMA with hybrid beamforming.