Spectral Efficiency Beamforming Scheme for UAV MIMO Communication via Budgeted Combinatorial Multi-Armed Bandit

Abstract

Unmanned aerial vehicles (UAVs) equipped with antenna arrays can deliver high-capacity, high-throughput, and low-latency communication services. Considering a UAV-assisted mmWave multi-input and multi-output (MIMO) system, a two-stage beamforming scheme based on a budgeted combinatorial multi-armed bandit (BC-MAB) is proposed to improve the system’s spectral efficiency (SE). The pre-beamformer design problem is initially formulated as a BC-MAB problem. In this framework, the reward is the received energy, while the cost corresponds to the energy consumed by each RF chain and the budget is represented by the residual energy of the UAV. To achieve a favorable trade-off between the number of communication slots and the energy acquired per slot, a pre-beamforming scheme based on the bang-per-buck ratio is introduced to optimize the number of activated RF chains, therefore maximizing the cumulative reward. The second stage utilizes the reduced-dimensional instantaneous channel state information to design and optimize the beamformer to achieve maximum system SE. The proposed scheme achieves more than 7.1% improvement in SE compared to the benchmark schemes. Simulations validate the superiority of the proposed scheme.

1. Introduction

Unmanned aerial vehicle (UAV)-assisted wireless communication has recently attracted significant attention and is regarded as a key enabler for the sixth generation (6G) of mobile networks [1,2,3,4]. Owing to their deployment flexibility and cost-efficiency, UAVs are deployed as aerial base stations (UAV-BSs) to provide extended coverage for terrestrial networks and improved service quality for ground users. As a result, UAV-assisted communication has become a prominent research topic in academia [5,6]. Millimeter-wave (mmWave) communication also represents a cornerstone technology for prospective 6G wireless networks. Compared to microwave frequencies, the mmWave band offers vast and contiguous available bandwidth that can effectively alleviate the spectrum scarcity issues faced by sub-6 GHz systems. Integrating mmWave communication into UAV-assisted systems can substantially support high-throughput and low-latency requirements [4]. The ability of UAVs to facilitate line-of-sight (LoS) links is a significant benefit for mmWave transmission, which is notoriously vulnerable to pronounced attenuation over non-line-of-sight (NLoS) paths. Furthermore, due to their compact size and limited payload capacity, UAVs impose strict constraints on the volume and weight of onboard equipment. The compact form factor of a large-scale antenna array on a UAV platform is made possible by the short wavelength of mmWave signals. This facilitates agile beamforming, which capitalizes on spatial multiplexing and substantial directional array gains. The inherent sparsity of mmWave channels, combined with the directional nature of beamforming, enables a substantial reduction in interference for UAV communication systems [7].

To date, a number of studies have investigated mmWave UAV-BS multi-input and multi-output (MIMO) communications, covering areas such as channel modeling and beamforming. A three-dimensional (3D) elliptical cylindrical channel model considering the mobility and flight altitude characteristics of UAV-BSs was proposed in [8]. This work further provides a detailed analysis of the model’s statistical properties, taking into account the effects of other environment reflections. Cheng and Li [9] introduced a stochastic model based on a 3D geometric structure to characterize wideband non-stationary air-to-ground MIMO channels. Similarly, a 3D geometry-based UAV-correlated MIMO model was proposed in [10]. By employing polarized geometric theory, the depolarization effects caused by terrestrial scattering environments were described based on a multi-cylinder geometric model. Furthermore, the authors of [11] developed a software-defined radio platform and conducted real-world channel measurements to characterize air-to-ground channels in various practical scenarios, including hovering, encircling, and linear topologies.

Numerous studies have also been conducted on beamforming techniques for mmWave UAV-BS MIMO systems. The application of hybrid beamforming in UAV-assisted massive MIMO communication systems was investigated in [12]. Xiao et al. [13] aimed to maximize the achievable sum rates under constraints including minimum per-user rate requirements, UAV-BS positioning limits, and constant-modulus constraints on beamforming vectors. A two-stage methodology was employed involving the introduction of an approximate beam pattern to solve the initial subproblems of deployment and gain allocation followed by the application of an artificial bee colony algorithm for beamforming optimization. Furthermore, the authors of [14] introduced a manifold matrix representation tailored for UAV platforms to support both MIMO communication and direction-of-arrival estimation. Zhu et al. [15] explored mmWave UAV-BS communication and proposed a 3D beamforming scheme based on phased uniform planar arrays to achieve flexible coverage. A mmWave massive MIMO network supported by multiple UAV-BSs was studied in [16]. By leveraging hybrid analog–digital beamforming, the system effectively compensates for the high channel path loss and mitigates cell interference. The work optimized UAV positioning, user assignment, and hybrid beamforming jointly to maximize the achievable sum rate while satisfying per-user minimum rate constraints. To address this high-dimensional non-convex problem, a sub-optimal solution was developed using alternating optimization. Reference [7] tackled the non-convex problem of co-optimizing UAV deployment, user clustering, and hybrid beamforming. Their proposed solution synergistically combined alternating optimization with successive convex and combinatorial methods. The existing approaches predominantly rely on channel state information (CSI). However, acquiring CSI introduces additional overhead, which can consequently degrade system performance. Furthermore, the ensuing growth in CSI dimensionality in massive antenna systems directly translates into prohibitive pilot overhead for channel estimation. Obtaining accurate CSI becomes increasingly challenging as the number of antennas increases, thereby impeding the advancement and practical implementation of the existing approaches.

A number of works have proposed two-stage beamforming (TSB) schemes to reduce the pilot overhead. In this work, slow-varying statistical CSI (SCSI) is utilized in the first stage to design a pre-beamformer, which reduced the dimensionality of the instantaneous CSI (ICSI) required in the second stage. The reduced-dimensional ICSI could then be acquired with considerably lower pilot overhead. In the second stage, the reduced-dimensional ICSI is used to design the beamformer. Considerable research attention has been devoted to the TSB approach within the domain of terrestrial MIMO communications. The two-stage design of the frequency-invariant beamformer was proposed in [17]. In [18], a cluster-group-based TSB scheme was developed for massive MIMO systems operating in multi-scattering-cluster channels. A large-system analysis was also conducted in [19] for a TSB system operating with limited channel feedback at the transmitter and incorporating a regularized zero-forcing precoder in its second stage. However, as summarized in Table 1, the aforementioned TSB schemes still require the acquisition of both SCSI and ICSI, which incurs additional overhead. While the required ICSI dimensionality has been substantially reduced and its acquisition overhead significantly lowered, obtaining SCSI still introduces pilot overhead.

Table 1. Summary of the existing literature in comparison to the present study.

References	Application of SCSI	Application of ICSI	Energy-Limited Environment	Application of TSB	Application of MAB	UAV Communication
[12,13,14,15,16]		✔				✔
[17,18,19]	✔	✔		✔
[20]		✔		✔	✔
Present paper		✔	✔	✔	✔	✔

To further reduce this overhead, a TSB design based on a multi-armed bandit (MAB) is considered. The MAB problem constitutes a specific category within the reinforcement learning models. In the MAB framework, a set of arms is defined, each yielding a reward when selected. The algorithm aims to maximize the cumulative reward over time by online learning from historical pull outcomes. Several studies have introduced MAB-based TSB methods. Reference [20] applied a combinatorial MAB (CMAB) approach to TSB in massive MIMO with frequency-division duplexing mode. The pre-beamforming design is modeled as a contextual MAB problem and subsequently recast into a 0–1 integer programming formulation, which is solved using a branch-and-bound algorithm. However, a critical limitation of these primarily ground-oriented studies is their general neglect of the stringent energy constraints that are crucial in UAV communications. Specifically, achieving higher SE in a single time slot typically comes at the cost of increased energy consumption. Yet, high per-slot SE does not necessarily guarantee high cumulative SE over an extended time interval. Under energy constraints, it becomes a challenging problem to simultaneously achieve high cumulative SE over a time interval while maintaining desirable per-slot SE. Consequently, the aforementioned studies designed for terrestrial communications are ill-suited for energy-constrained UAV scenarios.

Therefore, we investigated an MAB-based TSB scheme for a UAV-BS single-user downlink communication system. In the first stage, the pre-beamformer is designed to enlarge the energy of the received signal while minimizing the number of activated RF chains. The reduced-dimensional ICSI is then estimated after pre-beamforming. In the second stage, the low-dimensional ICSI is utilized to design the beamformer with the aim of maximizing the system spectral efficiency (SE). The specific contributions are summarized as follows:

(1) The pre-beamformer design problem is formulated as a budgeted combinatorial MAB (BC-MAB) problem. In this model, the UAV-BS acts as the agent, with its battery level serving as the budget. Each codeword in the beamforming codebook is treated as an arm, the energy consumption per RF chain is defined as the cost, and the received energy per slot (RES) constitutes the reward. In each slot, the UAV-BS leverages historical information to select multiple arms under the budget constraint, with the goal of maximizing the cumulative reward. By framing the pre-beamformer design as a BC-MAB problem, the proposed scheme eliminates the need for SCSI, thereby further reducing pilot overhead and improving overall SE. There exists a trade-off between the number of communication slots (NCS) and the RES. By formulating the pre-beamforming problem within the BC-MAB problem, the limited energy budget can be comprehensively accounted for, thereby achieving a favorable trade-off.

(2) The bang-per-buck ratio is introduced to efficiently solve the BC-MAB problem and achieve an effective trade-off between the NCS and the RES. It is challenging to achieve a favorable trade-off. Activating fewer RF chains reduces energy consumption and allows a larger NCS but may decrease the energy obtained per slot, ultimately reducing the cumulative SE during slots. Conversely, activating more RF chains increases the RES but leads to higher energy consumption and a smaller NCS, which also degrades the cumulative SE. The bang-per-buck ratio was adopted to achieve a good trade-off between these competing objectives. Reducing the number of activated RF chains not only increases the NCS, thereby improving cumulative SE, but also lowers the dimensionality of the ICSI. This reduction occurs because the number of columns in the ICSI matrix equals the number of activated RF chains. The decreased ICSI dimensionality subsequently reduces the pilot overhead, leading to enhanced system SE.

2. System Description and Problem Formulation

2.1. System Model

We consider a UAV-assisted mmWave MIMO link in which an N-antenna UAV-BS serves a multi-antenna ground user equipped with M elements, as shown in Figure 1. The transmission operates in downlink mode. A solar panel is incorporated into the UAV-BS to replenish its battery and provide energy sustainability. The respective positions of both the UAV-BS and user in 3D Cartesian coordinates are defined by $(x^t,y^t,h)$ and $({\widehat{x}}^t,{\widehat{y}}^t,0)$, where h represents the vertical height of the UAV-BS.

Figure 1. Illustration of a UAV mmWave MIMO system.

The communication process is illustrated in Figure 2. Given the relatively small variation in both the UAV and user positions within a time interval, the SCSI is assumed to remain approximately constant over this interval. Each time interval comprises multiple slots, during which the ICSI varies. A TSB scheme is employed for transmission in each slot to reduce the pilot overhead associated with ICSI acquisition. In the first stage, a pre-beamformer is designed to reduce the pilot overhead. In the second stage, the beamformer is designed to enhance system performance. The beamformer at the UAV-BS is connected to the pre-beamformer via $N_{RF}$ RF chains, while the number of RF chains is ${\widehat{N}}_{RF}$. During the first stage, a pre-beamformer ${\mathbf{F}}_{RF}^t\in {\mathbb{C}}^{N\times L^t}$ is designed, where $L^t$ denotes the number of activated RF chains. The UAV-BS transmits pilot signals to the user once the pre-beamformer design has been finalized. After receiving the signals, the user feeds back the reduced-dimensional ICSI, represented as ${\left({\mathbf{W}}^t\right)}^H{\left({\mathbf{H}}^t\right)}^H{\mathbf{F}}_{RF}^t$, where ${\mathbf{H}}^t\in {\mathbb{C}}^{N\times M}$ denotes the channel matrix between the UAV-BS and the user on the ground, ${\mathbf{W}}^t\in {\mathbb{C}}^{M\times N_s}$ is the combiner, and $N_s$ is the number of data streams. In the second stage, the obtained reduced-dimensional ICSI is utilized to design the beamformer ${\mathbf{F}}_{BB}^t\in {\mathbb{C}}^{L^t\times N_s}$, after which data transmission occurs. The signal received at the UAV-BS is

$${\mathbf{y}}^t={\left({\mathbf{W}}^t\right)}^H{\left({\mathbf{H}}^t\right)}^H{\mathbf{F}}_{RF}^t{\mathbf{F}}_{BB}^t{\mathbf{s}}^t+{\left({\mathbf{W}}^T\right)}^H{\mathbf{n}}^t,$$

(1)

where ${\mathbf{s}}^t\in {\mathbb{C}}^{N_s\times 1}$ denotes the transmitted signal in each slot t with $\mathbb{E}\left[{\left({\mathbf{s}}^t\right)}^H{\mathbf{s}}^t\right]={\mathbf{I}}_{N_s}$, and ${\mathbf{n}}^t\sim \mathcal{CN}(0,{\sigma }^2{\mathbf{I}}_M)$ represents the additive white Gaussian noise. In each slot t, the achievable SE can be expressed as

$$\begin{array}{cc}\hfill R^t={\Gamma }^t{log}_{2}det(\mathbf{I}+& {\displaystyle \frac{1}{{\sigma }^2}}{\left({\left({\mathbf{W}}^t\right)}^H{\mathbf{W}}^t\right)}^{-1}{\left({\mathbf{W}}^t\right)}^H{\left({\mathbf{H}}^t\right)}^H\hfill \\ \hfill & \times {\mathbf{F}}_{RF}^t{\mathbf{F}}_{BB}^t{\left({\mathbf{F}}_{BB}^t\right)}^H{\left({\mathbf{F}}_{RF}^t\right)}^H{\mathbf{H}}^t{\mathbf{W}}^t).\hfill \end{array}$$

(2)

where ${\Gamma }^t=\left(1-{\displaystyle \frac{{\tau }^t}{T_s}}\right)$, and ${\tau }_t$ is the pilot length. The maximum pilot overhead required for estimating the reduced-dimensional ICSI is $L^t$. $T_s$ is the number of symbols each slot.

Figure 2. The communication process.

2.2. Channel Model

The Rician channel model [21] was adopted. In this model, the channel is

$${\mathbf{H}}^t={\beta }^t\left(\sqrt{{\displaystyle \frac{K}{K+1}}}{\mathbf{H}}_{LoS}^t+\sqrt{{\displaystyle \frac{1}{K+1}}}{\mathbf{H}}_{NLoS}^t\right),$$

(3)

where ${\beta }^t={\beta }_0{\left(d^t\right)}^{-\alpha }$ is the distance-based large-scale path loss, ${\beta }_0$ is defined as the channel gain at the distance $d_0=1$ m, K denotes the Rician factor, $d^t$ is the distance between the ground user and the UAV, and $\alpha$ represents the path loss exponent. ${\mathbf{H}}_{LoS}^t=\mathbf{a}\left({\theta }^t\right)\mathbf{a}{\left({\widehat{\theta }}^t\right)}^H$ is the deterministic LoS channel components, where $\mathbf{a}(·)$ is the steering vector, and ${\theta }^t$ and ${\widehat{\theta }}^t$ are the angles of arrival and departure. ${\mathbf{H}}_{NLoS}^t$ is the NLoS channel components, which follow $\mathcal{CN}(\mathbf{0},\mathbf{R})$, where $\mathbf{R}$ denotes the covariance matrix of ${\mathbf{H}}_{NLoS}^t$. We characterize ${\mathbf{H}}_{NLoS}^t$ as a one-ring channel, where the scattering clusters surround the user and have a center angle $({\theta }^t,{\widehat{\theta }}^t)$ and an angular spread $(\Xi ,\widehat{\Xi })$. Therefore, $\mathbf{R}={\displaystyle \frac{1}{2\Xi }}{\int }_{{\theta }^t-\Xi }^{{\theta }^t+\Xi }{\int }_{{\widehat{\theta }}^t-\widehat{\Xi }}^{{\widehat{\theta }}^t+\widehat{\Xi }}\mathbf{a}\left(\theta \right)\mathbf{a}{\left(\widehat{\theta }\right)}^{H}d\theta d\widehat{\theta }$.

2.3. Energy Model

The total average power consumption of signal transmission at slot t is calculated as ${\tilde{P}}^t=P_{out}^t/{\alpha }_p+L^t{P}_c$[22], where $P_{out}^t={∥{\mathbf{F}}_{RF}^t{\mathbf{F}}_{BB}^t∥}_F^2$, ${\alpha }_p$ represents the power amplifier (PA) efficiency, and $P_c$ denotes the circuit power consumption of each RF chain. The residual energy $E^t$ at slot t is given by

$$E^t=E^{t-1}+E_{solar}-E_f-{\tilde{P}}^t{\Delta }_s,$$

(4)

where $E_{solar}$ denotes the recharge energy by solar panel, $E_f$ is the power consumed by the battery to maintain flight, and ${\Delta }_s$ is the time span in a slot. Denoted by ${\mathbb{I}}^t$, the indicator function is as follows:

$${\mathbb{I}}^t=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{E}^t>{\tau }_b,\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(5)

where ${\tau }_b$ denotes the minimum energy threshold required to ensure operational safety of the UAV-BS. When $E^t>{\tau }_b$, the UAV-BS engages in communication with the ground user. Otherwise, the UAV-BS suspends communication activities and initiates solar energy replenishment.

2.4. Problem Formulation

The discrete Fourier transform (DFT) codebook can effectively characterize the channel [23]. The n-th codeword of the codebook $\mathbf{U}$ ${\mathbf{u}}_n$ is given by

$${\mathbf{u}}_n=\sqrt{{\displaystyle \frac{1}{N}}}\left[1,e^{j2\pi {\displaystyle \frac{n}{N}}},\dots ,e^{j2\pi (N-1){\displaystyle \frac{n}{N}}}\right].$$

(6)

The pre-beamformer is designed by selecting codewords from the DFT codebook $\mathbf{U}$. Since the SE is positively correlated with the received energy, the pre-beamformer is designed to maximize the cumulative received energy. Note that this cumulative energy relies on both the RES and the NCS. The dimension of the pre-beamforming matrix, which sets the count of activated RF chains, presents a critical trade-off: an overly large dimension inversely affects the NCS component, leading to a reduction in the total harvested energy. Conversely, if too few RF chains are activated, the RES diminishes, which also adversely affects the cumulative energy. Therefore, it is crucial to optimize the column number in the pre-beamforming matrix to maximize the cumulative received energy. Our solution formulates the pre-beamformer design within a BC-MAB framework to address this trade-off between the RES and NCS, enabling an efficient balance between these competing components. Let $G^t={∥{\left({\mathbf{H}}^t\right)}^H{\mathbf{F}}_{RF}^t∥}_F^2$ denote the received energy at slot t and $col(·)$ the column number. Then, the problem of the pre-beamformer design maximizing the cumulative received energy during T slots is formulated as

$$\begin{array}{cc}\hfill {\mathcal{P}}_1:& \underset{{\mathbf{F}}_{RF}^t}{max}\sum _{t=1}^T{\mathbb{I}}_t{G}^t\hfill \end{array}$$

(7a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.N_s\le col\left({\mathbf{F}}_{RF}^t\right)\le N_{RF}.\hfill \end{array}$$

(7b)

In ${\mathcal{P}}_1$, to maximize the objective function, we need to make ${\mathbb{I}}_t$ be 1 as often as possible, which means the BS and the UAV should communicate in as many slots as possible.

After the pre-beamformer, the UAV-BS transmits the orthogonal pilot $\mathbf{X}\in {\mathbb{C}}^{N_s\times {\tau }_t}$ to the user, where ${\tau }_t$ denotes the pilot length at slot t. The user then estimates the reduced-dimensional ICSI and feeds it back to the UAV-BS. The number of columns in this reduced-dimensional ICSI matrix matches that of the pre-beamforming matrix, namely, the number of activated RF chains; i.e., $col\left({\left({\mathbf{W}}^t\right)}^H{\left({\mathbf{H}}^t\right)}^H{\mathbf{F}}_{RF}^t\right)=col\left({\mathbf{F}}_{RF}^t\right)$.

In the second stage, the beamformer is designed to maximize the SE with the reduced-dimensional ICSI. The beamformer is only designed when ${\mathbb{I}}_t=1$, and the problem of this beamformer design at slot t is

$$\begin{array}{cc}\hfill {\mathcal{P}}_2:& \underset{{\mathbf{F}}_{BB}^t}{max}{R}^t\hfill \end{array}$$

(8a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.P_{out}^t={∥{\mathbf{F}}_{RF}^t{\mathbf{F}}_{BB}^t∥}_F^2\le P,\hfill \end{array}$$

(8b)

where P denotes the maximum allowable transmit power.

With perfect CSI at the receiver, the user computes the combiner directly from the channel matrix ${\mathbf{H}}^t$. We firstly express the SVD of the channel as ${\mathbf{H}}^t=\mathbf{D}\Lambda \mathbf{V}$, where $\mathbf{D}=[{\mathbf{d}}_1,\dots ,{\mathbf{d}}_M]$ and $\mathbf{V}$ comprise the left and right singular vectors. The matrix $\Lambda$ is diagonal, with its non-zero entries consisting of the singular values arranged in descending order. Then, the combiner is given by

$${\mathbf{W}}^t=\sqrt{N_s}{\displaystyle \frac{\widehat{\mathbf{W}}}{\parallel \widehat{\mathbf{W}}{\parallel }_F^2}},$$

(9)

where

$$\widehat{\mathbf{W}}=[{\mathbf{d}}_1,\dots ,{\mathbf{d}}_{N_s}].$$

(10)

3. BC-MAB-Based TSB Scheme

3.1. Pre-Beamforming Design

We firstly formulate problem ${\mathcal{P}}_1$ as a BC–MAB problem. Subsequently, we present a pre-beamforming design based on the greedy policy.

3.1.1. BC-MAB Problem

The BC–MAB is a variant of the MAB in which, under a budget constraint, a subset of arms is selected to form a super-arm; each arm inside the super-arm is termed a base arm. In our formulation, the budget represents the total available resources, and the cost of each arm corresponds to its per-slot resource consumption. The BC–MAB problem is characterized by four key elements: arms, reward, budget, and cost.

Arm: The pre-beamforming matrix ${\mathbf{F}}_{RF}^t$ is constructed by $L^t$ codewords from the codebook; i.e., ${\mathbf{F}}_{RF}^t=\left[{\mathbf{u}}_{I^t\left(1\right)},\dots ,{\mathbf{u}}_{I^t\left(m\right)},\dots ,{\mathbf{u}}_{I^t\left(L^t\right)}\right]$, where ${\mathbf{u}}_{I^t\left(l\right)}$ denotes the $I^t\left(l\right)$-th codeword of codebook $\mathbf{U}$, $I^t\left(l\right)\in \{1,2,\dots ,N\}$. The codebook is defined as an arm space $\mathcal{A}$ and each codeword ${\mathbf{u}}_n\in \mathcal{A}$, $n=1,\dots ,N$, as an arm. The aggregation ${\mathcal{S}}^t=\{{\mathbf{u}}_{I^t\left(1\right)},\dots ,{\mathbf{u}}_{I^t\left(l\right)},\dots ,{\mathbf{u}}_{I^t\left(L^t\right)}\}$ is a super arm. The arm ${\mathbf{u}}_n$ is the base arm of ${\mathcal{S}}^t$ when $n\in \{I^t\left(1\right),\dots ,I^t\left(L^t\right)\}$.

Reward: The reward of super-arm ${\mathcal{S}}^t$ is defined as the total energy harvested within one slot. Note that the RETS equals the sum of the energies received over all the selected arms. Hence, the reward of base arm ${\mathbf{u}}_n\in {\mathcal{S}}^t$ is defined as $r_n^t={∥{\left({\mathbf{H}}^t\right)}^H{\mathbf{u}}_n∥}^2$.

Budget: In this paper, the budget is the available energy of the battery carried by the UAV. Hence, the budget $B^t$ at slot t is set to $E^{t-1}-{\tau }_b$.

Cost: The beamformer is connected to the pre-beamformer via RF chains. Hence, ${\mathbf{u}}_{I^t\left(l\right)}$ corresponds to the pre-beamforming vector for the l-th RF chain. At slot t, the energy consumption of the l-th RF chain is defined as the cost of arm ${\mathbf{u}}_{I^t\left(l\right)}$; i.e.,

$$c_n^t=\left\{\begin{array}{cc}\nu ,\hfill & \mathrm{if}n\in \{I^t\left(1\right),\dots ,I^t\left(l\right)\dots ,I^t\left(L^t\right)\},\hfill \\ c_n^{t-1},\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(11)

where $\nu ={\Delta }_s\left({∥{\mathbf{f}}_{BB,l}^{t-1}∥}_2^2/{\alpha }_p+P_c\right)$. ${\mathbf{f}}_{BB,l}^{t-1}$ denotes the l-th row of the beamforming matrix ${\mathbf{F}}_{BB}^{t-1}$. Since the energy consumption of the l-th RF chain cannot be measured at the beginning of time slot t, the value from the previous slot $t-1$ is utilized as an estimate for cost calculation.

3.1.2. The Proposed BC-MAB-Based Pre-Beamforming Algorithm

In the MAB problem, the upper confidence bound (UCB) value associated with each arm is used to estimate its potential for obtaining rewards. Unlike the traditional MAB formulation, the BC-MAB problem aims to maximize the cumulative reward sum with a limited budget. To this end, the bang-per-buck ratio is introduced. Arm ${\mathbf{u}}_n$ in slot t is defined as ${\displaystyle \frac{UC{B}_n^t}{{\overline{c}}_n^t}}$, where $UC{B}_n^t$ denotes the UCB value of arm ${\mathbf{u}}_n$ at slot t, and ${\overline{c}}_n^t$ represents the estimated cost of selecting arm ${\mathbf{u}}_n$ at slot t. The UCB value is positively correlated with the reward and hence positively related to the RES. A higher cost leads to a lower NCS. Therefore, the introduction of the bang-per-buck ratio helps achieve a trade-off between the RES and NCS. The UCB value of arm ${\mathbf{u}}_n$ at slot t is given by

$$UC{B}_n^t={\overline{r}}_n^t+\sqrt{{\displaystyle \frac{lnt}{N_n^t}}},$$

(12)

where ${\overline{r}}_n^t$ denotes the reward expectation of the arm ${\mathbf{u}}_n$ at slot t, and $N_n^t$ is the total selection count of arm ${\mathbf{u}}_n$ over the preceding t time slots. The calculations for ${\overline{r}}_n^t$ and $N_n^t$ are given as

$$\begin{array}{cc}\hfill & {\overline{r}}_n^t=\left\{\begin{array}{c}{\displaystyle \frac{{\overline{r}}_n^{t-1}{N}_n^{t-1}+r_n^t}{N_n^t}},\mathrm{if}{\mathbf{u}}_n\in {\mathcal{S}}^t,\\ {\overline{r}}_n^{t-1},\mathrm{otherwise},\end{array}\right.\hfill \\ \hfill & N_n^t=\left\{\begin{array}{c}{N}_n^{t-1}+1,\mathrm{if}{\mathbf{u}}_n\in {\mathcal{S}}^t,\\ N_n^{t-1},\mathrm{otherwise},\end{array}\right.\hfill \end{array}$$

(13)

The estimated cost of selecting arm ${\mathbf{u}}_n$ at slot t is given by

$${\overline{c}}_n^t=\left\{\begin{array}{c}{\displaystyle \frac{{\overline{c}}_n^{t-1}{N}_n^{t-1}+c_n^t}{N_n^t}},\mathrm{if}{\mathbf{u}}_n\in {\mathcal{S}}^t,\\ {\overline{c}}_n^{t-1},\mathrm{otherwise}.\end{array}\right.$$

(14)

The details of the greedy policy-based pre-beamforming algorithm in each t, $t=1,\dots ,T$, are presented in Algorithm 1. Specifically, in step 1, we calculate the bang-per-buck ratio of each arm by ${\displaystyle \frac{UC{B}_n^t}{{\overline{c}}_n^t}}$. In step 2, we sort the arms in non-increasing order of their bang-per-buck ratios at first, and then the new indices for all the arms $x_1,\dots ,x_N$ are obtained; i.e., ${\displaystyle \frac{UC{B}_{t,x_j}}{{\overline{c}}_{t,x_j}}}\ge {\displaystyle \frac{UC{B}_{t,x_{j^{\prime }}}}{{\overline{c}}_{t,x_{j^{\prime }}}}}$∀ pairs $j\le j^{\prime }$. In step 3, we calculate the number of slots in which each arm can be selected within the budget $N_{x_1}^{T-t},\dots ,N_{x_N}^{T-t}$. For instance, the budget for arm ${\mathbf{u}}_{x_1}$ is $B^t\left(1\right)=B^0$, and the number of the slots in which arm ${\mathbf{u}}_{x_1}$ can be selected is calculated as $N_{x_1}^{T-t}=min\left(T-t,⌊{\displaystyle \frac{B^t\left(1\right)}{{\overline{c}}_{t,x_1}}}⌋\right)$. Then, the remaining budget is updated by $B^t\left(2\right)=B^t\left(1\right)-{\overline{c}}_{x_1}^t{N}_{x_1}^{T-t}$. The $N_{x_2}^{T-t}$ is obtained similarly to $N_{x_1}^{T-t}$. After repeating the aforementioned operations, $N_{x_1}^{T-t},\dots ,N_{x_N}^{T-t}$ are obtained. In steps 4–10, we select the base arms. The arm ${\mathbf{u}}_{x_j}$ is selected as a base arm, satisfying the constraints that $N_{x_j}^{T-t}\ge 1$, $N_s\le col\left(S_t\right)\le N_{RF}$, and $UC{B}_{x_j}^t\ge \epsilon$, where $\epsilon =arg\underset{n}{max}\left(UC{B}_n^t\right)/20$. This is because when $UC{B}_{x_j}^t<\epsilon$, the reward gained is negligible compared to the energy consumption of the RF chain. In step 11, we pull the super arm and then update each UCB value. In step 12, we update ${\overline{r}}_n^t$ and $N_n^t$. In steps 13–14, we calculate the beamformer and update the cost of each arm. In step 15, the remaining budget is updated by $B^t=B^{t-1}-{\overline{c}}_n^t$.

Now we analyze the complexity of Algorithm 1. The complexity of step 1 is $\mathcal{O}\left(N\right)$. The complexity of step 2 is $\mathcal{O}(Nlog(N\left)\right)$. The complexity of step 3 is $\mathcal{O}\left(N\right)$. Steps 4–10 choose the super arm, whose complexity is $\mathcal{O}\left(N\right)$. The complexity of step 11 is less than $\mathcal{O}\left(N_{RF}\right)$. In step 12, the complexity is less than $\mathcal{O}\left(N_{RF}\right)$. In step 13, the complexity is less than $\mathcal{O}\left(N_{RF}\right)$. In step 14, we calculate the beamforming matrix and obtain their cost, and the complexity is upper-bounded by $\mathcal{O}(N_{RF}^2{\widehat{N}}_{RF}+N_{RF}{N}_{s}N+{\widehat{N}}_{RF}{N}_{s}M)$. Step 15 updates the budgets, and the complexity of is $\mathcal{O}\left(1\right)$. As a result, the computational complexity of Algorithm 1 is upper-bounded by $\mathcal{O}(Nlog\left(N\right)+N_{RF}^2{\widehat{N}}_{RF}$$+N_{RF}{N}_{s}N+{\widehat{N}}_{RF}{N}_{s}M)$. Thus, the proposed algorithm has cubic complexity.

Algorithm 1: BC-MAB-based beamforming algorithm

3.2. Beamforming Design

Let ${\mathbf{F}}_{BB}^t={\left({\left({\mathbf{F}}_{RF}^t\right)}^H{\mathbf{F}}_{RF}^t\right)}^{-{\displaystyle \frac{1}{2}}}{\widehat{\mathbf{F}}}_{BB}^t$ with ${\widehat{\mathbf{F}}}_{BB}^t$ a dummy variable, and then bring ${\mathbf{F}}_{BB}^t$ into ${\mathcal{P}}_2$. Due to the ${\left({\mathbf{W}}^t\right)}^H{\mathbf{W}}^t={\mathbf{I}}_{N_s}$, ${\mathcal{P}}_2$ can be transformed into

$$\begin{array}{cc}\hfill {\mathcal{P}}_3:& \underset{{\widehat{\mathbf{F}}}_{BB}^t}{max}{\Gamma }^t{log}_{2}det(\mathbf{I}+{\displaystyle \frac{1}{{\sigma }^2}}{\mathbf{H}}_e^t{\widehat{\mathbf{F}}}_{BB}^t{({\widehat{\mathbf{F}}}_{BB}^t)}^H{({\mathbf{H}}_e^t)}^H)\hfill \end{array}$$

(15a)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.{∥{\widehat{\mathbf{F}}}_{BB}^t∥}_F^2\le P,\hfill \end{array}$$

(15b)

where ${\mathbf{H}}_e^t={\left({\mathbf{W}}^t\right)}^H{\left({\mathbf{H}}^t\right)}^H{\mathbf{F}}_{RF}^t{\left({\left({\mathbf{F}}_{RF}^t\right)}^H{\widehat{\mathbf{F}}}_{RF}^t\right)}^{-{\displaystyle \frac{1}{2}}}$. The corresponding closed-form solution for ${\widehat{\mathbf{F}}}_{BB}^t$ is given by

$${\widehat{\mathbf{F}}}_{BB}^t=\sqrt{P}{\widehat{\mathbf{V}}}^t/{\parallel {\widehat{\mathbf{V}}}^t\parallel }_F,$$

(16)

where ${\widehat{\mathbf{V}}}^t$ is constructed by taking the first $N_s$ columns from the right singular vector matrix of ${\mathbf{H}}_e^t$. Once the optimal ${\widehat{\mathbf{F}}}_{BB}^t$ is obtained, the optimal beamformer can be derived as follows:

$$\begin{array}{cc}\hfill {\mathbf{F}}_{BB}^t& ={\left({\left({\mathbf{F}}_{RF}^t\right)}^H{\mathbf{F}}_{RF}^t\right)}^{-{\displaystyle \frac{1}{2}}}{\widehat{\mathbf{F}}}_{BB}^t\hfill \\ \hfill & ={\left({\left({\mathbf{F}}_{RF}^t\right)}^H{\mathbf{F}}_{RF}^t\right)}^{-{\displaystyle \frac{1}{2}}}\sqrt{P}{\widehat{\mathbf{V}}}^t/{\parallel {\widehat{\mathbf{V}}}^t\parallel }_F.\hfill \end{array}$$

(17)

4. Simulation Results

This section presents a comprehensive simulation-based evaluation of the proposed TSB scheme’s efficacy in a UAV-BS mmWave MIMO communication scenario. In this scenario, a UAV equipped with multiple antennas serves as a UAV-BS to provide downlink communication services for a single ground user with multiple antennas. The UAV follows a circular trajectory with a radius of 10 m at a constant speed of v. Aforementioned traditional methods, the joint hybrid precoding/combining (JHPC) scheme [24] and the simultaneous orthogonal matching pursuit (SOMP) scheme [25], are directly extended to the UAV mmWave communication system for comparison. The simulation parameters are shown in Table 2. We set the values of $K=10$ in accordance with [26], the values of $(\Xi ,\widehat{\Xi })$ and $T_s=100$ in accordance with [20], and the values of $N_s=2$ in accordance with [24]. In addition, we set the values of $P_c$, ${\alpha }_p$, $N_{RF}$, and ${\widehat{N}}_{RF}$ according to [22]. The maximum pilot overhead required for estimating the reduced-dimensional ICSI is $L^t$. The $E_f$ and $E_{solar}$ are given by

$$E_f={\Delta }_s\left(P_0\left(1+{\displaystyle \frac{3v^2}{U_{tip}^2}}\right)+{\displaystyle \frac{1}{2}}{d}_0\rho sG{v}^3+P_1{\left(\sqrt{(}1+{\displaystyle \frac{v^4}{4v_0^4}})-{\displaystyle \frac{v^2}{2v_0^4}}\right)}^{{\displaystyle \frac{1}{2}}}\right),$$

(18)

$$E_{solar}={\Delta }_s{\eta }_{s}S{G}_s({\alpha }_s-{\beta }_s{e}^{-{\displaystyle \frac{h}{{\delta }_s}}}),$$

(19)

where $v=3$ m/s is the horizontal velocity of the UAV. Reference [27] can be consulted for the values of other parameters.

Table 2. Simulation parameters.

Notation	Definition
M	10
$N_{RF}$	2
K	10
$(\Xi ,\widehat{\Xi })$	$(5^{\circ },5^{\circ })$
${\beta }_0$	20 dB
$\alpha$	2
${\alpha }_p$	0.35
$P_c$	240 mW
P	2 W
${\tau }_b$	${10}^{-3}$ Joule
$T_s$	100
T	200
$N_s$	2
${\tau }^t$	$L^t$
${\Delta }_s$	0.5 ms
$B^0$	1 Joule
v	3 m/s

Figure 3 demonstrates the convergence of the rewards achieved by the proposed scheme over slots. It can be observed that during the initial slots, the rewards are relatively small due to the necessity of exploring and initializing each arm. As the number of slots increases, the average reward gradually stabilizes, indicating the convergence behavior of the proposed scheme.

Figure 3. Reward achieved during slots while $N=16$ and SNR $=30$ dB.

Figure 4 and Figure 5 compare the SE at different signal-to-noise ratios (SNRs) when $N=16$ and $N=128$. The proposed scheme, the SOMP scheme, the JHPC scheme, and the fully digital precoding scheme all achieve lower SE. This is primarily due to the energy constraints of the UAV: the fully digital, SOMP, and JHPC schemes activate a larger number of RF chains, leading to higher energy consumption and consequently a smaller NCS, which results in inferior SE performance. Moreover, the proposed scheme does not require CSI for designing the pre-beamformer, which further reduces the pilot overhead for estimating the equivalent channel. As a result, the proposed scheme achieves higher SE with lower overhead. As shown in Figure 4 and Figure 5, the SE of the benchmark schemes decreases as the number of transmit antennas increases. This degradation occurs because the growing antenna count elevates the pilot overhead required for ICSI estimation, which in turn reduces the SE. In contrast, the proposed scheme effectively mitigates this overhead, thereby demonstrating its superior performance. The proposed scheme achieves a 7.1% improvement in SE compared to the benchmark schemes when $N=16$ and $N_{RF}=2$ and a 48.5% improvement when $N=128$ and $N_{RF}=2$.

Figure 4. The comparison of SE versus the SNR while $N=16$ and $N_{RF}=2$.

Figure 5. The comparison of SE versus the SNR while $N=128$ and $N_{RF}=2$.

The EE is given by

$$E{E}^t={\displaystyle \frac{R^t}{{\tilde{P}}^t}}.$$

(20)

Figure 6 and Figure 7 compare the energy efficiency (EE) at different SNR levels when $N_{RF}=10$. A comparative analysis reveals that the proposed scheme achieves superior EE, benchmarking against the fully digital, SOMP, and JHPC schemes. This performance gap stems from the energy constraints of the UAV platform: the aforementioned schemes activate a larger number of RF chains, resulting in higher energy consumption and consequently a reduced NCS. Particularly in the fully digital scheme, the absence of an analog module results in the number of RF chains being equal to the number of antennas. Consequently, this architecture exhibits the highest energy consumption among all the considered schemes. The proposed scheme achieves more than 9.8% improvement in EE compared to the benchmark schemes when $N=16$ and $N_{RF}=2$. As a result, they achieve lower EE than the proposed method.

Figure 6. The comparison of EE versus the SNR while $N=16$ and $N_{RF}=2$.

Figure 7. The comparison of EE versus the SNR while $N=128$ and $N_{RF}=2$.

Figure 8 and Figure 9 compare the SE and EE for different SNR values when $N=128$ and $N_{RF}=10$. The proposed scheme achieves more than 40% improvement in SE compared to the other schemes, while the EE of the proposed scheme is more than double those of the other algorithms. As corroborated by Figure 5 and Figure 7, the proposed and fully digital schemes exhibit stable SE and EE performance. This is because the activated RF chain count in our scheme is determined by the algorithm’s reward characteristics and remains independent of the parameter $N_{RF}$. Similarly, the SE and EE of the fully digital precoding scheme depend only on the number of antennas and are, therefore, also unaffected by the increase in $N_{RF}$. In contrast, as $N_{RF}$ increases, both the SOMP and JHPC schemes consume more energy, resulting in a lower NCS, and thus, lower SE and EE compared to Figure 5 and Figure 7. It can therefore be observed that as $N_{RF}$ increases, the performance of the SOMP and JHPC schemes correspondingly degrades. In contrast, both the proposed and the fully digital schemes remain largely unaffected by this parameter variation. Overall, these comparisons demonstrate the superiority of the proposed scheme.

Figure 8. The comparison of SE versus the SNR while $N=128$ and $N_{RF}=10$.

Figure 9. The comparison of EE versus the SNR while $N=128$ and $N_{RF}=10$.

Figure 10 and Figure 11 compare the SE and EE for different SNR values when $N_{RF}=10$ and $P=5$ W. The EE (SE) of the proposed scheme is more than double those of other algorithms. An increase in transmit power P elevates energy consumption. Consequently, as evidenced by Figure 8 and Figure 9, both the proposed and the compared schemes exhibit reduced SE and EE performance. Nevertheless, the proposed approach still outperforms all the compared methods.

Figure 10. The comparison of SE versus the SNR while $N=128$, $P=5$ W, and $N_{RF}=10$.

Figure 11. The comparison of EE versus the SNR while $N=128$, $P=5$ W, and $N_{RF}=10$.

5. Conclusions

In this paper, we proposed a BC-MAB-based TSB scheme for UAV-BS mmWave MIMO communications. The proposed method optimizes the number of activated RF chains in the first stage, achieving high EE while maintaining high SE. Furthermore, the scheme reduces pilot overhead and enhances the overall system SE. The proposed scheme achieves more than 7.1% improvement in SE compared to the benchmark schemes. The proposed scheme is designed for UAVs in uniform motion. When the UAV undergoes accelerated movement, the problem becomes considerably more complex. In our future work, we will consider more challenging scenarios, such as UAVs with variable velocities or multi-UAV communication systems.