Three-Dimensional Trajectory and Resource Allocation Optimization in Multi-Unmanned Aerial Vehicle Multicast System: A Multi-Agent Reinforcement Learning Method

Abstract

Unmanned aerial vehicles (UAVs) are able to act as movable aerial base stations to enhance wireless coverage for edge users with poor ground communication quality. However, in urban environments, the link between UAVs and ground users can be blocked by obstacles, especially when complicated terrestrial infrastructures increase the probability of non-line-of-sight (NLoS) links. In this paper, in order to improve the average throughput, we propose a multi-UAV multicast system, where a multi-agent reinforcement learning method is utilized to help UAVs determine the optimal altitude and trajectory. Intelligent reflective surfaces (IRSs) are also employed to reflect signals to solve the blocking problem. Furthermore, since the UAV’s onboard power is limited, this paper aims to minimize the UAVs’ energy consumption and maximize the transmission rate for edge users by jointly optimizing the UAVs’ 3D trajectory and transmit power. Firstly, we deduce the channel capacity of ground users in different multicast groups. Subsequently, the K-medoids algorithm is utilized for the multicast grouping problem of edge users based on transmission rate requirements. Then, we employ the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to learn an optimal solution and eliminate the non-stationarity of multi-agent training. Finally, the simulation results show that the proposed system can increase the average throughput by $14\%$ approximately compared to the non-grouping system, and the MADDPG algorithm can achieve a $20\%$ improvement in reducing the energy consumption of UAVs compared to traditional deep reinforcement learning (DRL) methods.

1. Introduction

Recently, unmanned aerial vehicles (UAVs) have attracted much attention due to their flexible position adjustment capability, which enhances the probability of line-of-sight (LoS) communication links to ground users (GUs). In wireless communication systems, UAVs can provide storage and computational resources to alleviate the communication pressure of the whole network. UAVs can be used as relays between GUs and ground base stations (GBSs) in a store-carry-forward manner. They can also serve as aerial base stations to improve the coverage of GBSs. Additionally, overloaded GBSs can offload the traffic to UAVs. Therefore, UAVs are widely employed in scenarios such as disaster management, traffic monitoring, and emergency rescue, assuming the roles of data coverage, data collection, and information transmission.

In this paper, UAVs are considered to act as aerial base stations. When utilized as an aerial mobile base station to serve GUs, UAVs possess the following advantages [1]:

• On-demand deployment: While conventional terrestrial base stations are fixed and immovable, UAVs are able to be deployed more flexibly and on-demand in accordance with GUs’ locations.
• Better communication quality: Instead of more obstacle blocking between GBSs and GUs, the air-to-ground link is dominated more by the LoS link.
• Mobility over time: UAVs have the capacity to move over time, adjusting their positions to satisfy the demands of GUs and enhance communication performance.

Meanwhile, the research about using UAVs as aerial base stations has encountered challenges, as follows:

• Trajectory design complexity: Since UAVs can move in multiple dimensions and need to be positioned to meet the needs of GUs; optimal deployment and trajectory design strategies need to be addressed.
• Energy limitation: How to optimize the performance with the restricted energy needs to be taken into account because UAVs consume energy both during their flight and communication.
• Signal blocking: In practical applications, the air-to-ground link is more likely to be blocked by territorial obstacles when UAVs fly at low altitudes in complicated environments.

Motivated by the above advantages and challenges, this paper considers a multi-cell cellular network in which UAVs act as aerial base stations to expand coverage and improve communication quality for edge GUs at a long distance. Furthermore, IRSs are deployed on the surface of ground buildings to reflect the UAVs’ signals to GUs.

For GUs, various users have different transmission rate requirements. Some GUs request high-latency-sensitivity services such as virtual reality and Internet of Vehicles (IoV), while others only request fundamental data computing services. In order to improve service efficiency, multicast channels with fixed transmitters have been extensively studied in wireless communications [2]. The capacity of a multicast channel is determined by the data rate of the GU with the worst channel quality in order for all users to successfully decode the common information from the transmitter. On one hand, the transmission rate is constrained if all the edge GUs are considered as one multicast group, which leads to more energy consumed by UAVs for hovering and transmission. On the other hand, when the UAV communicates with each GU in unicast mode, a higher transmission rate can be guaranteed. However, the UAV will consume more energy to travel longer distances. In order to trade off the two schemes, the GUs are divided into multiple multicast groups based on the transmission rate requirement in this paper. As a result, the average throughput of the entire system is improved since the transmission rate of each multicast group is determined by the GU with the worst channel condition within the group. UAVs can act as mobile transmitters [2] and leverage Orthogonal Frequency Division Multiple Access (OFDMA) technology to achieve multicast communication, thereby reducing energy consumption while overcoming user bottlenecks.

In contrast to conventional fixed GBSs, UAVs can adjust their positions horizontally and vertically over time, thus improving the wireless channel quality of users with different locations and transmission rate requirements. Therefore, optimizing the deployment location and trajectory of UAVs has become an important topic. The majority of current research focuses on optimizing UAV flight trajectory with a fixed altitude, ignoring the impact of NLoS links when UAVs fly at a low altitude. In terms of vertical altitude, for GUs with high transmission rate requirements, UAVs can appropriately shorten the communication link distance, thus increasing link capacity or saving transmit power. However, as shown in Figure 1, the lower the UAV’s altitude is, the more dominant the NLoS link will be [3]. Signals emitted by the aerial base station and received by the UAV propagate through the LoS link until reaching the urban environment, where additional loss is incurred due to shadowing and scattering caused by obstacles, such as buildings. Therefore, the 3D trajectory optimization problem needs to be tackled to design the optimal altitude over time.

In summary, this paper investigates the joint optimization problem of UAV 3D trajectory and power allocation in edge regions in a multi-UAV multicast network. GUs are categorized into multiple multicast groups, and each UAV serves one group at a time slot. The objective of this paper is to minimize the energy consumption of UAVs. To achieve this goal, GUs are first divided into different multicast groups depending on the relative distances and transmission rate requirements. This grouping issue is resolved by the K-medoids algorithm [4,5]. Next, it is important to find the 3D trajectory and power allocation that minimize UAVs’ energy consumption. This is known as an NP-hard problem because it involves coupled optimization variables, such as the UAVs’ association with the multicast group, the transmission power, the UAV data rate, and the UAV trajectory. The problem also consists of several non-convex constraints. Since multiple UAVs are considered in the system and each UAV needs to learn a policy, we utilize multi-agent deep reinforcement learning (MADRL) to solve the problem. MADRL can address the problem that the environment becomes non-stationary from the perspective of any individual agent with each agent’s policy updated [6]. For each agent, the Deep Deterministic Policy Gradient (DDPG) algorithm is applied which can learn policies in high-dimensional and continuous action spaces [7]. The main contributions in this paper are summarized as follows:

• We propose a multi-UAV multicast system assisted by IRSs in which we formulate the multicast grouping problem and an optimization problem aiming to minimize the UAVs’ energy consumption.
• We utilize the K-medoids algorithm [4,5] to solve the multicast grouping problem, and the MADRL framework is employed to efficiently obtain the optimal 3D trajectory and power allocation scheme and eliminate the non-stationarity of multi-agent training.
• The performance can be evaluated by the provided simulation results. We verify that the proposed system can improve the average throughput, and the MADDPG algorithm can reduce the energy consumption of UAVs effectively compared to traditional DRL methods.

The rest of the paper is organized as follows. The literature review is presented in Section 2. Section 3 introduces the system model. Section 4 states the problem description and formulates the optimization objective. Section 5 illustrates the solution, including the grouping algorithm and the optimization algorithm. The simulation results are discussed in Section 6. Finally, Section 7 concludes this paper. The methodology is shown in the following Figure 2.

2. Related Works

In light of the challenges mentioned in Section 1, we investigated some of the literature on UAVs serving as aerial base stations in wireless communication systems. The literature can be reviewed from two perspectives: the problem statement and the optimization methods.

As for the problem statement, Deng et al. [8] investigated UAVs as aerial base stations to multicast public information to GUs and proposed a machine learning method to jointly optimize the multicast grouping and trajectory planning scheme. However, they solely considered a single-UAV network. Chen et al. [9] studied multiple UAVs as aerial base stations to enhance the coverage of cellular networks and proposed a decentralized joint trajectory and power control (DTPC) algorithm to minimize the UAVs’ overall energy consumption. Heuristic algorithms were proposed to formulate the trajectory optimization problem to minimize the total travel time of the UAVs in the multi-cell network in [10,11], where UAVs are employed as flying base stations to serve GUs in collaboration with GBSs. However, none of the above papers considered the multicast grouping problem based on user features or a situation in which the UAV-GU communication link may be blocked in practical low-altitude environments. In [12,13,14,15,16,17], intelligent reflection surfaces (IRSs) were introduced into UAV systems to assist the transmission between UAVs and GUs. Nguyen et al. [18] deployed IRSs to combat air-to-ground (A2G) blockage events and derived closed-form expressions for Signal-to-Interference-Plus-Noise-Ratio (SINR) distributions. The literature shows that IRSs can reduce blockage effectively in the UAV-GU communication link.

As for optimization methods, heuristic algorithms are typically used to solve the trajectory optimization problem. In [10,11], a set of local search heuristic algorithms was proposed considering the curse of dimensionality problem. Xue et al. [19] proposed a heuristic algorithm based on alternating descent and successive convex approximation (SCA) to solve a joint 3D location and transmit power optimization problem. However, typical heuristic algorithms are more suitable for static optimization problems without taking historical data into consideration. In [20], the authors formulated the problem as Budgeted Multi-Armed Bandits (BMABs) to optimize the UAV trajectory and minimize battery consumption and used two Upper Confidence Bound (UCB) BMAB schemes to tackle the issue. Nowadays, with the development of artificial intelligence technology, more and more researchers use machine learning techniques to solve optimization issues. The interaction between the agent and the environment in reinforcement learning (RL) is more similar to the mechanism by which UAVs make decisions based on the observation. The Double Deep Q-Network (DDQN) and DDPG algorithms were utilized in [12] to solve the UAV trajectory optimization problem in the IRS-assisted UAV system. Fan et al. [21] proposed a novel multi-agent DRL method with global–local rewards for UAVs’ dynamic trajectory planning and data offloading decisions. In the multi-agent case, the intricate process of interactions between agents makes the environment constantly and dynamically changing. The non-stationarity will reduce the stability of the algorithm.

In summary, very little of the literature focuses on both the multicast grouping issue based on users’ characteristics and the joint optimization problem. Since machine learning techniques have become the new research trend, MADRL can be utilized to handle the issues in traditional DRL methods.

3. System Model

3.1. System Model

We consider a multi-cell cellular network, as shown in Figure 3, where multiple UAVs serve as aerial base stations to provide access to edge GUs that cannot be effectively served by GBSs. The set of edge GUs is represented by $\mathcal{N}=\left\{n\right|n=1,2,\dots ,N\}$. The GUs are divided into K multicast groups, with different transmission rate requirements for each group. The set of multicast groups is denoted as $\mathcal{K}=\left\{k\right|k=1,2,\dots ,K\}$. $N_k$ represents the number of GUs within the kth multicast group, satisfying $N={\sum }_{k\in \mathcal{K}}^{N_k},N_i\bigcap N_j=⌀,\forall i\ne j\in \mathcal{N}$, which means there’s no overlap between groups. The location of the ith GU is denoted as ${\mathbf{L}}_i^{GU}=(x_i,y_i),i\in \mathcal{N}$, and the transmission rate is denoted as ${\gamma }_i$. The two characteristics are used as metrics to classify multicast groups. When grouping has been completed, in order to satisfy the transmission rate requirement of all users in each group, the transmission rate requirement of the kth group is denoted as ${\gamma }_k=\underset{i\in N_k}{max}{\gamma }_i$.

UAVs fly within the multi-cell edge region and serve terrestrial multicast groups. Let $\mathcal{T}=\left\{t\right|t=1,2,\dots ,T\}$ denote the set of service time slots, which is also termed as an episode. The length of each time slot is defined as $\Delta t$. Assume that there are U UAVs, and the set of UAVs is represented by $\mathcal{U}=\left\{u\right|u=1,2,\dots ,U\}$. ${\mathbf{L}}_{u,t}^{UAV}=(x_{u,t},y_{u,t},h_{u,t})$ denotes the trajectory of the uth UAV at time slot t. Each multicast group is served by the closest UAV at each time slot. If the closest UAV is already occupied by its closest group, the second closest UAV will work. We assume that there are sufficient UAVs to ensure that each multicast group is served by one UAV at each time slot.

Meanwhile, in order to solve the signal-blocking problem when UAVs fly at low altitudes, an IRS is deployed on the surface of the building near each multicast group to avoid the NLoS link between UAVs and GUs. Denote $\mathcal{R}=\left\{r\right|r=r_1,r_2,\dots ,r_K\}$ as the set of IRSs. $r_k$ is the IRS corresponding to the kth multicast group, and its location is defined as ${\mathbf{L}}_{r_k}^{IRS}=(x_{r_k},y_{r_k},h_{r_k})$. For the IRS, we assume that a uniform planer array (UPA) consists of $M_c\times M_r$ passive reflection units (PRU). Each PRU can passively change its phase-shift with an independent reflection coefficient: $r_{r_k,m_c,m_r}=a{e}^{j{\theta }_{r_k,m_r,m_c}},\forall k\in K;\forall m_r\in 1,2,\dots ,M_r;\forall m_c\in 1,2,\dots ,M_c$ where $a\in [0,1]$ is the fixed reflection loss of the IRS and ${\theta }_{r_k,m_r,m_c}\in [-\pi ,\pi )$ is the phase shift inserted at PRU $(m_r,m_c)$ [12].

3.2. Transmission Model

The transmission model consists of two parts: the GBS-GU link and the UAV-GU link. UAVs provide supplemental services when the GBS is unable to meet the needs of the edge GUs. The service procedure of the uth UAV for the multicast group is shown in Figure 4. The uth UAV serves the kth multicast group at time slot t, during which other UAVs serve the other multicast groups. Over time, the UAV continues to fly to serve the next multicast group at time slot $t+1$.

3.2.1. GBS-GU Link

In the GBS-GU link, the ith GU is served by the closest GBS. The GBS-GU link channel is assumed according to [22], where it can be assumed to be a fading channel with a distance-dependent path loss with the exponent $\delta ⩾2$ and an additional random term ${\zeta }_{g,i}\sim Exp\left(1\right)$ accounting for small-scale fading. As a result, the received signal-to-noise-ratio (SNR) at the ith GU from the GBS can be expressed as

$${SNR}_i^G={\gamma }_i^G={\displaystyle \frac{G_G{P}_G{g}_0}{{\sigma }^2{(r_{g,i}^2+H_G^2)}^{\delta /2}}}$$

(1)

where $i\in \mathcal{N}$; $G_G$, $H_G$, $r_i$, and $P_G$ denote a fixed antenna gain, the height of the GBS, the distance between the ith GU and the GBS, and the transmit power of the GBS, respectively; $g_0={\left(\frac{c}{4\pi f_c}\right)}^2$ denotes the average channel power gain at a reference distance of $d_0=1m$; and ${\sigma }^2$ denotes the noise power. According to Shannon’s theorem, the transmission rate from the GBS to the ith GU is calculated as

$$R_{g,i}=B{log}_2(1+{\gamma }_i^G{\zeta }_i)$$

(2)

where B is the total transmission bandwidth.

3.2.2. GBS-GU Link Outage Probability

An outage occurs when the GBS is unable to meet the ith GU’s transmission rate requirement ${\gamma }_i$ because of the small-scale fading between the GBS and GUs. The outage probability is expressed as

$$\begin{array}{cc}\hfill P_{g,i}& =Pr\{R_{g,i}<{\gamma }_i\}\hfill \\ & =Pr\{B{log}_2(1+{\gamma }_i^G{\zeta }_{g,i})<{\gamma }_i\}\hfill \\ & =Pr\{{\zeta }_{g,i}<\frac{2^{{\gamma }_i/B}-1}{{\gamma }_i^G}\}\hfill \\ & =1-exp(-\frac{2^{{\gamma }_i/B}-1}{{\gamma }_i^G})\hfill \end{array}$$

(3)

3.2.3. UAV-GU Link

When GBSs are unable to provide reliable communication to the ith GU, the service is provided by UAVs in the multicast mode. In order to avoid NLoS links induced by shadowing and scattering caused by building complexes in urban environments, IRSs are introduced to reflect UAV-GU signals. As shown in Figure 5, the UAV-GU link can be replaced by the two UAV-IRS and IRS-GU links, both of which are LoS connections.

Similar to [22], we assume the corresponding antenna gain in direction $(\alpha ,\beta )$ as

$$G_U(\alpha ,\beta )=\left\{\begin{array}{cc}\hfill & G_0/{\Theta }_U^2,-{\Theta }_U⩽\alpha ⩽{\Theta }_U,-{\Theta }_U⩽\beta ⩽{\Theta }_U\hfill \\ \hfill & g_0\approx 0,otherwise\hfill \end{array}\right.$$

(4)

where $G_0=\frac{30000}{2^2}\ast {\left(\frac{\pi }{180}\right)}^2\approx 2.2846$ and ${\Theta }_U\in (0,\frac{\pi }{2})$. Thus, the ground coverage region of the UAV’s antenna main lobe corresponds to the disk region with a radius $r_{u,t}=h_{u,t}tan{\Theta }_U$ that is centered on the projection of the UAV on the ground. Determining the beamwidth ${\Theta }_U$, the coverage radius can be adjusted by changing the UAV’s altitude $h_{u,t}$ so that the GUs are located within the coverage area of the UAV.

When the signal reaches the urban environment, obstacles, such as building complexes, cause additional losses in the UAV-GU link. Define the outage probability at time slot t of the uth UAV with the ith GU in the kth multicast group as:

$$p_{u,i,t}=P_{NLoS}=1-\frac{1}{1+Cexp\left(-D\left(arctan\left(\frac{h_{u,t}}{d_{u,i,t}}\right)-C\right)\right)}$$

(5)

where $i\in N_k,k\in \mathcal{K}$; $d_{u,i,t}=\sqrt{{\left(h_{u,t}\right)}^2+{(x_i-x_{u,t})}^2+{(y_i-y_{u,t})}^2}$ is the distance between the ith GU and the uth UAV’s at time slot t; $arctan\left(\frac{h_{u,t}}{d_{u,i,t}}\right)$ denotes the elevation angle of the UAV to the ith GU; and D and C are constant values depending on the environment.

On one hand, when the UAV-GU link is unblocked, and assuming free space fading channel gain, the channel gain between the uth UAV and the ith GU at time slot t can be expressed as

$$g_{u,i,t}^{LoS}=g_0{d}_{u,i,t}^{-\delta }$$

(6)

where $\delta =2$ and $g_0={\left(\frac{c}{4\pi f_c}\right)}^2$. On the other hand, when the UAV-GU link is blocked, IRS is applied, as shown in processes ② and ③ of Figure 5.

(1)

UAV-IRS link

At time slot t, let ${\omega }_{1,u,r_k,t}=\frac{|x_{r_k}-x_{u,t}|}{d_{u,r_k,t}}$, ${\omega }_{2,u,r_k,t}=\frac{|y_{r_k}-y_{u,t}|}{d_{u,r_k,t}}$, and ${\omega }_{3,u,r_k,t}=\frac{|h_{r_k}-h_{u,t}|}{d_{u,r_k,t}}$ denote the cosine and sine of the horizontal angle of arrival (AoA) of the signal at the IRS $r_k$ from the uth UAV and the sine of the vertical AoA of the signal at the IRS $r_k$, respectively [12]. $d_{u,r_k,t}=\sqrt{{(h_{u,t}-h_{r_k})}^2+{(x_{u,t}-x_{r_k})}^2+{(y_{u,t}-y_{r_k})}^2}$ denotes the Euclidean distance between the uth UAV and IRS $r_k$ near the kth multicast group at time slot t. The channel gain between the uth UAV and IRS $r_k$ at time slot t can be expressed as

$$g_{u,r_k,t}=\sqrt{g_0{d}_{u,r_k,t}^{-\delta }}·{\Omega }_{u,r_k,t}$$

(7)

where $d_r$ and $d_c$ denote the length and the width of each UPA, respectively; ${\Omega }_{u,r_k,t}={[1,e^{-j\frac{2\pi }{\lambda }{d}_r{\omega }_{1,u,r_k,t}{\omega }_{3,u,r_k,t}},\dots ,e^{-j\frac{2\pi }{\lambda }(M_r-1)d_r{\omega }_{1,u,r_k,t}{\omega }_{3,u,r_k,t}}]}^T\otimes$ $[1,e^{-j\frac{2\pi }{\lambda }{d}_c{\omega }_{2,u,r_k,t}{\omega }_{3,u,r_k,t}},\dots ,$$e^{-j\frac{2\pi }{\lambda }(M_c-1)d_c{\omega }_{2,u,r_k,t}{\omega }_{3,u,r_k,t}}{]}^T$ represents the reflecting array response vector of the IRS [23].

(2)

IRS-GU link

Similar to the UAV-IRS link, we define the cosine and sine of the horizontal angle of arrival (AoA) of the signal at the ith GU from the IRS $r_k$ as ${\omega }_{1,r_k,i}=\frac{|x_{r_k}-x_i|}{d_{r_k,i}}$ and ${\omega }_{2,r_k,i}=\frac{|y_{r_k}-y_i|}{d_{r_k,i}}$. ${\omega }_{3,r_k,i}=\frac{h_{r_k}}{d_{r_k,i}}$ is the sine of the vertical AoA. $d_{r_k,i}=\sqrt{{\left(h_{r_k}\right)}^2+{(x_i-x_{r_k})}^2+{(y_i-y_{r_k})}^2}$ denotes the Euclidean distance between the IRS $r_k$ and the ith GU in the k multicast group. The channel gain from the IRS $r_k$ multicasting to the ith GU is

$$g_{r_k,i}=\sqrt{g_0{d}_{r_k,i}^{-\delta }}·{\Omega }_{r_k,i}$$

(8)

where ${\Omega }_{r_k,i}={[1,e^{-j\frac{2\pi }{\lambda }{d}_r{\omega }_{1,r_k,i}{\omega }_{3,r_k,i}},\dots ,e^{-j\frac{2\pi }{\lambda }(M_r-1)d_r{\omega }_{1,r_k,i}{\omega }_{3,r_k,i}}]}^T\otimes$ $[1,e^{-j\frac{2\pi }{\lambda }{d}_c{\omega }_{2,r_k,i}{\omega }_{3,r_k,i}},\dots ,$$e^{-j\frac{2\pi }{\lambda }(M_c-1)d_c{\omega }_{2,r_k,i}{\omega }_{3,r_k,i}}{]}^T$.

(3)

IRS-Assisted UAV-GU link

The channel gain of the UAV-GU link assisted by the IRS $r_k$ is given by

$$g_{u,i,t}^{IRS}=a{\left(g_{r_k,i}\right)}^T·M_{r_k,t}·g_{u,r_k,t}$$

(9)

where $M_{r_k,t}=diag(e^{j{\theta }_{r_k,1,1}^t},\dots ,e^{j{\theta }_{r_k,m_r,m_c}^t},\dots ,e^{j{\theta }_{r_k,M_r,M_c}^t})$ denotes the IRS reflection phase coefficient matrix.

Combining (5), (6) and (9), the achievable average channel gain and SNR at the ith GU are expressed as

$$g_{u,i,t}=(1-p_{u,i,t})g_{u,i,t}^{LoS}+p_{u,i,t}{g}_{u,i,t}^{IRS}$$

(10)

$${SNR}_{u,i,t}^U={\gamma }_{u,i,t}^U={\displaystyle \frac{G_U{P}_{u,t}{g}_{u,i,t}}{{\sigma }^2}}$$

(11)

where $P_{u,t}$ is the uth UAV’s transmit power at time slot t and ${\sigma }^2$ denotes the thermal noise power, which is linearly proportional to the allocated bandwidth [24].

We utilize OFDMA technology for multicasting. The transmission rate for the kth multicast group at time slot t is decided by the GU with the worst channel quality within the group:

$$R_{u,k,t}=\underset{i\in N_k}{min}\frac{B}{|N_k|}lo{g}_2(1+{\gamma }_{u,i,t}^U)$$

(12)

where $|N_k|$ denotes the number of GUs in the kth multicast group, and B is the total transmission bandwidth. According to (2), (3) and (12), the channel capacity of the ith GU in the kth multicast group can be calculated by

$$R_{i,t}=\underset{\mathrm{GBS}-\mathrm{GU}}{\underbrace{(1-P_{g,i})R_{g,i}}}+\underset{\mathrm{UAV}-\mathrm{IRS}-\mathrm{GU}}{\underbrace{P_{g,i}{R}_{u,k,t}}},i\in N_k$$

(13)

Therefore, the average throughput of the system is

$$throughput=\sum _{i\in N_k,k\in \mathcal{K}}{R}_{i,t}$$

(14)

4. Problem Formulation

4.1. Multicast Grouping

Our goal is to optimize the 3D trajectory of the UAVs based on the transmission rate requirements of different GUs while minimizing energy consumption. First, we divide the N GUs into K multicast groups based on the characteristics of the GUs. We assume that the GUs remain static during the grouping procedure. The characteristics of the ith GU is defined by ${\mathit{\varphi }}_i=\{{\mathbf{L}}_i^{GU},{\gamma }_i\},i\in N$, which denotes the location and transmission rate requirement, respectively. Let $x_{i,k}\in \{0,1\}$ indicate the correspondence between the GUs and multicast groups. $x_{i,k}=1$ indicates that the ith GU belongs to the kth multicast group. The multicast grouping problem can be formulated as

$$\begin{array}{cc}\hfill \mathbf{P}\mathbf{1}:& {\displaystyle \underset{\mathbf{X},\mathit{\psi }}{min}\sum _{k=1}^K\sum _{i=1}^N{x}_{i,k}{\parallel {\mathit{\varphi }}_i-{\mathit{\psi }}_k\parallel }^2}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill s.t.& x_{i,k}\in \{0,1\},\forall i\in N\hfill \end{array}$$

(16)

$$\begin{array}{cc}& {\sum }_{k=1}^K{x}_{i,k}=1,\forall i\in N\hfill \end{array}$$

(17)

$$\begin{array}{cc}& {\sum }_{i=1}^N{x}_{i,k}⩾S,\forall k\hfill \end{array}$$

(18)

where ${\mathit{\psi }}_k$ is the characteristics of the GU selected as the kth multicast group’s center. The constraints in (17) guarantee that a GU can only be in one multicast group. The constraints in (18) ensure that the number of GU within a group cannot exceed S.

4.2. Trajectory Optimization and Resource Allocation

4.2.1. UAV Energy Consumption

UAVs are mostly battery-powered with limited energy storage, so we aim to minimize the energy consumption of UAVs. At time slot t, the energy consumption of a UAV consists of the energy used to transmit signals to GUs and the energy for UAVs flying in the air. The hovering energy consumption can be neglected compared to both of them [9].

First, the delay of transmission from the uth UAV to the kth multicast group at time slot t can be denoted as $T_{u,k,t}=\frac{S_{u,k,t}}{R_{u,k,t}}, T_{u,k,t}⩽\Delta t$, where $S_{u,k,t}$ is the size of the transmission data; the transmission should be finished within the time slot. Therefore, the transmission energy consumption at time slot t can be calculated as

$$E_{u,m,t}=P_{u,t}{T}_{u,k,t}=P_{u,t}{\displaystyle \frac{S_{u,k,t}}{R_{u,k,t}}}$$

(19)

Then, according to [12], the flying propulsion consumption at time slot t is calculated as

$$E_{u,f,t}=\left(P_0(1+{\displaystyle \frac{3\parallel {\mathbf{v}}_{u,t}^2\parallel V_{max}^2}{U_{tip}^2}})+P_1{v}_{h,u,t}\right)\Delta t$$

(20)

where $P_0$ is the blade power; $U_{tip}$ is the tip speed of the rotor; $P_1$ is the descending/ascending power; $V_{max}$ is the achievable maximal speed of UAVs; ${\mathbf{v}}_{u,t}=(v_{x,u,t},v_{y,u,t},v_{h,u,t})$ denotes the normalized speed of the uth UAV at time slot t, where $\parallel {\mathbf{v}}_{u,t}\parallel ⩽1$ and $-1⩽v_{x,u,t},v_{y,u,t},v_{h,u,t}⩽1$. Therefore, the location of the uth UAV at time slot $t+1$ is

$${\mathbf{L}}_{u,t+1}^{UAV}={\mathbf{L}}_{u,t}^{UAV}+{\mathbf{v}}_{u,t}\Delta t$$

(21)

Furthermore, as for obstacle avoidance, we consider the collision between the UAVs. UAVs have to keep a minimum distance from each other at any time. In other words,

$$\parallel {\mathbf{L}}_{u,t}^{UAV}-{\mathbf{L}}_{u^{\prime },t}^{UAV}{\parallel }^2⩾d_{min}^2;\forall u\ne u^{\prime }\in U,t$$

(22)

where $d_{min}$ is the minimum distance UAVs should keep between each other.

In order to minimize the energy consumption in (19), the transmission rate $R_{u,k,t}$ needs to be maximized. The channel gain $g_u,i,t$ also needs to be maximized according to (11) and (12). The probability of the NLoS link $P_{NLoS}$ in (5) is a monotonically decreasing function as the UAV’s altitude increases. When the altitude $h_{u,t}$ is elevated, its elevation angle to the GU increases, and the probability that the LoS link is dominated increases. However, as the UAV is elevated and the distance between the UAV and the GU increases, the channel gain $g_{u,i,t}^{LoS}$ decreases. Therefore, in order to achieve the goal of minimizing energy consumption, altitude optimization needs to be tackled.

4.2.2. Joint Trajectory Optimization and Power Allocation Problem

Based on the above analysis, to minimize the total energy consumption of multiple UAVs, the joint optimization problem about the 3D trajectory and power allocation can be formulated as

$$\begin{array}{cc}\hfill \mathbf{P}\mathbf{2}:& \underset{V,P}{min}\sum _{t=1}^T\sum _{u=1}^U{E}_{u,m,t}+E_{u,f,t}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill s.t& \parallel {\mathbf{v}}_{\mathbf{u},\mathbf{t}}\parallel ⩽\mathbf{1};\hfill \end{array}$$

(24)

$$\begin{array}{cc}& P_{u,t}⩽P_{max};\hfill \end{array}$$

(25)

$$\begin{array}{cc}& T_{u,k,t}⩽\Delta t;\hfill \end{array}$$

(26)

$$\begin{array}{cc}& \parallel {\mathbf{L}}_{u,t}^{UAV}-{\mathbf{L}}_{u^{\prime },t}^{UAV}{\parallel }^2⩾d_{min}^2;\forall u\ne u^{\prime }\in U,t;\hfill \end{array}$$

(27)

where $V=\left\{{\mathbf{v}}_{u,t}\right|u\in U\}$ indicates the speed of the UAVs and $P=\left\{P_{u,t}\right|u\in U\}$ indicates the transmit power of the UAVs. The constraints in (24) ensure the normalized speed, and (25) guarantees that the UAV’s transmit power cannot exceed the maximal power. The constraints in (26) ensure that the data transmission at time slot t must be completed within it. The constraints in (27) guarantee that individual UAVs do not collide during their respective flights.

5. Proposed Solution

5.1. K-Medoids for Multicast Grouping

We utilize K-medoids, an improved algorithm of the K-means clustering algorithm, to solve the multicast grouing problem $\mathbf{P}\mathbf{1}$. In the K-medoids algorithm, when given the location of the cell edge GUs and the number of multicast groups K, we can find the corresponding multicast group centers and edge GUs contained in each multicast group [4,5].

First, the K medoids are randomly initialized, and the iteration is set as $t=0$. Subsequently, the medoids are continuously updated in iterations. When the medoids and their characteristics ${\mathit{\psi }}_k$ are given, $\mathbf{P}\mathbf{1}$ can be simplified as

$${\displaystyle \underset{\mathbf{X}}{min}\sum _{k=1}^K\sum _{i=1}^N{x}_{i,k}{\parallel {\mathit{\varphi }}_i-{\mathit{\psi }}_k\parallel }^2,s.t.\left(16\right)\left(17\right)\left(18\right)}$$

(28)

Problem (28) can be solved by the branch and bound method [5]. In iteration $t+1$, the medoid of each multicast group is updated to be the member point with the smallest criterion function, which is the distance of the user characteristics between one member point and the other member points. The medoid in the kth multicast group is updated as

$${\mathit{\psi }}_k^{t+1}=arg\underset{{\mathit{\varphi }}_i^t}{min}{\parallel {\mathit{\varphi }}_i^t-{\mathit{\varphi }}_j^t\parallel }^2,\forall i,j\in m_k;i\ne j$$

(29)

where $m_k$ denotes the set of members in the kth group. Repeat the above process until all the medoids do not change. The algorithm outputs the optimal medoids’ location ${\mathbf{L}}_k^{MG}=(x_k,y_k),\forall k\in \mathcal{K}$ and the GUs’ location in the kth multicast group ${\mathbf{L}}_{m_k}^{GU}=(x_i,y_i),i\in N_k,k\in K$.

The specific algorithm is shown in Algorithm 1.

Algorithm 1 Constrained K-medoids

Input:  GUs’ location ${\mathbf{L}}^{\mathbf{GU}}$, number of groups K Output:  optimal grouping strategy ${\mathbf{X}}^{\ast }$ 1: Initialization: set iteration t = 0 and randomly initialize ${\mathit{\psi }}_k^0$. 2: for the iteration $t=1,2,3,\dots$ 3:       Given ${\mathit{\psi }}_k^t$, solve (28) to find current optimal grouping strategy ${\mathbf{X}}^{\mathbf{t}}$. 4:       Update multicast group medoids ${\mathit{\psi }}_k^{t+1}$ using (29). 5: until medoids no longer change.

5.2. MADDPG for Optimization Problem

After solving the grouping problem, the joint trajectory optimization and power allocation problem is solved based on the best grouping strategy. In this section, we propose solving $\mathbf{P}\mathbf{2}$ with the MADDPG algorithm.

Each UAV acts as an agent that interacts with the environment over T time slots. At time slot t, an action $a_t$ is generated based on the state $s_t$ of the environment around the agent, and a reward $r_t$ is obtained for judging the action $a_t$ generated in the current state. The goal of each agent is to train a policy $\pi$ that can generate the action $a_t$ that makes the reward $r_t$ the highest based on the current state. Subsequently, at time slot $t+1$, the state $s_t$ transits into a new state $s_{t+1}$ due to the action $a_t$, and the process is repeated till the end of an episode.

5.2.1. State, Action, and Reward

In order to solve $\mathbf{P}\mathbf{2}$ with MADDPG, we define the state, action, and reward of each UAV, which acts as an agent at time slot t.

(1)

State

For each agent, the state includes information perceived from the environment based on which the agent determines its action and evaluates long-term rewards. Since the transmission rate is related to the distance between the UAV and GU, each agent’s perceived information includes the observed multicast groups and the locations of its member users along with the UAVs’ locations. Then, the state of the uth UAV can be formulated as

$${\mathbf{s}}_{u,t}=[{\mathbf{L}}_{1,t}^{UAV},{\mathbf{L}}_{2,t}^{UAV},\dots ,{\mathbf{L}}_{M_u,t}^{UAV},{\mathbf{L}}_{m_1,t}^{GU},{\mathbf{L}}_{m_2,t}^{GU},\dots ,{\mathbf{L}}_{m_{M_k},t}^{GU}]$$

(30)

which includes the locations of the $M_u$ closest neighbor UAVs that can be observed at the current time slot and the locations of GUs in the $M_k$ closest multicast group.

(2)

Action

In $\mathbf{P}\mathbf{2}$, we need to determine the speed and the power allocation of the UAV at time slot t. The action of the uth UAV can be formulated as

$${\mathbf{a}}_{u,t}=[{\mathbf{v}}_{u,t},P_{u,t}]$$

(31)

where (24) and (25) are satisfied.

(3)

Reward

The reward function determines the objective of the RL problem. The objective function of $\mathbf{P}\mathbf{2}$ is to minimize the energy consumption of the UAV, while the objective of RL is to maximize the reward. The reward of each agent can be set as negative energy consumption:

$$r_{u,t}=-(E_{u,m,t}+E_{u,f,t})$$

(32)

5.2.2. MADDPG Algorithm

The 3D trajectory and power allocation to be optimized in Problem $\mathbf{P}\mathbf{2}$ are both continuous variables. DDPG, based on an actor–critic architecture, is able to learn a deterministic policy for continuous actions, which can directly output the optimal action. However, in the multi-agent case, if each agent only considers its own observations and actions to learn its own policy using DDPG, the environment will become non-stationary from the perspective of any individual agent. This is due to the fact that the policies of other agents are also constantly updated and changing, and the sampled data of the agent does not follow a consistent probability distribution.

Therefore, Ref. [6] proposed a framework of centralized training and decentralized execution, which allows the critic to obtain the policy information of other agents during the training process. Only local information is needed when applying the actor to make decisions. The framework is shown in Figure 6. On one hand, the centralized critics $Q_u$ that MADDPG trains for each agent use the observed states and actions of all agents as input. As a result, critics are able to capture changes in all agents’ policies, thus eliminating non-stationarity [9]. The Q-value computed by the critic $Q_u$ is used to update the corresponding actor’s policy network ${\pi }_u$. On the other hand, when each agent is sufficiently trained, the actor can compute the policy independently based on the state, without the feedback of the critic and the state or action information of other agents. Hence, MADDPG facilitates fully decentralized execution. In general, MADDPG can be regarded as a centralized RL technique when training.

As shown in Figure 7, the training process of MADDPG includes data sampling, model training, and parameter updating. We define the state set, the action set, and the reward set from all of the agents as ${\mathbf{s}}_t=\{{\mathbf{s}}_{1,t},{\mathbf{s}}_{2,t},\dots ,{\mathbf{s}}_{U,t}\}$, ${\mathbf{a}}_t=\{{\mathbf{a}}_{1,t},{\mathbf{a}}_{2,t},\dots ,{\mathbf{a}}_{U,t}\}$, and ${\mathbf{r}}_t=\{r_{1,t},r_{2,t},\dots ,r_{U,t}\}$, respectively. According to [9], setting a specific reward in (32) for each agent will lead to increasing training complexity, so we set the reward of each agent to be the cumulative reward of all agents, ${\displaystyle \sum _{u=1}^U}{r}_{u,t}$. The reward in (32) is changed to be

$$r_{u,t}=-\sum _{u=1}^U(E_{u,m,t}+E_{u,f,t})$$

(33)

Each agent contains four networks during training. The actor network and target actor network share the same network architecture. The input is the current state ${\mathbf{s}}_{\mathbf{u},\mathbf{t}}$ of agent u, and the output is the action ${\mathbf{a}}_{u,t}$. The critic network has the same network architecture as the target critic network. The inputs are the states and actions of all the agents, and the output is the Q-value, which is defined as

$$Q_u^{\pi }(\mathbf{s},\mathbf{a})=\mathbb{E}\sum _{t=0}^{\infty }{\gamma }^t{r}_{u,t}$$

(34)

where $\gamma$ denotes the discount factor.

Let $\pi =\{{\pi }_1,{\pi }_2,\dots ,{\pi }_U\}$ denote the policies of U agents, which are fitted, respectively, by U actor networks parameterized by ${\mathit{\theta }}^{\pi }=\{{\theta }_1^{\pi },{\theta }_2^{\pi },\dots ,{\theta }_U^{\pi }\}$. As shown in Figure 7, the parameters of the actor are updated by policy gradient, which is calculated as

$${\nabla }_{{\theta }_u}J\left({\theta }_u\right)={\mathbb{E}}_{(\mathbf{s},\mathbf{a})\mathcal{D}}\left[{\nabla }_{{\mathbf{a}}_u}{Q}_u^{\pi }(\mathbf{s},\mathbf{a}){|}_{{\mathbf{a}}_u={\mathbf{\pi }}_u\left({\mathbf{s}}_u\right)}{\nabla }_{{\theta }_u}{\mathit{\pi }}_u\left({\mathbf{s}}_u\right)\right]$$

(35)

where $\mathcal{D}$ represents the experience replay buffer. Each state transition $({\mathbf{s}}_{u,t},{\mathbf{a}}_{u,t},r_{u,t},{\mathbf{s}}_{u,t+1})$ of agent u with other agents is stored in the buffer.

U critic networks are parameterized by ${\mathit{\theta }}^Q=\{{\theta }_1^Q,{\theta }_2^Q,\dots ,{\theta }_U^Q\}$. The updating process of the critic is shown in Figure 7. The gradient descent method is used to minimize the following loss function

$$\mathcal{L}\left({\theta }_u^Q\right)={\mathbb{E}}_{(\mathbf{s},\mathbf{a},\mathbf{r},{\mathbf{s}}^{\prime })\mathcal{D}}{[Q_u^{\pi }(\mathbf{s},\mathbf{a})-y]}^2$$

(36)

where $y=r_u+\gamma Q_u^{\prime {\pi }^{\prime }}({\mathbf{s}}^{\prime },{\mathbf{a}}^{\prime }){|}_{{\mathbf{a}}_u^{\prime }={\pi }_u^{\prime }\left({\mathbf{s}}_u\right)}$, $Q_u^{\prime {\pi }^{\prime }}$ and ${\pi }_u^{\prime }$ denote the target critic and the target actor, respectively. After the actor and critic are updated, we use a soft update to update the two target networks as

$$\begin{array}{c}\hfill {\mathit{\theta }}^{{\pi }^{\prime }}\leftarrow \tau {\mathit{\theta }}^{\pi }+(1-\tau ){\mathit{\theta }}^{{\pi }^{\prime }}\\ \hfill {\mathit{\theta }}^{Q^{\prime }}\leftarrow \tau {\mathit{\theta }}^Q+(1-\tau ){\mathit{\theta }}^{Q^{\prime }}\end{array}$$

(37)

where $\tau$ is typically set as $\tau =0.001$.

The specific steps are shown in Algorithm 2.

Algorithm 2 MADDPG for Optimization Problem

1: Initialization: Randomly set ${\mathit{\theta }}^{\pi }$, ${\mathit{\theta }}^{{\pi }^{\prime }}$, ${\mathit{\theta }}^Q$, ${\mathit{\theta }}^{Q^{\prime }}$; $\mathcal{D}=⌀$. 2: for  $\mathrm{episode}=1\mathrm{to}\max -\mathrm{episode}-\mathrm{num}$ do 3:    Reset the environment, and receive initial state $\mathit{s}=\{{\mathit{s}}_1,{\mathit{s}}_2,\dots ,{\mathit{s}}_U\}$. 4:    for $\mathrm{t}=1\mathrm{to}\max -\mathrm{episode}-\mathrm{length}\mathrm{T}$ do 5:      for each agent u, select action ${\mathit{a}}_u={\mathit{\pi }}_u\left({\mathit{s}}_u\right)+\mathrm{noise}$. 6:      Execute actions $\mathit{a}=\{{\mathit{a}}_1,{\mathit{a}}_2,\dots {\mathit{a}}_U\}$ and observe reward $\mathit{r}=\{r_1,r_2,\dots r_U\}$ and next state ${\mathit{s}}^{\prime }$. 7:      Push $(\mathit{s},\mathit{a},\mathit{r},{\mathit{s}}^{\prime })$ into replay buffer $\mathcal{D}$. 8:      $\mathit{s}\leftarrow {\mathit{s}}^{\prime }$. 9:      if $\mathrm{length}\mathrm{of}\mathcal{D}\mathrm{larger}\mathrm{than}\mathrm{given}\mathrm{length}$ then 10:         for $\mathrm{UAV}\mathrm{agent}\mathrm{u}=1\mathrm{to}\mathrm{U}$ do 11:           Sample a random batch of S samples ${\left\{({\mathit{s}}^j,{\mathit{a}}^j,{\mathit{r}}^j,{{\mathit{s}}^{\prime }}^j)\right\}}_{j=1,\dots ,S}$ from $\mathcal{D}$. 12:           Set $y^j=r_u^j+\gamma {Q^{\prime }}_u^{{\mathit{\pi }}^{\prime }}({\mathit{s}}^{\prime j},{\mathit{a}}^{\prime j}){|}_{{\mathit{a}}_u^{\prime j}={\mathit{\pi }}_u^{\prime }\left({\mathit{s}}_u^j\right)}$. 13:           Update critic ${\theta }_u^Q$ by minimizing the loss $\mathcal{L}\left({\theta }_u^Q\right)=\frac{1}{S}{\sum }_j{(Q_u^{\mathit{\pi }}({\mathit{s}}^j,{\mathit{a}}^j)-y^j)}^2$. 14:           Update actor ${\theta }_u^{\pi }$ using the sampled policy gradient ${\nabla }_{{\theta }_u^{\pi }}J\left({\theta }_u^{\pi }\right)\approx \frac{1}{S}{\sum }_j\left[{\nabla }_{{\mathit{a}}_u^j}{Q}^{\mathit{\pi }}({\mathit{s}}^j,{\mathit{a}}^j){|}_{{\mathit{a}}_u^j={\mathit{\pi }}_u\left({\mathit{s}}_u^j\right)}{\nabla }_{{\theta }_u^j}{\mathit{\pi }}_u\left({\mathit{s}}_u^j\right)\right]$. 15:         end for 16:         Update target network parameters by ${\mathit{\theta }}^{{\pi }^{\prime }}\leftarrow \tau {\mathit{\theta }}^{\pi }+(1-\tau ){\mathit{\theta }}^{{\pi }^{\prime }}$ and ${\mathit{\theta }}^{Q^{\prime }}\leftarrow \tau {\mathit{\theta }}^Q+(1-\tau ){\mathit{\theta }}^{Q^{\prime }}$. 17:      end if 18:    end for 19: end for

6. Simulation Results

In this section, we simulate the performance of MADDPG in the multi-UAV multicast system, comparing it with DDPG, Deep Q-Network (DQN), DDQN, and a classical RL algorithm Upper Confidence Bound (UCB) designed for Budgeted Multi-Armed Bandits (BMABs) problems. This paper considers a three-dimensional space with four cellular cells, each equipped with a GBS at the center. A total of 100 GUs are randomly distributed in the targeted area. The GUs are determined whether or not they are served by a UAV acting as an aerial base station according to (3). The GUs are divided into five multicast groups according to their locations and transmission rate requirements. There are five UAVs to serve them, and the initial positions of the UAVs are set as $(400,400,200), (400,-400,200),$$(-400,400,200), (-400,-400,200), \mathrm{and} (0,0,0)$, respectively. For the channel model, we set $C=5$ and $D=0.35$ in (5) [25]. The total transmission bandwidth and noise power are $B=2$ MHz and ${\sigma }^2=-100$ dBm, respectively. The maximal power, the UAV’s maximal speed, and the rotor tip speed are set as $P_{max}=500$ mW, $V_{max}=30$ m/s, and $U_{tip}=200$ m/s, respectively. We utilize PyTorch and Gym to model the environment and simulate the algorithms. In MADDPG, the actor and critic networks are both set as fully connected neural networks with [128, 64] neurons where the activation function of the two hidden layers is $ReLU$. We utilize $tanh$ as the activation function of the actor network’s output layer, which can restrict the output action within $(-1,1)$. For the optimizer, the Adam optimizer is applied to train the DNNs in the policy network and Q-Network. During the training, the length of an episode is set as $T=25$. The capacity of the buffer and the batch size are ${10}^5$ and 1024, respectively. The specific simulation settings are shown in

Table 1. Parameters settings on simulation.

Parameter	Value
Number of ground users N	100
Number of UAVs K	5
Number of multicast groups k	5
Blocking parameters $C,D$	5, 0.35
Bandwidth B	2 MHz
UAV maximal power $P_{max}$	500 mW
Noise power ${\sigma }^2$	−100 dBm
Blade power $P_0$	$\frac{12\times {30}^3\times 0.4^3}{8}\rho sG$
Descending/Ascending power $P1$	11.46
$V_{max}$, $U_{tip}$	30, 200
s, $\rho$, G	0.05, 1.225, 0.503 [26]
S, $d_{min}$	30, 20 m
Number of episodes	15,000
Length of an episode T	25
Batch size, Learning rate	1024, 0.001

.

In Figure 8, we plot the performance of Algorithm 1 to solve the multicast grouping problem. There are 160 edge GUs that cannot obtain reliable communication from the GBS in the figure and are classified into K = 8 multicast groups. As shown in Figure 8, there are seven cellular cells in the two-dimensional space of $(-500,500)\times (-500,500)$, and the red triangle in the center of each cell denotes the GBS. The scatters denote randomly distributed GUs in the space of $(-350,350)\times (-350,350)$, and scatters with the same color indicate that the users are classified into the same multicast group.

In Figure 9, we depict the average reward of different algorithms during 15,000 episodes. The average reward in each episode is set as the negative value of the average energy consumed per UAV within a time slot. We compare the training procedure of MADDPG with that of three DRL methods. For the discrete action space, we discretized the continuous action space, where the transmit power set is $P_{u,t}=P_{max}·\{0.1,0.2,\dots ,1\}$ and the UAV’s speed set is ${\mathit{v}}_{u,t}=\{[-1,-1,-1],[-1,-1,1],[-1,1,-1],[-1,1,1],[1,-1,-1],$$[1,-1,1],[1,1,-1],[1,1,1]\}$. As shown in Figure 9, the average reward of all algorithms increases with the number of episodes. The discrete action space of DQN is only a subset of that of DDPG. As a result, DQN converges more quickly compared to DDPG, but the final performance of DDPG is slightly higher than that of DQN. DDQN performs similarly to DQN. As for MADDPG, since the framework of centralized training and distributed execution can solve the issue of non-stationarity, MADDPG can achieve a higher average reward and save the UAVs’ energy more effectively.

The optimal 3D trajectory of the UAV obtained by solving $\mathbf{P}\mathbf{2}$ with MADDPG is plotted in Figure 10. The order in which each agent serves the multicast groups is basically the same, and different UAVs serve different multicast groups in the same time slot. For the sake of observation, only the trajectory of one UAV is plotted, and it can be seen that the UAVs decide the altitude that minimizes the energy consumption based on the served multicast group’s transmission rate requirement.

In Figure 11, we compare the average throughput of different algorithms when grouping GUs and not grouping. In contrast to communicating with all users by broadcasting, the average throughput is improved by multicast grouping considering the characteristics of the GUs’ respective transmission rate requirements. In that case, the average throughput is not restricted to the GUs with the worst channel but only to those within the multicast group. Meanwhile, MADDPG also performs well in improving the GUs’ throughput compared to DQN and DDPG.

In Figure 12, we plot the energy consumption as the number of multicast groups K varies under different algorithms. As K increases, UAVs need to travel longer distances to serve more multicast groups and consume more energy for transmission. However, if a smaller K is chosen, although the energy consumption decreases, the average throughput will also reduce. As a result, we chose an intermediate value $K=5$ to make a trade-off. As for different algorithms, MADDPG performs better than DQN, DDQN, DDPG, and the classical RL method UCB in reducing the energy consumption of UAVs. The results are consistent with those in Figure 9.

7. Conclusions

This paper investigates the optimization of grouping, joint 3D trajectories, and power allocation in multi-UAV multicast systems, aiming to minimize the energy consumption of UAVs. To achieve this goal, this paper first solves the multicast grouping problem for users with different transmission rate requirements using the constrained K-medoids algorithm. Then, due to the problem of non-stationarity caused by multiple agents training in the traditional DRL algorithm, this paper adopts MADDPG to eliminate the non-stationarity and set the negative value of UAVs’ energy consumption as the reward. Simulation results show that MADRL can effectively reduce the energy consumption of UAVs, and, at the same time, the combination of a multicast communication approach and using UAVs as aerial base stations can effectively improve the average throughput. Based on this work, we will investigate the performance of the proposed multi-UAV multicast system in other scenarios, such as vehicular environments.