Heterogeneous Multi-Agent Deep Reinforcement Learning for Cluster-Based Spectrum Sharing in UAV Swarms

Abstract

Unmanned aerial vehicle (UAV) swarms are widely applied in various fields, including military and civilian domains. However, due to the scarcity of spectrum resources, UAV swarm clustering technology has emerged as an effective method for achieving spectrum sharing among UAV swarms. This paper introduces a distributed heterogeneous multi-agent deep reinforcement learning algorithm, named HMDRL-UC, which is specifically designed to address the cluster-based spectrum sharing problem in heterogeneous UAV swarms. Heterogeneous UAV swarms consist of two types of UAVs: cluster head (CH) and cluster member (CM). Each UAV is equipped with an intelligent agent to execute the deep reinforcement learning (DRL) algorithm. Correspondingly, the HMDRL-UC consists of two parts: multi-agent proximal policy optimization for cluster head (MAPPO-H) and independent proximal policy optimization for cluster member (IPPO-M). The MAPPO-H enables the CHs to decide cluster selection and moving position, while CMs utilize IPPO-M to cluster autonomously under the condition of certain partial channel distribution information (CDI). Adequate experimental evidence has confirmed that the HMDRL-UC algorithm proposed in this paper is not only capable of managing dynamic drone swarm scenarios in the presence of partial CDI, but also has a clear advantage over the other existing three algorithms in terms of average throughput, intra-cluster communication delay, and minimum signal-to-noise ratio (SNR).

1. Introduction

As unmanned aerial vehicle (UAV) technology advances toward swarm intelligence and collaborative operations, UAV swarms have been increasingly applied in various fields, such as auxiliary communication, emergency response, target localization, reconnaissance exploration and internet of things (IoT) [1,2,3]. Recent studies demonstrate the versatility of UAVs in both fundamental tasks (e.g., autonomous parking occupancy detection [4]) and large-scale networked operations [5], highlighting the need for adaptive control in dynamic environments. Specially, flying ad hoc networks (FANETs), as a common organizational form of UAV swarms, can facilitate information interaction between UAVs and other devices based on the inter-UAV communication [6]. Furthermore, with the continuous development and upgrading of UAVs, various types of UAVs with different airborne capacities are included in the same group. Therefore, it becomes more practical to consider heterogeneous UAVs within FANETs [7].

However, as the number of UAVs within a network escalates, the conflict between the scarcity of spectrum resources and the necessity for reliable communication becomes increasingly acute. Spectrum sharing emerges as a potent strategy for resource utilization, aimed at facilitating dependable and secure communications within multi-UAV networks [8,9]. To achieve efficient spectrum sharing, the concept of coalition partitioning is introduced, which entails clustering the UAV swarm into distinct groups, particularly in scenarios involving multiple tasks [10]. In the context of a large-scale FANET, the adoption of clustering algorithms is imperative for effective networking and management [11]. Especially in heterogeneous scenarios, a single cluster consists of various types of UAVs, requiring efficient task completion while adhering to constraints related to latency, throughput, and other communication metrics.

In recent studies, clustering channels with similar interference characteristics have played a significant role in mitigating cross-channel interference and simplifying the demands of time synchronization, thereby lowering the overall system complexity [12,13]. In addition, clustering techniques have significantly enhanced the efficiency of spectrum utilization by partitioning spectrum resources into clusters [14,15]. Further, resource optimization within clusters has improved energy efficiency, allowing energy harvesting-supported device-to-device (EH-D2D) communication in multicast mode to utilize available energy more effectively, enhancing overall network performance [16]. And in [17], limited spectrum resources are effectively shared among a large number of users while simultaneously controlling interference and meeting quality of service (QoS) requirements. Moreover, clustering also achieves dynamic spectrum sharing. In [18], full-duplex user-centric ultra-dense networks adapt to changes in user numbers and locations through clustering schemes, dynamically adjusting the number and members of clusters to cope with dynamic network environments, maintaining the efficiency of spectrum sharing. It is evident that the application of clustering technology in complicated spectrum sharing networks is multifaceted. These effects will also collectively drive the development of UAV swarm communication networks toward greater efficiency and intelligence.

Many researchers have explored various clustering strategies for UAV swarms, with a focus on enhancing network performance, scalability, and adaptability [11,19,20,21,22,23,24]. A common clustering method is distance-based, such as K-means [19]. In contrast to the hard partitioning of K-means, an improved LB-FCM algorithm with a fuzzification parameter and a load balancing mechanism is proposed to enhance the algorithm’s flexibility and robustness in [20]. Further, UAVs are clustered according to an unsupervised scheme without pre-initializing the number in [21]. Swarm intelligence algorithm, like particle swarm optimization (PSO), is also adopted in [22], where a clustering routing protocol based on improved PSO algorithm determines the cluster head selection and clustering of FANETs in disaster response. To address the challenge of the mobility in UAV swarms, the UAV is clustered based on location prediction in [23], while the clustering problem is modeled as a graph cut problem in [24]. Moreover, in order to handle the multi-objective optimization in mobile edge computing (MEC), an iterative optimization algorithm based on penalty and block coordinate descent (BCD) is designed to tune the cluster head selection, positions, and transmit powers of UAVs in [11]. Recent studies further address critical challenges such as robustness in constrained environments [25] and energy-efficient control [26].

Nevertheless, the clustering approaches mentioned in the aforementioned articles mainly rely on a central control node, which limits the system’s ability to adapt to dynamic network conditions and recover from failures. To address this problem, distributed algorithms are proposed to enable autonomous clustering among UAVs. Coalition formation game (CFG) is used to jointly optimize task selection and spectrum allocation in UAV communication networks [27,28,29,30]. When the Global Navigation Satellite System (GNSS) is not functioning properly [29], CFG is also applied for clustering-based cooperative localization. Furthermore, a task scheduling-oriented clustering method for UAV swarms based on a dynamic super round has also been proposed in [31], which can effectively reduce the number of invalid clustering rounds. Moreover, recent decentralized frameworks, as exemplified in [32], have demonstrated progress in UAV swarm security enhancement through the systematic integration of machine learning methodologies. However, the above-mentioned distributed algorithm has high computational complexity, poor cooperation stability, lacks long-term planning capability, and has difficulty in adapting to dynamic environments.

Recently, reinforcement learning (RL) has been applied into the clustering of UAV swarms, significantly enhancing adaptive cooperative operation capability and task execution efficiency in complex environments [33,34,35]. Specially, due to the remarkable advantages of the multi-agent method, such as distributed decision-making, collaborative optimization, dynamic adaptation, and scalability, the authors in [33] researched the multi-agent RL algorithm, setting each UAV as an agent and determining the actions via an actor–critic network. However, the RL architecture suffers from insufficient heterogeneity and scalability. UAV swarms often contain heterogeneous individuals such as reconnaissance UAVs, transportation UAVs, and relay UAVs. There are significant differences in these heterogeneous individuals’ state and action spaces. Therefore, customized RL models are required, which exacerbates the computational and training complexity. Moreover, when the UAV swarm expands from a small scale (e.g., five UAVs) to a large scale (e.g., one hundred UAVs), the scale of RL parameters and the complexity of coordination increase superlinearly. To avoid the above drawback, the proximal policy optimization (PPO) algorithm [36], as an advanced RL method, offers significant advantages in addressing resource optimization problems for UAV swarms due to its stability, efficiency, and ease of implementation.

Motivated by the above, in this paper, we consider a heterogeneous UAV swarm, which needs to achieve spectrum sharing through reasonable clustering. Based on the multi-agent method and the PPO algorithm, a heterogeneous multi-agent deep reinforcement learning algorithm for UAV clustering (HMDRL-UC) is proposed to control the actions of UAVs in the presence of partial channel distribution information (CDI). The novel contributions of this paper can be summarized as follows:

• We consider a large-scale heterogeneous UAV swarm capable of meeting various mission requirements. The UAV swarm is divided into multiple small groups, and consists of two types of UAVs: cluster head (CH) and cluster member (CM). In particular, we study a practical setup, whereby the average communication delay for CMs and throughput for CHs are jointly optimized under the assumptions of knowing the partial CDI. Moreover, the link maintenance probability is discussed for CMs with the consideration of the mobility of CMs.
• We propose a novel distributed HMDRL-UC algorithm for heterogeneous UAV swarms, where each UAV acts as an intelligent agent. In detail, the HMDRL-UC is composed of two subalgorithms: the multi-agent PPO for CHs (MAPPO-H) and the independent PPO for CMs (IPPO-M). MAPPO-H is responsible for controlling the cluster selection and movement of CHs, while the IPPO-M is used to decide the cluster selection of CMs.
• A comprehensive performance analysis is carried out for the proposed HMDRL-UC algorithm, as well as a comparison with those of existing algorithms. The proposed HMDRL-UC algorithm demonstrates significant advantages in terms of average throughput, intra-cluster communication delay, and minimum signal-to-noise ratio (SNR). Moreover, it can optimize clustering for dynamic UAV swarm scenarios. Therefore, the HMDRL-UC algorithm is a highly promising clustering-based spectrum sharing candidate for multi-mission large-scale UAV swarm scenarios.

The outline of this article is arranged as follows. Section 2 introduces the system model and formulates the optimization problems. Section 3 proposes an HMDRL-UC algorithm. Simulation results are shown in Section 4. Finally, we summarize this paper in Section 5.

2. System Model and Problem Formulation

We consider a large UAV swarm to perform diversified reconnaissance missions, such as disaster monitoring, traffic management, environmental surveillance, etc. For example, the UAVs are equipped with a variety of sensing devices, including visible-light cameras, infrared cameras, or LiDAR, and they are supposed to conduct joint reconnaissance to obtain comprehensive reconnaissance results for a specific area. In order to improve the efficiency of mission completion, the UAVs are organized into small groups and maintain communication with each other for collaborative sensing and cooperative decision-making. Therefore, communication delay becomes one of the key factors in determining the operational efficiency of the system, especially in some mobile edge computing (MEC) networks [37,38]. However, due to the limitation of storage and processing capability onboard, some UAVs have to offload the collected data to the UAV with stronger ability. In this case, maximizing throughput can effectively improve the efficiency of data transmission. As shown in Figure 1, the UAV swarm consists of a control center, H cluster heads (CHs), and M cluster members (CMs). Among them, the control center has established stable communication links with CHs for swarm control and data transmission. CMs are divided into H clusters, where they collaborate to complete tasks and transmit data back to the corresponding CH. Nevertheless, given the rapid changes in tasks or environments, as well as the coordinated movement of the swarm, the relative positions of the CMs may undergo dynamic changes, which brings great challenges to the clustering of the UAV swarm. Consequently, the CMs are expected to cluster autonomously, while CHs adaptively select the service cluster and seek the optimal position.

For the sake of simplicity, we define the clustering strategies for CMs, the cluster selection strategies for CHs and the three-dimensional (3D) location of CHs as ${\mathcal{C}}^{\mathrm{M}}=\{C_1^{\mathrm{M}},C_2^{\mathrm{M}},\dots ,C_M^{\mathrm{M}}\}$, ${\mathcal{C}}^{\mathrm{H}}=\{C_1^{\mathrm{H}},C_2^{\mathrm{H}},\dots ,C_H^{\mathrm{H}}\}$ and ${\mathcal{L}}^H=\{(x_1,y_1,z_1),(x_2,y_2,z_2),\dots ,(x_H,y_H,z_H)\}$, respectively. Note that both the radius of the cluster and the number of CMs within the cluster will influence the communication quality of the UAV swarm. An excessively large cluster radius may compromise the reliability of communication links between the CH and its service CMs. Furthermore, an excessive number of CMs within the cluster can result in significant communication delays. Therefore, we define the hth ($h\in \{1,\dots ,H\}$) cluster radius, number of CMs within the hth ($h\in \{1,\dots ,H\}$) cluster, maximum cluster radius, and maximum number of CMs within a cluster as $r_h$, $n_h$, $r_{\max }$, and $n_{\max }$, respectively.

2.1. Link Maintenance Probability Prediction of CMs

During the task execution, the relative movement between CMs can affect the clustering results. Hence, it is necessary to predict the communication link maintenance probability by predicting UAVs’ movements. We assume that the CMs transmit Hello packets periodically, and during the interval, the CMs maintain a constant velocity. As shown in Figure 2, ${\mathrm{UAV}}_i$ is assumed to be stationary at time $t_0$, while ${\mathrm{UAV}}_j$ moves along the red line with a relative speed of $v_j^{t_0}$. Both successfully receive each other’s Hello messages, which include timestamps, transmitted carrier frequency, and signal power to enable motion estimation. The predicted flight trajectory $d_f$ of ${\mathrm{UAV}}_j$ and its relative speed $v_j^{t_0}$ are estimated using Doppler frequency shifts and received signal power variations across successive Hello packets, following the mobility prediction framework in [39]. Specifically, $v_j^{t_0}$ is derived from the Doppler shift $\Delta f$ as $v_j^{t_0}cos\theta =c\Delta f/f$, where $\theta$ is the angle between the relative velocity and the LOS path, and $d_f$ is calculated using the path loss model to track distance changes over time. As suggested by [39], the link maintenance probability between ${\mathrm{UAV}}_i$ and ${\mathrm{UAV}}_j$ can be expressed as

$${\eta }_{i,j}=\left\{\begin{array}{cc}{\displaystyle \frac{d_f}{v_j^{t_0}{T}_0}},\hfill & {\displaystyle \frac{d_f}{v_j^{t_0}}}<T_0,\hfill \\ 1,\hfill & {\displaystyle \frac{d_f}{v_j^{t_0}}}\ge T_0,\hfill \end{array}\right.$$

(1)

where $T_0$ denotes the time threshold.

2.2. Channel Model

The air-to-air communication channel gains follow the Nakagami-m distributed fading model [40]. It is presumed that the channel gains are independent and identically distributed (i.i.d.), and remain constant during the transmission of each packet [41]. Hence, the probability density functions (PDFs) of the channel gains $g_{i,j}$ between ${\mathrm{UAV}}_i$ and ${\mathrm{UAV}}_j$ can be expressed as

$$f_{g_{i,j}}\left(x\right)={\left({\displaystyle \frac{m_{i,j}}{{\mathsf{\Omega }}_{i,j}}}\right)}^{m_{i,j}}{\displaystyle \frac{x^{m_{i,j}-1}}{\Gamma \left(m_{i,j}\right)}}\exp (-{\displaystyle \frac{m_{i,j}x}{{\mathsf{\Omega }}_{i,j}}}),$$

(2)

where $m_{i,j}$ is the fading severity parameter, $\Gamma (·)$ represents the Gamma function, $g_{i,j}$ is a random variable with mean value ${\mathsf{\Omega }}_{|g_{i,j}{|}^2}=\mathrm{E}\left[\right|g_{i,j}{|}^2]$, and ${\mathsf{\Omega }}_{i,j}=\mathrm{E}\left[g_{i,j}\right]$. The Nakagami-m fading parameter $m_{i,j}$ is constrained to $m_{i,j}\ge 0.5$, ensuring the Gamma function $\Gamma \left(m_{i,j}\right)$ remains well defined. For $x=0$, $f_{g_{i,j}}\left(x\right)$ either attains a finite value (e.g., $m_{i,j}=1$) or represents a low-probability event (e.g., $m_{i,j}<1$), consistent with the physical interpretation of channel outages. Note that, in real-world UAV communication systems, $g_{i,j}=0$ corresponds to channel outages, which are inherently excluded during active communication phases.

In particular, because there are no obstacles in the sky, the path loss $L_{i,j}$ between ${\mathrm{UAV}}_i$ and ${\mathrm{UAV}}_j$ can be further described by the Friis free-space transmission formula. The standard Friis transmission equation defines the received power as

$$P_r=P_t·{\displaystyle \frac{G_t{G}_r{\lambda }^2}{{\left(4\pi d_{i,j}\right)}^2}},$$

(3)

where $P_t$ is the transmit power; $G_t$ and $G_r$ are the antenna gains of the transmitter and receiver, respectively; $\lambda ={\displaystyle \frac{c}{f}}$ is the wavelength; f denotes the center frequency of the carrier; $d_{i,j}$ is the distance between ${\mathrm{UAV}}_i$ and ${\mathrm{UAV}}_j$; and $c=3\times {10}^8\mathrm{m}/\mathrm{s}$. In our system model, we assume isotropic antennas (omnidirectional radiation pattern), i.e., $G_t=G_r=1$. The path loss $L_{i,j}$ is defined as the ratio of the transmitted power to the received power, which can be expressed as

$$L_{i,j}={\displaystyle \frac{P_t}{P_r}}={\left({\displaystyle \frac{4\pi d_{i,j}f}{c}}\right)}^2.$$

(4)

Therefore, the SNR from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$ is shown as

$${\mathsf{\Phi }}_{i,j}={\displaystyle \frac{{\eta }_{i,j}{P}_{i,j}{\left|g_{i,j}\right|}^2}{N_0{L}_{i,j}}},$$

(5)

where $N_0$ denotes the additive white Gaussian noise and $P_{i,j}$ represents the signal transmit power from ${\mathrm{UAV}}_i$ to ${\mathrm{UAV}}_j$. Then, the successful packet transmission probability can be expressed by

$$P({\mathsf{\Phi }}_{i,j}>\mu )=P(P_{i,j}>{\displaystyle \frac{\mu N_0{L}_{i,j}}{{\eta }_{i,j}{\left|g_{i,j}\right|}^2}}),$$

(6)

where $\mu$ denotes the SNR threshold. In this way, the average transmission delay ${\mathsf{\xi }}_{i,j}$ could be defined as

$${\mathsf{\xi }}_{i,j}={\overline{\mathsf{\Psi }}}_{i,j}(T_1+T_2),$$

(7)

where $T_1$ and $T_2$, respectively, denote the packet preparation delay and the packet transmission delay, and ${\overline{\mathsf{\Psi }}}_{i,j}$ represents the average number of retransmissions, which can be defined as follows:

$${\overline{\mathsf{\Psi }}}_{i,j}={\displaystyle \frac{1}{P({\mathsf{\Phi }}_{i,j}>\mu )}}.$$

(8)

2.3. Problem Formulation

In this paper, our objective is to seek proper clustering strategies ${\mathcal{C}}^M$ for CM, which minimizes the average intra-cluster communication delay and to identify optimal cluster selection strategy ${\mathcal{C}}^H$ and locations $\mathcal{L}$ for CHs to maximize overall cluster throughput, subject to the clustering constraints. The optimization problem can be described as follows:

$$\begin{array}{ccc}\hfill \left({\mathcal{P}}_1\right)& \underset{{\mathcal{C}}^M}{min}{\displaystyle \frac{1}{\left|C\right|}}\sum _{C^{\prime }\in \mathcal{C}}\sum _{\{i,j\}\in C^{\prime }}{\delta }_{i,j}{\tau }_{i,j},\hfill & \end{array}$$

(9)

$$\begin{array}{ccc}\hfill \left({\mathcal{P}}_2\right)& \underset{\{{\mathcal{C}}^H,\mathcal{L}\}}{max}\sum _{m\in \mathcal{M},n\in \mathcal{H}}{\sigma }_{m,n}{B}_{m,n}{log}_2(1+{\mathsf{\Phi }}_{m,n}),\hfill & \end{array}$$

(10)

subject to

$$\begin{array}{ccc}\hfill \left({\mathcal{S}}_1\right)& r_h\le r_{\max },\forall h,\hfill & \end{array}$$

(11)

$$n_h\le n_{\max },\forall h,$$

(12)

where $\left|C\right|$ denotes the cluster number of the cluster set $\mathcal{C}$; $C^{\prime }$ is one of $\mathcal{C}$; $\mathcal{M}$ represents the set of CMs; $\mathcal{H}$ represents the set of CHs; and $B_{m,n}$ is the signal bandwidth from ${\mathrm{CM}}_m$ to ${\mathrm{CH}}_n$. ${\delta }_{i,j}$ represents the communication probability of ${\mathrm{CM}}_i$ and ${\mathrm{CM}}_j$, which can be described as

$${\delta }_{i,j}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathrm{CM}}_i\mathrm{and}{\mathrm{CM}}_j\mathrm{are}\mathrm{in}\mathrm{the}\mathrm{same}\mathrm{cluster},\hfill \\ 0,\hfill & \mathrm{if}{\mathrm{CM}}_i\mathrm{and}{\mathrm{CM}}_j\mathrm{are}\mathrm{not}\mathrm{in}\mathrm{the}\mathrm{same}\mathrm{cluster}.\hfill \end{array}\right.$$

(13)

Similarly, the link status ${\sigma }_{m,n}$ can be expressed as

$${\sigma }_{m,n}=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathrm{CM}}_m\mathrm{is}\mathrm{linked}\mathrm{with}{\mathrm{CH}}_n,\hfill \\ 0,\hfill & \mathrm{if}{\mathrm{CM}}_m\mathrm{is}\mathrm{not}\mathrm{linked}\mathrm{with}{\mathrm{CH}}_n.\hfill \end{array}\right.$$

(14)

The communication delay ${\tau }_{i,j}$ can be described as

$${\tau }_{i,j}=\sum _{\{i^{\prime },j^{\prime }\}\in {\alpha }_{i,j}}{\mathsf{\xi }}_{i^{\prime },j^{\prime }},$$

(15)

where ${\alpha }_{i,j}=[(i,i_1),(i_1,i_2),\dots ,(i_n,j)]$ represents the shortest path between ${\mathrm{UAV}}_i$ and ${\mathrm{UAV}}_j$.

The problem given by $\left({\mathcal{P}}_1\right)$, $\left({\mathcal{P}}_2\right)$ and $\left({\mathcal{S}}_1\right)$ is a multi-objective nonlinear optimization problem. The commonly used method is to either decompose the joint optimization problem into smaller sub-problems for iterative optimization [11], resolve it through game theory [29], or utilize swarm intelligence methodologies [42]. However, the above methods may encounter high computational complexity and difficulties in handling high-dimensional, dynamically changing environments, etc. Therefore, in this paper, a distributed heterogeneous multi-agent deep reinforcement learning model called HMDRL-UC is proposed. Each UAV within the swarm is treated as an intelligent agent, autonomously making decisions through essential communications to address the highly dynamic and intricate scenarios of swarm movement. In particular, HMDRL-UC serves two types of UAVs: CHs and CMs, which are governed by MAPPO-H and IPPO-M, respectively. The tasks of CMs are to make decisions about clustering, while CHs are responsible for selecting the cluster to serve and for locating the optimal position.

3. Proposed Heterogeneous Multi-Agent Deep Reinforcement Learning Algorithm for UAV Clustering (HMDRL-UC)

In this section, we first introduce the framework of the HMDRL-UC, and its main parts. Then, the training procedure and complexity analysis are discussed for the HMDRL-UC.

3.1. The Framework of HMDRL-UC

As shown in Figure 3, the designed HMDRL-UC consists of two categories of agents: member agents and head agents, which server the CMs and the CHs, respectively. Both of them share the same environment. However, the HMDRL-UC allows member agents and head agents to have different observation spaces, actions spaces, and distinctive reward functions. The actions to be taken by agents are generated by an actor–critic structure. In detail, the actor network, called policy network, is responsible for action selection according to the current state, whereas the critic network predicts the future outcomes based on the action–state pairs to assesses the quality of the chosen actions. Furthermore, although the HMDRL-UC is a distributed framework, head agents employ a centralized training with a decentralized execution (CTDE) strategy, called the HAPPO-H algorithm, enabling them to share global information. Leveraging superior computational and communicative resources, head agents communicate with the control center during the centralized training phase, sharing their critic networks. This sharing of training data among head agents facilitates the identification of complex environments and globally optimal solutions. Specifically, after the model training is completed, a central control node is not required during the execution phase. In contrast, member agents, constrained by limited computational and communicative resources, adopt a fully decentralized independent training approach, called the IPPO-M algorithm, eliminating the need for communication with other agents during training. This approach enhances scalability and robustness. Each member agent can gain a precise understanding of the environment and enhance collaborative efficacy through the observation of others’ information. During the decentralized execution phase, member agents independently determine and select actions based on local observations.

3.2. The Design of IPPO-M

During the collective movement of UAV swarms, CMs must autonomously decide which cluster to join to facilitate collaborative communication. Considering the high scalability of a large number of CMs, yet their relatively weak on-board processing capabilities, we allow member agents to make decisions based on their individual local observations. Given that the actions mainly involve discrete choices, the IPPO algorithm framework is utilized to develop effective action strategies. Each CM is equipped with an IPPO-M subnetwork to obtain the optimal policy ${\pi }^m$ from the observation set $O^m$ to the action set ${\mathcal{A}}^m$ with the help of the reward function $R^m$.

3.2.1. Observation Set

As shown in Figure 2, the CMs periodically transmit Hello packets, which are utilized to estimate the velocity of neighboring UAVs and to forecast their movements. Further, by integrating the routing discovery protocol [43], the CM can acquire intra-cluster routing information and assess communication delays. To facilitate efficient clustering, the CM also broadcasts its cluster ID to neighboring areas. Consequently, the CM’s observation set encompasses both its own status and relevant information from neighboring agents. So, the observation set $O^m$ observed by a member agent ${\mathrm{UAV}}_m$ consists of fourteen parts: its own position $p_m$, its own velocity $v_m$, its own cluster ID $c_m$, its own communication delay $d_m$, the position $p_j$ of the neighbor ${\mathrm{UAV}}_j$, the velocity $v_j$ of the neighbor ${\mathrm{UAV}}_j$, the cluster ID $c_j$ of the neighbor ${\mathrm{UAV}}_j$, the link maintenance probability ${\eta }_{m,j}$ between ${\mathrm{UAV}}_m$ and ${\mathrm{UAV}}_j$, the path loss $L_{m,j}$ between ${\mathrm{UAV}}_m$ and ${\mathrm{UAV}}_j$, estimated mean value ${\mathsf{\Omega }}_{|g_{m,j}{|}^2}$, ${\mathsf{\Omega }}_{m,j}$ of the channel gain between ${\mathrm{UAV}}_m$ and ${\mathrm{UAV}}_j$, the signal transmit power $P_{m,j}$ from ${\mathrm{UAV}}_m$ to ${\mathrm{UAV}}_j$, the fading severity parameter $m_{m,j}$, and the additive white Gaussian noise $N_0$. Therefore, the observation set $O^m$ can be expressed as $O^m=\{p_m,v_m,c_m,d_m,p_j,v_j,c_j,{\eta }_{m,j},L_{m,j},{\mathsf{\Omega }}_{|g_{m,j}{|}^2},{\mathsf{\Omega }}_{m,j},P_{m,j},m_{m,j},N_0\}$.

3.2.2. Action Set

Once the member agents obtain an observation of the environment, the actor network will generate a specific action for member agents according to the policy ${\pi }^m$. In particular, the member agents ${\mathrm{UAV}}_m$ are supposed to select an action $a_m$ from the discrete action set ${\mathcal{A}}^m=\{1,\dots ,H\}$, where H is the number of clusters, and $a_m=h(h\in \{1,2,\dots ,H\})$ represents the member agent ${\mathrm{UAV}}_m$ selected to join the hth cluster.

3.2.3. Reward

After taking the action $a_m$, the member agent ${\mathrm{UAV}}_m$ will receive a reward $R^m$ from the critic network. The critic network will evaluate the value of action $a_m$ on the basis of the influenced environment and inform the member agent whether this is a good action. In conclusion, we encourage member agents to establish stable connections with neighbors, so as to participate in cluster cooperation. Meanwhile, we punish the isolated points and the delay between the member agent and its neighbors. In this way, each member agent seeks to perform the action that maximizes its cumulative reward. Therefore, the reward function of the member agent ${\mathrm{UAV}}_m$ is defined as

$$\begin{array}{c}\hfill R^m=\sum _{j\in ϵ\left(m\right)}{\eta }_{m,j}-{\zeta }_1{\mathrm{K}}_m-{\displaystyle \frac{{\zeta }_2}{\left|ϵ\right(m\left)\right|}}\sum _{f\in ϵ\left(m\right)}{\tau }_{m,f},\end{array}$$

(16)

where ${\zeta }_1$ and ${\zeta }_2$ are the discount factors, $ϵ\left(m\right)$ represents the neighbor set of member agent ${\mathrm{UAV}}_m$, $\left|ϵ\right(m\left)\right|$ denotes the number of neighbors, and ${\eta }_{m,j}$ is the link maintenance probability between member agent ${\mathrm{UAV}}_m$ and its neighbor ${\mathrm{UAV}}_j$ at time t. ${\mathrm{K}}_m$ indicates whether the member agent ${\mathrm{UAV}}_m$ is an isolated node, which can be described as

$${\mathrm{K}}_m=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}{\mathrm{UAV}}_m\mathrm{is}\mathrm{an}\mathrm{isolated}\mathrm{node},\hfill \\ 0,\hfill & \mathrm{if}{\mathrm{UAV}}_m\mathrm{is}\mathrm{not}\mathrm{an}\mathrm{isolated}\mathrm{node}.\hfill \end{array}\right.$$

(17)

3.3. The Design of MAPPO-H

Once CMs have formed stable clusters, the CHs first identify the cluster to be serviced, and then seek the most advantageous data transmit position to maximize the overall throughput of the UAV swarm. Therefore, the head agents have to determine two actions: the cluster to serve and the moving position. CHs generally possess more onboard processing capabilities and are less numerous; hence, we adopted the MAPPO algorithm to manage the actions of cluster head agents. Each CH is equipped with a MAPPO-H subnetwork to learn the optimal policy ${\pi }^h$ from the observation set $O^h$ to the action set ${\mathcal{A}}^h$ with the help of reward function $R^h$.

3.3.1. Observation Set

Because of the shared environment, the head agent is able to obtain the states of other head agents and also derives the states of member agents within the servicing cluster. Therefore, the head agent ${\mathrm{UAV}}_h$ prioritizes information about its own position $p_h$, its own cluster ID $c_h$, the cluster ID of other head agents $c_{\alpha \left(h\right)}$, and the position of its servicing member agents $p_{\beta \left(h\right)}$, where $\alpha \left(h\right)$ denotes the other head agents besides ${\mathrm{UAV}}_h$, and $\beta \left(h\right)$ represents the member agents that the head agent ${\mathrm{UAV}}_h$ serves. Meanwhile, the head agent ${\mathrm{UAV}}_h$ can obtain the link maintenance probability ${\eta }_{h,j}$ between ${\mathrm{UAV}}_h$ and ${\mathrm{UAV}}_j$, the path loss $L_{h,j}$ between ${\mathrm{UAV}}_h$ and ${\mathrm{UAV}}_j$, estimated mean value ${\mathsf{\Omega }}_{|g_{h,j}{|}^2}$ of the channel gain between ${\mathrm{UAV}}_h$ and ${\mathrm{UAV}}_j$, the signal transmit power $P_{j,h}$ from ${\mathrm{UAV}}_j$ to ${\mathrm{UAV}}_h$, and the additive white Gaussian noise $N_0$. Hence, the observation set of agent ${\mathrm{UAV}}_h$ can be described as $O^h=\{p_h,c_h,c_{\alpha \left(h\right)},p_{\beta \left(h\right)},{\eta }_{h,j},L_{h,j},{\mathsf{\Omega }}_{|g_{h,j}{|}^2},P_{h,j},N_0\}$.

3.3.2. Action Set

In contrast to member agents, the action set for head agents is a two-dimensional discrete space ${\mathcal{A}}^h=[a_{\mathrm{cluster}},a_{\mathrm{position}}]$, where $a_{\mathrm{cluster}}=h(h\in \{1,2,\dots ,H\}$ represents the servicing cluster of the head agent ${\mathrm{UAV}}_h$, and $a_{\mathrm{position}}=\{x_h^{\prime },y_h^{\prime },z_h^{\prime }\}$ denotes the moving position of the head agent ${\mathrm{UAV}}_h$.

3.3.3. Reward

Since the head agents can communicate and share training information with each other during the centralized training phase of the MAPPO algorithm, they have access to global information. To enhance training effectiveness, MAPPO-H incorporates a hybrid reward system. The rewards for head agents consist of both global and local components, motivating the head agents to maximize the overall throughput and maximize the minimum SNR with the member agents. The global and local reward functions can be expressed as

$$r_{global}=\sum _{h\in \mathcal{H}}\sum _{e\in \beta \left(h\right)}{B}_{e,h}{log}_2(1+{\mathsf{\Phi }}_{e,h}),$$

(18)

$$r_{h,local}=min\sum _{e\in \beta \left(h\right)}\left({\mathsf{\Phi }}_{e,h}\right),$$

(19)

where $\mathcal{H}$ represents the set of CHs, $\beta \left(h\right)$ denotes the member agents that the head agent ${\mathrm{UAV}}_h$ serves, and $B_{e,h}$ and ${\mathsf{\Phi }}_{e,h}$ are the signal bandwidth and SNR from ${\mathrm{CM}}_e$ to ${\mathrm{CH}}_h$. In general, the global reward $r_{global}$ motivates the head agents to optimize the overall throughput. Only the head agent with the highest reward can serve the cluster, which avoids multiple head agents serving the same cluster simultaneously. Meanwhile, the local reward $r_{h,local}$ prompts the head agents to optimize the minimum SNR with the member agents, thereby minimizing the likelihood of communication disruptions. Therefore, the total reward for head agents can be defined as

$$R^h={\gamma }^h\ast r_{global}+(1-{\gamma }^h)\ast r_{h,local},$$

(20)

where ${\gamma }^h\in [0,1]$ is used to make a trade-off between the global and local goals.

3.4. Training Procedure

The training procedure of the proposed HMDRL-UC algorithm is described in Algorithm 1. First, we initiate actor networks ${\pi }^m$ and critic networks $V^m$ for member agents with parameters ${\theta }^m$ and ${\omega }^m$, while initiating actor networks ${\pi }^h$ and critic networks $V^h$ for head agents with parameters ${\theta }^h$ and ${\omega }^h$. Then, the agents interact with the environment to obtain the trajectories of the member agent ${\gamma }^m={\{O_t^m,a_t^m,R_t^m\}}_{t=1}^T$ and the head agent ${\gamma }^h={\{O_t^h,a_t^h,R_t^h\}}_{t=1}^T$ at the time step t, where $a_t^m$ is the action taken by the member agent ${\mathrm{UAV}}_m$ based on the observation set $O_t^m$ using the policy ${\pi }^m$, i.e., $a_t^m\sim {\pi }^m\left(a_t^m\right|O_t^m)$. Similarly, $a_t^h\sim {\pi }^h\left(a_t^h\right|O_t^h)$. After executing the actions $a_t^m$ and $a_t^h$, the rewards $R_t^m$ and $R_t^h$ and the next observation sets $O_{t+1}^m$ and $O_{t+1}^h$ are generated, respectively. Finally, the data $D^m={\{O_t^m,a_t^m,V^m\left(O_t^m\right),A_t^m\}}_{t=1}^T$ and $D^h={\{O_t^h,a_t^h,V^h\left(O_t^h\right),A_t^h\}}_{t=1}^T$ are, respectively, stored in the memory buffers for member agents and head agents, where $V^m$ and $V^h$ are the value estimates for critic networks of member agents and head agents, and $A_t^m$ and $A_t^h$ are the advantage functions of member agents and head agents. According to [40], $A_t^m$ and $A_t^h$ can be estimated via generalized advantage estimation (GAE) as follows:

$$\begin{array}{c}\hfill A_t^m={\displaystyle \sum _{l=1}^T}{\left({\rho }_1{\lambda }_1\right)}^l{\chi }_{t+l}^m,\end{array}$$

(21)

$$\begin{array}{c}\hfill A_t^h=\sum _{l=1}^T{\left({\rho }_2{\lambda }_2\right)}^l{\chi }_{t+l}^h,\end{array}$$

(22)

where ${\chi }_{t+l}^m=R^m+{\rho }_1{\widehat{V}}^m\left(O_{t+1}^m\right)-{\widehat{V}}^m\left(O_t^m\right)$, ${\chi }_{t+l}^h=R^h+{\rho }_2{\widehat{V}}^h\left(O_{t+1}^h\right)-{\widehat{V}}^h\left(O_t^h\right)$, ${\lambda }_1,{\lambda }_2\in [0,1]$ are the key hyperparameters for GAE that balance bias and variance in the advantage estimation, ${\rho }_1,{\rho }_2\in [0,1]$ are the discount factors that control the weight of future rewards, and ${\widehat{V}}^m$ and ${\widehat{V}}^h$ are the target local critics of member agent ${\mathrm{UAV}}_m$ and head agent ${\mathrm{UAV}}_h$, respectively.

In the practical implementation, the policy gradient for each actor network is calculated as

$$\begin{array}{c}\hfill △{\theta }^m={▽}_{{\theta }^m}\widehat{\mathbb{E}}\left\{H({\kappa }_t^m\left({\theta }^m\right),A_t^m)\right\},\end{array}$$

(23)

$$\begin{array}{c}\hfill △{\theta }^h={▽}_{{\theta }^h}\widehat{\mathbb{E}}\left\{H({\kappa }_t^h\left({\theta }^h\right),A_t^h)\right\},\end{array}$$

(24)

where $\widehat{\mathbb{E}}\{·\}$ is the sample average, and

$$\begin{array}{cc}\hfill H({\kappa }_t^m\left({\theta }^m\right),A_t^m)& =min({\displaystyle \frac{{\pi }^m\left(a_t^m\right|O_t^m)}{{\pi }_{old}^m\left(a_t^m\right|O_t^m)}}{A}_t^m,\mathrm{clip}({\displaystyle \frac{{\pi }^m\left(a_t^m\right|O_t^m)}{{\pi }_{old}^m\left(a_t^m\right|O_t^m)}},\hfill \\ \hfill & 1-ϵ,1+ϵ)A_t^m),\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill H({\kappa }_t^h\left({\theta }^h\right),A_t^h)& =min({\displaystyle \frac{{\pi }^h\left(a_t^h\right|O_t^h)}{{\pi }_{old}^h\left(a_t^h\right|O_t^h)}}{A}_t^h,\mathrm{clip}({\displaystyle \frac{{\pi }^h\left(a_t^h\right|O_t^h)}{{\pi }_{old}^h\left(a_t^h\right|O_t^h)}},\hfill \\ \hfill & 1-ϵ,1+ϵ)A_t^h).\hfill \end{array}$$

(26)

In Equations (25) and (26), ${\pi }_{old}^m\left(a_t^m\right|O_t^m)$ and ${\pi }_{old}^h\left(a_t^h\right|O_t^h)$ denote the current policy with parameters ${\theta }_{old}^m$, ${\omega }_{old}^m$, ${\theta }_{old}^h$, and ${\omega }_{old}^h$; ${\pi }^m\left(a_t^m\right|O_t^m)$ and ${\pi }^h\left(a_t^h\right|O_t^h)$ represent the updated policy with parameters ${\theta }^m$, ${\omega }^m$, ${\theta }^h$, and ${\omega }^h$; and the clip function $\mathrm{clip}$ restricts the probability ratio ${\displaystyle \frac{{\pi }^m\left(a_t^m\right|O_t^m)}{{\pi }_{old}^m\left(a_t^m\right|O_t^m)}}$ and ${\displaystyle \frac{{\pi }^h\left(a_t^h\right|O_t^h)}{{\pi }_{old}^h\left(a_t^h\right|O_t^h)}}$ into the interval $[1-ϵ,1+ϵ]$.

Meanwhile, the policy gradient for each critic network is calculated as

$$\begin{array}{c}\hfill △{\omega }^m={▽}_{{\omega }^m}\widehat{\mathbb{E}}\left\{{(Y_t^m-V^m\left(O_t^m\right))}^2\right\},\end{array}$$

(27)

$$\begin{array}{c}\hfill △{\omega }^h={▽}_{{\omega }^h}\widehat{\mathbb{E}}\left\{{(Y_t^h-V^h\left(O_t^h\right))}^2\right\},\end{array}$$

(28)

where $Y_t^m=A_t^m+{\widehat{V}}^m\left(O_t^m\right)$, $Y_t^h=A_t^h+{\widehat{V}}^h\left(O_t^h\right)$.

In the period of training, we iteratively update ${\theta }^m$, ${\theta }^h$, ${\omega }^m$, and ${\omega }^h$ with the gradient in Equations (23), (24), (27) and (28), respectively. Then, ${\theta }_{old}^m$, ${\theta }_{old}^h$, ${\omega }_{old}^m$, and ${\omega }_{old}^h$ are updated iteratively until the training process terminates.

Algorithm 1 Network Training Stage of HMDRL-UC Algorithm

1: Initiate ${\pi }_{old}^m$ with ${\theta }_{old}^m\leftarrow {\theta }^m$, ${\pi }_{old}^h$ with ${\theta }_{old}^h\leftarrow {\theta }^h$, $V^m$ with ${\omega }_{old}^m\leftarrow {\omega }^m$, $V^h$ with ${\omega }_{old}^h\leftarrow {\omega }^h$, $D^m$ for member agents, and $D^h$ for head agents 2: for iteration $i=1,2,\dots ,I$ 3:    for episode $t=1,2,\dots ,T$ 4:       Member agents execute actions according to ${\pi }_{old}^m\left(a_t^m\right|O_t^m)$ 5:       Head agents execute actions according to ${\pi }_{old}^h\left(a_t^h\right|O_t^h)$ 6:       Get the rewards $R_t^m$, $R_t^h$ and the next states $O_{t+1}^m$, $O_{t+1}^h$ 7:    end for 8:    Get trajectories ${\gamma }^m$ and ${\gamma }^h$ for member agents and head agents 9:    Compute advantages ${\left\{A_t^m\right\}}_{t=1}^T$ and ${\left\{A_t^h\right\}}_{t=1}^T$ using (21) and (22) 10:    Compute $Y_t^m$ and $Y_t^h$ 11:    Store data $D^m$ and $D^h$ 12:    for $k=1,2,\dots ,K$ 13:       for member agent $m=1,2,\dots ,M$ 14:          Apply gradient ascent on ${\theta }^m$ using (23) 15:          Apply gradient descent on ${\omega }^m$ using (27) 16:       end for 17:       for head agent $h=1,2,\dots ,H$ 18:          Apply gradient ascent on ${\theta }^h$ using (24) 19:          Apply gradient descent on ${\omega }^h$ using (28) 20:       end for 21:    end for 22:    Update ${\theta }_{old}^m\leftarrow {\theta }^m$ and ${\omega }_{old}^m\leftarrow {\omega }^m$ for each member agent 23:    Update ${\theta }_{old}^h\leftarrow {\theta }^h$ and ${\omega }_{old}^h\leftarrow {\omega }^h$ for each head agent 24: end for

3.5. Complexity Analysis

As suggested by [36,44], the complexity of the PPO algorithm is primarily determined by the forward and backward propagation of the neural network, and is related to the dimensions of the state space and action space, the number of layers and nodes per layer, as well as the number of samples per update. The proposed HMDRL-UC algorithm consists of IPPO-M algorithm and MAPPO-H algorithm. In the IPPO-M algorithm, the forward propagation of the actor network ${\pi }^m$ generates a trajectory ${\gamma }^m$, and back propagation calculates the policy gradient, with a computational complexity of $\mathcal{O}(I\ast B\ast T\ast L_{\mathrm{actor}}^m\ast N_{\mathrm{actor}}^m)$, where I is the iteration number; B is the number of trajectories per iteration, i.e., the batch size; T is the length of each trajectory; $L_{\mathrm{actor}}^m$ is the number of layers in the actor network ${\pi }^m$; and $N_{\mathrm{actor}}^m$ is the number of hidden units per layer in the actor network ${\pi }^m$. The advantage functions is calculated with a computational complexity of $\mathcal{O}(B\ast T)$. The process of the critic network $V^h$ is similar to the actor network ${\pi }^m$, with a computational complexity of $\mathcal{O}(I\ast B\ast T\ast L_{\mathrm{critic}}^m\ast N_{\mathrm{critic}}^m)$, where $L_{\mathrm{critic}}^m$ is the number of layers in the critic network $V^m$, and $N_{\mathrm{critic}}^m$ is the number of hidden units per layer in the critic network $V^m$. Therefore, if there are M member agents in the heterogeneous UAV swarm, the computational complexity of the IPPO-M algorithm is $\mathcal{O}(M\ast I\ast B\ast T\ast (L_{\mathrm{actor}}^m\ast N_{\mathrm{actor}}^m+L_{\mathrm{critic}}^m\ast N_{\mathrm{critic}}^m))$. Different from the IPPO-M algorithm, all the head agents in the MAPPO-H algorithm share a single critic network. Hence, if there are H head agents, the computational complexity of the MAPPO-H algorithm is $\mathcal{O}(I\ast B\ast T\ast (H\ast L_{\mathrm{actor}}^h\ast N_{\mathrm{actor}}^h+L_{\mathrm{critic}}^h\ast N_{\mathrm{critic}}^h))$, where $L_{\mathrm{actor}}^h$ and $N_{\mathrm{actor}}^h$ are the number of layers and the number of hidden units per layer in the actor network ${\pi }^h$; $L_{\mathrm{critic}}^h$ and $N_{\mathrm{critic}}^m$ are the number of layers and the number of hidden units per layer in the critic network $V^h$. In general, the ultimate computational complexity of the proposed HMDRL-UC algorithm is given by $\mathcal{O}(I\ast B\ast T\ast (M\ast L_{\mathrm{actor}}^m\ast N_{\mathrm{actor}}^m+M\ast L_{\mathrm{critic}}^m\ast N_{\mathrm{critic}}^m+H\ast L_{\mathrm{actor}}^h\ast N_{\mathrm{actor}}^h+L_{\mathrm{critic}}^h\ast N_{\mathrm{critic}}^h))$.

4. Simulation and Analysis

In this section, we evaluate the proposed HMDRL-UC algorithm from various simulation perspectives. Firstly, we conducted a performance comparison between the proposed HMDRL-UC algorithm and the existing algorithms. Then, we investigated the impact of different numbers of CMs on the performance of the proposed HMDRL-UC algorithm. Finally, we analyzed the influence of different speeds of CHs on the convergence speed of the HMDRL-UC algorithm.

4.1. Experiment Setting

We considered a heterogeneous UAV swarm uniformly distributed in the mission area. It is assumed that the UAVs in the swarm are divided into two types: CHs and CMs. The CHs were deployed at an altitude of $300\mathrm{m}$, while the altitude of CMs ranged from $200\mathrm{m}$ to $250\mathrm{m}$. The relative speed of CMs ranged from $0\mathrm{m}/\mathrm{s}$ to $20\mathrm{m}/\mathrm{s}$, and the velocity of CHs was about $100\mathrm{m}/\mathrm{s}$. The transmit power of CMs was $20\mathrm{dBm}$, and the SNR threshold of CHs was defined as $0\mathrm{dB}$. The central frequency of the carrier was set to be $2.4\mathrm{GHz}$, while the bandwidth was $10\mathrm{MHz}$. The background noise power was $-100\mathrm{dBm}$. The packet preparation delay and the packet transmission delay were set as $T_1=0\mathsf{\mu }\mathrm{s}$ and $T_2=50\mathsf{\mu }\mathrm{s}$. The Hello interval was configured to last $1\mathrm{s}$ [27]. The maximum cooperative communication distance for CMs was set as $400\mathrm{m}$ [43]. The fading severity parameter was set to be 2 [40].

In the MAPPO-H algorithm, the initial learning rate was set to 0.007, the hidden layer dimension of the Multi-Layer Perceptron (MLP) was 128, the dimension of actor network was 256, and the dimension of critic network was 256. In the IPPO-M network, the learning rate was 0.0007, the hidden layer dimension of MLP was 64, the dimension of actor network was 128, and the dimension of critic network was 256. The batch size was set to 3200. Both actor network and critic network utilized the Adaptive Moment Estimation (Adam) optimizer, with the learning rate linearly decaying to 0.01 of its initial value. A total of 128 parallel environments were established for the simulation, which was conducted using Python 3.7 on a CPU (Intel Xeon Gold 6430, ARM Cortex-A72) and GPU (NVIDIA RTX A5000).

Figure 4 shows the clustering results under different numbers of clusters. We considered four scenarios: 2 CHs and 20 CMs in an area of $1\mathrm{km}\times 2\mathrm{km}$, 4 CHs and 40 CMs in an area of $2\mathrm{km}\times 2\mathrm{km}$, 6 CHs and 60 CMs in an area of $2\mathrm{km}\times 3\mathrm{km}$, and 9 CHs and 90 CMs in an area of $3\mathrm{km}\times 3\mathrm{km}$. We mainly compared the performance of the algorithms based on the following three metrics: average throughput, average intra-cluster delay, and average minimum SNR. Average throughput reflects the ability of CHs to collect information sent back from CMs, with higher values indicating better performance. Moreover, average intra-cluster delay reflects the performance of cooperative communication within clusters, where lower values indicate more efficient communication. Finally, the average minimum SNR reflects the ability of CHs to serve edge member agents, with higher values indicating better service quality.

4.2. Training Efficiency and Resource Consumption Analysis

Table 1. Training time and resource consumption of HMDRL-UC under different UAV swarm Sscales.

Scenario	Training Time (CPU)	Training Time (GPU)	GPU Memory (GB)	CPU Utilization	Energy (Wh)	FLOPs (G)
2 CHs + 20 CMs	2.1 h	0.8 h	8–10	$\le 70$% (64 threads)	420 (CPU) 240 (GPU)	0.8
4 CHs + 40 CMs	5.3 h	1.9 h	10–12	$\le 75$% (64 threads)	1060 (CPU) 570 (GPU)	2.1
6 CHs + 60 CMs	12.6 h	4.5 h	12–14	$\le 80$% (64 threads)	2520 (CPU) 1350 (GPU)	5.3
9 CHs + 90 CMs	19.8 h	6.7 h	15–17	$\le 85$% (64 threads)	3960 (CPU) 2010 (GPU)	10.5

summarizes the training time, GPU/CPU resource consumption, and energy efficiency of the proposed HMDRL-UC algorithm under different UAV swarm scales. In general, we can observe that the training time and resource consumption of HMDRL-UC exhibit scalability across different UAV swarm scales. For the smallest scenario (2 CHs and 20 CMs), training completes in 2.1 h on CPU and 0.8 h on GPU, while the largest scenario (9 CHs and 90 CMs) requires 19.8 h (CPU) and 6.7 h (GPU), indicating a near-linear increase in computational demand with swarm size. GPU acceleration reduces training time by 60–66% compared to CPU, albeit with higher memory usage (8–17 GB). CPU utilization remains moderate ($\le 85$% with 64 threads), suggesting efficient parallelism. Energy consumption scales proportionally, reaching 3960 Wh (CPU) and 2010 Wh (GPU) for the largest configuration. In addition, the algorithm’s inference phase demonstrates hardware adaptability: on resource-constrained platforms like ARM Cortex-A72, IPPO-M requires only 0.12 Wh and 0.8 MFLOPs per inference, making it viable for embedded UAV systems with limited power budgets.

While HMDRL-UC requires substantial computational resources during centralized training, the trained models can be deployed on resource-constrained UAV hardware for decentralized execution. For instance, the IPPO-M actor network (128-dimensional hidden layer) occupies only 1.2 MB of memory and requires 0.3 ms per inference on an ARM Cortex-A72 embedded processor. By leveraging model quantization and pruning techniques, we further reduced the model size to 0.5 MB with negligible performance degradation. Additionally, the distributed nature of HMDRL-UC eliminates the need for real-time global coordination, making it feasible for UAVs with limited energy budgets (e.g., 10 W power consumption during inference).

4.3. Comparison

We provide numerical results to compare the performance of the proposed HMDRL-UC algorithm with the iterative coverage efficient clustering (CECA) algorithm [11], energy-efficient swarm intelligence-based clustering algorithm based on particle swarm optimization (SIC-PSO) [42], coalition formation game (CFG) algorithm [29], and deep deterministic policy gradient (DDPG) [45] under different numbers of UAV swarm clusters. It is worth to emphasize that, both the CECA algorithm and CFG algorithm focus on static scenarios of UAV swarms. While the SIC-PSO algorithm and the DDPG algorithm can handle clustering issues in dynamic UAV swarm scenarios, it requires prior Channel State Information (CSI). In contrast, the proposed HMDRL-UC algorithm not only addresses dynamic UAV swarm environments, but also requires only estimated channel gain distribution patterns, i.e., CDI.

Figure 5 compares the performance of the proposed HMDRL-UC algorithm with the other four algorithms in terms of average throughput, average intra-cluster delay, and average minimum SNR. It is clearly visible from the figure that as the number of UAV swarm clusters increases, both the average cluster throughput and the average minimum SNR slightly decrease, while the average intra-cluster delay gradually rises. This observation shows that as the number of clusters increases, the potential for clustering becomes more complex, leading to greater challenges in clustering. Hence, this can result in irregular cluster shapes, increased routing paths, and larger cluster areas. By contrast, in Figure 5a,c, the average throughput and average minimum SNR of the HMDRL-UC algorithm are obviously better than those achieved by the other four algorithms. Moreover, in Figure 5b, it can be clearly observed that the HMDRL-UC algorithm achieves the lowest average intra-cluster delay. From the above simulation results, the proposed HMDRL-UC algorithm achieved the best performances. This is because the HMDRL-UC algorithm utilizes a multi-agent deep reinforcement learning approach, where each agent observes the environment and extracts useful information through a deep neural network, learning their own action strategies. This enables a more effective handling of complex decision-making problems. Moreover, compared to the DDPG algorithm, HMDRL-UC employs a divide-and-conquer strategy by decoupling the optimization processes of MAPPO-H and IPPO-M, significantly reducing model complexity while achieving dynamic environment adaptation through partial CDI utilization instead of relying on complete CSI. Compared to the other algorithms, the proposed HMDRL-UC algorithm demonstrates greater robustness and scalability.

As quantitatively demonstrated in

Table 2. Performance comparison of HMDRL-UC with baseline algorithms (9 CHs and 90 CMs).

Algorithm	Average Throughput (Mbps)	Average Intra-Cluster Delay (ms)	Minimum SNR (dB)	Dynamic Adaptation	Required Channel Information	Complexity
HMDRL-UC	11.9	3.2	0.88	Yes	Partial CDI	High
DDPG D [45]	11.5	3.4	0.85	Yes	Full CSI	High
CECA [11]	7.8	3.9	0.35	No	Full CSI	Moderate
CFG [29]	8.5	4.0	0.33	No	Partial CDI	Moderate
SIC-PSO [42]	9.7	3.8	0.42	Yes	Full CSI	Moderate

, the proposed HMDRL-UC algorithm delivers substantial performance enhancements across all critical metrics compared to baseline methods. In a representative scenario with 9 CHs and 90 CMs within a $3\mathrm{km}\times 3\mathrm{km}$ operational area, HMDRL-UC achieves remarkable throughput improvements of 3.5%, 52.6%, 40.0%, and 22.7% over DDPG, CECA, CFG, and SIC-PSO, respectively. The algorithm also demonstrates superior latency performance, reducing average intra-cluster delay by 5.9%, 17.9%, 20.0%, and 15.8% compared to the same baselines. Furthermore, HMDRL-UC maintains a 3.5%, 151.4%, 166.7%, and 109.5% advantage in minimum SNR preservation over DDPG, CECA, CFG, and SIC-PSO, respectively. Notably, HMDRL-UC pioneers dynamic adaptation capabilities in environments with partial CDI, addressing critical limitations of existing approaches: CECA and CFG are restricted to static operational modes, while SIC-PSO and DDPG depend on complete CSI. Moreover, the HMDRL-UC, through a heterogeneous design combining lightweight IPPO-M and the centralized training of MAPPO-H, achieves significantly enhanced performance, thereby validating its balanced advantage in managing the trade-off between complexity and performance. These systemic enhancements collectively confirm the effectiveness of our heterogeneous multi-agent framework in harmonizing network performance optimization, environmental adaptability, and practical deployment viability for large-scale UAV swarms.

4.4. Effects of Different Numbers of CMs

Figure 6 shows the cumulative reward values ${\displaystyle \sum _{m=1}^M}{R}^m$ of member agents under different numbers of CMs. We conducted simulations in a $2\mathrm{km}\times 2\mathrm{km}$ scenario, with 20, 40, and 60 CMs grouped into 4 clusters. As can be observed from the figure, convergence occurs at 80 iterations for the 20 CMs case, at 260 iterations for the 40 CMs case, and at 700 iterations for the 60 CMs case. Moreover, the more cluster members there are, the slower the convergence speed becomes, the lower the cumulative reward value after convergence, and the greater the volatility. This is because as the number of CMs increases, each UAV has more potential communication partners, requiring the agent to process more environmental information to make decisions. This results in a larger state space, leading to slower convergence and greater instability. In addition, the increased complexity of UAV ad hoc networks inevitably leads to an increase in communication hops, which in turn causes a corresponding increase in latency. As a result, the cumulative reward value of member agents decreases.

4.5. Effects of Different Speeds of CH

Figure 7 investigates the cumulative reward values ${\displaystyle {\displaystyle \sum _{h=1}^H}}{R}^h$ of head agents under different speeds of CH. We conducted simulations with CH speeds of 100 m/s, 150 m/s, and 200 m/s, respectively. As shown in the figure, when the CH’s speed is 100 m/s, 150 m/s, and 200 m/s, the algorithm converges within 170, 150, and 131 iterations, respectively. It can also be observed that at the beginning stage, the CH’s cumulative reward is the highest when the speed is 150 m/s, followed by 200 m/s. However, at convergence, the cumulative reward values are similar when the speed is 100 m/s and 150 m/s, followed by 200 m/s.This is because as the CH’s speed increases, the CH will reach the optimal data transmission and reception position more quickly, and the algorithm will converge faster accordingly. However, since the action space of the CH is discrete, the step size for moving in space also increases with the speed. Therefore, at 200 m/s, the cluster head may be more likely to deviate from the optimal data transmission position.

5. Conclusions

We propose a novel algorithm based on heterogeneous multi-agent deep reinforcement learning, named HMDRL-UC, for cluster-based spectrum sharing in large-scale heterogeneous UAV swarms. The HMDRL-UC algorithm consists of two parts: MAPPO-H and IPPO-M. The MAPPO-H algorithm equips CHs with decision-making capabilities for cluster selection and mobility planning, whereas CMs employ the IPPO-M algorithm to perform decentralized clustering operations based on CDI. Simulation results show that, compared to the other existing algorithms, our proposed HMDRL-UC algorithm can handle dynamic spectrum sharing scenarios under CDI conditions, achieving higher throughput and SNR while effectively reducing communication delay. While this work primarily focuses on role-based heterogeneity (CHs vs. CMs) and mobility dynamics, future research will extend HMDRL-UC to handle more complex heterogeneous scenarios, such as UAVs with varying computational capabilities, communication bandwidths, or energy constraints. Incorporating these factors could further enhance the algorithm’s practicality in real-world multi-mission UAV swarms. In general, the proposed HMDRL-UC algorithm is a promising clustering-based spectrum sharing algorithm for multi-mission UAV swarm scenarios.