Hierarchical Role-Based Multi-Agent Reinforcement Learning for UHF Radiation Source Localization with Heterogeneous UAV Swarms

Highlights

What are the main findings?

• The proposed heterogeneous multi-UAV deep reinforcement learning (HMUDRL) algorithm achieves high-precision localization and wide-area monitoring of an Ultra High Frequency (UHF) radiation source through a hierarchical architecture within heterogeneous unmanned aerial vehicle (UAV) swarms, improving localization success rate and reducing localization error.
• We demonstrate that the HMUDRL algorithm substantially reduces communication overhead and enhances system efficiency through an intra-cluster information aggregation mechanism.

What are the implications of the main finding?

• The proposed algorithm provides a practical technical solution to address key challenges faced by UAVs in UHF radiation source localization, including payload constraints, limited endurance, and adaptability to dynamic environments, thereby advancing the development of intelligent spectrum sensing technologies.
• The application of this algorithm enables electromagnetic spectrum monitoring scenarios involving a broader range of frequency bands and diverse types of radiation sources.

Abstract

With the continuous proliferation of radio frequency devices, electromagnetic environments in various regions are becoming increasingly complex. Effective monitoring of the electromagnetic environment and identification of interference sources have thus become critical tasks for maintaining order in the electromagnetic spectrum. In recent years, rapid advances in UAV technology have spurred exploration of UAV-based electromagnetic spectrum monitoring as a novel approach. However, the limited payload capacity and endurance of UAVs constrain their monitoring capabilities. To address these challenges, we propose HMUDRL, a distributed heterogeneous multi-agent deep reinforcement learning algorithm. By leveraging cooperative operation between cluster-head UAVs (CH) and cluster-monitoring UAVs (CM) within a heterogeneous UAV swarm, HMUDRL enables high-precision detection and wide-area localization of UHF radiation source. Furthermore, we integrate a minimum-gap localization algorithm that exploits the spatial distribution of multiple CM to accurately pinpoint anomalous radiation sources. Simulation results validate the effectiveness of HMUDRL: in the later stages of training, the success rate of localizing target radiation sources converges to 96.1%, representing an average improvement of 1.8% over baseline algorithms; localization accuracy, measured by root mean square error (RMSE), is enhanced by approximately 87.3% compared to baselines; and communication overhead is reduced by more than 80% relative to homogeneous architectures. These results demonstrate that HMUDRL effectively addresses the challenges of data transmission control and sensing-localization performance faced by UAVs in UHF spectrum monitoring.

1. Introduction

With the continuous proliferation of radio frequency (RF) devices, electromagnetic environments in various regions have become increasingly complex. Effectively monitoring these environments and identifying interference sources has thus become a critical task for maintaining electromagnetic spectrum order [1,2]. The UHF band is widely used in aviation navigation, maritime communications, railway dispatching, and emergency broadcasting due to its favorable propagation characteristics, yet it is also highly vulnerable to illegal transmissions and malicious interference. Conventional ground-based monitoring systems often struggle to promptly localize interference sources in complex terrains such as urban or mountainous areas due to line-of-sight blockages and limited coverage [3,4].

UAV-based spectrum monitoring has emerged as a powerful alternative due to its flexible deployment and extensive spatial coverage [5]. However, existing multi-agent reinforcement learning (MARL) methods for UAV-based source localization typically adopt homogeneous agent designs or centralized critic architectures [6,7,8], which face several practical bottlenecks in real-world monitoring scenarios. Most current studies focus on small-scale monitoring setups with homogeneous UAV swarms; as the number of UAVs and monitoring targets increases, information exchange via fully connected or dense k-NN communication topologies incurs prohibitively high communication overhead, making system scalability challenging. Moreover, constrained by limited onboard power and payload capacity, a single UAV platform struggles to simultaneously perform spectrum sensing, data processing, and relaying tasks, thereby restricting its operational capabilities in practice. Additionally, few approaches jointly coordinate motion planning with high-precision geometric estimation, leading to suboptimal localization accuracy.

To address these challenges, this paper proposes HMUDRL, a hierarchical heterogeneous MARL framework specifically designed for UHF radiation source localization. Compared with prior works, our approach introduces fundamental innovations in three aspects:

• We propose a role-based heterogeneous architecture that explicitly distinguishes CH, which are responsible for network coordination, from CM, which are dedicated to signal sensing, thereby enabling task-specific policy learning.
• We adopt a two-level hierarchical decision-making structure in which CHs aggregate intra-cluster information. This design reduces inter-agent communication by over 80% compared to homogeneous baselines while maintaining strong collaborative performance.
• We fuse angle-of-arrival (AOA) and received signal strength (RSS) measurements from multiple CMs to jointly optimize triangulation geometry and refine position estimates. Experimental results show a 96.1% localization success rate and an 87.3% reduction in RMSE compared to state-of-the-art MARL baselines, demonstrating the effectiveness and practicality of our method for spectrum monitoring tasks.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 presents the system model. Section 4 formulates and analyzes the problem. Section 5 details the proposed HMUDRL algorithm. Simulation results are provided in Section 6. Finally, Section 7 concludes the paper.

2. Related Work

Current approaches to electromagnetic spectrum monitoring using heterogeneous UAVs primarily rely on clustering strategies combined with heuristic and reinforcement learning algorithms for sensing and detection. This section reviews the key advancements in these methods and identifies existing research gaps.

2.1. Clustering Strategies

A single-node UAV exhibits limitations in terms of coverage, timeliness, and accuracy in electromagnetic spectrum monitoring. With the continuous improvement of communication network capacity, the number of UAVs operating simultaneously within a network can be significantly increased. Clustering strategies effectively mitigate channel congestion during monitoring data transmission among multiple UAVs, ensuring high-quality communication links between CH and CM, thereby facilitating rapid and reliable transmission of localization data. Furthermore, clustering enables CM within each subgroup to achieve favorable spatial geometric distributions, which enhances collaborative coverage over larger monitoring areas and supports sustained tracking of mobile or sporadic radiation sources. By leveraging heterogeneous CH to aggregate sensing data from CM, redundant data transmission is reduced, improving overall system efficiency. Gao et al. [9] integrated frequency-domain energy detection, frequency-domain noise estimation, and Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to develop a spectrum sensing method capable of accurately detecting multiple unknown signals. Various clustering strategies have been shown to enhance network performance, scalability, and robustness in UAV swarms. Conventional distance-based clustering algorithms, such as K-means, have demonstrated effectiveness in addressing cooperative spectrum sensing under low signal-to-noise ratio (SNR) conditions [10,11,12]. Elliott et al. [13] employed a Growing Self-Organizing Map (SOM) algorithm to adaptively mine spectrum data, enabling more accurate awareness of dynamic spectrum situations.

2.2. Heuristic Algorithms

Traditional heuristic algorithms have demonstrated notable effectiveness in addressing monitoring and localization problems in specific environments [14,15]. Heuristic algorithms are optimization methods based on empirical rules or simulations of natural phenomena. Their core principle is to rapidly obtain acceptable solutions through approximate-optimal search strategies when a complete problem model is unavailable or computational resources are limited. Compared with exact algorithms (e.g., exhaustive search or dynamic programming), heuristics generally exhibit lower time complexity and greater robustness, making them particularly suitable for high-dimensional, nonlinear, and multi-constrained complex optimization tasks, such as UAV path planning, spectrum sensing scheduling, and radiation source localization. Awadhesh et al. [16] integrated Particle Swarm Optimization (PSO) with fuzzy logic to develop a bio-inspired metaheuristic hybrid model that dynamically adjusts UAV flight trajectories based on real-time RSS and AOA, thereby accelerating localization convergence while reducing energy consumption. Spandana et al. [17] proposed a Hybrid Colliding Bodies Galaxy Swarm Optimization (HCBGSO) algorithm that simulates celestial gravitational and collision mechanisms to dynamically reposition UAVs in three-dimensional space, significantly improving coverage efficiency and localization stability for sporadic radiation sources. Other studies have employed Genetic Algorithms (GA) or Differential Evolution (DE) to post-process Time Difference of Arrival (TDOA) or AOA measurements, enhancing localization accuracy. For instance, Chen et al. [18] introduced a GA-TDOA algorithm that searches the solution space of nonlinear equations via genetic operations to estimate optimal radiation source coordinates, effectively mitigating localization drift caused by multipath interference. Moreover, heuristic algorithms naturally support multi-objective optimization frameworks. For example, the Hybrid Particle Swarm Optimization (HPSO) algorithm [19] simultaneously optimizes clustering structure, flight paths, and sensing schedules in UAV swarms, significantly extending system endurance while maintaining high localization success rates. Despite their strong performance in specific scenarios, heuristic algorithms suffer from several limitations: their efficacy is highly sensitive to parameter tuning and initial solution quality, and they generally lack theoretical convergence guarantees. More critically, conventional heuristic approaches are typically designed for “one-shot” problem solving and struggle to adapt to dynamically changing electromagnetic environments, such as those involving mobile emitters or sudden interference and cannot effectively leverage historical experience for knowledge transfer.

2.3. Reinforcement Learning Algorithms

In recent years, Reinforcement Learning (RL) has emerged as a powerful tool for addressing autonomous control and cooperative optimization problems in complex dynamic environments, owing to its remarkable performance in sequential decision-making tasks. Electromagnetic spectrum monitoring and radiation source localization involve highly uncertain environments, characterized by fluctuating signal strength, mobile interference sources, terrain occlusion, among others—where traditional model-based or rule-based approaches often fall short. In contrast, RL enables agents to learn optimal policies through continuous interaction with the environment by maximizing cumulative rewards, without requiring an accurate prior model of the environment. Single-agent RL algorithms, such as Deep Q-Networks (DQN), Deep Deterministic Policy Gradient (DDPG), and Proximal Policy Optimization (PPO), have been applied to path planning and spectrum sensing scheduling for individual UAVs. Ebrahimi et al. [20] employed Q-learning to improve localization accuracy for multiple targets while minimizing time and path length. Shurrab et al. [21] developed a DQN-based UAV target localization framework, leveraging DQN as a scalable extension of Q-learning that replaces the conventional Q-table with a neural network to handle large or continuous state spaces, thereby enhancing model generalization and adaptability in dynamic environments. Recently, Guan et al. [22] proposed a token-specific deep reinforcement learning (TS-DRL) framework for the capacitated electric vehicle routing problem, achieving significant energy savings by dynamically adapting routing policies to node-specific characteristics. As monitoring tasks evolve toward larger coverage, higher precision, and greater dynamism, single-agent frameworks are increasingly inadequate for multi-UAV cooperation. Consequently, Multi-Agent Reinforcement Learning (MARL) has become a focal point of research. Its key advantage lies in enabling complex tasks beyond the capability of a single agent through distributed decision-making and coordinated interaction among multiple agents—particularly well-suited for cooperative monitoring and localization using heterogeneous UAV swarms. Hou et al. [23] adopted the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which employs a centralized Critic network that utilizes the states and actions of all agents to evaluate policy value, while each agent maintains its own decentralized Actor network to generate actions. This architecture mitigates the non-stationarity caused by multi-agent interactions while preserving decentralized execution—aligning well with the requirement for collaborative decision-making and independent operation in UAV swarm monitoring. Qin et al. [24] proposed the Multi-Agent Soft Actor-Critic (MASAC) algorithm based on soft policy gradient theory. Building upon MADDPG, MASAC incorporates entropy regularization and utilizes multiple Critic networks to reduce value estimation bias. By maximizing policy entropy, the algorithm encourages exploration of potentially optimal actions, thereby enhancing robustness and generalization. In [25], the Counterfactual Multi-Agent Policy Gradients (COMA) algorithm was employed, which introduces a counterfactual baseline to address the credit assignment problem in multi-agent settings. This mechanism effectively differentiates the contribution of each agent to the collective task outcome. Wang et al. [26] applied the Multi-Agent Advantage Actor-Critic (MAA2C) algorithm, which combines independent Actor networks for each agent with a shared Critic network. This design integrates distributed policy learning with centralized value estimation. By leveraging advantage function estimation to reduce variance in value assessment and supporting parallel training, MAA2C achieves high training efficiency, making it suitable for rapid deployment in large-scale UAV swarms.

2.4. Research Gaps and Our Approach

Compared with traditional clustering and heuristic algorithms, data-driven approaches such as reinforcement learning offer stronger generalization and online adaptation capabilities [27]. However, existing MARL methods still face critical challenges when applied to electromagnetic source localization tasks using heterogeneous UAV swarms. To address these limitations, we propose the HMUDRL framework for radiation source localization. Unlike the homogeneous multi-agent setup in [8], which assumes identical agent capabilities and employs centralized training, HMUDRL explicitly incorporates role-based heterogeneous modeling by introducing dedicated CH and CM agents, each equipped with distinct observation spaces, action spaces, and reward structures. The centralized critic network in [23] suffers from input dimensionality that grows linearly with swarm size, leading to poor scalability. In contrast, HMUDRL adopts a decentralized two-level PPO architecture that eliminates dependence on global state information, thereby supporting large-scale deployment. Similarly, while the approach in [24] requires each agent to maintain multiple policy networks, resulting in high computational overhead, our method assigns only one lightweight PPO network per agent type, significantly improving efficiency. Furthermore, unlike [28], which relies on a single UAV or a centrally coordinated swarm for radio source localization, HMUDRL enables fully distributed decision-making without access to global information, a crucial advantage in communication-constrained environments. The COMA algorithm used in [25] requires complete action logs from all agents to compute counterfactual baselines, incurring substantial communication costs. By comparison, HMUDRL’s cluster-level information aggregation mechanism is designed to reduce inter-agent messaging by more than 80%. Moreover, the shared critic network in [26] enforces a uniform value function across all agents, which fails to accommodate diverse objectives in heterogeneous teams and often biases optimization toward a single goal. Our differentiated policy learning mechanism effectively avoids this issue. Recent work such as [29] focuses on wideband spectrum sensing but does not address localization policy learning. Meanwhile, the survey in [27] identifies hierarchical coordination and communication efficiency as open challenges in MARL-based UAV control, both of which are directly addressed by HMUDRL’s hierarchical architecture and role-specific reward mechanisms. Built upon the flexible PPO framework [8,30] and enhanced with a tailored two-level policy structure, HMUDRL uniquely balances localization accuracy, communication efficiency, and scalability in dynamic UHF monitoring scenarios.

The limitations summarized and systematized in

Table 1. Comparison of MARL Approaches for Heterogeneous, Scalable UAV Coordination.

Algorithm	Supports Heterogeneous Agents	Supports Hierarchical Architecture	Communication Mechanism	Scalability
MADDPG [23]	Yes Limited support	Yes with modifications	Centralized critic; all-to-all message passing during training	Poor Critic input scales quadratically
MASAC [24]	Yes Limited support	Theoretically feasible but inefficient	Centralized training with shared critics; dense inter-agent messaging	Moderate High parameter overhead
COMA [25]	Yes Limited support	Yes with modifications	Centralized critic with counterfactual baseline; full joint-action enumeration	Poor Exponential in agents
MAA2C [26]	Yes Limited support	Yes with modifications	Fully decentralized; no explicit coordination	Moderate Weak collaboration
MAPPO [8]	Yes Highly flexible	Yes with modifications	Global state for centralized critic	Moderate Global critic causes memory bottleneck
HMUDRL (Ours)	Yes	Yes	Cluster-based aggregation; CH broadcasts fused info	Good Complexity decoupled from swarm size

expose a critical gap in current MARL research. Although existing methods may theoretically accommodate heterogeneity or hierarchical structures through ad hoc modifications, none natively support role-aware and scalable coordination under the stringent communication, computational, and functional constraints inherent in UAV-based spectrum monitoring. In contrast, HMUDRL is explicitly designed around the operational realities of heterogeneous UAV swarms: CH and CM agents exhibit distinct sensing capabilities, action spaces, and mission objectives. To align with this role-based decomposition, our framework co-designs the architecture, communication protocol, and learning mechanisms, thereby eliminating the need for global state sharing or fully connected message passing. Information within each cluster is aggregated locally at the CH, and only compact fused estimates are broadcast, significantly reducing bandwidth consumption while preserving collaborative accuracy. Furthermore, by decoupling the policy networks for CH and CM, HMUDRL enables independent optimization of distinct reward signals, effectively overcoming the convergence bias commonly observed in shared-critic approaches. This native integration of heterogeneity, hierarchical structure, and communication efficiency not only resolves the scalability bottlenecks present in prior methods but also establishes a new paradigm for multi-agent reinforcement learning in physically constrained, multi-role autonomous systems.

• We establish a large-scale, distributed cooperative sensing and localization framework for heterogeneous UAV swarms to enhance collaborative localization efficiency.
• The HMUDRL algorithm integrates a branched network architecture to improve computational and communication efficiency, while jointly leveraging received signal power differences and AOA measurements to enhance localization accuracy.
• HMUDRL employs a multi-faceted reward mechanism that effectively balances monitoring coverage, localization accuracy, and operational efficiency.

3. System Model

Here, we consider establishing a heterogeneous UAV swarm to perform spectrum monitoring tasks, aiming to overcome the inherent limitations of individual UAVs, including constraints on payload weight and type, limited onboard power supply duration, and the difficulty of simultaneously achieving long operational endurance, wide monitoring coverage, and high-quality communication performance. As shown in Figure 1, to enhance monitoring efficiency, the UAV swarm is organized into multiple subgroups, comprising an information processing station, several CH, and a large number of CM. The information processing station maintains stable communication links with all CH and is responsible for overall swarm coordination and integrated data processing. CH possess strong data processing and transmission capabilities; they manage their respective CM within each subgroup, aggregate monitoring data, and relay it to the central base station. In contrast, CM are equipped with advanced sensing capabilities; through cooperative operation among multiple CM, a large monitoring coverage area is achieved, and collected data are transmitted back to their associated CH. However, due to the collective motion of the UAV swarm, the relative positions of CM may change dynamically, posing significant challenges to cluster stability and formation. Therefore, during the monitoring process, it is desirable that CM can autonomously form clusters, while CH can adaptively select their associated CM and determine optimal positioning strategies in real time.

3.1. CH–CM Link Establishment and Maintenance Model

Within the monitoring task area, we consider $M$ CH and $N$ CM, all moving independently within the region. Each CM is associated with exactly one CH. To reduce the risk of collisions in regions with high CM density, the number of CM per cluster is limited to $N_U$. A CH is considered undersubscribed if the number of CM currently associated with it is less than $N_U$. CH and their associated CM exchange Hello messages periodically over a dedicated control channel at a fixed interval $T_B$ [31]. Each Hello message contains the sender’s identifier, association status, carrier frequency, and signal power information. When a CM is already associated with a CH, it responds with a Hello message to maintain the link. Conversely, when a CM is unaffiliated with any cluster, it broadcasts a Hello message toward nearby CH to request association and actively join one of them. Only undersubscribed CH are permitted to accept new association requests.

Based on the received Hello messages, CH and CM can establish and update their association lists. If the $i$th CH and the $j$th CM appear in each other’s association lists and successfully exchange Hello messages within a predefined time window of $\eta T_B\left(\eta \ge 1\right)$ periods, a stable association link is considered established between them. Conversely, if no Hello message is received from the counterpart within $\eta T_B\left(\eta \ge 1\right)$ consecutive periods, the association link is deemed broken. A larger $\eta$ value enhances robustness against transient link interruptions caused by occasional Hello packet loss, thereby improving cluster stability. However, an excessively large $\eta$ may delay the network’s response to topology changes induced by UAV mobility. The optimal choice of $\eta$ depends on specific factors such as the number of UAVs involved in the mission and the network configuration requirements. Furthermore, by leveraging the carrier frequency and received signal power information contained in the Hello messages, the relative position between a linked CH–CM pair can be estimated using Doppler shift and signal power variation. This positional estimate enables dynamic adjustment of the CH’s location to maintain cluster structural stability.

As illustrated in Figure 2, assume that the shaded area centered at a CH represents its broadcast coverage region with radius $R$. When the distance $d_1$ between CM₁ and the CH is less than $R$, CM₁ can successfully receive the broadcast Hello message from the CH and establish a link. Conversely, if the distance $d_2$ between CM₂ and the CH exceeds $R$, CM₂ cannot receive the broadcast message and thus cannot form a link with the CH.

Assuming that CHs and CMs are independently and uniformly distributed within the task region $A$ of area $S$, with $N_h$ CH and $N_m$ CM, the spatial accessibility probability, that a randomly located CM lies within the coverage disk of at least one CH is given by:

$$P_{cover}={\displaystyle \frac{\pi R^2}{S}},$$

(1)

If each CH can associate with at most $N_{\max }$ CM, and currently $k$ CM are associated with a given CH, then the average load per CH is ${\overline{n}}_c={\displaystyle \frac{N_m}{N_h}}$, when ${\overline{n}}_c<N_{\max }$, most CH are in an undersubscribed state. Under random association, by Poisson approximation:

$$P_{unsat}=P\left(Poisson\left({\overline{n}}_c\right)<N_{\max }\right)={\displaystyle \sum _{k=0}^{N_{\max }-1}\frac{{\overline{n}}_c^k{e}^{-{\overline{n}}_c}}{k!}},$$

(2)

Let $P_{unsat}$ denote the probability that a CH is undersubscribed with respect to its associated CM. Suppose the success probability of receiving a Hello message in each period is $P_{hello}$, which depends on channel quality, multipath effects, and path loss, among other factors. If a stable link is established only when Hello messages are successfully received for $T$ consecutive periods, then the probability of establishing a stable link is given by $P_{stable}=P_{hello}^T$, where $P_{hello}$ is a function of distance. According to the free-space path loss model:

$$P_{hello}\left(d\right)=\left\{\begin{array}{cc}1,\hfill & d\le R_0,\hfill \\ {(R_0/d)}^n,\hfill & d>R_0.\hfill \end{array}\right.$$

(3)

where $R_0$ is the reliable communication radius (slightly less than $R$), and $n$ is the path loss exponent. For a CM to successfully join a CH, there must exist at least one CH within its communication range that is both undersubscribed and maintains a stable link. The CM will select either the nearest undersubscribed CH or the one with the strongest signal. Thus, the probability of successful link establishment is given by:

$$P_{link}=1-{\displaystyle \prod _{i=1}^{N_h}\left[1-P_{cover,i}\cdot P_{unsat,i}\cdot P_{stable,i}\right]},$$

(4)

Within the task region, the probability that a single CM successfully establishes a stable association link with a CH is given by:

$$P_{link}=\left({\displaystyle \frac{\pi R^2}{S}}\right)\cdot \left({\displaystyle \sum _{k=0}^{N_{\max }-1}\frac{{\overline{n}}^k{e}^{-\overline{n}}}{k!}}\right)\cdot P_0^T,$$

(5)

where $P_0$ is the probability of successfully receiving a Hello message within one period in the coverage area, and $T$ is the number of consecutive successful periods required to establish a stable link.

3.2. UAV Kinematic Model

This section provides a detailed description of the UAV kinematic model, focusing on variations in both speed and heading to facilitate electromagnetic spectrum source localization. We first define the position update equation and then specify the sets of speed and heading used to update the UAV’s position. For analytical tractability, UAV motion is modeled in a two-dimensional plane, and the UAV’s position–velocity state is given by:

$$Q_i\left(t\right)=\left(x_i\left(t\right),y_i\left(t\right),v_{xi}\left(t\right),v_{yi}\left(t\right)\right),$$

(6)

where $Q_i\left(t\right)$ represents the position and velocity state of UAV $i$ at time $t$, in which $x_i\left(t\right)$ and $y_i\left(t\right)$ denote the coordinates of UAV $i$ at time $t$, $x_i\left(t\right)$ and $y_i\left(t\right)$ represent the velocity components along the $x$- and $y$-axes, respectively.

The acceleration ${\dot{v}}_i\left(t\right)$ of the UAV is given by:

$${\dot{v}}_i\left(t\right)={\left[\Delta v_{xi}\left(t\right),\Delta v_{yi}\left(t\right)\right]}^T,$$

(7)

where $\Delta v_{xi}\left(t\right)$ and $\Delta v_{yi}\left(t\right)$ denote the changes in velocity components of the UAV $i$ at time $t$, In practical systems, these changes are controlled by the flight control system and updated at a fixed time step $\Delta t$. Based on Equation (7), the UAV state update is given by:

$$\begin{gathered} Q_i\left(t+1\right)=\left(x_i\left(t+1\right),y_i\left(t+1\right),v_{xi}\left(t+1\right),v_{yi}\left(t+1\right)\right) \\ =\left(\left(x_i\left(t\right)+v_{xi}\left(t+1\right)\cdot \Delta t\right),\left(y_i\left(t\right)+v_{yi}\left(t+1\right)\cdot \Delta t\right),\right. \\ \left.\left(v_{xi}\left(t\right)+\Delta v_{xi}\left(t\right)\right),\left(v_{yi}\left(t\right)+\Delta v_{yi}\left(t\right)\right)\right), \end{gathered}$$

(8)

Each UAV is subject to three physical constraints, among which the velocity constraint is defined as follows:

$$v_{\min }\le \Vert v\left(t\right)\Vert \le v_{\max },$$

(9)

where $\Vert v\left(t\right)\Vert =\sqrt{v_x{\left(t\right)}^2+v_y{\left(t\right)}^2}$ denotes the magnitude of the UAV’s velocity, $v_{\min }$ and $v_{\max }$ represent the minimum (typically zero) and maximum flight speeds of the UAV, respectively.

The acceleration constraint is defined as follows:

$$\left\{\begin{array}{l}\left|\Delta v_x\left(t\right)\right|\le a_{\max }\cdot \Delta t\hfill \\ \left|\Delta v_y\left(t\right)\right|\le a_{\max }\cdot \Delta t\hfill \end{array}\right.,$$

(10)

where $a_{\max }$ denotes the maximum acceleration of the UAV.

Additionally, the position constraint can be defined as:

$$\left\{\begin{array}{l}{x}_{\min }\le x\left(t\right)\le x_{\max }\hfill \\ y_{\min }\le y\left(t\right)\le y_{\max }\hfill \end{array}\right.,$$

(11)

3.3. Spectrum Sensing Model

In the monitoring task region, spectrum sensing is performed by CM to detect UHF radiation source. The small-scale fading channel between the source and CM follows the Nakagami-m distribution [32]. Therefore, the probability density function (PDF) of the small-scale fading amplitude $\left|h\right|$ can be expressed as:

$$f_{\left|h\right|}\left(r\right)={\displaystyle \frac{2m^m}{\mathsf{\Gamma }\left(m\right){\mathsf{\Omega }}^m}}{r}^{2m-1}\exp \left(-{\displaystyle \frac{m}{\mathsf{\Omega }}}{r}^2\right),r\ge 0,$$

(12)

where $r$ denotes the small-scale fading amplitude $\left|h\right|$, $\Omega$ represents the average power of the fading signal, i.e., $\Omega =E\left[|h{|}^2\right]$, $E\left[\cdot \right]$ denotes expectation, $\Gamma \left(m\right)$ is the Gamma function with $\Gamma \left(m\right)=\left(m-1\right)!$, and $m$ is the shape parameter that quantifies the severity of small-scale fading.

For the UHF radiation source, when sensed by a CM, the path loss from the source $s$ to the CM can be modeled using a log-distance path loss model, given by:

$$L_{cm,s}\left(d_{cm,s}\right)=L_0+10n{\log }_{10}\left({\displaystyle \frac{d_{cm,s}}{d_0}}\right)+X_{\sigma },$$

(13)

where $d_{cm,s}$ is the Euclidean distance between the radiation source $s$ and the CM, $L_0$ is the path loss at the reference distance $d_0=1$(in dB), which is calculated as $L_0=20{\log }_{10}\left({\displaystyle \frac{4\pi d_0}{\lambda }}\right)$, $\lambda =c/f$ is the wavelength with $c$ being the speed of light and $f$ the operating frequency, $n$ is the path loss exponent, and $X_{\sigma }$ is a zero-mean Gaussian random variable with standard deviation $\sigma$.

When multiple CM simultaneously monitor the radiation source $s$ the received signal power at the $i$th CM can be determined based on its precise location and the measured signal strength differences, enabling estimation of the source’s coordinates $(x_s,y_s)$ and the path loss exponent $n$. Specifically, the received signal power at the $i$th CM is given by:

$$P_{r,i}=P_{t,s}+G_t+G_{r,i}-L_{i,s}\left(d_{i,s}\right),$$

(14)

where $P_{t,s}$ is the transmit power of the radiation source, $G_t$ is the transmitter antenna gain, $G_{r,i}$ is the receiver antenna gain of the $i$th CM, and $L_{i,s}\left(d_{i,s}\right)$ denotes the path loss from the source to the $i$th CM. Assuming the first CM is selected as a reference, the difference in signal strength between the $i$th CM and the reference CM is defined as:

$$\Delta P_i=P_{r,i}-P_{r,1},$$

(15)

Substituting Equation (14), we obtain the sensing model expression:

$$\Delta P_i=\left(G_{r,i}-G_{r,1}\right)-10n{\log }_{10}\left({\displaystyle \frac{d_i}{d_1}}\right)+\left(X_{\sigma ,i}-X_{\sigma ,1}\right),$$

(16)

If all CMs have identical receiving antenna gains, i.e., $G_{ri}=G_{r1}$, then:

$$\Delta P_i=-10n{\log }_{10}\left({\displaystyle \frac{d_i}{d_1}}\right)+\left(X_{\sigma ,i}-X_{\sigma ,1}\right),$$

(17)

For localization of the radiation source s, at least three or more CM are required. The distance from each CM to the source can be estimated using the measured received power $P_{r,i}\left(t\right)$ [33], given by:

$$d_i\left(t\right)={10}^{{\displaystyle \frac{P_{r,i}\left(t\right)-P_1}{-10\cdot n}}},$$

(18)

where $d_i\left(t\right)$ represents the distance from the $i$th CM to the radiation source at time $t$. Based on the logarithmic relationship between $P_{r,i}\left(t\right)$ and distance, the distance can be estimated accordingly. However, due to the limited sensing range of each CM, measurements are only valid under the condition:

$$d_i\left(t\right)\le d_{\max },$$

(19)

where $d_{\max }$ denotes the maximum monitoring coverage radius of a CM. Under the constraints of Equations (18) and (19), the set of effective measurement data from CM at time $t$ is defined as:

$${ℂ}_{CM}\left(t\right)=\left\{i\left|P_{r,i}\left(t\right)>P_{\min }\right.\right\},$$

(20)

where $P_{\min }$ is the minimum detectable signal power threshold. For triangulation-based localization, at least three spatially distributed CM are required. Thus, accurate localization of the radiation source $s$ is feasible only when $\left|{ℂ}_{CM}\left(t\right)\right|\ge 3$, i.e., when three or more CM simultaneously acquire valid signal measurements $P_{r,i}\left(t\right)$.

The CM estimates the AOA ${\varphi }_{i,s}$ of the signal from the radiation source $s$ via its onboard sensing system, and combines this with its own position information to construct a nonlinear observation equation for estimating the source location $x_s=\left(x_s,y_s\right)$. In this work, the Cramér–Rao Lower Bound (CRLB) [34] is introduced as a theoretical performance benchmark. It describes the fundamental limit of estimation accuracy: for any unbiased estimator of the source location $x_s$, the trace of the covariance matrix of the estimation error cannot be lower than the CRLB. A smaller CRLB value indicates a more favorable distribution of CM for high-precision localization.

3.4. Communication Model

The path loss between a CH and a CM during communication transmission can be expressed as:

$$L_{h,m}\left(d_{h,m}\right)=L_0+10n{\log }_{10}\left({\displaystyle \frac{d_{h,m}}{d_0}}\right)+X_{\sigma },$$

(21)

where $d_{h,m}$ denotes the distance between the CH and the CM. The SNR received by the CM from the CH is given by:

$$SN{R}_{h,m}={\displaystyle \frac{P_{t,h}|g_{h,m}{|}^2}{N_{0}B}},$$

(22)

where $P_{t,h}$ is the transmit power of the CH, $g_{h,m}$ is the channel gain, $N_0$ is the noise power spectral density, and $B$ is the communication bandwidth. According to Shannon’s theorem, the maximum achievable data rate between the CH and CM is:

$$R_{h,m}=B{\log }_2\left(1+SN{R}_{h,m}\right),$$

(23)

and the total throughput of the CH is defined as:

$$R_h^{total}={\displaystyle \sum _{m\in {\mathcal{C}}_h}{R}_{h,m}}.$$

(24)

Assume that the entire heterogeneous UAV swarm consists of $H$ CH and $M$ CM, which are divided into $K$ clusters. Each cluster contains one CH and multiple CM. Let ${\mathcal{G}}_k$ denote the set of all UAVs in the $k$th cluster, where $k\in \left\{1,2,\dots ,K\right\}$:

$${\mathcal{G}}_k=\{C{H}_k,C{M}_{k,1},C{M}_{k,2},\dots ,C{M}_{k,M_k}\},$$

(25)

where $M_k$ represents the number of CM in the $k$th cluster. The communication delay within cluster $k$, denote $T_{h,m,k}$, can be expressed as:

$$T_{h,m,k}=T_{prop,k}+T_{proc,k}+T_{queue,k}+\left(1+{\mathsf{\Psi }}_{h,m,k}\right)\times T_{trans,k},$$

(26)

where $T_{prop,k}$ is the propagation delay, determined by the distance between the CH and its associated CM, $T_{proc,k}$ is the processing delay, including encoding and decoding time, $T_{queue,k}$ is the queuing delay, dependent on network load and queue length, ${\mathsf{\Psi }}_{h,m,k}$ is the average retransmission count, $T_{trans,k}$ is the single transmission delay, determined by packet size and bandwidth. Assuming $P_{success}\left(\gamma \right)$ denotes the probability of successful packet delivery under a given SNR condition $\gamma$, which can be derived from metrics such as bit error rate (BER), we have:

$${\mathsf{\Psi }}_{h,m,k}={\displaystyle \frac{1}{P_{success}\left(\gamma \right)}}-1,$$

(27)

When the link state between CH and CM is denoted by ${\sigma }_{h,m}$, it is defined as:

$${\sigma }_{h,m}=\left\{\begin{array}{cc}1,\hfill & CMlinkedwithCH,\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.$$

(28)

The overall communication delay $T_{total}$ is defined as the weighted average of intra-cluster communication delays across all clusters:

$$T_{total}={\displaystyle \frac{{\displaystyle \sum _{k=1}^K\left({\displaystyle \sum _{m\in {\mathcal{G}}_k}{\sigma }_{h,m}\cdot w_{h,m,k}{T}_{h,m,k}}\right)}}{{\displaystyle \sum _{k=1}^K\left({\displaystyle \sum _{m\in {\mathcal{G}}_k}{\sigma }_{h,m}\cdot w_{h,m,k}}\right)}}},$$

(29)

where $w_{h,m,k}$ is the communication weight between the CH and CM in the $k$th cluster, reflecting the relative importance or traffic load of the link.

4. Problem Formulation and Analysis

In this work, the objective is to achieve distributed localization of UHF radiation source by leveraging a heterogeneous UAV swarm, through coordinated collaboration among diverse UAV types, enabling efficient and high-precision localization. Specifically, the CH maintain stable communication links with the central control base station and manage their associated CM within each cluster. The CM are equipped with high-sensitivity sensing devices and are responsible for detecting and localizing UHF radiation source within their assigned task regions.

To address the constraints on individual UAVs in terms of endurance, payload, and performance, we adopt a cooperative strategy based on the Dynamic Range-Sensitive Source Localization (DRSS) method [35], which jointly estimates the source location $x_s=\left(x_s,y_s\right)$ and path loss exponent $n$ by solving a nonlinear system of equations that incorporates both signal strength and AOA measurements. Using a weighted least squares approach, the estimation is performed when at least three or more CM simultaneously detect the source. By combining Equations (6) and (9)–(11), along with the sensing and communication constraints, the goal is to maximize the total duration $S\left(t\right)$ during which the radiation source s is continuously localized. This leads to the following optimization problem for adjusting UAV motion and information transmission:

$$\underset{{\left\{Q_i\left(t\right)\right\}}_{t=1}^T,\forall i}{\max }{\displaystyle \sum _{t=1}^{T}S}\left(t\right),$$

(30)

$$S\left(t\right)=\{\begin{array}{cc}1,& \mathit{if}\left|{\mathbb{C}}_{\mathit{CM}}\right(t\left)\right|\ge 3,\\ 0,& \mathit{otherwise}.\end{array}$$

(31)

The objective of this optimization is to find, under kinematic and communication constraints, the optimal trajectory $Q_i\left(t\right)$ for each UAV over the time horizon $T$, such that the cumulative duration $S\left(t\right)$, representing the number of time steps where at least three CM are actively detecting the source—is maximized. This ensures continuous three-point localization of the radiation source.

Simultaneously, under dynamic constraints, we aim to determine suitable clustering strategies and CH positions to maximize the overall network throughput while minimizing the end-to-end communication delay, thereby reducing the source localization latency. Based on Equations (22) and (29), the optimization problem can be further formulated as:

$$\max {\displaystyle \sum _{h,m\in {\mathcal{G}}_k}{\sigma }_{h,m}\cdot B\cdot {\log }_2\left(1+SN{R}_{h,m}\right)},$$

(32)

$$\min T_{total}=\min \left({\displaystyle \frac{{\displaystyle \sum _{k=1}^K\left({\displaystyle \sum _{m\in {\mathcal{G}}_k}{w}_{h,m,k}{T}_{h,m,k}}\right)}}{{\displaystyle \sum _{k=1}^K\left({\displaystyle \sum _{m\in {\mathcal{G}}_k}{w}_{h,m,k}}\right)}}}\right).$$

(33)

Equations (30), (32) and (33) represent multi-objective non-linear optimization problems. While traditional methods such as nonlinear programming [35], genetic algorithms [36], particle swarm optimization [37], and interactive evolutionary algorithms [38] have been widely used for static or slowly varying scenarios and offer good performance in fixed environments, they suffer from poor adaptability and learning capability when faced with dynamic and complex environments. These methods require reinitialization and full search upon environmental changes, making them inefficient for real-time decision-making. To overcome these limitations, we propose HMUDRL, which enables autonomous decision-making by both CH and CM. By integrating dynamic environment perception, adaptive clustering, and joint optimization of mobility and communication, HMUDRL achieves robust and efficient distributed localization of UHF radiation source in highly dynamic scenarios.

5. Distributed Heterogeneous Multi-Agent Deep Reinforcement Learning Algorithm for UAV Clustering

In this section, we first introduce the HMUDRL framework and its key components, followed by a discussion of its training procedure. While existing hierarchical and cooperative multi-agent reinforcement learning approaches, such as MAPPO [8] and QMIX [39], adopt the centralized training with decentralized execution paradigm and have achieved notable progress in various domains, they are not specifically designed for physically constrained tasks like electromagnetic source localization, where sensing and communication are tightly coupled. Our proposed HMUDRL framework is not a straightforward adaptation of existing MARL paradigms; rather, it represents a domain informed architectural innovation tailored to UHF source seeking missions. By co-designing agent roles, communication topology, and reward structure, HMUDRL effectively addresses the core challenges inherent in decentralized, heterogeneous, and perception driven localization scenarios.

5.1. HMUDRL Framework

As illustrated in Figure 3, the proposed HMUDRL is composed of two types of agents: one serving CH and the other serving CM. These agents share the same environment but possess distinct observation spaces, action spaces, and reward functions tailored to their respective roles. The intelligent decision-making process is implemented via an actor-critic architecture: the actor network selects actions based on the current state, thereby executing the policy, while the critic network evaluates the anticipated future rewards, assessing the quality of the selected actions. HMUDRL adopts a distributed framework where each heterogeneous UAV agent operates independently. CH agents employ the Hierarchical Multi-Agent PPO (HMPPO) algorithm, leveraging their computational and communication capabilities to exchange global information with the central control center and share a common critic network. In contrast, CM agents utilize the Decentralized Distributed Multi-Agent PPO (DDMPPO) algorithm, focusing on cooperative spectrum sensing and localization tasks for UHF radiation source. This modular design enables effective coordination between heterogeneous agents while allowing them to interact dynamically with the environment. During actual operation, each CM only needs to communicate with its associated CH, eliminating the need for inter-agent communication among all UAVs. This reduces computational and communication overhead significantly. Moreover, the distributed architecture allows the two types of agents to be trained separately, enhancing the scalability and robustness of the HMUDRL framework.

5.2. HMPPO Design

In the UHF radiation source localization scenario, the objective is to collaboratively discover an optimal policy through distributed coordination among heterogeneous UAVs, enabling the CM to effectively locate radiation sources while minimizing communication overhead and latency. To achieve this goal, we propose HMPPO based on a decentralized control strategy for CH. The design of the CH control strategy focuses on maintaining stable cluster formation, balancing the workload among CH, ensuring high-quality communication links, low-latency transmission, and sufficient coverage.

5.2.1. Observation Set

As shown in Figure 2, each CH periodically broadcasts Hello messages for topology maintenance and establishes or updates associations with nearby CM. The observation set of the CH includes its own state as well as global contextual information. Specifically, the observation vector consists of eight key variables: the position of the CH $Q_h$, the displacement $S_h$ relative to the centroid of all CM, the distance $L_h$ from the estimated radiation source location $s^{\prime }$, the spatial dispersion $S_{mt}$ of the associated CM, and the average inter-CH distance $D_h$. The full observation set is thus expressed as:

$$O^h=\left\{Q_h,S_h,L_h,S_{mt},D_h\right\},$$

(34)

5.2.2. Action Set

The CH adjusts its movement action $\Delta Q_h$ based on environmental observations, which can be subdivided into five options: stay still, move north, south, east, or west, denoted as $a_t\in \left\{0,1,2,3,4\right\}$. Only one action is executed at each time step; thus, $a_t$ is a discrete multiclass random variable with probability distribution determined by parameters $\pi =\left({\pi }_0,{\pi }_1,{\pi }_2,{\pi }_3,{\pi }_4\right)$, satisfying:

$$P(a_t=i)={\pi }_i,{\displaystyle \sum _{i=0}^4{\pi }_i}=1,{\pi }_i\ge 0,$$

(35)

To map the state information in Equation (34) to specific actions, we design an Actor-Network that takes the observation set and the advantage value from the Critic-Network as input. The network outputs unnormalized scores (logits) for each action, $z=f\left(O^h;\theta \right)$, where $f\left(\cdot \right)$ is a neural network function with output dimension 5 and $\theta$ denotes the learnable parameters. The action probabilities are then computed via the softmax function:

$${\pi }_i={\displaystyle \frac{\exp \left(z_i\right)}{{\displaystyle \sum _{j=0}^4\exp \left(z_j\right)}}},$$

(36)

where $z_i$ is the logit corresponding to action $i$. Finally, the action $a_t$ is selected stochastically according to the probability ${\pi }_i$.

5.2.3. Reward

Based on the observable state variables of the CH, including AOA estimates reported by CM, RSS measurements, and relative geometric relationships between CH and CM, we design a reward function to guide the CH in learning optimal clustering and mobility policies. At each time step $t$, the scalar reward $r_t^h$ for a CH is defined as a composite of three components: localization confidence reward $r_{loc}^h$, coverage quality reward $r_{cov}^h$, and exploration diversity reward $r_{dir}^h$.

The localization confidence reward $r_{loc}^h$ is designed to reflect the accuracy and reliability of source localization based on AOA measurements from multiple CM. It is determined jointly by angular diversity and the number of valid AOA observations, and is expressed as:

$${\kappa }_t=\{\begin{array}{cc}0,& N_t<2,\\ \left({\displaystyle \frac{N_t}{N_m}}\right)(1-{\displaystyle \frac{2}{N_t(N_t-1)}}{\displaystyle {\displaystyle \sum }_{1\le i<j\le N_t}}|\cos ({\varphi }_{mi}-{\varphi }_{mj})|),& N_t\ge 2.\end{array}$$

(37)

where $N_t$ denotes the number of CM that successfully report valid AOA values at time $t$, ${\varphi }_{mi}$ is the AOA measured by the $i$th CM, and $N_m$ is the total number of CM in the cluster.

The coverage quality reward $r_{cov}^h$ encourages the CH to maintain its associated CM within effective communication and sensing ranges. It is defined as:

$$r_{cov}^h={\displaystyle \frac{1}{N_m}}{\displaystyle \sum _{i=1}^{N_m}\exp \left(-\frac{|\parallel Q_i^m-c^h\parallel -R_0|}{{\sigma }_{cov}}\right),}$$

(38)

where $Q_i^m$ is the position of the $i$th CM, $c^h$ is the centroid of the CH, $R_0$ is the maximum communication radius, and ${\sigma }_{cov}$ is a scaling parameter.

The exploration diversity reward $r_{dir}^h$ promotes decentralized movement of CH to avoid clustering and enhance spatial coverage. It is formulated as:

$$r_{dir}=\tan \mathrm{h}\left({\displaystyle \frac{1}{N_h}}{\displaystyle \sum _{j=1}^{N_h}\frac{\parallel Q_{j,t}^h-Q_{j,t-1}^h\parallel }{v_{scale}}}\right),$$

(39)

where $N_h$ is the total number of CH, $Q_{j,t}^h$ is the position of the $j$th CH at time $t$ and $v_{scale}$ is a normalized velocity scale factor. The total reward function for the CH is then given by:

$$r_t^h={\omega }_{loc}^h\cdot r_{loc}^h+{\omega }_{cov}^h\cdot r_{cov}^h+{\omega }_{dir}^h\cdot r_{dir}^h.$$

(40)

where ${\omega }_{loc}^h$, ${\omega }_{cov}^h$ and ${\omega }_{dir}^h$ are weighting coefficients that balance the contributions of localization confidence, coverage quality, and exploration diversity, respectively.

5.3. DDMPPO Design

Considering the limited payload and operational duration of individual CM platforms, we aim to highlight their monitoring capabilities by optimizing information processing and networking transmission on a single platform, enabling lightweight and compact design for CM. By deploying a large number of such UAVs in a distributed manner, we ensure stable communication links with their associated CH while achieving real-time, persistent, and high-accuracy localization of UHF radiation source across the target region.

5.3.1. Observation Set

The observation set for each CM primarily includes: Its own position $Q_m$, Distance $d_{h,m}$ to its associated CH, RSS $P_r$, RSS variation $\Delta P_r$, AOA ${\varphi }_m$ of the incoming signal from the radiation source. Thus, the observation vector is defined as:

$$O^m=\left\{Q_m,d_{h,m},P_r,\Delta P_r,{\varphi }_m\right\},$$

(41)

5.3.2. Action Set

The action of a CM is mainly defined as a movement action $\Delta Q_m$. The actual motion decision is determined based on the measured AOA ${\varphi }_m$ and the received signal power $P_r$ When at least three CM simultaneously detect the radiation source, i.e., $\left|{\mathbb{C}}_{\mathit{CM}}\right(t\left)\right|\ge 3$, the CM can collaboratively perform triangulation to achieve accurate localization of the source.

5.3.3. Reward

A scalar reward signal is generated based on the state and action of the CM to provide feedback for learning. The total reward is composed of multiple components, including: RSS increase reward $r_{\Delta P_r}^m$, encouraging movement toward regions with higher signal strength, Distance maintenance reward $r_d^m$, rewarding actions that maintain a suitable distance from the CH, Valid AOA reward $r_{\varphi }^m$, incentivizing accurate AOA measurements. The RSS-based reward is formulated as:

$$r_{\varphi }^m=\{\begin{array}{cc}1-\frac{\Delta {\varphi }_{\min }}{\pi },& \mathit{if}N\ge 3,\\ 0,& \mathit{otherwise},\end{array}$$

(42)

where $\Delta {\varphi }_{\min }={\min }_{i\ne j}|{\varphi }_i-{\varphi }_j|$ denotes the minimum absolute angular difference between AOA measurements from different CM. A smaller $\Delta {\varphi }_{\min }$ indicates higher angular consistency among CM, leading to lower rewards, while $\Delta {\varphi }_{\min }$ approaching $\pi /2$ yields the maximum reward, promoting diverse angular coverage. The total reward function for CM is then expressed as:

$$r_t^m={\omega }_{\Delta P_r}^m\cdot r_{\Delta P_r}^m+{\omega }_d^m\cdot r_d^m+{\omega }_{\varphi }^m\cdot r_{\varphi }^m.$$

(43)

where ${\omega }_{\Delta P_r}^m$, ${\omega }_d^m$ and ${\omega }_{\varphi }^m$ are weighting coefficients that balance the contributions of signal strength improvement, distance maintenance, and angular diversity, respectively.

5.4. Training Process

In the proposed HMUDRL framework, both HMPPO (for CH) and DDMPPO (for CM) are implemented based on Algorithm 1, the PPO algorithm. The PPO method introduces a clipped objective function via a trust region approach [30] to constrain the magnitude of policy updates at each step, thereby enhancing training stability. The clipped surrogate objective is defined as:

$$L^{CLIP}\left(\theta \right)={\mathbb{E}}_t\left[\min \left(r_t\left(\theta \right)A_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ)A_t\right)\right],$$

(44)

where $r_t(\theta )={\displaystyle \frac{{\pi }_{\theta }^{\mathrm{new}}(a_t|s_t)}{{\pi }_{\theta }^{\mathrm{old}}(a_t|s_t)}}$ denotes the probability ratio between the new and old policies, $\theta$ represents the policy parameters (i.e., neural network weights), $A_t$ is the advantage function estimating the relative benefit of the current action, and $ϵ$ controls the step size of policy update by clipping $r_t\left(\theta \right)$ within the interval $\left(1-ϵ,1+ϵ\right)$. The overall objective of the PPO algorithm [30] is formulated as:

$$L^{CLIP+VF+S}\left(\theta \right)=L^{CLIP}\left(\theta \right)-c_1{L}^{VF}\left(\phi \right)+c_{2}S\left[{\pi }_{\theta }\right]\left(s_t\right),$$

(45)

where $L^{VF}\left(\phi \right)$ is the value loss function, defined as $L^{VF}\left(\phi \right)={\mathbb{E}}_t\left[{\left(V_{\phi }\left(s_t\right)-V_t^{\mathrm{target}}\right)}^2\right]$, representing the squared error between the estimated value and the target value $V_t^{\mathrm{target}}=A_t+V_{\phi ,old}\left(s_t\right)$, $S\left[{\pi }_{\theta }\right]\left(s_t\right)$ is the entropy term, given by $S[{\pi }_{\theta }]\left(s_t\right)=-{\mathbb{E}}_{a_t~{\pi }_{\theta }(\cdot {|s}_t)}\log {\pi }_{\theta }{(a}_t{|s}_t)$, which encourages exploration and prevents premature convergence to local optima, $c_1$ and $c_2$ are hyperparameters balancing the contributions of value loss and entropy regularization.

For the HMPPO algorithm, the primary goal is to optimize cluster formation and mobility strategies for CH while maintaining high-quality communication links. To achieve this, the advantage function is estimated using the Generalized Advantage Estimation (GAE) method [31], which computes the advantage as:

$$A_t^h={\displaystyle \sum _{l=0}^T{\left(\gamma \lambda \right)}^l{\delta }_{t+l}^h},$$

(46)

where $\gamma$ is the discount factor, $\lambda$ is the GAE parameter that balances bias and variance in the advantage estimate, and ${\delta }_t^h$ is the temporal difference (TD) error defined as:

$${\delta }_t^h=r_t^h+\gamma V\left(s_{t+1}^h\right)-V(s_t^h),$$

(47)

With $r_t^h$ being the reward received by the CH at time step $t$, $r_t^h={\displaystyle \sum _d\log \frac{\pi \left(a_t\right|s_t)}{{\tilde{p}}_d\left(s_t\right)}}$, where ${\tilde{p}}_d(s_t)={e^{\mu \cdot o_d}/\sum }_{{d^{\prime }}^{e^{\mu \cdot o_{d^{\prime }}}}}$, $o_d$ and $o_{d^{\prime }}$ are the observation scores, $\mu$ is the scaling factor controlling the distribution of observation probabilities, $V\left(s_t^h\right)$ and $V\left(s_{t+1}^h\right)$ denote the critic network’s value estimates for the current and next states, respectively.

For the DDMPPO algorithm, its primary objective is to optimize the sensing performance, SNR, and energy consumption of CM. The advantage function for CM is expressed as:

$$A_t^m={\displaystyle \sum _{l=0}^T{\left(\gamma \lambda \right)}^l{\delta }_{t+l}^m,}$$

(48)

where ${\delta }_t^m$ denotes the TD error computed for CM, defined as:

$${\delta }_t^m=r_t^m+\gamma V\left(s_{t+1}^m\right)-V(s_t^m),$$

(49)

With $r_t^m$ being the reward obtained by CM at time step $t$, $V\left(s_t^m\right)$ the value estimate of the current state from the critic network, and $V\left(s_{t+1}^m\right)$ the value estimate of the next state.

In this work, we employ a HMUDRL framework that integrates CH and CM into a unified distributed learning architecture. To reduce decision complexity and effectively implement distributed cooperative control, we design a dual-branch policy network structure: one branch outputs control actions (e.g., cluster formation and mobility decisions) for CH, while the other branch generates monitoring-related actions (e.g., sensing and energy management) for CM. This modular architecture enables independent optimization of control and sensing tasks, with distinct action spaces and objectives for each agent type. Based on Equation (44), we reformulate the total policy loss function for HMUDRL as the sum of two separate clipped surrogate objectives:

$$L_h^{CLIP}\left(\theta \right)={\mathbb{E}}_t\left[\min \left(r_t^h\left(\theta \right)A_t^h,\mathrm{clip}(r_t^h(\theta ),1-ϵ,1+ϵ)A_t^h\right)\right],$$

(50)

$$L_m^{CLIP}\left(\theta \right)={\mathbb{E}}_t\left[\min \left(r_t^m\left(\theta \right)A_t^m,\mathrm{clip}(r_t^m(\theta ),1-ϵ,1+ϵ)A_t^m\right)\right],$$

(51)

where $L_h^{CLIP}\left(\theta \right)$ and $L_m^{CLIP}\left(\theta \right)$ represent the clipped policy losses for CH and CM, respectively. The clipping parameter $ϵ$ limits the magnitude of policy updates to prevent large, destabilizing changes during training. Each branch operates independently, ensuring stable and coordinated updates across the two agent types.

$$L_{total}^{CLIP}\left(\theta \right)=L_h^{CLIP}\left(\theta \right)+L_m^{CLIP}(\theta ),$$

(52)

To ensure effective learning in the heterogeneous system, we apply entropy regularization to both agent types, promoting exploration and preventing premature convergence. The entropy terms are defined as:

$$S[{\pi }_{\theta ,h}]\left(s_t\right)=-{\mathbb{E}}_{a_t~{\pi }_{\theta ,h}(\cdot {|s}_t)}\log {\pi }_{\theta ,h}{(a}_t{|s}_t),$$

(53)

$$S[{\pi }_{\theta ,m}]\left(s_t\right)=-{\mathbb{E}}_{a_t~{\pi }_{\theta ,m}(\cdot {|s}_t)}\log {\pi }_{\theta ,m}{(a}_t{|s}_t),$$

(54)

The total policy entropy is then combined as:

$$S[{\pi }_{\theta }]\left(s_t\right)=S[{\pi }_{\theta ,h}]\left(s_t\right)+S[{\pi }_{\theta ,m}]\left(s_t\right),$$

(55)

By jointly optimizing both branches using Equation (55), we unify the exploration pressure across the system, avoiding imbalance where one branch may converge prematurely while the other remains under-explored. Furthermore, due to the distinct state spaces, action spaces, reward functions, and objectives of CH and CM, we design separate critic networks for each agent type and compute their respective value losses:

$$L_h^{VF}\left({\phi }_h\right)=\mathbb{E}t\left[{\left(V_{{\phi }_h}\left(s_t^h\right)-V_{t,h}^{\mathrm{target}}\right)}^2\right],$$

(56)

$$L_m^{VF}\left({\phi }_m\right)=\mathbb{E}t\left[{\left(V_{{\phi }_m}\left(s_t^m\right)-V_{t,m}^{\mathrm{target}}\right)}^2\right],$$

(57)

where $V_{t,h}^{\mathrm{target}}$ and $V_{t,m}^{\mathrm{target}}$ are the target values for the CH and CM critic networks, respectively. The total value loss is then given by:

$$L_{total}^{VF}\left(\phi \right)={\alpha }_h{L}_h^{VF}\left({\phi }_h\right)+{\alpha }_m{L}_m^{VF}({\phi }_m),$$

(58)

where ${\alpha }_h$ and ${\alpha }_m$ are weighting coefficients balancing the contributions of the two critic networks. Based on the overall objective function of the PPO algorithm in Equation (45), the complete objective function for the HMUDRL algorithm incorporates the policy loss, value loss, and entropy regularization terms, as shown in Equation (59):

$$L^{HMUDRL}(\theta ,\varphi )=L_{total}^{CLIP}\left(\theta \right)-L_{total}^{VF}\left(\varphi \right)+\beta S[{\pi }_{\theta }]\left(s_t\right).$$

(59)

where $\beta$ is the weight coefficient for the entropy term. By appropriately tuning the weight coefficients, the trade-off among policy improvement, value estimation accuracy, and exploration can be effectively balanced, enabling efficient optimization of the heterogeneous UAV monitoring system.

Algorithm 1 Training Process of HMUDRL Algorithm
1:	Initialize policy networks ${\pi }_{\theta }^h$ for CH agents and ${\pi }_{\theta }^m$ for CM agents, value networks $V_{{\phi }_h}$ for CH agents and $V_{{\phi }_m}$ for CM agents, ${\theta }_{old}^h$$\leftarrow {\theta }^h$$,{\theta }_{old}^m$$\leftarrow {\theta }^m$$,{\phi }_{old}^h$$\leftarrow {\phi }^h$$,{\phi }_{old}^m$$\leftarrow {\phi }^m$$,D^h$ for CH agents and $D^m$ for CM agents
2:	for iteration $i=1,2,\dots ,I$ do
3:	for episode $t=1,2,\dots ,T$ do
4:	for each CH agent $h$ do
5:	Observe state $s_t^h$
6:	Select action $a_t^h$$~{\pi }_{\theta ,h}^{old}\left(\left.\cdot \right|O_t^h\right)$
7:	end for
8:	for each CM agent $m$ do
9:	Observe state $s_t^m$
10:	Select action $a_t^m$$~{\pi }_{\theta ,m}^{old}\left(\left.\cdot \right|O_t^m\right)$
11:	end for
12:	Execute actions $a_t^h$$,a_t^m$ in environment
13:	Observe rewards $R_t^h$$,R_t^m$ and next states $O_{t+1}^h$$,O_{t+1}^m$
14:	Store transition $(O_t^h$$,a_t^h$$,R_t^h$$,O_{t+1}^h)$ and $(O_t^m$$,a_t^m$$,R_t^m$$,O_{t+1}^m)$ in $D^h$ and $D^m$
15:	end for
16:	Compute advantages $A_t^h$ and $A_t^m$ using Equations (46) and (48)
17:	Compute target values $V_{t,h}^{\mathrm{target}}$ and $V_{t,m}^{\mathrm{target}}$
18:	for $k$$=1,2,\dots ,K$ do
19:	for each CH agent $h$ do
20:	Compute policy losses $L_h^{CLIP}\left(\theta \right)$ using Equation (50)
21:	Compute entropy bonus $S[{\pi }_{\theta ,h}]\left({\mathrm{s}}_{\mathrm{t}}\right)$ using Equation (53)
22:	Compute value loss $L_h^{VF}\left({\phi }_h\right)$ using Equation (56)
23:	Update ${\theta }^h$ and ${\phi }^h$
24:	end for
25:	for each CM agent $m$ do
26:	Compute policy losses $L_m^{CLIP}\left(\theta \right)$ using Equation (51)
27:	Compute entropy bonus $S[{\pi }_{\theta ,m}]\left(s_t\right)$ using Equation (54)
28:	Compute value loss $L_m^{VF}\left({\phi }_m\right)$ using Equation (57)
29:	Update ${\theta }^m$ and ${\phi }^m$
30:	end for
31:	Compute total loss $L^{HMUDRL}(\theta ,\phi )$ using Equation (59)
32:	end for
33:	Update ${\theta }_{old}^h$$\leftarrow {\theta }^h$$,{\phi }_{old}^h$$\leftarrow {\phi }^h$ for all CH agents
34:	Update ${\theta }_{old}^m$$\leftarrow {\theta }^m$$,{\phi }_{old}^m$$\leftarrow {\phi }^m$ for all CM agents
35:	end for

6. Simulation Results

This section presents the simulation experiments of the proposed HMUDRL algorithm. First, we simulate the monitoring and sensing environment described in the paper. Subsequently, numerical simulations are conducted to evaluate the performance of the proposed method in terms of localization accuracy, reward value, number of steps to first detection, average interaction frequency, and other metrics. The results are compared and analyzed against baseline algorithms.

6.1. Experimental Setup and Parameters

The simulation is configured in a 1000 × 1000 m² mission area, where CH and CM are deployed. The deployment altitude of CH ranges from 200 to 400 m, while that of CM ranges from 100 to 300 m. The speed of CH varies between 0 and 22 m/s, and that of CM ranges from 0 to 14 m/s. The transmission power of CH is set to 20 dBm, with a carrier frequency of 3.0 GHz and a bandwidth of 10 MHz. The Hello message broadcasting interval is set to 1 s [40]. The maximum control range of each CH is defined as 500 m. Each CM is equipped with a 2.4 GHz UHF radiation source detector, which consists of eight measurement units arranged at 45° intervals for omnidirectional sensing to ensure reliable signal reception. Considering the payload capacity and energy constraints of UAVs, and to verify the practical deployability of our approach on resource constrained UAV platforms, which typically lack dedicated GPU hardware, all reinforcement learning training and evaluation experiments were conducted using a lightweight training scheme. The proposed HMUDRL framework was implemented in Python 3.11.9 and PyTorch 2.4.1, without any GPU acceleration, and trained entirely on the CPU of a standard desktop computer equipped with an Intel Core i5-1130G7 processor and 16 GB RAM, thereby clearly demonstrating its computational efficiency. Each training episode lasts 10 s, discretized into 100 time steps (each step lasting 0.1 s). For convenience in simulation, the speeds of both CH and CM are uniformly set to 10 m/s (1 m per step). The weights of different rewards in the heterogeneous UAV monitoring system are determined through the training process, and the specific task scenario is adjusted to achieve a balance between control and sensing objectives. The simulation parameters are summarized in

Table 2. Hyperparameter setting.

System Parameters	Numerical Settings	Description
$m$	3	number of CH
$n$	14	number of CM
episodes	1000	number of training episodes
$T$	100	time steps per episode
${\alpha }_a$	0.0001	actor learning rate
${\alpha }_c$	0.001	critic learning rate
$\gamma$	0.99	discount factor
$\lambda$	0.95	GAE lambda
$ϵ$	0.1	policy update clipping parameter
${\alpha }_h$	0.6	weight of CH reward in joint reward
${\alpha }_m$	0.4	weight of CM reward in joint reward
${\omega }_{loc}^h$	0.6	weight of localization confidence reward
${\omega }_{cov}^h$	0.2	weight of coverage quality reward
${\omega }_{dir}^h$	0.2	weight of exploration diversity reward
${\omega }_{\Delta P_r}^m$	0.5	weight of RSS gain reward
${\omega }_d^m$	0.2	weight of distance-based reward to CH
${\omega }_{\varphi }^m$	0.3	weight of valid AOA reward
$P_0$	-20	reference RSS value at 1 m
$n$	2.0	path loss exponent
${\sigma }_{RSS}$	2.0	RSS measurement standard deviation (dB)
${\sigma }_{AOA}$	5.0	AOA measurement noise standard deviation (°)
${\gamma }_{AOA}$	−70.0	minimum RSS threshold for AOA estimation (dBm)
$D_{S{S}^{\text{'}}}$	20	localization success threshold (m)
$D_{safe}$	1	safe distance (m)

.

The weights of individual reward components, such as the coverage reward, localization accuracy reward, and communication cost penalty, were determined through a grid search over a reasonable range and further refined by incorporating UAV maneuverability constraints and UHF propagation characteristics. Starting from balanced initial weights, we iteratively adjusted them to prioritize successful localization while ensuring feasible trajectories and low communication overhead. HMUDRL exhibits robust performance under moderate variations in these weights.

6.2. Experimental Results and Analysis

This study establishes a multi-intelligent UAV monitoring platform, deploying three CH and fourteen CM. In each experiment, the UHF radiation source is randomly placed within the experimental area, and the CH control the CM to perform localization tasks. The system records key metrics throughout the process. As shown in Figure 4, the red solid line represents the trajectory of the CH during the localization process; the blue dot and dashed line indicate the movement path of the CM during localization; the yellow star marks the actual location of the UHF radiation source; the green “×” denotes the initial position of the CM; and concentric circles around the radiation source within the region illustrate changes in color representing the gradient change of the radiation field strength. Figure 4a illustrates the localization trajectories under RSS-only conditions (i.e., without AOA information). After completing the full HMUDRL algorithm training, both the CH and CM effectively converge toward the radiation source. CM perform spiral search motions around the source, ultimately achieving precise localization with the final estimated source position closely matching the true radiation source location. Figure 4b shows the case where the CM is randomly deployed within the designated area, but fails to locate the source due to the lack of dynamic coordination mechanisms. In this scenario, localization performance deteriorates significantly compared to when the HMUDRL algorithm is employed. Figure 4c presents the situation where the clustering optimization function of the CH is disabled. Without dynamic cluster adjustment, the CH remains fixed at a predefined position (e.g., uniformly distributed), which limits the ability of CM using DMPPO to explore optimally. As a result, the overall localization efficiency may be compromised. Figure 4d demonstrates the performance of the HMUDRL algorithm under high RSS and AOA noise conditions. Despite the more challenging environment, the system’s localization accuracy slightly decreases, yet it still achieves relatively accurate source positioning. This highlights the robustness and adaptability of the HMUDRL algorithm in complex and noisy monitoring environments.

Figure 5 Comparison of the moving average total rewards over 1000 training episodes between the proposed HMUDRL algorithm and four baseline methods (MADDPG, MASAC, COMA, and MAA2C). The x-axis represents the number of episodes during training, while the y-axis shows the smoothed cumulative reward values, reflecting the stability and convergence performance of each algorithm’s policy optimization throughout the prolonged learning process.

Overall, the HMUDRL algorithm (red curve) demonstrates a significant learning advantage in the early stages of training, with its reward value rapidly increasing and stabilizing after approximately the 150th episode, ultimately settling around 180. This indicates that the algorithm possesses rapid convergence capabilities and strong policy exploration efficiency. In contrast, although the MASAC algorithm (purple curve) also exhibits a fast initial growth rate, it suffers from significant reward fluctuations and noticeable oscillations in later stages, suggesting instability in its policy. The MAA2C algorithm (green curve) maintains a relatively stable growth trend throughout the training process, with reward values ranging between 70 and 90, demonstrating good robustness; however, its maximum reward level is notably lower than that of HMUDRL, indicating room for improvement in cooperative decision-making capability. The COMA algorithm (orange curve) experiences severe reward volatility, especially with several significant declines during the mid-training phase, suggesting that its counterfactual baseline mechanism struggles to effectively model individual contributions in complex heterogeneous environments, leading to unstable policy learning. The MADDPG algorithm (blue curve) performs the worst, with its reward values hovering around 25 without significant changes as training progresses, indicating that its centralized Critic network faces severe dimensionality issues when handling large-scale heterogeneous UAV swarms, resulting in significant value estimation errors and difficulties in policy updates. Notably, by introducing a hierarchical architecture fusion mechanism, the HMUDRL algorithm successfully achieves differentiated policy learning for two types of agents: CH and CM, thereby avoiding performance bottlenecks caused by unified policy spaces in traditional MARL approaches. Moreover, the hierarchical reward design further enhances the policy’s responsiveness to environmental observations, enabling continuous optimization of localization behaviors in dynamic electromagnetic environments. The HMUDRL algorithm not only leads significantly in final reward levels but also outperforms existing mainstream multi-agent reinforcement learning algorithms in terms of training stability and convergence speed, fully validating its effectiveness and superiority in the task of heterogeneous UAV collaborative localization.

Table 3. Comparison of RMSE, localization success rate, and number of steps to first detection among the proposed algorithm and four baseline methods over 1000 episodes.

Algorithm Name	RMSE Mean (m)	RMSE Median (m)	RMSE Max (m)	RMSE Min (m)	Localization Success Rate (%)	Steps to First Detection
HMUDRL	39.34	22.83	678.11	0.89	96.1%	6
MADDPG [23]	620.43	666.15	1125.30	12.06	93.7%	5
MASAC [24]	395.88	297.76	1774.58	13.15	94.6%	8
COMA [25]	135.02	30.07	890.99	0.36	96.3%	6
MAA2C [26]	82.70	28.89	992.92	0.76	94.6%	8

presents a comparative analysis of the proposed HMUDRL algorithm against four baseline methods, in terms of root mean square error (RMSE), localization success rate, and the number of steps to first detection over 1000 training episodes. The results demonstrate that HMUDRL significantly outperforms all baselines in the UHF radiation source localization task. In terms of RMSE, HMUDRL achieves a mean error of 39.34 m, substantially lower than MADDPG (620.43 m), MASAC (395.88 m), and COMA (135.02 m), and only slightly better than MAA2C (82.70 m). More importantly, HMUDRL exhibits the lowest median RMSE among all methods at 22.83 m, indicating consistent high-precision performance. Its maximum localization error is 678.11 m, which is notably lower than those of MASAC (1774.58 m) and MADDPG (1125.30 m), reflecting improved robustness under worst-case scenarios. Regarding efficiency, HMUDRL requires only 6 steps on average to first detect the target—outperforming all baselines except MADDPG (5 steps). Crucially, this rapid detection is achieved without compromising accuracy, as HMUDRL maintains a high localization success rate, underscoring its effective exploration capability in dynamic environments. These advantages stem from the proposed hierarchical architecture: CH employ the HMPPO algorithm to optimize cluster formation and mobility control, while CM utilize the DDMPPO algorithm to balance sensing performance and energy consumption. This dual-policy design, combined with a hierarchical reward fusion mechanism, mitigates the curse of dimensionality in policy updates. Furthermore, differentiated entropy regularization prevents premature convergence in either agent branch. In contrast, MADDPG suffers from dimensionality explosion in its centralized critic network, leading to biased value estimation. MASAC’s increased parameter complexity tends to induce policy oscillations. MAA2C, which shares a single critic across heterogeneous agents, fails to account for their distinct objectives and observation spaces, thereby limiting localization accuracy. These limitations collectively highlight the superiority of HMUDRL in addressing the challenges of heterogeneous multi-UAV cooperative localization.

Figure 6 further reveals the characteristics of the data distribution. The RMSE distribution of the HMUDRL algorithm is notably more concentrated, with a narrow interquartile range (IQR), indicating high consistency and robustness in localization performance. In contrast, the boxplots for MADDPG and MASAC exhibit significantly elongated boxes accompanied by numerous outliers, e.g., MADDPG reaches a maximum RMSE of 1125 m, demonstrating substantial error fluctuations and poor stability. Although COMA and MAA2C display relatively compact distributions compared to MADDPG and MASAC, their boxplots are visibly wider than that of HMUDRL, suggesting degraded localization accuracy. Notably, COMA achieves a localization success rate (96.3%) comparable to HMUDRL (96.1%); however, its mean RMSE (135.02 m) is approximately 3.4 times higher than that of HMUDRL (39.34 m), revealing inadequate performance in complex or noisy environments despite frequent detection. It can be stated that the HMUDRL algorithm achieves a breakthrough in the integrated performance of accuracy, stability, and efficiency through architectural innovation and mechanism optimization.

According to the convergence of the HMUDRL algorithm and the MASAC baseline within the first 200 episodes as shown in Figure 5, this study selected 200 training episodes to conduct a critical comparative analysis of the computational efficiency between the proposed HMUDRL framework and four other baseline algorithms (MADDPG, MASAC, COMA, and MAA2C). The primary metric assessed is the duration per training episode (in seconds), which directly reflects the algorithm’s training runtime and serves as an effective proxy for inference latency during online deployment. In Figure 7, the results clearly demonstrate that over the entire span of 200 training episodes, HMUDRL, along with the three baseline algorithms MASAC, COMA, and MAA2C, showed similar durations per episode, consistently remaining within a narrow range of approximately 0.7 to 1.3 s. In contrast, MADDPG exhibited significantly higher durations, averaging around 8 to 9 s per episode, which is an order of magnitude greater.

Moreover, as shown in Table 2, HMUDRL achieves significantly higher localization accuracy and success rate compared to MASAC, COMA, and MAA2C. This superior performance, combined with its comparable computational efficiency, makes HMUDRL stand out overall. From a practical deployment perspective, this low per-episode duration directly translates into ultra-low inference latency during online operation. Given that our experiments were conducted on a standard CPU without GPU acceleration, the observed latency of approximately 1 s for HMUDRL holds strong promise for real-time applications on resource-constrained UAV platforms. Such low latency ensures that the UAV swarm can rapidly respond to dynamic changes in the electromagnetic environment, enabling timely repositioning and accurate target localization.

Analysis of internal information exchange frequency within UAV swarms, different network structures exhibit significant variations in communication overhead. For heterogeneous architectures, CM must upload AOA estimates and RSS information to CH, which perform source localization estimation based on this data and issue control or clustering instructions to the CM. Simultaneously, different CH share positions or estimation results to achieve cross-cluster cooperation. In our experimental setup, we simplify by assuming that at each time step, each CM sends state information (including AOA and RSS) once to its CH, and any pair of CH communicates once if they establish a connection. By contrast, in k-NN type homogeneous structures, all drones are considered the same type, with each drone exchanging information only with its nearest 5 neighbors due to communication range limitations. In fully connected homogeneous structures, every drone directly communicates with every other drone, forming a completely connected network.

Table 4. Average number of internal information exchanges per episode and corresponding communication savings ratio under different network structures.

Different Numbers of Clusters	Exchange Frequency (Times/Episode)		Communication Savings Ratio (%)
	Heterogeneous (CH + CM)	Homogeneous (k-NN, k = 5)	Homogeneous (Full Connected)	vs. k-NN, k = 5	vs. Full Connected
1CH + 8CM	900	4500	7200	80.0%	87.5%
3CH + 14CM	1500	8500	27,200	82.4%	94.5%
5CH + 20CM	2100	12,500	60,000	83.2%	96.5%
7CH + 30CM	3100	18,500	133,200	83.2%	97.7%

presents the average number of information exchanges per episode among UAV swarms under different architectures, along with the corresponding communication savings ratios. In the minimal configuration (1CH + 8CM), the heterogeneous architecture requires approximately 900 exchanges per episode, significantly less than the 4500 exchanges needed in the k-NN homogeneous structure and the 7200 exchanges in the fully connected homogeneous structure. This translates into communication savings of 80.0% and 87.5%, respectively. As the system scales up to the maximum configuration (7CH + 30CM), the number of exchanges in the heterogeneous architecture increases to 3100, whereas it skyrockets to 18,500 for k-NN and an overwhelming 133,200 for fully connected architectures. At this scale, the heterogeneous architecture’s communication savings ratio escalates to 83.2% compared to k-NN and 97.7% compared to fully connected structures. This trend underscores the significant advantage of heterogeneous architectures in reducing internal information exchange frequency within UAV swarms, especially when deploying large-scale UAV systems, leading to substantial reductions in communication overhead and improved overall system efficiency. Furthermore, these findings validate from a communication efficiency perspective the critical role of the proposed HMUDRL algorithm in optimizing UAV swarm collaborative localization. Specifically, in executing UHF radiation source localization tasks under complex electromagnetic environments, the HMUDRL algorithm achieves notable improvements in system-wide energy efficiency and scalability by implementing rational role division and sparse communication mechanisms, ensuring high positioning accuracy while drastically enhancing overall performance.

Figure 8 intuitively illustrates that, as the number of UAVs increases, the heterogeneous architecture requires significantly fewer information exchanges compared to homogeneous architectures. This disparity primarily stems from the more efficient communication strategy inherent in the heterogeneous design, wherein CH aggregate and broadcast information on behalf of their member CM, thereby substantially reducing redundant message transmissions across the swarm.

To quantitatively assess the contribution of each core component in the proposed HMUDRL framework, we conduct a controlled ablation study by systematically removing one module at a time while keeping all other hyperparameters and training procedures unchanged. Specifically, we evaluate three variants:

• HMUDRL w/o Hierarchical Control (HC): disables the hierarchical control mechanism, allowing agents to act solely based on local observations without high-level coordination;
• HMUDRL w/o Dynamic Clustering (DC): replaces dynamic clustering with static, pre-defined clusters, thereby removing the adaptability to source mobility;
• HMUDRL w/o AOA: excludes AOA measurements from agent observations, relying only on RSS.

Figure 9 presents the cumulative distribution functions (CDFs) of localization error for the full HMUDRL model and its ablated versions after 1000 episodes. As expected, the complete HMUDRL consistently outperforms all variants across the majority of the error spectrum, confirming the synergistic benefit of integrating hierarchical control, dynamic clustering, and RSS + AOA. Notably, the removal of AOA leads to the most significant performance degradation, particularly in the high-accuracy regime (error < 5 m), underscoring the critical role of directional information in fine-grained localization. Similarly, disabling dynamic clustering results in slower convergence and reduced robustness under source movement, as evidenced by the rightward shift of its CDF curve.

An interesting observation is that the variant w/o HC exhibits a slightly higher cumulative probability than the full model in the sub-2 m region. We attribute this to transient over-correction by the high-level controller during the final refinement phase, which can occasionally perturb otherwise accurate estimates. However, this local advantage does not translate into global superiority: beyond 10 m, the full HMUDRL demonstrates markedly better performance, achieving a 90th-percentile error of 66.86 m, compared to 97.54 m for w/o HC as shown in

Table 5. Performance evaluation of HMUDRL and ablated variants in terms of median error, 90% error, and mean absolute error (MAE).

Method	Median Error (m)	90% Error (m)	MAE (m)
HMUDRL	22.83	66.86	43.54
HMUDRL w/o HC	32.11	97.54	67.95
HMUDRL w/o DC	22.75	66.96	50.12
HMUDRL w/o AOA	574.91	833.92	559.53

. This confirms that hierarchical coordination is essential for efficient global search and long-tail error suppression.

Quantitative metrics further validate these trends, the full HMUDRL achieves the lowest median error (22.83 m), 90% error (66.86 m), and MAE (43.54 m) among all configurations. These results collectively demonstrate that each proposed component, hierarchical control, dynamic clustering, and AOA integration, provides distinct and complementary gains, and their joint design is key to the overall performance.

6.3. Comparative Analysis of Model Complexity

In electromagnetic spectrum monitoring and localization tasks, the model complexity of MARL algorithms directly impacts training efficiency, communication overhead, and deployment feasibility. The proposed HMUDRL algorithm incorporates a hierarchical architecture along with RSS variation modeling and AOA based sensing mechanisms, resulting in a distinct complexity profile compared to classical MARL approaches such as MADDPG [23], MASAC [24], COMA [25], and MAA2C [26]. In terms of parameter scale, HMUDRL adopts a heterogeneous dual-network structure: CH agents employ a standard Actor-Critic network with an input dimension of 13 and hidden layer size of 128; CM agents incorporate a gating fusion module based on a 128-dimensional multilayer perceptron (MLP), along with an action prior module that maps AOA angles softly. The action space size is set to 5, and the CH state dimension is 13, yielding approximately $P_{CH}=2\times \left(13\times 128+128\times 128+128\times 5\right)\approx \mathrm{40,000}$ parameters. The CM state dimension is 11, including gating logic, leading to about $P_{CM}\approx \mathrm{45,000}$. For a system with $N_{CH}=3$ and $N_{CM}=14$, the total parameter count is approximately 750,000.

In contrast, MADDPG and MASAC adopt a centralized training decentralized execution (CTDE) framework, where each agent maintains an individual policy network and shares a centralized Q-network. The input dimension of the centralized critic scales quadratically with the number of agents and their state-action spaces. For example, with $N=17$ and $d=13$, the critic input dimension reaches 306, and per-layer parameters can exceed ${10}^5$, leading to total parameters up to ${10}^6$ scale. COMA reduces variance via counterfactual baselines but requires the critic to evaluate all possible joint actions, resulting in computational complexity of $O\left(A^N\right)$. MAA2C is fully decentralized, each agent independently trains its own Actor and Critic, but lacks inter-agent collaboration, making it unsuitable for cooperative tasks. From a computational complexity perspective, HMUDRL decouples global coordination into CH-level macro-deployment and CM-level local sensing and motion planning. This hierarchical design avoids modeling high-dimensional joint action spaces, keeping model complexity at $O\left(N\right)$, in contrast to MADDPG’s $O\left(N^2\right)$ and COMA’s $O\left(A^N\right)$. Thus, HMUDRL demonstrates superior scalability and practicality for large-scale electromagnetic spectrum monitoring tasks.

7. Conclusions

This paper proposes HMUDRL, a distributed heterogeneous multi-agent deep reinforcement learning framework specifically designed for UHF electromagnetic radiation source localization in UAV swarm-based spectrum monitoring missions. By introducing a hierarchical architecture, HMUDRL categorizes UAVs into two distinct types of agents, CH and CM, each assigned unique roles, observation spaces, and reward mechanisms. This design enables efficient collaboration, significantly reduces communication overhead, and supports scalable decision-making. Experimental results demonstrate that HMUDRL achieves a localization success rate of 96.1% over 1000 test episodes, with an average RMSE of only 39.34 m, substantially outperforming four baseline algorithms in both accuracy and robustness. Notably, compared to these baselines, HMUDRL reduces the average localization error by approximately 87.3%, while its cluster-based information aggregation mechanism cuts inter-agent communication volume by more than 80%. These results strongly validate the practical deployment potential of HMUDRL in real-world scenarios. For instance, the system can be used in urban environments for illegal broadcast detection by rapidly localizing unauthorized UHF transmitters; it can assist regulatory authorities in identifying sources of harmful interference during spectrum enforcement; and in emergency response situations such as earthquakes or wildfires, it can locate distress signals from trapped individuals using compact UHF beacons. The current implementation is limited to 2D simulations with idealized channel models, Gaussian sensor noise assumptions, and lacks hardware-in-the-loop validation; future work will address these gaps by incorporating 3D flight dynamics, realistic RF propagation environments, and real-time UAV testbed experiments.