Role-Structured Multi-Agent Pursuit–Evasion with Potential Game Constraints for Heterogeneous Airship–UAV Systems

Highlights

What are the main findings?

• We propose a role-structured pursuit framework that couples internal potential-game coordination among pursuers with external general-sum competition against evasive targets.
• The assignment-aware centralized critic and staged reassignment curriculum improve training stability and pursuit effectiveness over representative baselines in three-dimensional multi-target simulations with dynamic-level validation.

What are the implications of the main findings?

• The framework provides a practical learning approach for heterogeneous airship–UAV collaborative pursuit under kinematic and mission constraints.
• Explicit role and assignment conditioning offers a scalable design direction for multi-agent problems that involve dynamic task grouping and cooperative–adversarial coupling.

Abstract

Cooperative pursuit–evasion with heterogeneous agents poses a training challenge that flat multi-agent reinforcement learning methods handle poorly: the pursuer team must coordinate internally while competing against adversarial targets, and the two forms of coupling require different learning signals. We present a potential-game-constrained role-structured tracking framework: a centralized training, decentralized execution algorithm for airship-guided unmanned aerial vehicle teams. It decomposes the multi-agent interaction into an internal potential game among pursuers and an external general-sum game against independently controlled targets, and pairs role-structured critics with multi-head attention over heterogeneous agent tokens and a two-stage task-assignment solver embedded as critic conditioning. The simulation results in a three-dimensional environment show that the proposed framework maintains high capture success in multi-target scenarios where standard baselines degrade substantially. A Gazebo-based visual simulation with full rigid-body dynamics confirms that the learned policy transfers to a higher-fidelity simulator after continuation training with a cascaded PID inner-loop controller.

1. Introduction

Persistent surveillance and wide-area security missions increasingly rely on mixed-composition aerial teams that combine high-endurance platforms with agile Unmanned Aerial Vehicles (UAVs). A common operational pattern assigns the endurance platform—such as an airship—to maintain broad situational awareness and coordinate task allocation, while UAVs execute local interception and terminal capture maneuvers [1,2]. Although this division of labor extends mission endurance and spatial coverage, it couples three decision layers that must remain mutually consistent: global guidance from the airship, UAV-to-target assignment, and per-UAV pursuit control. The coupling intensifies in multi-target scenarios, where target motion, workload balance, and capture windows evolve concurrently; a reassignment that appears locally reasonable can still degrade team-level pursuit coordination [3,4,5].

This paper studies such a setting: an airship-guided multi-UAV pursuer team against multiple evasive targets (Figure 1). The airship provides regional coverage guidance and assignment support; the UAVs carry out direct pursuit and capture. Relative to homogeneous swarm formulations, this role asymmetry introduces two additional training constraints. First, the airship must improve group-level geometry without overriding UAV terminal behavior. Second, assignment updates must improve responsiveness without destabilizing training through abrupt changes in local observation semantics and shaped rewards. Assignment design therefore cannot be treated as a purely external scheduling module—it is an integral part of the learning system.

Figure 1. Research scenario illustration. An airship at high altitude provides guidance and assignment support to a UAV pursuer layer operating in a three-dimensional airspace, while multiple evasive targets maneuver at lower altitudes. Dashed blue links indicate guidance/assignment information flow, solid blue arrows indicate pursuit actions, and red arrows indicate target escape motion.

We formulate the problem as a role-structured general-sum stochastic game. At the mission level, the pursuer and target teams interact adversarially; internally, the pursuer team satisfies a potential-game coordination constraint enforced through reward shaping and assignment-consistent utilities. The stochastic game view captures adversarial tracking and escape dynamics in sequential decision-making [6], while the potential-game perspective aligns decentralized pursuer improvements with team-level progress [7]. Separating these two forms of coupling, cooperative consistency among pursuers and adversarial competition between teams, makes the formulation tractable for role-asymmetric pursuit.

Multi-Agent Reinforcement Learning (MARL) under the centralized training, decentralized execution (CTDE) paradigm suits this problem because it permits shared training-time information without requiring inter-agent communication at execution times [8,9,10]. However, the present scenario aggregates several known MARL difficulties: partial observability, shifting interaction neighborhoods induced by reassignment, and cross-role asymmetry in observation and action spaces. Recent Heterogeneous-Agent Reinforcement Learning (HARL) frameworks [11] and maximum-entropy multi-type actor–critic methods [12] offer role-structured training abstractions, yet, to the best of our knowledge, they have not jointly addressed airship-guided coverage control, assignment-aware reward shaping, and curriculum-managed multi-target reassignment in one framework. Addressing these three aspects within a single learning framework requires an integrated design that tightly couples the game-theoretic formulation, the critic architecture, and the assignment schedule.

To this end, this paper presents Potential-game-constrained Role-structured Tracking with Dynamic Assignment (PRTDA), a CTDE learning framework built on the dual game formulation above. PRTDA maintains decentralized per-agent actors and role-structured centralized critics, augments critic inputs with assignment encodings, and applies staged curriculum transitions that progressively release the target policy and activate dynamic reassignment. The airship policy is trained toward coverage-guidance quality and assignment support; each UAV policy is trained for assignment-conditioned pursuit execution. The design prioritizes training stability under changing coordination structure over raw representational capacity.

The main contributions are as follows:

• We introduce a role-structured game formulation for heterogeneous airship–UAV pursuit–evasion that decomposes team coordination into an internal potential game among pursuers and an external general-sum game against evasive targets.
• We develop an assignment-aware CTDE training framework with role-structured centralized critics, assignment-conditioned critic states, and staged curriculum transitions for dynamic reassignment.
• In three-dimensional multi-target simulations with kinematic and mission constraints, we demonstrate that the proposed method yields more stable training and better pursuit performance than representative MARL baselines, and validate the learned policy in a Gazebo environment with rigid-body dynamics and cascaded PID control.

   The remainder of the paper is organized as follows: Section 2 reviews related work on pursuit–evasion theory, learning-based UAV tracking, MARL for role-asymmetric systems, and dynamic task allocation. Section 3 presents the system formulation. Section 4 details the PRTDA algorithm. Section 5 reports simulations, comparative evaluation, and limitations. Section 6 concludes the paper.

2. Related Work

We review four lines of work that define the context of this study: pursuit–evasion theory, learning-based UAV tracking, heterogeneous MARL, and dynamic task assignment.

2.1. Pursuit–Evasion and Cooperative Tracking Foundations

Stochastic game theory provides the classical framework for sequential adversarial interaction in multi-agent tracking [6]. Geometric differential-game analyses extend this foundation to multi-pursuer scenarios, characterizing capture feasibility, relative geometry, and coordinated pursuit structure under simplified kinematic assumptions [13]. Reachability-based pursuit–evasion bounds [14,15] and path-planning formulations [16] extend this geometric reasoning to bounded-speed cooperative engagements. Recent potential-game models provide coordination guarantees for multi-pursuer encirclement under simplified dynamics [17]. The interpretable quantities produced by these analyses (inter-agent distances, encirclement angles, approach rates) can serve as reward-shaping signals, safety constraints, or diagnostic metrics in learning-based systems.

For aerial tracking, model-predictive coordinated standoff methods demonstrate that bounded-speed, bounded-turn vehicles can maintain observation geometry around moving targets while respecting vehicle dynamics [18]. These methods provide strong local performance guarantees under fixed task structure and known coordination objectives. When UAV-to-target assignment may be revised online, however, the task decomposition itself becomes time-varying, and fixed-structure control designs cannot accommodate the resulting non-stationarity. A learning formulation is therefore needed that preserves the geometric structure of classical pursuit while adapting to changing grouping decisions.

2.2. Learning-Based Multi-UAV Tracking and Heterogeneous Pursuit

Reinforcement learning (RL) has been applied to multi-UAV tracking to address nonlinear motion, partial observability, and multi-target coordination. Zhou and coauthors developed deep RL methods for multi-target tracking, cooperative guidance, and joint communication–action learning in UAV swarms, demonstrating that learned policies can outperform heuristic baselines under dynamic target behavior [3,4,5]. Yue et al. proposed a factored maximum-entropy approach for cooperative multi-target tracking that exploits structured value decomposition across agents [19]. Kouzehgar et al. studied decentralized heterogeneous UAV-swarm pursuit in a multi-target MARL setting, showing that explicit heterogeneity modeling in observations, dynamics, and policy architecture yields measurable coordination gains [2]. Airship-oriented multi-agent coverage planning extends this heterogeneous paradigm to endurance platforms that coordinate with faster UAV agents [20]. Recent surveys catalog the rapid growth of UAV tracking methods and identify persistent open challenges in robustness, scalability, and communication efficiency [1]. Recent work has extended cooperative multi-UAV pursuit–evasion to three-dimensional environments with online planning and sim-to-real transfer [21], and heterogeneous multi-robot reinforcement learning methods have demonstrated that encoding agent-type differences into the policy architecture improves coordination in mixed teams [22]. A common limitation across these studies is the assumption of a fixed or slowly varying coordination structure; reassignment-induced training instability has received little direct analysis.

2.3. MARL for Role-Asymmetric Cooperative–Adversarial Systems

Reinforcement learning [23] provides the single-agent foundation; CTDE decouples training-time information access from execution-time observability and remains the dominant paradigm for cooperative and mixed cooperative–competitive MARL [8]. Non-stationarity and credit assignment are the two principal difficulties in this paradigm; counterfactual baselines and structured critics offer partial remedies [9,24]. Airship-guided pursuit intensifies both difficulties because role asymmetry demands critic architectures that can represent qualitatively different value functions for the airship, UAVs, and targets, while all agents still execute decentralized policies.

Empirical studies of strong CTDE baselines such as Multi-Agent Proximal Policy Optimization (MAPPO) reveal that training stability depends as much on implementation-level choices (learning rates, normalization, clipping) as on the nominal algorithm class [10,25]. Benchmarking studies further confirm that algorithmic differences are often smaller than implementation-level effects across cooperative tasks [26,27]. Trust-region cooperative updates [28] and game-theoretic convergence analyses [29] improve robustness in fixed-structure settings, but multi-target pursuit amplifies sensitivity because each reassignment event shifts local observation semantics and reward interpretation simultaneously [30]. Heterogeneous-agent RL frameworks such as HARL [11] and maximum-entropy heterogeneous actor–critic methods [12] supply role-structured abstractions compatible with the airship–UAV–target setting. These observations motivate combining maximum-entropy training with assignment-aware critic conditioning and curriculum scheduling when group structure changes over time.

2.4. Dynamic Task Assignment for UAV Swarms

Tracking quality in multi-target scenarios depends jointly on local pursuit control and the distribution of agents across targets. Recent reviews survey a broad spectrum of assignment mechanisms for multi-UAV systems, including optimization-based methods, auctions, and distributed heuristics, and consistently identify tension between computational optimality and real-time responsiveness [31]. SAR-oriented task-allocation reviews [32] and event-triggered dynamic task-allocation methods [33] address parts of this tension, but in airship-guided pursuit, this tension is acute: assignment updates must be frequent enough to react to target maneuvers yet stable enough to prevent oscillatory switching that degrades learning.

Scalable alternatives have been proposed to relax this tension. Two-stage greedy auctions reduce repeated assignment cost for large swarms while preserving acceptable allocation quality [34], and deep RL has been applied to task assignment to enable long-horizon sequential reasoning beyond myopic geometric heuristics [35]. PRTDA does not learn assignment end-to-end; instead, it schedules reassignment as a curriculum component within the MARL training loop. This design isolates the effect of reassignment on critic learning stability and provides a controlled baseline against which fully learned allocation policies can later be compared.

Taken together, the four research threads reviewed above supply the individual components—game-theoretic models, learned pursuit policies, role-structured MARL, and dynamic assignment mechanisms. To the best of our knowledge, prior formulations have not integrated them to jointly address heterogeneous roles, adversarial multi-target pursuit, and assignment-induced training instability in the airship–UAV setting considered here. PRTDA occupies this gap by combining potential-game-constrained pursuer coordination with role-structured CTDE training and curriculum-managed reassignment.

3. System Formulation

The operational scenario involves three agent classes at distinct altitude layers: a stratospheric airship H providing persistent wide-area surveillance from high altitudes, N UAVs $U_1,\dots ,U_N$ conducting pursuit at intermediate altitudes, and M evasive targets $T_1,\dots ,T_M$ operating at lower altitudes. The coordination problem is formulated in three-dimensional space, where each agent plans its horizontal position, heading, and altitude.

3.1. Agent Kinematics and Dynamics

We define a bounded mission volume $\Omega ={[0,L]}^2\times [z_{min},z_{max}]$ containing the three agent classes, where ${[0,L]}^2$ is the horizontal extent and $[z_{min},z_{max}]$ is the altitude band. The full agent set is

$$\mathcal{N}=\left\{H\right\}\cup \mathcal{U}\cup \mathcal{T},$$

(1)

where $\mathcal{U}=\{U_1,\dots ,U_N\}$ and $\mathcal{T}=\{T_1,\dots ,T_M\}$. The cooperative pursuer coalition is $\mathcal{P}=\left\{H\right\}\cup \mathcal{U}$; the adversarial side consists of $\mathcal{T}$. Each agent class operates within a role-specific altitude band: the airship is held at a fixed high altitude $z_H$; UAVs maneuver within $[z_U^{min},z_U^{max}]$; targets operate within $[0,z_T^{max}]$.

Each agent $i\in \mathcal{N}$ has state $(p_i,{\psi }_i,{\gamma }_i,v_i)$: position $p_i={[x_i,y_i,z_i]}^{\top }\in \Omega$, yaw angle ${\psi }_i\in [0,2\pi )$ measured clockwise from geographic north, flight-path angle ${\gamma }_i\in [-{\gamma }_{max},{\gamma }_{max}]$ controlling the climb/dive rate, and scalar speed $v_i\ge 0$. The horizontal heading unit vector corresponding to yaw $\psi$ is

$$\widehat{d}\left(\psi \right)={[sin\psi ,cos\psi ]}^{\top }.$$

(2)

All agents share a normalized action interface. Each policy outputs a three-dimensional command $a_i={[{\tilde{v}}_i,{\tilde{\omega }}_i,{\tilde{\omega }}_{\gamma ,i}]}^{\top }\in {[-1,1]}^3$, which the environment maps to physical controls:

$$v_i={\displaystyle \frac{{\tilde{v}}_i+1}{2}}{v}_i^{max},{\omega }_i={\tilde{\omega }}_i{\omega }_i^{max},{\omega }_{\gamma ,i}={\tilde{\omega }}_{\gamma ,i}{\omega }_{\gamma ,i}^{max},$$

(3)

where $(v_i^{max},{\omega }_i^{max},{\omega }_{\gamma ,i}^{max})$ are role-dependent speed, yaw rate, and pitch rate limits. With discrete time step $\Delta t$, the kinematic update proceeds in three stages. The yaw and flight-path angles are updated first:

$${\psi }_{i,t+1}=\left({\psi }_{i,t}+\Delta t{\omega }_{i,t}\right)mod2\pi ,$$

(4)

$${\gamma }_{i,t+1}=\mathrm{clip}\left({\gamma }_{i,t}+\Delta t{\omega }_{\gamma ,i,t},-{\gamma }_{max},{\gamma }_{max}\right).$$

(5)

The flight-path angle decomposes the total speed into horizontal and vertical components, $v_{h,i}=v_{i}cos{\gamma }_i$ and $v_{z,i}=v_{i}sin{\gamma }_i$, coupling climb authority with horizontal pursuit speed. The position update is then

$$\begin{array}{ccc}\hfill {[x_i,y_i]}_{t+1}^{\top }& =& {\mathrm{clip}}_{XY}\left({[x_i,y_i]}_t^{\top }+\Delta t{v}_{h,i,t}\widehat{d}\left({\psi }_{i,t+1}\right)\right),\hfill \\ \hfill z_{i,t+1}& =& {\mathrm{clip}}_Z\left(z_{i,t}+\Delta t{v}_{z,i,t}\right),\hfill \end{array}$$

(6)

where ${\mathrm{clip}}_{XY}$ and ${\mathrm{clip}}_Z$ enforce horizontal boundary containment and role-specific altitude limits, respectively. The position update uses the already-updated yaw ${\psi }_{i,t+1}$ and flight-path angle ${\gamma }_{i,t+1}$, so all angular changes and the resulting displacement occur within the same time step. The speed–yaw rate–pitch rate parameterization enforces turn-rate and climb-rate consistency and prevents unrealistic heading jumps that would arise from direct velocity commands.

3.2. Assignment

UAV-to-target task allocation is defined by a mapping

$$\rho :\{1,\dots ,N\}\to \{1,\dots ,M\},$$

(7)

where $\rho \left(i\right)$ denotes the target index assigned to UAV $U_i$. When $N\ge M$, we require $\rho$ to be surjective so that every target receives at least one assigned UAV. This mapping partitions $\mathcal{U}$ into M disjoint groups

$${\mathcal{U}}_m\left(\rho \right)=\{U_i\in \mathcal{U}:\rho \left(i\right)=m\},m=1,\dots ,M,$$

(8)

each responsible for the cooperative capture of target $T_m$. The assignment may be held fixed throughout an episode or updated periodically based on the evolving geometry; the solver and curriculum schedule that govern these transitions are described in Section 4.

From a game-theoretic perspective, $\rho$ plays a structural role beyond combinatorial scheduling. It determines which sub-problem each UAV participates in, which peers appear in its local observation, and which potential terms enter its shaped reward. A reassignment event therefore simultaneously alters three quantities for every affected UAV: the observation content (new assigned-target geometry), the reward signal (recomputed counterfactual potential), and the group composition (different teammates in ${\mathcal{U}}_m$). This triple coupling is the root cause of the training instability observed in multi-target pursuit: the learning problem becomes non-stationary from the perspective of any agent whose group membership changes.

3.3. Capture Condition

Target $T_m$ is declared captured at step t when every UAV in its assigned group satisfies the three-dimensional proximity criterion simultaneously:

$$T_m\mathrm{captured}\mathrm{at}t⟺\forall U_i\in {\mathcal{U}}_m\left(\rho \right):{\parallel p_{U_i,t}-p_{T_m,t}\parallel }_3\le r_c,$$

(9)

where ${\parallel ·\parallel }_3$ denotes the Euclidean distance in three-dimensional space and $r_c$ is the capture radius. Only UAVs participate in capture; the airship contributes to coverage guidance and formation quality but is not required to enter the capture neighborhood. An episode terminates successfully when all M targets are captured or when it is truncated at a fixed time horizon $T_{max}$ otherwise.

3.4. Dual Game Structure: Internal Potential Game and External General-Sum Competition

The multi-agent interaction is modeled as a partially observed Markov game (POMG) [6,29]. A POMG extends a Markov decision process (MDP) [23] to multiple agents with private observations: each agent selects actions based solely on its local observation history, the environment transitions according to the joint action, and each agent receives its own reward signal. Formally, the POMG is defined by the tuple

$$\mathcal{G}=⟨\mathcal{N},\mathcal{S},{\left\{{\mathcal{O}}_i\right\}}_{i\in \mathcal{N}},{\left\{{\mathcal{A}}_i\right\}}_{i\in \mathcal{N}},P,{\left\{r_i\right\}}_{i\in \mathcal{N}},\gamma ⟩,$$

(10)

where $\mathcal{N}$ is the set of agents defined in (1); $\mathcal{S}$ is the global state space containing the three-dimensional positions, headings, flight-path angles, and speeds of all agents together with the current assignment $\rho$; ${\mathcal{O}}_i$ is the local observation space of agent i, a partial view of $\mathcal{S}$ restricted to local geometry and assigned-group information (detailed in Section 4.1); ${\mathcal{A}}_i={[-1,1]}^3$ is the continuous action space (normalized speed, yaw rate, and pitch rate); $P:\mathcal{S}\times {\prod }_i{\mathcal{A}}_i\to \Delta \left(\mathcal{S}\right)$ is the stochastic transition kernel defined by the kinematic update (4)–(6); $r_i:\mathcal{S}\times {\prod }_i{\mathcal{A}}_i\to \mathbb{R}$ is the scalar reward function; and $\gamma \in (0,1)$ is the discount factor. Under the centralized training, decentralized execution (CTDE) paradigm, agents execute factorized policies ${\pi }_i(a_i\mid o_i)$

$$\pi (\mathbf{a}\mid \mathbf{o})=\prod _{i\in \mathcal{N}}{\pi }_i(a_i\mid o_i),$$

(11)

while critics at training time have access to the full state $s\in \mathcal{S}$.

This game has a two-level structure that we exploit for tractable learning.

3.4.1. External Level: General-Sum Team Competition

The pursuer coalition $\mathcal{P}$ and target set $\mathcal{T}$ pursue opposing objectives. Each side seeks to maximize its own team-level discounted return:

$$J^{\mathcal{P}}({\pi }_{\mathcal{P}},{\pi }_{\mathcal{T}})={\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{T_{max}}}{\gamma }^t{\displaystyle \sum _{i\in \mathcal{P}}}{r}_i^{\mathcal{P}}(s_t,{\mathbf{a}}_t)\right],$$

(12)

$$J^{\mathcal{T}}({\pi }_{\mathcal{P}},{\pi }_{\mathcal{T}})={\mathbb{E}}_{\pi }\left[{\displaystyle \sum _{t=0}^{T_{max}}}{\gamma }^t{\displaystyle \sum _{m=1}^M}{r}_m^{\mathcal{T}}(s_t,{\mathbf{a}}_t)\right],$$

(13)

where ${\pi }_{\mathcal{P}}={\left\{{\pi }_i\right\}}_{i\in \mathcal{P}}$ and ${\pi }_{\mathcal{T}}={\left\{{\pi }_m\right\}}_{m=1}^M$ denote the policy sets of the two teams. Training optimizes policies toward a Markov–Nash solution concept in which neither team can unilaterally improve:

$${\pi }_{\mathcal{P}}^{*}\in arg\underset{{\pi }_{\mathcal{P}}}{max}{J}^{\mathcal{P}}({\pi }_{\mathcal{P}},{\pi }_{\mathcal{T}}^{*}),{\pi }_{\mathcal{T}}^{*}\in arg\underset{{\pi }_{\mathcal{T}}}{max}{J}^{\mathcal{T}}({\pi }_{\mathcal{P}}^{*},{\pi }_{\mathcal{T}}).$$

(14)

The interaction is general-sum rather than zero-sum because $r_i^{\mathcal{P}}$ includes potential-based shaping and a shared team reward term that have no counterpart in $r_m^{\mathcal{T}}$, so $J^{\mathcal{P}}+J^{\mathcal{T}}\ne \mathrm{const}$ across policy profiles [6]. A zero-sum formulation would preclude these shaped cooperative signals, forcing all pursuer credit to come through a single adversarial channel and slowing down coordinated group closure.

3.4.2. Internal Level: Potential-Game Pursuer Coordination

Within $\mathcal{P}$, we impose a cooperative structure through a team potential function $\Phi (s;\rho )$ that aggregates formation-quality metrics over all target groups (the specific terms comprising $\Phi$ are defined in Section 4). Each pursuer $i\in \mathcal{P}$ receives a shaped credit signal computed as a one-step difference reward:

$$r_i^{\Delta }=\left(\Phi (s_{t+1};\rho )-\Phi (s_t;\rho )\right)-\left({\Phi }^{-i}(s_{t+1};\rho )-{\Phi }^{-i}(s_t;\rho )\right),$$

(15)

where ${\Phi }^{-i}$ denotes the potential recomputed after removing agent i’s contribution. The first parenthesized term measures the team-wide potential progress; the second measures the potential progress that would have occurred without agent i. Their difference isolates agent i’s marginal contribution to team improvement.

This construction satisfies the potential-game alignment property [7]. When agent i unilaterally changes its action from $a_i$ to $a_i^{\prime }$ while all other agents hold their actions fixed, the resulting change in $r_i^{\Delta }$ equals the change in $\Phi$:

$$r_i^{\Delta }(a_i,a_{-i})-r_i^{\Delta }(a_i^{\prime },a_{-i})=\Phi \left(s_{t+1}\left(a_i\right)\right)-\Phi \left(s_{t+1}\left(a_i^{\prime }\right)\right).$$

(16)

The equality holds because ${\Phi }^{-i}$ excludes agent i from its computation, so it is invariant to $a_i$; similarly, $\Phi \left(s_t\right)$ and ${\Phi }^{-i}\left(s_t\right)$ depend only on the current state and cancel when differencing. The alignment guarantee means that any unilateral improvement by a pursuer also improves the team potential. This reduces incentive misalignment: a pursuer cannot improve its own shaped reward by directly degrading team-level formation quality under the fixed-assignment condition.

The total pursuer reward combines the potential-aligned credit with a shared team term:

$$r_i^{\mathcal{P}}=r_{\mathrm{team}}+{\lambda }_{\Delta }{r}_i^{\Delta },i\in \mathcal{P},$$

(17)

where $r_{\mathrm{team}}$ captures global mission events (synchronization progress and terminal capture bonuses) and ${\lambda }_{\Delta }$ scales the marginal credit signal. The shared term aligns all pursuers toward mission-level success; the difference term provides the agent-specific gradient information needed to resolve credit assignment within each target group ${\mathcal{U}}_m$.

For target $T_m\in \mathcal{T}$, the reward encourages evasion and penalizes terminal failure:

$$r_m^{\mathcal{T}}={\displaystyle \frac{d_m^{\mathrm{near}}}{d_{max}}}-c_{\mathrm{cap}}{⊮}_{\mathrm{cap},m}-c_{\mathrm{oob}}{⊮}_{\mathrm{oob},m},$$

(18)

where $d_m^{\mathrm{near}}$ is the distance from $T_m$ to its nearest pursuer, $d_{max}$ normalizes to $[0,1]$, ${⊮}_{\mathrm{cap},m}$ indicates capture, and ${⊮}_{\mathrm{oob},m}$ penalizes boundary violations.

The dual game structure separates two learning pressures that would otherwise conflict. Externally, general-sum competition drives the pursuer team to improve against adaptive learning targets that actively probe coordination weaknesses. Internally, the potential-game constraint channels individual pursuer improvements through the team potential, so that per-agent policy updates remain aligned with group-level formation objectives across all M target groups.

4. The PRTDA Algorithm

The dual game formulation in Section 3 specifies what each agent should optimize but does not prescribe how to instantiate the optimization. Translating the formulation into a working learning system requires decisions at three levels: the observation encoding must accommodate variable fleet sizes under a single shared-backbone network; the critic must fuse heterogeneous information from the airship, the assigned target, and peer UAVs without breaking the potential-game alignment established by the reward design; and the training loop must schedule assignment updates and curriculum transitions so that the shaped reward signal remains informative as group composition evolves.

This section describes how PRTDA resolves each of these concerns through a modular pipeline. The presentation is organized in six parts: observation and action spaces, target assignment, actor–critic architecture, the training pipeline with pseudocode, computational cost, and convergence.

4.1. Observation and Action Space Design

All observations are normalized by the mission-volume diagonal $d_{max}=\sqrt{2L^2+{(z_{max}-z_{min})}^2}$ or the relevant role-specific maximum, and angular quantity use $(sin,cos)$ pairs. Table 1 summarizes the layout for each role.

Table 1. Observation space structure per agent role (M targets, N UAVs, $K=N-1$ peer slots).

Role	Component	Dim	Content
H	Self position	2	$[x_H,y_H]/L$ (horizontal; altitude fixed)
	Self kinematics	3	speed, $sin{\psi }_H$, $cos{\psi }_H$
	Coverage anchor	6	$G_{\mathrm{cov}}$ position (2), distance (1), unit vector (2), speed diff (1)
$U_i$	Self base	14	Position (3D), speed, heading ($sin\psi$, $cos\psi$), flight-path angle ($sin\gamma$, $cos\gamma$), target 3D dist, horiz/vert aspect
	Peer slots	$12K$	Per-peer: 3D dist to target, speed, heading, flight-path angle, 3D inter-peer geometry
	Peer mask	K	Validity indicator per slot
	Airship relative	3	Unit vector and distance to H
	Group rank	1	Normalized proximity rank in ${\mathcal{U}}_{\rho \left(i\right)}$
$T_m$	Self state	10	Position (3D), speed, heading (sin/cos), flight-path angle (sin/cos)
	Nearest threat	2	3D distance and aspect (masked beyond $d_{\mathrm{sense}}$)

The UAV observation warrants further detail. Each $o_{U_i}$ is organized around the assigned target $T_{\rho \left(i\right)}$: base features encode $U_i$’s own state relative to $T_{\rho \left(i\right)}$, peer slots carry features of same-group teammates ${\mathcal{U}}_{\rho \left(i\right)}\setminus \left\{U_i\right\}$ sorted by proximity, and the peer mask flags which slots are occupied. Because the peer slots are permutation-equivariant and exclude agents from other target groups, the observation naturally handles variable-size groups and restricts each UAV to its local tactical context. The action space is shared across all roles: $a_i={[{\tilde{v}}_i,{\tilde{\omega }}_i,{\tilde{\omega }}_{\gamma ,i}]}^{\top }\in {[-1,1]}^3$, mapped to physical speed, yaw rate, and pitch rate via (3).

Centralized critics receive a structured state instead of a flat concatenation. For query agent $U_i$, this state comprises H’s observation paired with its action, $T_{\rho \left(i\right)}$’s observation paired with its action, $U_i$’s base features with its own action, peer slot features with peer actions, the peer mask, and a two-dimensional episode summary encoding capture progress. Grouping inputs by role lets each sub-encoder specialize, whereas a monolithic state vector would force the critic to discover role boundaries from raw dimensions.

4.2. Two-Stage Assignment with Curriculum Scheduling

Given the current three-dimensional positions and headings, the assignment solver computes $\rho$ by minimizing a composite cost. The cost of assigning UAV $U_i$ to target $T_m$ is

$$C_{im}=w_d{\displaystyle \frac{\parallel p_{U_i}-p_{T_m}\parallel }{d_{max}}}+w_h{\displaystyle \frac{|\Delta {\psi }_{im}|}{\pi }}+w_s⊮[{\rho }_{\mathrm{prev}}\left(i\right)\ne m],$$

(19)

where $\Delta {\psi }_{im}$ is the heading error between $U_i$ and the bearing toward $T_m$, $(w_d,w_h,w_s)$ weight distance, heading alignment, and switching cost, and ${\rho }_{\mathrm{prev}}$ is the previous assignment. Including the switching penalty suppresses oscillatory reassignment that would otherwise disrupt shaped rewards and observation semantics.

The solver operates in two stages.

Stage A guarantees coverage: it solves a minimum-cost injective matching from targets to UAVs via the Hungarian algorithm on $C^{\top }$, assigning exactly one UAV per target. When $N<M$ (fewer UAVs than targets), a greedy distinct-target fallback is used.

Stage B assigns the remaining $N-M$ UAVs greedily, selecting for each the target minimizing $C_{im}+w_l{n}_m$, where $n_m$ is the current load on $T_m$ and $w_l$ is a load-balancing penalty. A candidate assignment ${\rho }^{\prime }$ replaces $\rho$ only when the total cost improvement exceeds a gain threshold ${\delta }_g$:

$${\displaystyle {\sum }_{i=1}^N}{C}_{i,\rho \left(i\right)}-{\sum }_{i=1}^N{C}_{i,{\rho }^{\prime }\left(i\right)}>{\delta }_g.$$

(20)

The assignment is managed through staged curriculum transitions to prevent training instability. In the first phase, $\rho$ is fixed to a predetermined allocation (e.g., $[0,0,0,1,1]$ for 5 UAVs and 2 targets), giving each UAV a stable observation context and consistent reward signal. After the release step $t_{\rho }^{\mathrm{release}}$, the solver is activated and $\rho$ is updated periodically. In preliminary experiments, training all agents jointly from the start with a reduced learning rate led to unstable critic estimates, because the rapidly evolving target policy produced high-variance TD targets. Reducing the learning rate further slowed pursuer convergence without resolving the instability. The two-phase curriculum avoids these issues by presenting a stationary reward landscape during the initial phase. The release step $t_{\rho }^{\mathrm{release}}=\mathrm{100,000}$ was selected by monitoring training curves: at this point, the capture success rate under fixed assignment plateaus, indicating that the pursuit policies are mature enough to tolerate adversarial target training. Concurrently, the target policy follows a separate curriculum: targets initially execute a deterministic circular trajectory, providing predictable geometry for the pursuers to learn basic approach and closure behavior. The target policy is released at step $t_T^{\mathrm{release}}$, after which each target is driven by an independent actor network trained with the evasion reward (18).

4.3. Role-Structured Actor–Critic Architecture

All agents follow the CTDE paradigm: actors condition only on local observations, while critics access a structured global state during training. We adopt a shared-backbone, count-specific-head design that maintains one set of feature extractors across all UAV counts $k\in \{1,\dots ,N_{max}\}$ but routes final outputs through a head selected by the runtime count $k=\left|\mathcal{U}\right|$. Representation learning (shared backbone) is thus decoupled from count-dependent action calibration (per-head), eliminating the need to retrain feature extractors when the fleet size changes.

4.3.1. UAV Actor

The UAV actor ${\pi }_U$ processes the observation $o_{U_i}$ defined in Table 1 through three stages: self encoding, peer encoding, and trunk evaluation. The self encoder is a single linear layer $f_{\mathrm{self}}:{\mathbb{R}}^{14}\to {\mathbb{R}}^{d_e}$ applied to the base features $u_{\mathrm{base}}$. For the peer branch, each occupied peer slot $j\in \{1,\dots ,K\}$ is independently encoded by a shared multi-layer perceptron (MLP) $f_{\mathrm{slot}}:{\mathbb{R}}^{12}\to {\mathbb{R}}^{d_e}$. The encoded peer tokens are aggregated through masked mean–max pooling:

$$e_{\mathrm{peer}}=g_{\mathrm{pool}}\left({\displaystyle \frac{1}{\left|\mathcal{V}\right|}}{\displaystyle \sum _{j\in \mathcal{V}}}{f}_{\mathrm{slot}}\left(x_j\right),\underset{j\in \mathcal{V}}{max}{f}_{\mathrm{slot}}\left(x_j\right)\right),$$

(21)

where $\mathcal{V}\subseteq \{1,\dots ,K\}$ is the set of valid peer slots (indicated by the peer mask), $x_j$ is the j-th peer feature vector, and $g_{\mathrm{pool}}:{\mathbb{R}}^{2d_e}\to {\mathbb{R}}^{d_e}$ is a linear projection. Mean pooling captures average group geometry; max pooling preserves extreme-case information such as the most distant peer. A trunk MLP $f_{\mathrm{trunk}}:{\mathbb{R}}^{2d_e}\to {\mathbb{R}}^{d_h}$ processes the concatenation $[e_{\mathrm{self}};e_{\mathrm{peer}}]$ to produce a latent representation $h\in {\mathbb{R}}^{d_h}$. Count-specific heads ${\mu }_k,{\sigma }_k:{\mathbb{R}}^{d_h}\to {\mathbb{R}}^3$ then output Gaussian parameters for the selected count k:

$$a_{U_i}=tanh\left({\mu }_k\left(h\right)+{\sigma }_k\left(h\right)\odot \epsilon \right),\epsilon \sim \mathcal{N}(0,I),$$

(22)

and the log-probability follows the standard tanh-squashed correction [36]. During training, all UAVs share a single actor instance.

4.3.2. Airship and Target Actors

They are plain two-layer MLPs that map their respective observations to Gaussian parameters through the same count-head mechanism as the UAV actor. Because these roles have a single instance per team, count conditioning is relevant only when the critic’s value landscape shifts with k.

4.3.3. UAV Critic

The U-role critic $Q_U$ operates on the structured state described in Section 4.1, where each information source is processed by a dedicated branch encoder:

$$\begin{array}{cc}\hfill z_H& =f_H\left([o_H;a_H]\right)\in {\mathbb{R}}^{d_H},\hfill \\ \hfill z_T& =f_T\left([o_{T_{\rho \left(i\right)}};a_{T_{\rho \left(i\right)}}]\right)\in {\mathbb{R}}^{d_T},\hfill \\ \hfill z_U& =f_U\left([z_{U,\mathrm{self}};z_{U,\mathrm{peer}}]\right)\in {\mathbb{R}}^{d_U},\hfill \end{array}$$

(23)

where $z_{U,\mathrm{self}}$ encodes the query UAV’s base features and action, and $z_{U,\mathrm{peer}}$ aggregates peer slot features (with their actions) through the same masked mean–max pooling as the actor. Token projections map each branch output to a common fusion dimension $d_f$:

$${\widehat{z}}_R=W_R{z}_R\in {\mathbb{R}}^{d_f},R\in \{H,U,T\}.$$

(24)

A multi-head attention (MHA) block with $n_{\mathrm{head}}$ heads fuses the three tokens:

$$[{\widehat{z}}_H^{\prime };{\widehat{z}}_U^{\prime };{\widehat{z}}_T^{\prime }]=\mathrm{MHA}\left([{\widehat{z}}_H;{\widehat{z}}_U;{\widehat{z}}_T]\right),$$

(25)

followed by layer normalization and a feedforward residual. The fused U-token ${\widehat{z}}_U^{\prime }$ is extracted as a query-centric readout and concatenated with a two-dimensional episode summary $[\mathrm{progress},1-\mathrm{progress}]$ that tracks capture advancement. A count-specific Q-head $Q_k:{\mathbb{R}}^{d_f+2}\to \mathbb{R}$ produces the scalar value estimate, and twin networks $Q_{{\theta }_1},Q_{{\theta }_2}$ with clipped double-Q mitigate overestimation bias. Figure 2 illustrates the full architecture.

Figure 2. Network architecture of PRTDA. (Left): The UAV actor encodes self features and variable-length peer slots via masked mean–max pooling, and then routes through a shared trunk to count-specific output heads selected by $k=\left|\mathcal{U}\right|$. (Right): The UAV critic uses separate branch encoders for H, U, and T inputs, fuses them via multi-head attention, and extracts the query-centric U-token for count-specific Q-heads. Solid arrows show the main data flow; dashed arrows indicate mask or head-selection signals.

4.3.4. Airship and Target Critics

They follow the same token-fusion structure but treat all N UAV slots as a single peer set (no self/peer split), since these roles observe the entire UAV fleet rather than a single target group.

4.3.5. Multi-Target Zero-Shot Transfer

Training uses only single-target scenarios ($M=1$) with varying fleet sizes $k\in \{2,\dots ,N_{max}\}$, where each count k trains its corresponding count-specific head. At evaluation time, multi-target scenarios ($M>1$) are handled without retraining through a group-isolation mechanism. The assignment solver partitions the N UAVs into M groups $\{{\mathcal{U}}_1,\dots ,{\mathcal{U}}_M\}$ with sizes $k_1,\dots ,k_M$. Each group’s observation is then reconstructed to include only same-group peers and the assigned target, with inter-group information masked out. The actor selects the count-specific head corresponding to $k_m=\left|{\mathcal{U}}_m\right|$ for each group m, so each sub-team executes the same single-target pursuit policy learned during training. Because the masked mean–max pooling (21) naturally handles variable-length peer sets, no architectural change or zero-padding is required for different group sizes. This design enables zero-shot transfer from single-target training to multi-target evaluation: the 7-UAV-vs-3-target scenario is decomposed into three isolated sub-problems (e.g., groups of sizes 3, 2, 2), each using the pre-trained head for its respective count.

4.4. Training Pipeline

Each training iteration alternates between environment interaction and parameter updates. We first define the team potential that produces the shaped rewards and then describe the soft actor–critic (SAC)-based update rules.

4.4.1. Team Potential Function

The potential $\Phi (s;\rho )$, referenced in (15), aggregates five terms over all target groups:

$$\Phi (s;\rho )=w_{\mathrm{cov}}{\varphi }_{\mathrm{cov}}+w_{\mathrm{app}}{\varphi }_{\mathrm{app}}+w_{\mathrm{ring}}{\varphi }_{\mathrm{ring}}+w_{\mathrm{enc}}{\varphi }_{\mathrm{enc}}-{\lambda }_{\mathrm{safe}}{\varphi }_{\mathrm{safe}}.$$

(26)

Each component targets a distinct coordination objective:

• Coverage (${\varphi }_{\mathrm{cov}}$) drives the airship toward a smoothed coverage anchor $G_{\mathrm{cov}}$. The raw anchor blends three positional cues via normalized weights $({\lambda }_T,{\lambda }_U,{\lambda }_F)$:

  $$G_{\mathrm{cov}}^{\mathrm{raw}}={\lambda }_T{\overline{p}}_T+{\lambda }_U{\overline{p}}_U+{\lambda }_F{\tilde{p}}_F,{\tilde{p}}_F={\displaystyle \sum _{i=1}^N}{w}_i{p}_{U_i},w_i={\displaystyle \frac{exp\left({\beta }_F\parallel p_{U_i}-{\overline{p}}_T\parallel /d_{max}\right)}{{\sum }_{j}exp\left({\beta }_F\parallel p_{U_j}-{\overline{p}}_T\parallel /d_{max}\right)}},$$

  (27)

  where ${\overline{p}}_T$ is the target centroid, ${\overline{p}}_U$ is the active-UAV centroid, and ${\tilde{p}}_F$ is a softmax-weighted position biased toward UAVs far from targets (controlled by temperature ${\beta }_F$). An exponential moving average with rate $\eta$ suppresses oscillation: $G_{\mathrm{cov},t}=(1-\eta )G_{\mathrm{cov},t-1}+\eta G_{\mathrm{cov}}^{\mathrm{raw}}$. The coverage potential is then

  $${\varphi }_{\mathrm{cov}}=-{\displaystyle \frac{\parallel p_H-G_{\mathrm{cov}}\parallel }{d_{\mathrm{step}}}},$$

  (28)

  where $d_{\mathrm{step}}=max(v_U^{max},v_H^{max})\Delta t$ is normalized by the maximum single-step displacement.
• Approach (${\varphi }_{\mathrm{app}}$) measures aggregate UAV-to-target distance, normalized by team size to prevent the reward scale from growing with N:

  $${\varphi }_{\mathrm{app}}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\displaystyle \frac{\parallel p_{U_i}-p_{T_{\rho \left(i\right)}}\parallel }{d_{\mathrm{step}}}}.$$

  (29)
• Capture ring (${\varphi }_{\mathrm{ring}}$) applies a log-softplus barrier near the capture radius $r_c$ with temperature $\tau$, providing a smooth gradient transition as UAVs cross the capture boundary:

  $${\varphi }_{\mathrm{ring}}=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}log\left(1+\mathrm{softplus}\left(\frac{d_i-r_c}{\tau }\right)\right),$$

  (30)

  where $d_i=\parallel p_{U_i}-p_{T_{\rho \left(i\right)}}\parallel$. When $d_i\ll r_c$, this term is close to zero; as $d_i$ increases beyond $r_c$, the penalty magnitude increases smoothly, supplying a persistent yet non-dominant far-field gradient that complements the linear ${\varphi }_{\mathrm{app}}$.
• Encirclement (${\varphi }_{\mathrm{enc}}$) penalizes angular co-alignment of UAV-to-target bearing vectors within each group, encouraging spatial dispersion:

  $${\varphi }_{\mathrm{enc}}=-{\displaystyle \frac{1}{M}}{\displaystyle \sum _{m=1}^M}\parallel {\displaystyle \sum _{i\in {\mathcal{U}}_m}}{e}^{-{\beta }_d{d}_i/d_{max}}{\widehat{u}}_i{\parallel }^2,$$

  (31)

  where ${\widehat{u}}_i$ is the unit vector from $U_i$ toward $T_{\rho \left(i\right)}$, and the exponential weighting ${\beta }_d$ reduces contributions from distant UAVs that have not yet closed in.
• Safety (${\varphi }_{\mathrm{safe}}$) penalizes boundary violations and inter-pursuer collisions:

  $${\varphi }_{\mathrm{safe}}={\displaystyle \sum _{i\in \mathcal{P}}}max\left(0,\frac{{\delta }_s-d_i^{\mathrm{bdr}}}{{\delta }_s}\right)+{\displaystyle \sum _{i<j}}max\left(0,\frac{d_{\mathrm{col}}-d_{ij}}{d_{\mathrm{col}}}\right),$$

  (32)

  where $d_i^{\mathrm{bdr}}$ is pursuer i’s minimum distance to any world boundary, ${\delta }_s$ is the safety margin, $d_{ij}$ is the inter-pursuer distance, and $d_{\mathrm{col}}$ is the collision threshold.

4.4.2. Weight Selection

Each component of $\Phi$ is designed with built-in normalization: ${\varphi }_{\mathrm{cov}}$ is divided by the maximum single-step displacement $d_{\mathrm{step}}$, ${\varphi }_{\mathrm{app}}$ and ${\varphi }_{\mathrm{ring}}$ are averaged over the UAV count N, and ${\varphi }_{\mathrm{enc}}$ is averaged over the target count M. The weights, therefore, reflect task priority rather than unit differences. We set $w_{\mathrm{ring}}$ as the largest because entering the capture region is the direct success criterion, $w_{\mathrm{app}}$ and $w_{\mathrm{cov}}$ at moderate levels to guide approach and coverage at a longer range, and $w_{\mathrm{enc}}$ small because angular dispersion plays an auxiliary role. These values were selected from a small set of pilot runs and then held fixed for all fleet sizes and all compared algorithms; no per-method tuning was performed.

The team reward $r_{\mathrm{team}}$ in (17) combines a synchronization progress bonus that penalizes lagging UAVs, a per-step time penalty that incentivizes faster capture, and a terminal capture bonus triggered when every group simultaneously satisfies the capture condition.

4.4.3. Soft Update

Each role maintains twin Q-networks $Q_{{\theta }_1},Q_{{\theta }_2}$ with target copies $Q_{{\overline{\theta }}_1},Q_{{\overline{\theta }}_2}$. Given a transition $(s,\mathbf{a},\mathbf{r},s^{\prime })$ sampled from the replay buffer, the temporal-difference (TD) target for agent i is

$$y_i=r_i+\gamma \left(\underset{j\in \{1,2\}}{min}{Q}_{{\overline{\theta }}_j}(s^{\prime },{\mathbf{a}}^{\prime })-\alpha log{\pi }_i(a_i^{\prime }\mid o_i^{\prime })\right),a_i^{\prime }\sim {\pi }_i(·\mid o_i^{\prime }),$$

(33)

where ${\mathbf{a}}^{\prime }$ denotes the joint next action assembled under the role-wise update rule, with $a_i^{\prime }$ resampled for the queried agent and other components set by the corresponding policy branch. The target is clipped to $[-y_{max},y_{max}]$ to prevent divergent bootstrapping. The critic loss is the mean-squared Bellman error:

$${\mathcal{L}}_Q\left({\theta }_j\right)=\mathbb{E}\left[{\left(Q_{{\theta }_j}(s,\mathbf{a})-y_i\right)}^2\right],j\in \{1,2\}.$$

(34)

The actor maximizes the soft value:

$${\mathcal{L}}_{\pi }\left(\varphi \right)=\mathbb{E}\left[\alpha log{\pi }_{\varphi }(a_i\mid o_i)-\underset{j}{min}{Q}_{{\theta }_j}(s,\mathbf{a})\right],$$

(35)

where only the queried agent’s action is resampled from the current policy.

4.4.4. Batch Aggregated U-Actor Update

Because all UAVs share one actor, gradients from all k active UAVs are accumulated in a single backward pass instead of k sequential ones. Observations ${\left\{o_{U_i}\right\}}_{i=1}^k$ are stacked along the agent dimension, the actor loss is evaluated in one forward call, and a single optimizer step is applied. This batching improves update efficiency by avoiding k separate optimizer steps, although total compute still grows with batch-agent tensor size.

4.4.5. Entropy Temperature

A separate log-temperature $log{\alpha }_k$ is maintained for each active count k, trained to satisfy $\mathbb{E}[-log\pi (a\mid o)]\ge \overline{\mathcal{H}}$, where $\overline{\mathcal{H}}$ is the target entropy:

$${\mathcal{L}}_{\alpha }\left(k\right)=-{\alpha }_k\mathbb{E}\left[log\pi (a\mid o)+\overline{\mathcal{H}}\right],{\alpha }_k=max\left(e^{log{\alpha }_k},{\alpha }_{min}\right).$$

(36)

Maintaining separate temperatures lets the exploration rate adapt to each fleet size independently; an $\alpha$ calibrated for a large team would otherwise suppress exploration when only a few UAVs are active.

Target networks are updated via Polyak averaging $\overline{\theta }\leftarrow (1-{\tau }_{\mathrm{soft}})\overline{\theta }+{\tau }_{\mathrm{soft}}\theta$ after each gradient step. The full training procedure is summarized in Algorithm 1.

Algorithm 1 PRTDA Training

Require:  Parallel environments ${\left\{E_j\right\}}_{j=1}^P$, replay buffer $\mathcal{B}$, curriculum thresholds $t_{\rho }^{\mathrm{release}},t_T^{\mathrm{release}}$   1: Initialize actors ${\pi }_H,{\pi }_U,{\pi }_T$ and twin critics $Q_H,Q_U,Q_T$ with target copies   2: Initialize per-count temperatures ${\left\{{\alpha }_k\right\}}_{k=1}^{N_{max}}$   3: for $t=1,2,\dots ,t_{\mathrm{total}}$do   4:       // Environment interaction (parallel)   5:       for each env j in parallel do   6:              Collect actions ${\mathbf{a}}^j\leftarrow {\left\{{\pi }_r\left(o_i^j\right)\right\}}_{i\in \mathcal{N}}$ for each agent   7:              Step $(s^{\prime j},{\mathbf{r}}^j,{\mathrm{done}}^j)\leftarrow E_j.\mathrm{step}\left({\mathbf{a}}^j\right)$   8:       end for   9:       // Assignment and reward 10:      if $t<t_{\rho }^{\mathrm{release}}$ then 11:             Keep fixed $\rho$ 12:      else 13:             Compute ${\rho }^{\prime }\leftarrow \mathrm{TwoStageAssign}\left(C\right)$; accept if gain $>{\delta }_g$ 14:      end if 15:      if $t<t_T^{\mathrm{release}}$ then 16:              Override $a_{T_m}$ with curriculum policy (e.g., circular) 17:       end if 18:       Compute $\Phi (s;\rho ),{\Phi }^{-i}(s;\rho )$ and shaped rewards via (15)–(17) 19:       Store transitions in $\mathcal{B}$ 20:       // Parameter update (every ${\Delta }_{\mathrm{update}}$ steps) 21:       if $tmod{\Delta }_{\mathrm{update}}=0$ then 22:              Sample batch $\left\{(s,\mathbf{a},\mathbf{r},s^{\prime })\right\}\sim \mathcal{B}$ 23:              for each role $R\in \{H,U,T\}$ do 24:                    Compute TD targets (33) with count-selected ${\alpha }_k$ 25:                    Update critics via (34) with gradient clipping 26:                    Update actor via (35) (batch aggregated for $R=U$) 27:                    Update ${\alpha }_k$ via (36) 28:              end for 29:              Polyak-update all target networks 30:       end if 31: end for

4.5. Computational Complexity

We express per-update cost in terms of batch size B, UAV capacity $N_{max}$, target count M, and network hidden dimension d.

4.5.1. Critic Forward Pass

The branch encoders for H, U, and T each perform $O\left(d^2\right)$ work per sample. The U-branch peer pooling iterates over $N_{max}-1$ slots, contributing $O\left(N_{max}{d}^2\right)$ per sample. The multi-head attention fusion over three tokens costs $O\left(d_f^2\right)$ per sample (constant in $N_{max}$ because the token count is fixed at three). The Q-head MLP adds $O\left(d_q^2\right)$, where $d_q$ is the Q-head hidden dimension. The total per-sample critic cost is therefore $O(N_{max}{d}^2+d_f^2+d_q^2)$, and the per-batch cost is $O\left(B(N_{max}{d}^2+d_f^2+d_q^2)\right)$.

A monolithic critic that concatenates all agent observations into a flat vector of dimension $O\left((N+M+1)d_{\mathrm{obs}}\right)$ would incur per-sample cost $O\left({\left(N{d}_{\mathrm{obs}}\right)}^2\right)$, growing quadratically with fleet size. The structured token-fusion design replaces this quadratic scaling with a linear term $O\left(N_{max}{d}^2\right)$ at the peer-pooling stage and a constant-cost fusion over the fixed set of role tokens.

4.5.2. Actor Forward Pass

The UAV actor encodes $N_{max}-1$ peer slots at $O\left(d_e^2\right)$ each, performs mean–max pooling in $O\left(N_{max}{d}_e\right)$, and evaluates the trunk MLP in $O\left(d_h^2\right)$. Per-batch cost is $O\left(B(N_{max}{d}_e^2+d_h^2)\right)$. The H and T actors are standard MLPs at $O\left(B{d}_h^2\right)$ each.

4.5.3. Assignment Solver

The Hungarian algorithm on the $M\times N$ cost matrix runs in $O(min{(M,N)}^2·max(M,N))$. With M targets and N UAVs, this is at most $O\left(M^{2}N\right)$ and is computed once per reassignment period (typically every $k_{\mathrm{assign}}$ steps), not every gradient step. The greedy Stage B fill adds $O\left(\right(N-M\left)M\right)$.

4.6. Convergence Analysis

Because PRTDA includes curriculum releases and reassignment updates, the full training process is non-stationary. This subsection therefore provides phase-local theoretical guarantees under fixed conditions, rather than a global convergence guarantee for the full outer general-sum learning dynamics.

Assumption 1. 

Let curriculum switch times be $\{{\tau }_0,{\tau }_1,\dots \}$ and define phase ν on $t\in [{\tau }_{\nu },{\tau }_{\nu +1})$. Within each phase, the game tuple in (10) is treated as stationary with fixed transition kernel and rewards; namely, no curriculum release or reassignment-triggered distribution shift occurs inside the phase. For internal-game analysis, assignment ρ is fixed and unilateral deviation is defined with other agents’ actions held fixed.

Proposition 1 

(Internal potential alignment under fixed phase conditions). Under fixed ρ and fixed opponents in a phase, the difference reward (15) satisfies the exact potential-alignment identity in (16) for unilateral deviations. Hence, any strict unilateral improvement in $r_i^{\Delta }$ strictly increases Φ. For finite internal action sets, every improvement path terminates at a pure Nash equilibrium. For continuous actions, the above implies monotone ascent of potential values; with bounded-above Φ, the potential-value sequence converges, but this alone does not imply global policy convergence.

Proof. 

From (15), terms depending only on $s_t$ cancel under unilateral deviation, and ${\Phi }^{-i}$ is invariant to $a_i$ by construction. Therefore, the utility change equals the potential change, which is the defining exact-potential condition [7]. For finite action sets, the classical finite-improvement property yields termination at a pure Nash equilibrium. For continuous actions, strict improvements generate a monotone sequence of potential values. Since each component in (26) is non-positive under the stated definitions, $\Phi$ is bounded above, so potential values converge. □

Proposition 2 

(Role-wise soft policy iteration in the stationary/tabular limit). Fix a phase ν and fix opposing-role policies. Then, each role $R\in \{H,U,T\}$ faces a stationary entropy-regularized MDP. In the tabular/exact-backup limit, soft policy iteration converges to $(Q_R^{*},{\pi }_R^{*})$ [37].

Proof. 

Under fixed opponents and fixed phase dynamics, each role’s Bellman operator is stationary; the soft policy evaluation/improvement iterations satisfy the standard SAC tabular convergence result [36,37]. Double-Q clipping is used as a stabilization device against overestimation [38]; in deep function approximation with replay and non-stationary outer updates, this supports empirical stability but does not establish global convergence of the full multi-agent game. □

5. Simulations and Discussion

All experiments train and evaluate PRTDA in a custom three-dimensional pursuit–evasion simulator. Section 5.1 describes the environment and training configuration. Section 5.2 characterizes pursuit performance on a multi-target generalization task through trajectory analysis. Section 5.3 benchmarks PRTDA against two standard MARL baselines across six fleet sizes, and Section 5.4 isolates the contribution of the structured attention critic.

5.1. Experimental Setup

The environment and training hyperparameters are listed in Table 2 and Table 3. All reported results are averaged over 3 independent random seeds per configuration (fleet size × algorithm). Each seed controls environment initialization, network weight initialization, and replay sampling order. Success rates and episode lengths are reported as mean ± standard deviation across these seeds. No systematic hyperparameter search was performed; the learning rates and network dimensions in Table 3 were selected from a small set of candidates based on single-target convergence speed, and then held fixed for all fleet sizes and baselines. All baselines share the same training budget ($\mathrm{400,000}$ steps × 8 parallel environments), observation interface, and reward structure to ensure a fair comparison. Table 4 summarizes the implementation details of the compared baselines.

Table 2. Simulation environment parameters.

Parameter	Value
Arena (horizontal); altitude bands	$1000\mathrm{m}\times 1000\mathrm{m}$; UAV $[0,600]\mathrm{m}$, target $[0,600]\mathrm{m}$, airship fixed at $600\mathrm{m}$
Capture radius $r_c$ (3D)	$35\mathrm{m}$
Fleet composition	$N\in \{2,\dots ,7\}$, $M\in \{1,3\}$, 1 airship
Speed limits $(v_H^{max},v_U^{max},v_T^{max})$	$(8,12,10)$ (m/s)
Yaw rate limits $({\omega }_H^{max},{\omega }_U^{max},{\omega }_T^{max})$	$(0.30,0.50,0.40)\mathrm{rad}/\mathrm{s}$
Pitch rate limits $({\omega }_{\gamma ,U}^{max},{\omega }_{\gamma ,T}^{max})$; ${\gamma }_{max}$	$(0.30,0.20)\mathrm{rad}/\mathrm{s}$; $1.20\mathrm{rad}$
Episode max length; time step $\Delta t$	300 steps; $0.5\mathrm{s}$
Potential weights $(w_{\mathrm{cov}},w_{\mathrm{app}},w_{\mathrm{ring}},w_{\mathrm{enc}})$	$(4,7,5,0.2)$
Safety weight ${\lambda }_{\mathrm{safe}}$; safety margin ${\delta }_s$	$1.0$; $3\mathrm{m}$
Curriculum	Fixed-target (circular) phase: 0–$\mathrm{100,000}$ steps; free-target: >100,000

Table 3. Training and network hyperparameters.

Parameter	Value
Training steps; parallel envs	$\mathrm{400,000}$; 8
Replay buffer capacity; batch size	$\mathrm{200,000}$ transitions; 512
Warmup steps; update frequency ${\Delta }_{\mathrm{update}}$	2000; every 10 steps
Discount factor $\gamma$; Polyak $\tau$	$0.97$; $0.005$
Learning rates $({\eta }_{\pi },{\eta }_Q,{\eta }_{\alpha })$	$(1\times {10}^{-4},1\times {10}^{-4},5\times {10}^{-5})$
Target entropy $\overline{\mathcal{H}}$; temperature floor ${\alpha }_{min}$	$-2.0$; $0.05$
Gradient clip; TD target clip	$5.0$; $50.0$
Actor MLP hidden dim	$256\times 2$ layers
Critic trunk MLP hidden dims	$[256,256]$
Token dim $(H/T/U)$	$64/64/128$
MHA heads; fusion dim	4; 128

Table 4. Baseline implementation details. All methods share the same observation interface, reward function, and training budget.

	PRTDA (attn)	PRTDA (MLP)	MATD3	MADDPG	MAPPO
Critic type	Role-token MHA	Flat MLP	Per-agent MLP	Per-agent MLP	Role-token $V\left(s\right)$
Actor type	Shared, count heads	Shared, count heads	Shared, count heads	Per-agent determ.	Shared, count heads
Reward	$r_{\mathrm{team}}+{\lambda }_{\Delta }{r}_i^{\Delta }$	Same	Same	Same	Same
Training budget	$\mathrm{400,000}\times 8$ envs	Same	Same	Same	Same

5.2. Pursuit Performance Evaluation

We evaluate multi-target pursuit in 3D airspace with two configurations, $N=5$ UAVs against $M=3$ targets and $N=8$ UAVs against $M=4$ targets. Training uses only single-target scenarios ($M=1$) with varying k, so both configurations are out-of-distribution generalization tests. At deployment, the assignment solver partitions the fleet into sub-groups and each sub-group activates the corresponding count-specific head, with inter-group observations masked (Section 4.3).

Figure 3 shows a representative 5-vs-3 episode. Figure 3a–d give 3D trajectory snapshots at $t=0,48,97,145$. The five UAVs converge on three targets in both horizontal and vertical dimensions while the airship holds its high-altitude coverage position. The assignment timeline in Figure 3e shows that the solver redistributes UAVs as groups complete captures. The mean 3D distance curves in Figure 3f show Group $\to T_1$ dropping below $r_c$ first around step 60, followed by Group $\to T_0$ around step 120 and Group $\to T_2$ around step 140. Over 50 evaluation episodes, PRTDA achieves $98\%$ capture success with a mean episode length of $134\pm 38$ steps.

Figure 3. Representative 3D pursuit episode with $N=5$, $M=3$. (a–d) Trajectory snapshots at $t=0,48,97,145$. The airship (blue) holds high altitude while UAVs pursue three targets in 3D airspace. Filled markers show current positions and faded trails show past trajectories. Triangles mark agent positions and squares mark target positions; colors follow the assignment groups shown in the legend. (e) UAV–target assignment timeline. (f) Mean group-to-target 3D distance. The gray line indicates $r_c$.

Figure 4 shows a larger-scale 8-vs-4 episode. The solver distributes eight UAVs across four targets. Around step 80, several reassignment events occur in Figure 4e as freed UAVs from captured targets are redirected. Figure 4a–d give 3D snapshots at $t=0,51,103,154$, and the distance curves in Figure 4f confirm that all four groups converge below $r_c$ by episode end. Over 50 evaluation episodes, the 8-vs-4 configuration reaches $100\%$ capture success with a mean episode length of $133\pm 35$ steps. The learned policy thus scales to larger fleet and target counts without retraining.

Figure 4. Representative 3D pursuit episode with $N=8$, $M=4$. (a–d) Trajectory snapshots at $t=0,51,103,154$. Eight UAVs coordinate pursuit of four independently evading targets in 3D airspace. Triangles mark agent positions and squares mark target positions; colors follow the assignment groups shown in the legend. (e) UAV–target assignment timeline. (f) Mean group-to-target 3D distance. All four groups converge below $r_c$.

5.3. Comparative Evaluation Against Baselines

Three MARL baselines are evaluated alongside PRTDA. MATD3 is a multi-agent extension of TD3 [38] that uses per-agent critics trained under the same potential-shaped reward but without role structure. MADDPG [8] is a CTDE actor–critic baseline with deterministic policies and per-agent centralized critics. MAPPO [25] is a multi-agent PPO variant with a centralized state-value critic $V\left(s\right)$; we equip it with the same role-structured token branches as the PRTDA critic, minus action injection, and train it under identical curriculum and budget. A fifth variant, PRTDA(MLP), replaces the structured attention critic with a flat MLP over concatenated role tokens while keeping all other components identical; its full ablation is presented in Section 5.4. Figure 5 shows training curves and final success rates for all five methods across fleet sizes $k\in \{2,3,4,5,6,7\}$; Table 5 provides the corresponding numerical values.

Figure 5. Training performance comparison across five methods. (a) Capture success rate at $\mathrm{400,000}$ training steps for fleet sizes $k\in \{2,\dots ,7\}$. The MAPPO bars are zero and therefore coincide with the x-axis. (b–g) Training curves for each k; shaded bands show one standard deviation over a 21-step rolling window. The vertical dashed line marks the $\mathrm{400,000}$-step snapshot.

Table 5. Capture success rates (%) at the $\mathrm{400,000}$-step snapshot, averaged over a $\pm 5000$-step window and consistent with the bar chart in Figure 5. Bold entries indicate the highest value per fleet size.

Algorithm	$\mathit{k}=2$	$\mathit{k}=3$	$\mathit{k}=4$	$\mathit{k}=5$	$\mathit{k}=6$	$\mathit{k}=7$
PRTDA (attn)	63	92	98	100	99	99
MATD3 †	69	57	90	100	99	84
MADDPG	39	76	79	79	76	78
PRTDA (MLP)	55	80	88	89	90	90
MAPPO	0	0	0	0	0	0

^† Same shared actor structure as PRTDA; per-agent MLP critic without role-token fusion.

PRTDA(attn) achieves above $90\%$ for all $k\ge 3$. The sharpest contrast with MATD3 appears at $k=3$, where MATD3 reaches only $57\%$ against PRTDA’s $92\%$, a gap of 35 percentage points. At this fleet size, three UAVs coordinate pursuit of a single target, and the critic must resolve which agent closes in, which encircles, and which trails. At $k=7$, MATD3 falls to $84\%$ against PRTDA’s $99\%$, indicating that the per-agent design does not scale to larger fleets under the same potential reward.

MADDPG converges early at intermediate k but plateaus at $39\%$ for $k=2$, where a two-UAV fleet yields sparse training signal. At $k\in \{4,5\}$, MADDPG is competitive with PRTDA, but this window narrows at both small and large k, where the absence of role structure becomes the binding constraint.

MAPPO fails to learn effective pursuit at any fleet size, with $0\%$ capture success after $\mathrm{800,000}$ training steps across all k. Two factors explain this gap. First, the capture condition requires all assigned UAVs to simultaneously enter the capture radius, making success an AND-type joint event with sparse reward. The $V\left(s\right)$-based advantage in MAPPO cannot evaluate whether a specific joint action moved the team closer to capture, whereas the $Q(s,\mathbf{a})$ critic in PRTDA provides sharper credit assignment for coordinated maneuvers. Second, MAPPO is on-policy and discards collected trajectories after each PPO update. In our setting, near-capture transitions are rare but carry high information value. The off-policy replay buffer in PRTDA allows for the repeated reuse of these rare events, which is important when successful coordination is initially infrequent.

Taken together, Table 5 shows that PRTDA(attn) leads across five of six fleet sizes at the $\mathrm{400,000}$-step snapshot. The only exception is $k=2$, where a two-UAV fleet provides limited scope for role differentiation and MATD3 holds a marginal advantage. For all $k\ge 3$, PRTDA(attn) reaches or exceeds $90\%$ capture success, while each baseline falls short at one or more fleet sizes. MADDPG’s deterministic policy produces elevated training variance under the potential reward, particularly at small k, and its final success rates stabilize around 76–$79\%$ for $k\ge 3$ without further improvement. The results indicate that role-structured conditioning is most consequential when the pursuer team faces non-trivial internal assignment decisions: for the role-asymmetric airship–UAV configuration studied here, this is already the case at $k=3$ and becomes more pronounced as the fleet grows.

Beyond success rate, capture speed is relevant for time-critical missions. The shared team reward includes a per-step penalty that incentivizes faster capture. Table 6 reports mean episode lengths for successful captures. PRTDA(attn) achieves the shortest capture times across all fleet sizes. At $k=7$, PRTDA completes capture in 125 steps ($62.5\mathrm{s}$), approximately $15\%$ faster than MATD3 (147 steps) and $34\%$ faster than MADDPG (189 steps).

Table 6. Mean episode length (steps) for successful captures, averaged over 50 evaluation episodes per fleet size. Lower is better. “—” indicates fewer than 5 successful episodes.

Algorithm	$\mathit{k}=2$	$\mathit{k}=3$	$\mathit{k}=4$	$\mathit{k}=5$	$\mathit{k}=6$	$\mathit{k}=7$
PRTDA (attn)	248	215	182	156	138	125
MATD3	261	243	198	165	152	147
MADDPG	—	256	231	212	198	189
PRTDA (MLP)	255	238	201	178	161	148

5.4. Ablation Study

5.4.1. Attention Module

To isolate the contribution of structured attention fusion in the PRTDA critic, PRTDA(attn) is specially compared with PRTDA(MLP) in Table 7.

Table 7. Ablation study on attention critic: capture success rates (%) at intermediate and final checkpoints, and steps to first reach $90\%$ success. PRTDA(MLP) replaces the structured attention fusion with a flat MLP over concatenated role tokens while keeping all other components identical.

	$\mathit{k}=2$	$\mathit{k}=3$	$\mathit{k}=4$	$\mathit{k}=5$	$\mathit{k}=6$	$\mathit{k}=7$
Success rate at $\mathrm{300,000}$ steps (%)
PRTDA (attn)	73	88	68	75	69	72
PRTDA (MLP)	52	68	61	67	71	64
Success rate at $\mathrm{400,000}$ steps (%)
PRTDA (attn)	63	92	98	100	99	99
PRTDA (MLP)	55	80	88	89	90	90
Steps to first reach $90\%$ success ($\times {10}^3$)
PRTDA (attn)	—	312	335	321	332	326
PRTDA (MLP)	—	—	—	—	334	336

“—” indicates that the method did not reach $90\%$ success within $\mathrm{400,000}$ steps.

At $\mathrm{400,000}$ steps, PRTDA(attn) consistently outperforms PRTDA(MLP) across all fleet sizes (Table 7), with gaps of 8–12 percentage points at $k\in \{3,4,5\}$. PRTDA(MLP) fails to reach the $90\%$ success threshold within the training horizon at $k\in \{3,4,5\}$, reaching only $80\%$, $88\%$, and $89\%$, respectively, while PRTDA(attn) achieves above $90\%$ at all $k\ge 3$. At $k=3$, PRTDA(attn) first crosses the $90\%$ threshold at step $\mathrm{312,000}$; PRTDA(MLP) does not reach this level within the $\mathrm{400,000}$ training horizon. This gap is apparent throughout mid-training: at $\mathrm{300,000}$ steps, PRTDA(attn) achieves $88\%$ at $k=3$ against $68\%$ for PRTDA(MLP), a 20-percentage-point margin.

The convergence advantage is most pronounced at $k=3$, where the diversity of role assignments—one UAV per target or asymmetric splits of two versus one—requires the critic to disentangle the contributions of heterogeneous agent types. A flat MLP over concatenated tokens treats role-differentiated inputs uniformly, so this disentanglement must emerge purely from the data. The MHA fusion layer encodes the structured relationship between the airship coverage signal and UAV pursuit states as an explicit inductive bias; this is consistent with the observed faster early-stage convergence.

5.4.2. Difference Reward

To evaluate the contribution of the potential-based difference reward, we train a variant PRTDA(no-diff) that sets the difference reward scale ${\lambda }_{\Delta }=0$, so each pursuer receives only the shared team reward without individual credit assignment. All other components (attention critic, curriculum schedule, network architecture) remain identical to PRTDA(attn). Table 8 compares capture success rates at the $\mathrm{300,000}$ and $\mathrm{400,000}$ checkpoints.

Table 8. Ablation study on difference reward: capture success rates (%) at intermediate and final checkpoints. PRTDA(no-diff) removes individual credit assignment (${\lambda }_{\Delta }=0$) while keeping all other components identical.

	$\mathit{k}=2$	$\mathit{k}=3$	$\mathit{k}=4$	$\mathit{k}=5$	$\mathit{k}=6$	$\mathit{k}=7$
Success rate at $\mathrm{300,000}$ steps (%)
PRTDA (attn)	73	88	68	75	69	72
PRTDA (no-diff)	0	0	0	0	0	0
Success rate at $\mathrm{400,000}$ steps (%)
PRTDA (attn)	63	92	98	100	99	99
PRTDA (no-diff)	0	0	0	0	0	0

Without the difference reward, PRTDA(no-diff) fails to learn effective pursuit at any fleet size, reaching $0\%$ capture success at both the $\mathrm{300,000}$ and $\mathrm{400,000}$ checkpoints across all k. The shared team reward alone provides only a terminal capture bonus and a small per-step synchronization signal. Because the capture condition requires all assigned UAVs to simultaneously enter the capture radius, this reward is extremely sparse during early training when coordinated closure has not yet emerged. The difference reward derived from the shaped potential $\Phi$ gives each UAV a dense, per-step credit signal that reflects its marginal contribution to approach, encirclement, and ring closure progress. Removing this signal leaves each agent unable to distinguish whether its own action improved the team state, and the policy remains at random-walk level throughout training. This result shows that the potential-based credit assignment is not merely helpful but essential for learning cooperative pursuit under the simultaneous-capture constraint.

5.5. Assignment Solver Runtime

To assess the computational overhead of the two-stage assignment solver, we measure single-call CPU wall-clock times on an Intel i7-13700K processor, averaged over 1000 random 3D position configurations per $(N,M)$ pair. For the operational configuration ($N=7$, $M=3$), each solver call completes in approximately $0.39\mathrm{ms}$, which is negligible relative to the $0.5\mathrm{s}$ physical time step. Table 9 reports solver times for larger configurations to characterize scalability.

Table 9. Assignment solver wall-clock time (ms) per single call, averaged over 1000 trials with random 3D positions.

N\ M	5	6	7	8	9	10
10	0.058	0.058	0.055	0.054	0.052	0.049
20	0.328	0.830	0.236	0.085	0.085	0.406
30	0.114	0.115	0.297	0.929	0.478	0.120

All configurations complete within $1\mathrm{ms}$, confirming that the two-stage solver (Hungarian matching plus greedy load-balanced fill) remains real-time feasible for swarm scales well beyond the fleet sizes evaluated in this paper. The non-monotonic variation across $(N,M)$ combinations reflects the interaction between the Hungarian algorithm’s complexity (sensitive to the $M\times N$ cost matrix shape) and the greedy fill stage (which processes $N-M$ remaining UAVs).

5.6. UAV Dropout Robustness

To evaluate resilience to mid-mission agent loss, we test PRTDA under UAV dropout during active pursuit. Starting from an $N=8$ fleet pursuing a single target, we randomly remove 1–5 UAVs at a uniformly sampled step in $[30,80]$. The remaining UAVs treat the dropout as a reassignment event: the count-specific actor head switches to match the new fleet size, and the observation is rebuilt with the reduced peer set. The dropped UAV holds its last position with zero velocity. Capture success is evaluated over the remaining active UAVs only. Table 10 reports the results averaged over 10 episodes per configuration.

Table 10. UAV dropout robustness: capture success rate and mean episode length after mid-episode removal of 1–5 UAVs from an $N=8$ fleet.

UAVs Dropped	0	1	2	3	4	5
Remaining UAVs	8	7	6	5	4	3
Success rate (%)	100	100	100	100	100	90
Mean steps	101	98	107	102	114	133

PRTDA maintains $100\%$ capture success when up to four UAVs are removed mid-episode, and $90\%$ when five are removed (leaving only three active). The mean episode length increases modestly from 101 to 133 steps as the fleet shrinks, reflecting the reduced pursuit capacity. This robustness stems from the shared-backbone count-head architecture: each fleet size k has a dedicated actor head trained during the dynamic-k curriculum, so the transition after dropout is equivalent to a standard reassignment event that the policy has already learned to handle. No retraining or special dropout-aware mechanism is required.

5.7. Gazebo Visual Simulation

We further deploy PRTDA in a Gazebo-based visual simulation with full rigid-body dynamics. The converged actor network from kinematic training serves as the initial policy. A cascaded PID controller bridges the policy output and the rotor-level actuators. The outer loop tracks the velocity and yaw rate references produced by the actor network, generating attitude and thrust setpoints. The inner loop regulates roll, pitch, and yaw rates at $100\mathrm{Hz}$, with gravity compensation added to the thrust channel. The policy is then refined through a continuation training phase that feeds back physical quantities from the Gazebo engine, allowing adaptation to effects absent in the kinematic model. The simulation runs at $100\mathrm{ms}$ integration steps with real-time 3D rendering.

Figure 6 shows two views from an 8-vs-3 pursuit episode. In Figure 6a, eight quadrotor UAVs converge toward three target spheres while the airship holds a higher altitude. Figure 6b gives an overhead perspective, where the airship is positioned near its coverage anchor $G_{\mathrm{cov}}$. The UAVs maintain stable flight attitudes throughout the maneuver, confirming that the continuation training successfully bridges the gap between the kinematic model and the higher-fidelity simulator.

Figure 6. Gazebo visual simulation of an 8-vs-3 pursuit episode with full rigid-body dynamics. (a) Perspective view showing eight quadrotor UAVs pursuing three target spheres with the airship at higher altitude. (b) Overhead view of the same episode, with the airship tracking its coverage anchor $G_{\mathrm{cov}}$. The orange quadrotors denote the UAV pursuers, the gray ellipsoid denotes the airship, and the colored spheres denote the targets.

Overall, the Gazebo experiment validates a two-stage workflow where the kinematic phase learns cooperative coordination and the continuation phase adapts the policy to aerodynamic effects and actuator limits. This approach offers a practical path toward deploying PRTDA on physical multi-rotor platforms.

5.8. Limitations

Several simplifying assumptions constrain the scope of the current results. First, the simulator assumes ideal state information within each agent’s observation range, without modeling sensor noise, measurement latency, or communication delays that arise in physical UAV deployments. Second, environmental disturbances such as wind gusts and turbulence are not modeled; these would affect both kinematic accuracy and observation quality. These limitations are consistent with the scope of a method-focused contribution; addressing them in physical flight tests and higher-fidelity simulators is a priority for future work.

6. Conclusions

We presented PRTDA, a learning framework for cooperative pursuit–evasion in role-asymmetric airship–UAV teams operating in three-dimensional environments. The formulation separates pursuer-side coordination, enforced through difference rewards derived from a shaped team potential, from competitive interaction with independently controlled targets. A role-structured centralized critic with attention-based fusion captures cross-role dependencies, while assignment-conditioned value estimation and curriculum scheduling reduce non-stationarity during training.

Experimental results show that PRTDA outperforms representative baselines across all tested fleet sizes, with the advantage most pronounced in smaller teams where coordination difficulty is highest. Ablation studies confirm that both the structured attention critic and the difference reward contribute to training efficiency. The learned policy generalizes to unseen multi-target configurations through the group-isolation mechanism. A Gazebo-based visual simulation further validates the approach under rigid-body dynamics with a cascaded PID inner-loop controller.

Deploying the trained policy on physical multi-rotor platforms with onboard inference is the natural next step. Extending the evaluation to larger fleets and more complex evasion strategies and closing the sim-to-real gap through sensor noise and wind disturbance modeling remain open research directions.