Task-Guided Multi-UAV Cooperative Multi-Target Tracking with Gaussian Process-Based Value Correction

Highlights

What are the main findings?

• A task-guidance-conditioned Dec-POMDP framework is formulated to jointly represent limited sensing and communication, target visibility, obstacle and safety constraints, neighborhood information, and task-guidance signals in cooperative multi-UAV multi-target tracking.
• The proposed HGP-MAPPO method combines hierarchical task guidance with Gaussian-process-based critic correction, improving persistent target observation, coordinated task handover, and value-estimation stability in standard- and dense-obstacle scenarios.

What are the implications of the main findings?

• Hierarchical task guidance can reduce the decision-coupling complexity of cooperative multi-target tracking by enabling differentiated UAV behaviors for target association, task handover, recovery, and observation-geometry maintenance.
• Gaussian-process-based residual correction provides an uncertainty-aware approach to mitigating local value-estimation bias, offering a more stable learning framework for partially observable and dynamically constrained multi-UAV tracking tasks.

Abstract

The cooperative tracking of multiple ground targets by multiple UAVs remains challenging under partial observability, limited communication, and obstacle constraints, owing to complex target association, difficult task handover, strong coupling among low-level continuous control decisions, and unstable critic value estimation. To address these issues, this paper proposes a hierarchical-guidance and Gaussian-process-corrected multi-agent proximal policy optimization method, termed HGP-MAPPO. Built upon the centralized-training and decentralized-execution paradigm, HGP-MAPPO introduces low-frequency task-guidance signals derived from target-association information, task handover and recovery cues, task priorities, and desired observation geometry. These guidance signals are incorporated as conditional inputs into the low-level actor–critic framework, thereby reducing the policy learning difficulty in jointly handling target tracking, occlusion recovery, obstacle avoidance, and smooth control. Moreover, to alleviate local estimation bias in the neural-network critic under complex partially observable conditions, a Gaussian-process-based residual correction mechanism is designed. Specifically, the posterior mean is used to compensate for value residuals, while the posterior uncertainty adaptively regulates the correction intensity, improving the stability of value evaluation and policy optimization. A sparse inducing-point approximation is adopted to control the training-stage computational cost, while the Gaussian-process module is removed during decentralized execution and, therefore, introduces no additional online inference overhead. Experiments are conducted in standard-obstacle and densely obstructed multi-UAV multi-target tracking scenarios, with DDPG-MHSA, MAPPO, MADDPG, and MATD3 adopted as baselines. The experimental results demonstrate that HGP-MAPPO achieves faster training convergence, higher average episode rewards, and improved target retention rates. It also effectively reduces UAV–target distance fluctuations and the mean absolute temporal-difference (TD) error. Ablation studies further confirm the contributions of task-guidance signals, Gaussian-process residual correction, and uncertainty-aware weighting to cooperative tracking performance and training stability.

1. Introduction

With the rapid development of unmanned aerial vehicle (UAV) platforms, onboard sensing, and cooperative-control technologies, multi-UAV systems have shown considerable potential in dynamic target surveillance, disaster search and rescue, regional security, and emergency reconnaissance [1,2]. Compared with single-UAV systems, multiple UAVs can provide wider mission coverage, sensing redundancy, and improved robustness to local occlusion and limited individual sensing capability [3,4]. However, practical tracking missions are usually subject to time-varying target motion, restricted sensing and communication ranges, limited fields of view, environmental obstacles, mission boundaries, and inter-UAV safety constraints [5,6]. These factors make persistent and reliable cooperative tracking of multiple moving targets particularly challenging.

Unlike target capture, point-reaching, or static coverage, cooperative multi-target tracking emphasizes continuous and effective observation over the mission horizon [7]. UAVs must maintain appropriate observation geometry while dynamically adapting target associations and observation responsibilities when targets become occluded, temporarily lost, or reappear. Meanwhile, obstacle avoidance, inter-UAV separation, mission-boundary constraints, and smooth continuous control must be satisfied simultaneously. Under limited sensing and communication, each UAV, therefore, operates under pronounced partial observability and must coordinate task-level decisions such as target association, handover, and recovery with low-level motion control [8].

Existing studies have investigated trajectory prediction, cooperative scheduling, sensing–communication integration, task allocation, and motion optimization for multi-UAV tracking [9,10,11,12,13]. Learning-based approaches have also been introduced to improve tracking and control adaptability [14,15]. Nevertheless, many optimization-based methods rely on explicit environmental models, predefined target-motion assumptions, or repeated replanning, while task allocation and continuous control are often handled separately. Such separation becomes restrictive when target visibility and observation responsibilities change frequently.

Multi-agent reinforcement learning (MARL) provides a data-driven framework for cooperative decision-making in dynamic environments. In particular, centralized training with decentralized execution (CTDE) exploits global information during training while retaining decentralized policies during execution [16,17,18]. Attention mechanisms, graph neural networks, and centralized guidance have further improved inter-agent interaction modeling and joint value estimation [19,20,21,22], and MARL has been applied to multi-UAV surveillance, target search, trajectory optimization, resource allocation, and safety-constrained control [23,24,25,26]. However, CTDE and interaction-attention mechanisms do not explicitly specify the time-varying cooperative role of each UAV. The decentralized policy must still infer target association, observation priority, handover and recovery requirements while simultaneously learning continuous motion control, which increases exploration difficulty and can aggravate training instability under partial observability and multi-agent non-stationarity [27,28,29].

Hierarchical reinforcement learning and task- or goal-conditioned policy learning can reduce long-horizon decision complexity by introducing subgoals, options, or contextual task information [30,31,32,33,34,35,36,37]. Centralized-teacher/decentralized-student approaches can also exploit privileged information to facilitate decentralized policy learning [21,38]. However, conventional hierarchical methods generally require an independently optimized high-level policy, whereas teacher–student methods typically involve additional supervision or policy distillation. Moreover, conventional task representations are often designed for predefined goals or tasks rather than rapidly changing cooperative roles induced by target occlusion, handover, recovery, and responsibility transfer.

Another important issue is the accuracy of the centralized Critic. Abrupt changes in target visibility, communication neighborhoods, and observation responsibilities can produce localized distribution shifts, causing systematic value-estimation errors even when the neural Critic provides an effective global approximation. Distributional, calibrated, and ensemble Critics improve value estimation through return-distribution modeling, calibration, or network disagreement [39,40,41], while uncertainty-aware reinforcement-learning studies further highlight the influence of insufficient state coverage and distribution shift [42,43]. Nevertheless, these methods do not explicitly model the structured local residual between the prediction of a given centralized Critic and its target value. Gaussian processes (GPs) provide nonlinear local prediction, together with posterior uncertainty [44,45], but directly applying a standard GP to an increasing global state space and an unbounded training dataset would incur substantial computational and memory costs.

From a control-theoretic perspective, input-to-state stability analysis and sliding-mode control provide rigorous tools for consensus tracking, trajectory regulation, and disturbance rejection under model uncertainty and external disturbances [46,47]. These methods are particularly suitable for the physical-control layer, but they generally rely on explicit dynamic models or prescribed reference objectives and do not directly address dynamic target association, task handover, recovery, and observation-responsibility allocation under partial observability. They are, therefore, complementary to the cooperative decision-making problem considered in this work.

The above analysis reveals two closely related challenges: reducing the coupling between dynamically changing cooperative tracking responsibilities and low-level continuous control without introducing an independently trained high-level policy, and improving localized Critic value estimation under abrupt changes in target visibility and cooperative roles. Addressing either issue alone is insufficient because task coordination affects the state distribution experienced by the policy, while inaccurate value estimation directly influences temporal-difference errors, advantage estimation, and policy updates.

To address these challenges, this paper proposes Hierarchical Guidance and Gaussian-Process-Corrected Multi-Agent Proximal Policy Optimization (HGP-MAPPO) for persistent multi-UAV multi-target tracking under partial observability. Low-frequency task-guidance variables describing target association, task priority, handover and recovery cues, and desired observation geometry are incorporated into both the decentralized Actors and the centralized Critic. Unlike conventional hierarchical reinforcement learning, these variables are not generated by an independently optimized high-level policy and require neither an additional high-level reward nor a separate policy-update process.

Furthermore, a sparse GP residual correction mechanism is introduced for the conditional centralized Critic. Rather than replacing the neural Critic, the GP models localized value residuals in a fixed-dimensional value-feature space. Its posterior mean provides residual compensation, while posterior uncertainty regulates the correction magnitude. A bounded residual buffer, sparse inducing points, and periodic updates are adopted to maintain computational tractability. The GP module is used only during centralized training and is removed together with the Critic during decentralized execution. Hence, it introduces no additional execution-stage inference overhead.

Although HGP-MAPPO incorporates established concepts such as CTDE, task-conditioned policy learning, and uncertainty-aware value estimation, its methodological novelty lies in their problem-specific formulation and integration for persistent multi-UAV multi-target tracking. Specifically, dynamic cooperative-role information conditions both decentralized policy execution and centralized value estimation, while a sparse GP provides uncertainty-regulated correction of localized Critic residuals.

The main contributions of this paper are summarized as follows:

(1)

A task-guidance-conditioned decentralized partially observable Markov decision process (Dec-POMDP) is formulated for persistent multi-UAV multi-target tracking under limited sensing and communication, obstacle and boundary constraints, inter-UAV safety requirements, and dynamically changing cooperative roles.

(2)

A lightweight task-guidance-conditioned MAPPO framework is developed, in which target association, task priority, handover/recovery cues, and observation geometry condition both the decentralized Actors and the centralized Critic without introducing an independently optimized high-level policy or teacher–student distillation process.

(3)

A sparse GP residual correction mechanism is developed for the centralized Critic. Posterior-mean compensation and uncertainty-adaptive regulation are combined with sparse inducing points, periodic updates, and stop-gradient operation to improve value-estimation stability while maintaining computational tractability and decentralized-execution efficiency.

(4)

Comparative, ablation, parameter-sensitivity, safety, computational-cost, and scalability experiments demonstrate the effectiveness of HGP-MAPPO in the evaluated 4-UAV/4-target and 6-UAV/6-target settings.

The remainder of this paper is organized as follows. Section 2 reviews the related work. Section 3 formulates the cooperative multi-target tracking problem. Section 4 presents the proposed HGP-MAPPO method. Section 5 reports the experimental results, and Section 6 concludes the paper.

2. Related Work

2.1. Multi-UAV Cooperative Control, Robust Control, and Multi-Target Tracking

Multi-UAV cooperative control and multi-target tracking aim to maintain persistent observation of multiple dynamic targets through coordinated motion, task allocation, and information sharing. Compared with single-UAV systems, multi-UAV systems provide wider coverage, sensing redundancy, and improved tolerance to local occlusion and limited individual sensing capability. However, practical tracking missions involve several tightly coupled requirements, including dynamic target association, observation-responsibility adjustment, task handover, target recovery, obstacle avoidance, inter-UAV separation, and continuous motion control [48].

Sun et al. systematically reviewed UAV-based moving-target tracking methods and categorized them into detection-based, following-based, and cooperative tracking approaches, highlighting challenges such as rapid target maneuvers, target occlusion, limited fields of view, and multi-platform coordination [9,49]. Zhou et al. proposed a swarm-based cooperative tracking method that integrates trajectory prediction, cooperative scheduling, and collision avoidance [10]. Xiang et al. jointly optimized power control, path planning, and target association to reduce interference-induced performance degradation in multi-UAV tracking systems [11]. These studies have advanced trajectory prediction, cooperative scheduling, and sensing–communication integration.

Nevertheless, such optimization-based methods commonly rely on explicit environmental models, predefined target-motion assumptions, or repeated trajectory and assignment optimization. When target visibility changes frequently because of occlusion, reappearance, or trajectory intersections, the target-association and scheduling solutions may need to be recomputed. This repeated replanning can reduce online adaptability and tracking continuity. Moreover, optimizing target association or communication resources does not directly provide a continuous decentralized policy for handling temporary target loss, responsibility transfer, and obstacle avoidance simultaneously.

Other studies have focused on coverage, search, and task allocation. Zhang et al. unified coverage and tracking and formulated task allocation as a minimum-cost maximum-flow problem [12]. Wang et al. developed a distributed method based on improved quantum particle swarm optimization to enhance cooperative tracking through motion and coordination optimization [13]. Xia et al. proposed a MARL-based UAV swarm tracking method using historical and current observations [14], while Ma et al. applied deep reinforcement learning to single-UAV target tracking under disturbances [15].

These methods improve task allocation, tracking accuracy, energy efficiency, trajectory optimization, or individual-UAV robustness. However, task allocation and low-level control are often treated as separate stages. An assignment that is optimal at the task level may become infeasible when field-of-view maintenance, obstacle avoidance, inter-UAV separation, and smooth continuous actions must be satisfied simultaneously. In addition, single-UAV or accuracy-oriented learning methods do not explicitly address dynamic observation-responsibility transfer among multiple UAVs. Therefore, the coupled relationships among target association, handover, occlusion recovery, observation geometry, and continuous cooperative control remain insufficiently addressed.

From the control-theoretic perspective, input-to-state stability (ISS) analysis and sliding-mode control provide important foundations for multi-agent coordination, trajectory tracking, and disturbance rejection. Di Ferdinando et al. investigated robust quantized sampled-data leaderless consensus tracking for nonlinear multi-agent systems from an ISS perspective [46]. Their analysis addresses the effects of sampled-data implementation, information quantization, and external perturbations. Farid et al. studied robust control of multi-rotor UAVs in harsh wind fields and demonstrated the effectiveness of sliding-mode control for trajectory regulation and disturbance rejection [47].

These control-theoretic approaches are particularly suitable for low-level stabilization, trajectory tracking, consensus regulation, and disturbance rejection but generally rely on explicit dynamic models, reference trajectories, or predefined consensus objectives. Therefore, they do not directly address dynamic target association, task handover, target recovery, or observation-responsibility redistribution under partial observability. More broadly, although existing studies have made substantial progress in cooperative scheduling, task allocation, sensing–communication integration, trajectory optimization, and robust control, dynamic cooperative roles and continuous control are often handled separately or through repeated model-based optimization. This limitation motivates the task-guidance-conditioned cooperative decision framework developed in HGP-MAPPO.

2.2. Multi-Agent Reinforcement Learning and Centralized Training with Decentralized Execution

Multi-UAV tracking can be formulated as a cooperative multi-agent decision-making problem. MARL enables agents to learn coordinated policies through interaction with the environment, but its performance is affected by non-stationarity, partial observability, and multi-agent credit assignment [27]. Because each agent changes its policy during training, the effective environment observed by the other agents also changes, which can destabilize policy evaluation and optimization.

Hao et al. analyzed exploration in multi-agent settings and showed that concurrent policy updates aggravate non-stationarity [28,29]. To improve inter-agent interaction modeling, Qi et al. proposed a cascaded graph-attention mechanism [19], while Wei et al. incorporated graph attention into value decomposition to improve joint value estimation [20]. These methods improve the aggregation of interaction information and allow agents to assign different importance to their neighbors.

Nevertheless, graph- and attention-based interaction modeling mainly determines how information from other agents should be aggregated. It does not explicitly indicate the current target association, observation priority, handover status, recovery requirement, or desired observation geometry of each UAV. Therefore, even with an attention mechanism, the decentralized policy must still infer its time-varying cooperative role and continuous control behavior simultaneously from local observations and delayed rewards. Attention modeling can improve relational representation, but it does not by itself resolve the coupling between task-level coordination and low-level motion control.

Centralized training with decentralized execution (CTDE) is widely used to mitigate partial observability and non-stationarity. A centralized Critic can exploit global information during training, while the decentralized Actors use only locally available information during execution. Zhao et al. proposed a centralized-teacher-with-decentralized-students framework to guide decentralized policy learning [21]. Hu et al. combined graph neural networks with centralized value functions for large-scale cooperative control [22]. These studies demonstrate the benefits of centralized information and structured interaction modeling.

However, CTDE does not automatically reduce the semantic complexity of decentralized decision-making. In persistent multi-target tracking, the Actor must still infer target association, task priority, handover, and occlusion recovery while simultaneously learning obstacle avoidance and continuous control from local observations. Moreover, centralized-teacher approaches typically require a separately optimized teacher, action supervision, or policy distillation, which are not required by the proposed task-guidance mechanism.

MARL has also been applied to multi-UAV surveillance, target search, trajectory optimization, resource allocation, and safety-constrained control. Yun et al. developed a multi-UAV surveillance method that balances coverage and communication constraints [23]. Hou et al. considered cooperative target search using UAV swarms [24]. Ning et al. jointly optimized UAV trajectories and quality of service through MARL [25], while Termehchi et al. incorporated safety constraints into joint trajectory and channel-allocation optimization [26]. These studies demonstrate the suitability of MARL for continuous, constrained, and multi-objective UAV control.

Nevertheless, surveillance, search, resource allocation, and trajectory optimization do not fully capture the requirements of persistent multi-target tracking, which requires continuous target visibility, dynamic observation- responsibility transfer, target recovery, and observation-geometry regulation. Although representative algorithms such as MADDPG, MATD3, MAPPO, and attention-enabled multi-agent methods are widely used for continuous multi-agent control, standard MAPPO and attention-based methods do not explicitly provide dynamic task-role information to the Actors. Consequently, the policy must jointly infer cooperative responsibilities and continuous actions, increasing exploration difficulty. Moreover, abrupt changes in target visibility, communication neighborhoods, and observation responsibilities may cause localized value-estimation errors in the centralized Critic.

HGP-MAPPO addresses these limitations through low-frequency task-guidance conditioning and Gaussian-process-based Critic correction. The guidance variables explicitly provide target association, task priority, handover/recovery cues, and desired observation geometry to reduce the semantic burden on the decentralized Actors, while the GP module corrects localized value-estimation residuals during centralized training. Both the centralized Critic and GP module are removed during decentralized execution.

2.3. Task Guidance and Conditional Policy Learning

In complex multi-UAV multi-target tracking, relying solely on raw observations and delayed scalar rewards makes policy learning difficult because of long-horizon credit assignment and strong coupling between task-level decisions and continuous control. Each UAV must simultaneously infer dynamic tracking responsibilities, including target association, handover, recovery, and observation requirements, while generating continuous motion actions. Introducing structured task information can, therefore, reduce exploration difficulty and alleviate the need to infer cooperative semantics implicitly.

Hierarchical reinforcement learning decomposes long-horizon problems through subgoals, options, or temporally extended actions. Pateria et al. reviewed hierarchical reinforcement-learning methods and showed that temporal abstraction can improve policy-search efficiency [30]. Yang et al. combined high-level subtask planning with low-level policies for sequential control [31], while Luo et al. improved the coordination between subgoal generation and policy learning through goal-conditioned hierarchical reinforcement learning [32]. Liu et al. applied a hierarchical architecture combining high-level planning and low-level control to UAV navigation [33]. However, conventional hierarchical reinforcement learning typically requires an independently optimized high-level policy and additional high-level rewards or termination mechanisms, increasing implementation and temporal credit-assignment complexity. Moreover, high-level decision errors may propagate to the low-level controller, while learning task retention, handover, and recovery can itself require substantial exploration.

Task-conditioned and goal-conditioned learning provide an alternative means of incorporating task information. Wang et al. introduced task logic and action primitives to improve long-horizon decision switching [34], while Zhou et al. used human feedback to guide subgoal selection [50]. In multi-agent settings, Mai et al. employed task-specific teachers for knowledge transfer [38]. Li et al. integrated task and interaction representations to improve adaptation across tasks [35]. Zhu et al. considered variable-entity and variable-task settings through Transformer-based task modeling [36], and Chen et al. proposed attention transfer across different agent scales [37].

Most task-conditioned approaches represent tasks using identifiers, goal vectors, logical specifications, or learned embeddings. Although effective for predefined tasks or cross-task transfer, such representations do not explicitly describe rapidly changing cooperative roles such as target handover, recovery, and observation-priority adjustment. Centralized-teacher/decentralized- student methods can exploit privileged information during training but generally require a separately trained teacher, action supervision, or policy distillation. Moreover, teacher actions do not necessarily provide explicit representations of these dynamic tracking semantics.

In contrast, HGP-MAPPO does not introduce an independently trained high-level policy, teacher policy, or policy-distillation process. Instead, time-varying cooperative tracking information, including target association, task priority, handover/recovery cues, and desired observation geometry, is represented as low-frequency guidance variables that condition both the decentralized Actors and the centralized Critic. HGP-MAPPO is, therefore, better characterized as a lightweight task-guidance-conditioned CTDE architecture rather than a conventional hierarchical reinforcement-learning or teacher–student framework, reducing task–control coupling while preserving decentralized execution.

2.4. Value Function Estimation and Gaussian-Process Correction

In Actor–Critic methods, Critic accuracy directly affects temporal-difference errors, advantage estimation, and policy-update stability. In persistent multi-UAV tracking, abrupt changes in target visibility, communication neighborhoods, and observation responsibilities may cause localized value-estimation errors. Existing studies improve value estimation through distributional modeling, calibration, ensembles, and uncertainty estimation. Duan et al. proposed Distributional Soft Actor–Critic to model return distributions and mitigate Q-value overestimation [39]. Dorka et al. introduced adaptive Critic calibration to improve the consistency between value estimation and policy optimization [40]. Hoel et al. developed Ensemble Quantile Networks to represent both return distributions and predictive uncertainty [41]. However, distributional Critics model the overall return distribution rather than localized residuals, ensemble methods require multiple Critic networks, and calibration methods do not explicitly capture the remaining residual structure. Consequently, these approaches do not directly target the localized errors induced by occlusion, reappearance, topology changes, and observation-responsibility transfer.

In partially observable and distribution-shift settings, value errors are also associated with limited state coverage and model uncertainty. Wu et al. proposed uncertainty-aware model-based reinforcement learning using ensemble dynamics models to reduce unreliable interactions [42]. Prudencio et al. identified distribution shift and extrapolation error as important limitations of offline reinforcement-learning value estimation [43]. These studies demonstrate the importance of uncertainty awareness, but their objectives differ from correcting localized residuals of a centralized MARL Critic during on-policy training.

Gaussian processes provide local nonlinear prediction together with principled posterior uncertainty. Lin et al. combined Gaussian processes with teacher-advice modeling to improve the reliability of external guidance [44], while Zhou et al. used Gaussian processes for safe-region estimation in dynamic systems [45]. These studies show that GP posterior uncertainty can be used to regulate the influence of predictions. However, their GP models are primarily used for teacher reliability or safety-region estimation rather than for correcting the residual of a centralized neural Critic.

Directly replacing a neural Critic with a standard GP is impractical for long-horizon multi-agent training because its computational and memory costs increase rapidly with the dataset size and state dimension. HGP-MAPPO, therefore, retains the neural Critic for global value approximation and employs a sparse GP only to model localized value residuals in a fixed-dimensional feature space. Posterior-mean compensation and uncertainty-aware regulation are used to correct these residuals while maintaining computational tractability.

Unlike distributional or ensemble value methods, HGP-MAPPO neither models the complete return distribution nor estimates uncertainty through multiple Critic networks. Instead, it formulates Critic correction as an uncertainty-regulated sparse residual-learning problem used only during centralized training. The GP module is removed together with the Critic during decentralized execution, introducing no additional execution-stage overhead. This design complements the task-guidance-conditioned policy by improving localized value-estimation stability without altering decentralized execution.

2.5. Focused Comparison with Related Frameworks

Although HGP-MAPPO incorporates established concepts from CTDE, task-conditioned learning, hierarchical guidance, and uncertainty-aware value estimation, it differs from representative related frameworks in the formulation of dynamic task guidance and localized value correction, as summarized in Table 1.

Table 1. Focused comparison between HGP-MAPPO and representative related method families. ✓ denotes explicit support, ∆ denotes partial or method-dependent support, and – denotes that the function is not explicitly included in the representative formulation.

Method Family	Learned High-Level Policy	Dynamic Task Guidance	Centralized Critic	Decentralized Execution	GP/ Uncertainty Correction	Task Handover	Occlusion Recovery	Persistent Multi-Target Tracking
MAPPO with shaped rewards	–	–	✓	✓	–	–	–	∆
Hierarchical RL with subgoals	✓	∆	∆	∆	–	∆	–	∆
Goal-/task-conditioned MARL	–	∆	∆	✓	–	∆	∆	∆
Centralized teacher/decentralized student	–	–	∆	✓	–	∆	∆	∆
Uncertainty-aware or ensemble Critic	–	–	✓	∆	✓	–	–	–
HGP-MAPPO	–	✓	✓	✓	✓	✓	✓	✓

As summarized in Table 1, HGP-MAPPO differs from standard MAPPO and conventional hierarchical or task-conditioned methods by explicitly representing time-varying cooperative roles through low-frequency guidance variables without introducing an independently optimized high-level policy. Unlike centralized-teacher/decentralized-student approaches, it requires neither teacher-generated actions nor policy distillation. Moreover, unlike distributional or ensemble Critic methods, the GP module does not replace the neural Critic or model the complete return distribution but instead performs uncertainty-regulated correction of localized Critic residuals. Therefore, HGP-MAPPO combines dynamic task-guidance conditioning with localized residual value correction for persistent multi-UAV multi-target tracking under partial observability.

3. Problem Formulation and Dec-POMDP-Based Modeling

3.1. Scenario Description and Fundamental Assumptions

This paper considers cooperative tracking of multiple mobile ground targets by multiple homogeneous UAVs in a bounded, two-dimensional, partially observable environment containing static obstacles, mission boundaries, and potential occluders. Under limited sensing and communication, the UAV team is required to maintain persistent target observation while handling temporary occlusion and target loss. Meanwhile, obstacle avoidance, inter-UAV safety separation, and smooth control must be satisfied to ensure safe and feasible cooperative tracking.

The mission area is assumed to be locally flat, with UAV horizontal motion and ground-target motion represented in a two-dimensional plane and the UAVs operating at a fixed altitude. Terrain elevation, altitude variation, and three-dimensional terrain-induced occlusion are, therefore, not explicitly modeled. Extension to non-flat or large-scale environments would require terrain-aware target dynamics, three-dimensional UAV kinematics, and three-dimensional visibility modeling.

To reduce the coupling between task-level coordination and continuous control, target association, handover/recovery cues, task priorities, and desired observation geometry are abstracted as low-frequency task-guidance signals and used as conditional inputs to the Actor and Critic rather than as outputs of an independently optimized high-level policy. Under the CTDE paradigm, global information is available to the centralized Critic during training, whereas each UAV executes its policy using only local observations, neighbor communication summaries, and its assigned guidance signals. Ground targets are treated as exogenous dynamic entities, with their true states used primarily for environment updates, reward computation, and evaluation rather than being directly provided to the decentralized Actor.

3.2. Multi-UAV Multi-Target Cooperative Tracking Task and Environment Modeling

To facilitate the subsequent conditional Dec-POMDP formulation, this section provides a unified description of the basic entities, UAV kinematics, ground-target dynamics, and environmental constraints involved in the multi-UAV multi-target cooperative tracking task. The environmental model distinguishes among constraints on UAV motion, constraints on ground-target motion, and structures that affect line-of-sight visibility. These effects may originate from the same physical structure but are not required to be identical. In this paper, the modeling framework focuses only on the components directly associated with lower-level continuous control, whereas target prediction and high-level task allocation are not investigated as independent research objectives.

3.2.1. Definition of the Cooperative Tracking Task

Assume that there are N UAVs and M mobile ground targets in the mission area, whose sets are, respectively, defined as

$$U=\{u_1,u_2,\dots ,u_N\},G=\{g_1,g_2,\dots ,g_M\}.$$

(1)

where $u_i$ denotes the i-th UAV, and $g_j$ denotes the j-th ground target. At any time instant, t, if $g_j$ lies within the effective sensing range of $u_i$ and satisfies the constraints on the field of view, occlusion, and observation quality, $u_i$ is regarded as being in an effective observation state with respect to $g_j$. Define the binary visibility variable $z_{i,j,t}\in \{0,1\}$. Then, target $g_j$ is considered to be maintained under tracking by the system at time t if the following condition is satisfied:

$${\displaystyle \sum _{i=1}^N}{z}_{i,j,t}\ge 1.$$

(2)

If a target is not effectively observed by any UAV for several consecutive time steps, it is regarded as lost. Once the target is effectively observed again by any UAV, target recovery is considered to be achieved. The above definitions provide the basis for subsequent modeling of target maintenance, occlusion recovery, and cooperative tracking policies.

3.2.2. UAV Kinematics and Lower-Level Action Definition

Considering that this paper focuses on lower-level continuous control and conditional value estimation rather than complex three-dimensional flight dynamics, a two-dimensional discrete-time kinematic model at a fixed flight altitude is adopted to describe UAV motion, following the commonly used simplification in UAV navigation and control modeling [51]. The state and lower-level action of the i-th UAV at time t are, respectively, defined as

$$s_{i,t}^u={\left[x_{i,t}^u,y_{i,t}^u,v_{i,t}^u,{\psi }_{i,t}^u\right]}^{\top },a_{i,t}^L={\left[a_{i,t}^{acc},{\omega }_{i,t}\right]}^{\top },$$

(3)

where $(x_{i,t}^u,y_{i,t}^u)$ denotes the UAV position, $v_{i,t}^u$ denotes the flight velocity, and ${\psi }_{i,t}^u$ denotes the heading angle; ⊤ presents the transpose operation; $a_{i,t}^{acc}$ and ${\omega }_{i,t}$ represent the linear acceleration control input and the angular velocity control input, respectively. Let

$$p_{i,t}^u={\left[x_{i,t}^u,y_{i,t}^u\right]}^{\top }.$$

(4)

Then, the discrete-time motion update of the UAV can be summarized as

$$\left\{\begin{array}{cc}\hfill v_{i,t+1}^u& =clip\left(v_{i,t}^u+a_{i,t}^{acc}\Delta t,v_{\min }^u,v_{\max }^u\right),\hfill \\ \hfill {\psi }_{i,t+1}^u& ={\psi }_{i,t}^u+{\omega }_{i,t}\Delta t,\hfill \\ \hfill p_{i,t+1}^u& =p_{i,t}^u+\Delta t{v}_{i,t+1}^u{\left[\cos {\psi }_{i,t+1}^u,\sin {\psi }_{i,t+1}^u\right]}^{\top },\hfill \end{array}\right.$$

(5)

where $\Delta t$ denotes the control time step, and $clip(·)$ is used to constrain the admissible speed range of the UAV. To ensure action executability, the linear acceleration and angular velocity are subject to maximum magnitude constraints, respectively. This model enables the lower-level policy to achieve target tracking, obstacle avoidance, safety separation maintenance, and smooth control by regulating the UAV velocity and heading.

3.2.3. Exogenous Dynamic Description of Ground Targets

This paper does not focus on predicting ground-target motion. Instead, ground targets are modeled as exogenous dynamic entities in the environment. The position of the j-th ground target at time t is defined as

$$p_{j,t}^g={[x_{j,t}^g,y_{j,t}^g]}^{\top },$$

(6)

where $x_{j,t}^g$ and $y_{j,t}^g$ denote the horizontal and vertical coordinates of the target, respectively.

Let ${\mathcal{B}}^g=\{B_1^g,B_2^g,\dots ,B_{K_g}^g\}$ denote the set of ground non-traversable regions, where $K_g$ is the number of regions that constrain ground-target motion. The traversable ground region is defined as

$${\mathrm{\Omega }}_{\mathrm{free}}^g=\mathrm{\Omega }\setminus {\displaystyle \bigcup _{k=1}^{K_g}}{B}_k^g,p_{j,t}^g\in {\mathrm{\Omega }}_{\mathrm{free}}^g.$$

(7)

To ensure the basic physical plausibility of the dynamic environment, the target motion is assumed to satisfy bounded-velocity and position-continuity constraints:

$$∥p_{j,t+1}^g-p_{j,t}^g∥\le v_{\max }^g\Delta t,$$

(8)

where $v_{\max }^g$ denotes the maximum moving speed of the target. This constraint is used only to limit the magnitude of target position variations and does not imply that the UAVs can directly access the true motion model of the targets. In the simulation environment, target trajectories can be generated using random perturbations, predefined trajectories, hybrid maneuvers, or scenario scripts. During execution, the lower-level Actor can only obtain information such as the relative position of the target, the estimated relative velocity, the visibility status, the observation quality, and the most recent observation time from local observations.

3.2.4. Entity-Specific Obstacles, Boundaries, and Safety Constraints

Let the mission area be a bounded two-dimensional region, $\mathrm{\Omega }$. To distinguish the different effects of environmental structures, three sets are introduced. Let ${\mathcal{B}}^u=\{B_1^u,B_2^u,\dots ,B_{K_u}^u\}$ denote aerial exclusion regions, ${\mathcal{B}}^g=\{B_1^g,B_2^g,\dots ,B_{K_g}^g\}$ denote ground non-traversable regions, and ${\mathcal{B}}^{\mathrm{occ}}=\{B_1^{\mathrm{occ}},B_2^{\mathrm{occ}},\dots ,B_{K_{\mathrm{occ}}}^{\mathrm{occ}}\}$ denote the set of structures that may block the line of sight between UAVs and targets, where $K_{\mathrm{occ}}$ denotes the number of such occluding structures.

A safety buffer is introduced around each aerial exclusion region. Accordingly, the collision-free region for UAV motion is defined as

$${\mathrm{\Omega }}_{\mathrm{free}}^u=\mathrm{\Omega }\setminus {\displaystyle \bigcup _{k=1}^{K_u}}{B}_k^{u,\mathrm{safe}},p_{i,t}^u\in {\mathrm{\Omega }}_{\mathrm{free}}^u,$$

(9)

where $B_k^{u,\mathrm{safe}}$ denotes the safety-expanded region associated with the k-th aerial obstacle or no-fly region.

The admissible ground-target region is separately defined as

$${\mathrm{\Omega }}_{\mathrm{free}}^g=\mathrm{\Omega }\setminus {\displaystyle \bigcup _{k=1}^{K_g}}{B}_k^g,p_{j,t}^g\in {\mathrm{\Omega }}_{\mathrm{free}}^g.$$

(10)

The visibility between UAV $u_i$ and target $g_j$ is affected by the occlusion set ${\mathcal{B}}^{\mathrm{occ}}$. In particular, the target is regarded as geometrically occluded if the line segment connecting ${\mathit{p}}_{i,t}^u$ and ${\mathit{p}}_{j,t}^g$ intersects any region in ${\mathcal{B}}^{\mathrm{occ}}$. This geometric condition is combined with the sensing-range, field-of-view, and observation-quality requirements when determining the binary visibility variable $z_{i,j,t}$.

The three obstacle sets are not required to be identical. An aerial-only obstacle or no-fly region may constrain UAV motion without restricting a ground target, whereas a low ground barrier may restrict target motion without affecting a UAV operating above it. Similarly, a structure may affect line-of-sight visibility without being treated as an impassable region for both entity types. In the experiments of this paper, the same static structures may share geometric footprints across these roles, but their effects on UAV motion, target motion, and visibility are evaluated separately.

If a UAV exceeds the mission boundary or enters an aerial safety-exclusion region, it incurs a boundary violation or an obstacle collision risk. In addition, the minimum inter-UAV safety-distance constraint is given by

$$∥p_{i,t}^u-p_{k,t}^u∥\ge d_{\mathrm{safe}}^u,i\ne k.$$

(11)

These entity-specific definitions allow the framework to represent aerial-only obstacles, ground-only obstacles, and visibility-only occluders. They support the subsequent construction of local observations, visibility states, reward terms, and safety constraints without requiring all environmental structures to affect UAVs and ground targets in the same way.

3.3. Partially Observable Decision Model with Task Guidance

Based on the task and environment modeling presented in Section 3.2, this section further formulates a partially observable decision model with task guidance. Since an individual UAV cannot access the complete global state during execution, task guidance is incorporated into the Dec-POMDP framework. Accordingly, the multi-UAV multi-target cooperative tracking task is formulated as a task-guided partially observable multi-agent continuous control problem.

3.3.1. Neighborhood Communication and Decentralized Execution Information Structure

To satisfy the requirement of decentralized execution, each UAV exchanges only local summary information with neighboring UAVs within its communication range [52]. The communication neighborhood of the i-th UAV at time t is defined as

$${\mathcal{N}}_{i,t}^{com}=\left\{u_k\mid ∥p_{i,t}^u-p_{k,t}^u∥\le R_{com},k\ne i\right\}.$$

(12)

where $R_{com}$ denotes the communication radius. $u_i$ can obtain summary information from neighboring UAVs, including relative positions, relative velocities, currently attended targets, the set of visible targets, observation states, and handover cues. It should be emphasized that neighborhood communication information is not equivalent to the global state, and the communication neighborhood dynamically changes with the positions of the UAVs. Therefore, the information available to the i-th UAV during execution consists of its local observation and the task guidance signal associated with itself:

$${\xi }_{i,t}^L=\left(o_{i,t},c_{i,t}^H\right).$$

(13)

This information structure reflects the basic requirement of decentralized execution in this paper: during real-time control, each UAV does not access the complete global state, but outputs continuous control actions based on local observations, neighborhood summaries, and task conditions.

3.3.2. Definition and Generation of Task Guidance Signals

To reduce the learning difficulty associated with target assignment, task handover, occlusion recovery, observation coordination, and continuous control, low-frequency task-guidance signals are introduced as conditional inputs to the lower-level Actor–Critic architecture. For UAV i, the local task-guidance signal is defined as

$$c_{i,t}^H=\left(m_{i,t},h_{i,t},{\omega }_{i,t}^H,{\xi }_{i,t}^H\right),$$

(14)

where $m_{i,t}$ denotes the target-association cue, $h_{i,t}$ denotes the handover or recovery state, ${\omega }_{i,t}^H$ denotes the task-priority weight, and ${\xi }_{i,t}^H$ denotes the desired observation geometry. The aggregated guidance signal supplied to the conditional centralized Critic is

$$C_t^H=\left\{c_{1,t}^H,c_{2,t}^H,\dots ,c_{N,t}^H\right\}.$$

(15)

The task-guidance signal is generated through a deterministic rule-based interface rather than an independently learned high-level policy. Only information available during decentralized execution is used, including the current local observation, locally available neighborhood summaries, the previous target association, and locally maintained target-loss information. No future target trajectory or privileged centralized training information is used.

(1) Target Association

The candidate set of UAV i contains targets that are currently visible, reported as visible in the available neighborhood summary, or associated with the UAV at the preceding guidance-update instant:

$${\mathcal{C}}_{i,t}=\left\{j|z_{i,j,t}=1\vee {\widehat{z}}_{i,j,t}^{\mathrm{nei}}=1\vee j=m_{i,t-K_H}\right\},$$

(16)

where ${\widehat{z}}_{i,j,t}^{\mathrm{nei}}$ denotes the locally available neighborhood visibility summary, and $K_H$ is the guidance-update interval.

Each candidate target is evaluated according to

$$\begin{array}{cc}\hfill S_{i,j,t}=& {\alpha }_z{z}_{i,j,t}+{\alpha }_d\left(1-{\overline{d}}_{i,j,t}^{ug}\right)+{\alpha }_c{q}_{i,j,t}+{\alpha }_l{\overline{L}}_{i,j,t}\hfill \\ & -{\alpha }_{\mathrm{sw}}\mathbb{I}\left(j\ne m_{i,t-K_H}\right)-{\alpha }_{\mathrm{safe}}{\rho }_{i,j,t}^{\mathrm{safe}},\hfill \end{array}$$

(17)

where ${\overline{d}}_{i,j,t}^{ug}$ is the normalized UAV–target distance, $q_{i,j,t}$ is the observation-coverage deficit, ${\overline{L}}_{i,j,t}$ is the normalized target-loss duration, and ${\rho }_{i,j,t}^{\mathrm{safe}}$ is the assignment-related safety risk. The target-association cue is then selected deterministically as

$$m_{i,t}=\arg \underset{j\in {\mathcal{C}}_{i,t}}{\max }{S}_{i,j,t}.$$

(18)

(2) Handover, Recovery, and Task Priority

The discrete state $h_{i,t}\in \{0,1,2\}$ represents normal tracking, task handover, and target recovery, respectively. Normal tracking is retained while the associated target remains visible. A handover is activated when the target remains lost for at least $N_{\mathrm{ho}}$ consecutive guidance updates and another UAV has a more suitable observation condition. Recovery is activated when the target remains unobserved for at least $N_{\mathrm{rec}}$ guidance updates. In the recovery state, the most recently observed target position and velocity are used to generate a short-horizon constant-velocity reference. The consecutive-update thresholds introduce hysteresis and suppress frequent switching among the three states.

The normalized task-priority weight associated with the selected target is defined as

$${\omega }_{i,t}^H={\displaystyle \frac{ϵ+{\beta }_q{q}_{i,m_{i,t},t}+{\beta }_l{\overline{L}}_{i,m_{i,t},t}+{\beta }_z\left(1-{\overline{z}}_{i,m_{i,t},t}\right)+{\beta }_h{\chi }_{i,m_{i,t},t}^{hr}}{{\displaystyle \sum _{j\in {\mathcal{C}}_{i,t}}\left[ϵ+{\beta }_q{q}_{i,j,t}+{\beta }_l{\overline{L}}_{i,j,t}+{\beta }_z\left(1-{\overline{z}}_{i,j,t}\right)+{\beta }_h{\chi }_{i,j,t}^{hr}\right]}}},$$

(19)

where ${\overline{z}}_{i,j,t}$ denotes the locally available visibility ratio, and ${\chi }_{i,j,t}^{hr}$ indicates whether target j is involved in a handover or recovery event.

(3) Desired Observation Geometry

The desired observation geometry is represented by

$${\xi }_{i,t}^H=\left(r_{i,t}^{*},{\theta }_{i,t}^{*},{\psi }_{i,t}^{*}\right),$$

(20)

where $r_{i,t}^{*}$ is the desired UAV–target distance, ${\theta }_{i,t}^{*}$ is the desired observation bearing, and ${\psi }_{i,t}^{*}$ is the desired viewing direction. The desired distance is selected within the reliable observation range. The observation bearings of UAVs assigned to the same target are initially distributed over different angular sectors, after which the nearest feasible bearing satisfying flight- region, inter-UAV safety-distance, and line-of-sight constraints is selected. The viewing direction points from the selected observation position toward the associated target. During recovery, the short-horizon predicted target position is used instead of the unavailable current position.

(4) Low-Frequency Update and Information Scope

The task-guidance signal is updated every $K_H$ lower-level control steps and is held constant between two consecutive guidance updates. The lower-level Actor generates continuous actions at every interaction step using the current local observation and the most recently updated task-guidance signal. In all experiments, the guidance parameters are fixed as $K_H=5$, $N_{\mathrm{ho}}=2$, $N_{\mathrm{rec}}=3$, $({\alpha }_z,{\alpha }_d,{\alpha }_c,{\alpha }_l,{\alpha }_{\mathrm{sw}},{\alpha }_{\mathrm{safe}})=(1.0, 0.8, 0.8, 0.6, 0.2, 1.2)$, $({\beta }_q,{\beta }_l,{\beta }_z,{\beta }_h)=(1.0, 0.8, 0.8, 1.2)$, and $ϵ=0.1$, and remain unchanged throughout all training and evaluation scenarios.

Given identical locally available observations, neighborhood summaries, historical target-loss states, and previous associations, the interface produces the same guidance signal. Therefore, the task-guidance interface is an engineered and reproducible coordination mechanism rather than an independently optimized high-level policy. Its role is to incorporate explicit low-frequency cooperative information into both decentralized action generation and centralized value estimation.

3.3.3. Conditional Dec-POMDP Formulation

Based on the above definitions and following the classical modeling framework of decentralized partially observable decision processes, this paper formulates the multi-UAV multi-target cooperative tracking task as a task-guidance-conditioned Dec-POMDP:

$$\mathcal{M}=⟨\mathcal{S},\mathcal{O},{\mathcal{C}}^H,{\mathcal{A}}^L,\mathcal{P},{\mathcal{R}}^L,\gamma ⟩,$$

(21)

where $\mathcal{S}$ denotes the global state space, $\mathcal{O}$ denotes the local observation space, ${\mathcal{C}}^H$ denotes the task-guidance signal space, ${\mathcal{A}}^L$ denotes the lower-level continuous action space, $\mathcal{P}$ denotes the state-transition probability function, ${\mathcal{R}}^L$ denotes the lower-level reward function, and $\gamma \in (0, 1)$ denotes the lower-level discount factor.

During centralized training, the global state is represented as

$$S_t=\left(X_t^u,P_t^g,E_t,C_t^{\mathrm{com}},H_t\right),$$

(22)

where $X_t^u$ contains the states of all UAVs, $P_t^g$ contains the states of all ground targets, $E_t$ denotes the environmental information, $C_t^{\mathrm{com}}$ denotes the communication relationships among UAVs, and $H_t$ contains the historical tracking and task summaries. The complete target states and environmental information are used only by the centralized Critic, reward computation, and performance evaluation during training. They are unavailable to an individual UAV during decentralized execution.

During execution, UAV i uses only the locally available information ${\xi }_{i,t}^L$ defined in (13). Accordingly, the shared lower-level decentralized policy is expressed as

$$a_{i,t}^L\sim {\pi }_{\theta }^L\left(·\mid o_{i,t},c_{i,t}^H\right),$$

(23)

where $\theta$ denotes the parameters of the shared Actor. Let

$$A_t^L=\left\{a_{1,t}^L,a_{2,t}^L,\dots ,a_{N,t}^L\right\}$$

(24)

denote the joint lower-level action. The system transition is described by

$$S_{t+1}\sim \mathcal{P}\left(S_{t+1}\mid S_t,A_t^L,C_t^H\right).$$

(25)

The task-guidance signal does not directly serve as a physical control input. Instead, it indirectly affects the system evolution by conditioning the continuous actions generated by the lower-level policy. The physical state transition is ultimately determined by the UAV control actions, target motion, obstacle constraints, and communication-topology changes.

During centralized training, the conditional centralized Critic estimates the value function using the global state and the aggregated task-guidance signal:

$$V_{\varphi }\left(S_t,C_t^H\right),$$

(26)

where $\varphi$ denotes the Critic parameters and

$$C_t^H=\left\{c_{1,t}^H,c_{2,t}^H,\dots ,c_{N,t}^H\right\}.$$

(27)

The lower-level cooperative policy is optimized according to the team reward defined in Section 3.3.4, where the corresponding reward formulation and optimization objective are presented in detail.

The implementation correspondence: To establish a direct correspondence between the above conditional Dec-POMDP formulation and the simulation implementation, the local observation, task-guidance signal, continuous action, and centralized training state are represented by fixed-dimensional numerical vectors. The exact feature components, dimensions, and preprocessing procedures are summarized in Table 2.

Table 2. Implementation details of the conditional Dec-POMDP.

Category	Feature	Components	Dimension	Preprocessing
Actor input	Self state	${\overline{x}}^u,{\overline{y}}^u,{\overline{v}}^u,\sin {\psi }^u,\cos {\psi }^u$	5	Position normalized by map size; speed by $v_{\max }^u$
	Target	$\Delta {\mathbf{p}}^{ug},\Delta {\mathbf{v}}^{ug},z,q^{\mathrm{obs}},{\overline{\tau }}^{\mathrm{last}}$	$7M=28$	Position by $R_{\mathrm{sen}}$; velocity by $v_{\mathrm{rel}}^{\max }$; time by $T_{\mathrm{loss}}^{\max }$
	Environment	Nearest-obstacle features and four boundary distances	$3K_o+4=16$	Obstacle features by $R_{\mathrm{sen}}$; boundary distances by map size
	Neighbors	$\Delta {\mathbf{p}}^{uu},\Delta {\mathbf{v}}^{uu},{\mathbf{e}}^m,{\mathbf{e}}^z,s^{\mathrm{obs}},h$	$(N-1)(6+2M)=42$	Normalized relative states; binary and one-hot encoding
Task guidance	Local guidance	${\mathbf{e}}^m,{\mathbf{e}}^h,{\omega }^H,{\overline{r}}^{*},{\theta }^{*},{\psi }^{*}$	$M+9=13$	One-hot encoding; $r^{*}/R_{\mathrm{sen}}$; sine/cosine encoding for angles
Actor output	Continuous action	$a^{\mathrm{acc}},\omega$	2	Outputs in $[-1,1]$, then scaled to physical limits
Critic input	UAV states	Normalized states of all UAVs	$5N=20$	Same preprocessing as the Actor self-state
	Target states	Positions and velocities of all targets	$4M=16$	Position normalized by map size; velocity by $v_{\max }^g$
	Environment	Fixed obstacle slots and four boundary variables	$3K_{\max }+4=34$	Normalized values; unavailable obstacle slots zero-padded
	Communication	Flattened communication-adjacency matrix	$N^2=16$	Binary encoding
	Tracking history	Coverage deficit and normalized target-loss duration	$2M=8$	Clipped to $[0,1]$
	System guidance	Concatenation of all local guidance vectors	$N(M+9)=52$	Same preprocessing as the local guidance vector
Reward	Reward terms	$r^{\mathrm{trk}},r^{\mathrm{fov}},r^{\mathrm{rec}},p^{\mathrm{obs}},p^{\mathrm{col}},p^{\mathrm{sm}},p^{\mathrm{eng}}$	7	Weighted scalar summation

For example, under the 4-UAV/4-target configuration, $N=4$ and $M=4$. Each target and neighboring UAV is assigned a fixed-index feature slot. The $K_o=4$ nearest obstacles are retained as fixed-index slots in each local observation, whereas the centralized environmental representation is padded to $K_{\max }=10$ obstacle slots. For other team sizes, the corresponding UAV- and target-related feature slots are adjusted according to N and M, while the same feature ordering, padding rules, and normalization procedures are retained.

The input of the shared decentralized Actor is represented as

$$x_{i,t}^{\mathrm{Actor}}=\left[o_{i,t}^{\mathrm{self}},o_{i,t}^{\mathrm{tar}},o_{i,t}^{\mathrm{env}},o_{i,t}^{\mathrm{nei}},{\overline{c}}_{i,t}^H\right],$$

(28)

where the five feature blocks have dimensions 5, $7M$, $3K_o+4$, $(N-1)(6+2M)$, and $M+9$, respectively. Therefore, for $N=4$, $M=4$, and $K_o=4$, the final Actor input dimension is

$$d_{\mathrm{Actor}}=5+28+16+42+13=104.$$

(29)

The input of the conditional centralized Critic is represented as

$$x_t^{\mathrm{Critic}}=\left[{\overline{X}}_t^u,{\overline{P}}_t^g,{\overline{E}}_t,C_t^{\mathrm{com}},{\overline{H}}_t,{\overline{C}}_t^H\right],$$

(30)

where the six feature blocks correspond to all UAV states, all target states, the fixed-length environmental representation, the communication-adjacency matrix, the target-level tracking-history summary, and the aggregated task-guidance signal, respectively. For $N=4$, $M=4$, and $K_{\max }=10$, the final Critic input dimension is

$$d_{\mathrm{Critic}}=20+16+34+16+8+52=146.$$

(31)

The Actor outputs a two-dimensional normalized continuous action ${\tilde{a}}_{i,t}^L\in {[-1,1]}^2$, consisting of linear-acceleration and angular-velocity commands. Before being applied to the UAV dynamics, the two outputs are scaled by their corresponding physical limits.

The implemented reward consists of seven components: target tracking, field-of-view maintenance, target recovery, obstacle/boundary avoidance, inter-UAV separation, control smoothness, and energy consumption. The same reward definition and weighting coefficients are used for HGP-MAPPO, all baseline methods, and all ablation variants.

3.3.4. Lower-Level Reward and Optimization Objective

Since this paper focuses on the lower-level continuous control policy under task-guidance conditions, the reward function considers target tracking, field-of-view maintenance, target recovery, obstacle/boundary avoidance, inter-UAV separation, control smoothness, and energy consumption. Let $r_{i,t}^L$ denote the lower-level reward of the i-th UAV at time t, and let

$$R_t^L={\displaystyle \sum _{i=1}^N}{r}_{i,t}^L$$

(32)

denote the team reward of the system. The general form of the lower-level reward for an individual UAV is given by

$$r_{i,t}^L={\alpha }_1{r}_{i,t}^{trk}+{\alpha }_2{r}_{i,t}^{fov}+{\alpha }_3{r}_{i,t}^{rec}-{\alpha }_4{p}_{i,t}^{obs}-{\alpha }_5{p}_{i,t}^{col}-{\alpha }_6{p}_{i,t}^{sm}-{\alpha }_7{p}_{i,t}^{eng}.$$

(33)

where $r_{i,t}^{\mathrm{trk}}$ denotes the target-tracking reward, $r_{i,t}^{\mathrm{fov}}$ denotes the field-of-view maintenance reward, and $r_{i,t}^{\mathrm{rec}}$ denotes the target-recovery or rediscovery reward; $p_{i,t}^{\mathrm{obs}}$ denotes the obstacle and boundary penalty, $p_{i,t}^{\mathrm{col}}$ denotes the inter-UAV collision-risk penalty, $p_{i,t}^{\mathrm{sm}}$ denotes the control-smoothness penalty, and $p_{i,t}^{\mathrm{eng}}$ denotes the energy-consumption penalty. The coefficients ${\alpha }_1,\dots ,{\alpha }_7$ represent the corresponding reward weights.

Based on the team reward defined above, the optimization objective is to learn a shared lower-level policy, ${\pi }_{\theta }^L$, conditioned on the task-guidance signals by maximizing the expected discounted team return:

$$J\left(\theta \right)={\mathbb{E}}_{{\pi }_{\theta }^L}\left[{\displaystyle \sum _{t=0}^{T-1}}{\gamma }^t{R}_t^L\right].$$

(34)

where T denotes the episode horizon and $\gamma$ denotes the discount factor. It should be emphasized that the optimization variable in the above objective is only the lower-level policy parameter $\theta$. The task guidance signals participate in trajectory generation and value estimation as conditional variables, while their generation process is not independently optimized.

4. HGP-MAPPO: Task-Guidance-Conditioned Policy Learning and Residual Value Correction

4.1. Overall Framework of the Proposed Method

Based on the task-guidance-conditioned Dec-POMDP formulated in Section 3, this paper proposes HGP-MAPPO for partially observable multi-UAV multi-target cooperative tracking. As shown in Figure 1, HGP-MAPPO extends MAPPO by introducing low-frequency task-guidance conditioning and a Gaussian-process-based residual correction mechanism [53]. The former reduces the coupling between time-varying cooperative responsibilities and continuous control, whereas the latter improves localized Critic value estimation under complex partially observable conditions. The methodological contribution lies in integrating these two mechanisms within the CTDE framework for persistent cooperative multi-target tracking.

Figure 1. Overall framework of the proposed HGP-MAPPO for multi-UAV cooperative multi-target tracking.

The task guidance is not generated by an independently optimized high-level policy. Instead, target association, task priority, handover/recovery cues, and desired observation geometry are represented as conditional variables for the Actor and Critic. Therefore, HGP-MAPPO does not require an additional high-level reward, termination rule, teacher policy, or policy-distillation process. Its distinctions from conventional hierarchical, task-conditioned, and teacher–student frameworks are summarized in Table 1.

HGP-MAPPO consists of a task-guidance interface, a shared decentralized Actor, a conditional centralized Critic, and a GP residual correction module. The Actor generates continuous actions from local observations, neighbor communication summaries, and local guidance signals, while the centralized Critic exploits the global training state and system-level guidance information for value estimation. Rather than replacing the neural Critic, the sparse GP models localized residuals between Critic predictions and target values, with posterior information used to regulate residual correction.

HGP-MAPPO follows centralized training with decentralized execution. During training, the conditional centralized Critic and GP residual correction module assist value estimation and MAPPO policy optimization. During execution, both the Critic and GP module are removed, and each UAV independently generates continuous actions using only its local observation, neighbor communication summary, and associated task-guidance signal. Consequently, the proposed framework preserves decentralized execution without additional GP inference or centralized communication overhead. The following subsections detail the task-guidance-conditioned Actor–Critic architecture, policy optimization, and GP residual correction mechanism.

4.2. Lower-Level Actor-Critic Architecture Under Task Guidance Conditions

Based on the execution-stage information structure defined in Section 3, this section develops a lower-level Actor-Critic architecture conditioned on task guidance. In this architecture, the lower-level Actor generates continuous control actions based on local information during decentralized execution, while the lower-level Critic estimates the value function during centralized training by exploiting the global training state and system-level task guidance signals.

According to the execution-stage information structure defined in Section 3.3, the lower-level Actor takes $o_{i,t}$ and $c_{i,t}^H$ as inputs. To integrate local state information with task conditions, the lower-level Actor first encodes the two inputs and obtains the lower-level control feature conditioned on task guidance:

$$h_{i,t}^{\pi }=F_{\theta }\left(o_{i,t},c_{i,t}^H\right).$$

(35)

where $F_{\theta }(·)$ denotes the feature extraction and fusion network of the lower-level Actor, and $h_{i,t}^{\pi }$ denotes the fused policy feature. This feature incorporates both the UAV local observation information and the conditional information provided by the task guidance signal, such as target association, handover recovery, or desired observation geometry. The lower-level Actor outputs a continuous action distribution rather than selecting from a discrete action set. For the i-th UAV, the action generation process can be expressed as

$$a_{i,t}^L\sim {\pi }_{\theta }^L\left(·\mid h_{i,t}^{\pi }\right),a_{i,t}^L={\left[a_{i,t}^{acc},{\omega }_{i,t}\right]}^{\top }.$$

(36)

where $a_{i,t}^{acc}$ and ${\omega }_{i,t}$ denote the linear acceleration and angular velocity control inputs, respectively. The action consists of the linear acceleration and angular velocity, which are used to control the velocity and heading variations of the UAV in the two-dimensional plane. During training, policy exploration can be performed by sampling from the action distribution, whereas during execution, the mean action or deterministic output can be adopted to improve control stability. All UAVs share the same lower-level Actor network parameters, $\theta$. Therefore, the joint lower-level policy of the system can be written as

$${\pi }_{\theta }^L\left(A_t^L\mid O_t,C_t^H\right)={\displaystyle \prod _{i=1}^N}{\pi }_{\theta }^L\left(a_{i,t}^L\mid o_{i,t},c_{i,t}^H\right).$$

(37)

where $O_t$ denotes the set of local observations of all UAVs. The parameter-sharing mechanism helps improve sample efficiency and enables different UAVs to generate differentiated behaviors under the same policy architecture according to their respective local observations and task guidance signals.

Unlike the lower-level Actor, the lower-level Critic is used only during the training stage. To exploit the global information available during centralized training, this paper constructs a lower-level conditional centralized Critic, which takes the global training state $S_t$ and the system-level task guidance signal $C_t^H$ as inputs to estimate the lower-level state value conditioned on task guidance, as shown:

$$V_{\varphi }^L\left(S_t,C_t^H\right)=G_{\varphi }\left(S_t,C_t^H\right).$$

(38)

where $G_{\varphi }(·)$ denotes the lower-level Critic network, and $\varphi$ represents the Critic parameters. This value function is used for subsequent advantage estimation and policy updating, and is not invoked during execution.

In summary, the lower-level Actor-Critic architecture integrates task-level coordination requirements into continuous control through task guidance signals. This enables the Actor to generate task-conditioned actions during decentralized execution and allows the Critic to perform centralized value estimation under task guidance during training. The resulting architecture provides a basis for the subsequent MAPPO-based policy optimization and GP-based residual value correction.

4.3. Lower-Level Policy Optimization Based on MAPPO

After constructing the lower-level Actor-Critic architecture conditioned on task guidance, this paper adopts the MAPPO framework to optimize the lower-level shared policy. During training, the lower-level Actor interacts with the environment to generate trajectory data, while the lower-level conditional Critic estimates the state value conditioned on task guidance. Based on the estimated value, the advantage function is constructed to update the shared policy. During training, the system first interacts with the environment using the current lower-level policy and collects a rollout of length $T_{\mathrm{step}}$. For each time step, t, the sampled trajectory can be represented as

$$\mathcal{D}={\left\{S_t,C_t^H,O_t,A_t^L,R_t^L,S_{t+1},C_{t+1}^H,O_{t+1}\right\}}_{t=0}^{T_{\mathrm{step}}-1}.$$

(39)

where $S_t$ denotes the global state available during training, $C_t^H$ denotes the system-level task guidance signal, $O_t=\{o_{1,t},o_{2,t},\dots ,o_{N,t}\}$ denotes the set of local observations of all UAVs, $A_t^L=(a_{1,t}^L,a_{2,t}^L,\dots ,a_{N,t}^L)$ denotes the joint lower-level action, and $R_t^L$ denotes the immediate lower-level team reward. It should be noted that $S_t$ and $C_t^H$ are used only for value estimation during training, whereas the lower-level Actor still relies solely on the individual UAV’s own local observation and task guidance signal during execution.

Based on the lower-level conditional Critic $V_{\varphi }^L(S_t,C_t^H)$, the temporal-difference error can be constructed as

$${\delta }_t^L=R_t^L+\gamma V_{\varphi }^L\left(S_{t+1},C_{t+1}^H\right)-V_{\varphi }^L\left(S_t,C_t^H\right).$$

(40)

The temporal-difference (TD) error reflects the estimation bias of the current Critic with respect to the lower-level state value conditioned on task guidance. The lower-level advantage function is further obtained using generalized advantage estimation (GAE) [54]:

$${\widehat{A}}_t^L={\displaystyle \sum _{l=0}^{T_{\mathrm{step}}-t-1}}{\left(\gamma \lambda \right)}^l{\delta }_{t+l}^L.$$

(41)

where $\lambda$ denotes the GAE parameter, which is used to achieve a trade-off between bias and variance. Since this paper adopts a team reward and a centralized conditional value function, the resulting advantage estimates are used to guide the parameter update of the shared Actor for all UAVs. To constrain the magnitude of policy updates, MAPPO adopts the PPO clipped surrogate objective. For the i-th UAV, the probability ratio between the new and old policies is defined as

$${\rho }_{i,t}\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }^L\left(a_{i,t}^L\mid o_{i,t},c_{i,t}^H\right)}{{\pi }_{{\theta }_{old}}^L\left(a_{i,t}^L\mid o_{i,t},c_{i,t}^H\right)}}.$$

(42)

where ${\theta }_{old}$ denotes the old policy parameters used for sampling the trajectory. Based on this probability ratio, the clipped policy objective of the lower-level Actor can be written as

$${\mathcal{L}}_{actor}\left(\theta \right)={\mathbb{E}}_{\mathcal{D}}\left[{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\min \left({\rho }_{i,t}\left(\theta \right){\widehat{A}}_t^L,\mathrm{clip}\left({\rho }_{i,t}\left(\theta \right),1-\epsilon ,1+\epsilon \right){\widehat{A}}_t^L\right)\right].$$

(43)

where $\epsilon$ denotes the clipping coefficient. This objective improves the stability of the training process by constraining the magnitude of policy changes between the new and old policies. The training objective of the lower-level Critic is to minimize the error between the value prediction and the return target. Let ${\widehat{R}}_t^L$ denote the discounted return computed from the sampled trajectory or the GAE-based bootstrap return. Then, the Critic loss is defined as:

$${\mathcal{L}}_{critic}\left(\varphi \right)={\mathbb{E}}_{\mathcal{D}}\left[{\left(V_{\varphi }^L\left(S_t,C_t^H\right)-{\widehat{R}}_t^L\right)}^2\right].$$

(44)

This loss encourages the lower-level conditional Critic to learn the long-term return conditioned on the given global training state and task guidance signal. Since the task guidance signal changes the control objective and reward structure under the same physical state, the Critic needs to receive both $S_t$ and $C_t^H$ as inputs to improve the accuracy of value estimation.

To encourage the policy to maintain a moderate level of exploration in the early stage of training, an entropy regularization term can be incorporated into the Actor objective. Combining the policy objective, the value function loss, and the entropy regularization term, the training loss function of the lower-level MAPPO can be expressed as

$${\mathcal{L}}_{MAPPO}=-{\mathcal{L}}_{actor}\left(\theta \right)+c_v{\mathcal{L}}_{critic}\left(\varphi \right)-c_e\mathcal{H}\left({\pi }_{\theta }^L\right).$$

(45)

where $c_v$ and $c_e$ denote the weights of the value loss and entropy regularization term, respectively, and $\mathcal{H}\left({\pi }_{\theta }^L\right)$ denotes the entropy of the lower-level policy. During training, the Actor and Critic parameters are updated by minimizing the loss function. Specifically, the Actor is stably updated according to the clipped policy objective, while the Critic improves its state-value estimation capability based on the value regression objective.

Overall, the lower-level MAPPO optimization process consists of trajectory sampling, value estimation, advantage computation, clipped Actor update, and value regression update of the Critic. This mechanism provides a stable baseline training framework for the lower-level continuous control policy. However, under complex conditions such as target occlusion, task handover, target reappearance, and changes in the communication neighborhood, relying solely on the neural-network-based Critic may still result in localized value estimation bias. To further enhance the stability of value estimation, the next section introduces a Gaussian-process-based residual correction mechanism built upon the lower-level conditional Critic.

4.4. Gaussian-Process-Based Residual Correction for the Lower-Level Conditional Critic

In Section 4.3, the lower-level conditional Critic estimates the task-guidance-conditioned state value and provides advantage estimates for updating the shared Actor. However, target occlusion, task handover, target reappearance, and changes in the communication neighborhood may cause localized abrupt variations in state transitions and reward feedback. Consequently, the neural-network-based Critic may exhibit local value-estimation bias in complex, partially observable state regions. To mitigate this problem, an inducing-point sparse Gaussian-process regression approximation is introduced to model and compensate for the residual errors of the lower-level conditional Critic.

4.4.1. Sparse GP Residual Modeling and Implementation

To avoid directly applying the GP to the high-dimensional global state, the input of the residual model is extracted from the last hidden layer of the lower-level conditional Critic:

$$x_t^V=F_V\left(S_t,C_t^H\right)\in {\mathbb{R}}^{d_v},$$

(46)

where $F_V(·)$ denotes the value-feature extractor of the conditional Critic, and $d_v=64$ is the fixed dimension of the GP input feature. The value-feature dimension remains unchanged for different team sizes, although the input dimension and encoding cost of the centralized Critic may increase as the global-state dimension increases.

The GP training target is defined as the detached residual between the return target and the current Critic estimate:

$$e_t=\mathrm{sg}\left[{\widehat{R}}_t^L-V_{\varphi }^L\left(S_t,C_t^H\right)\right],$$

(47)

where $\mathrm{sg}[·]$ denotes the stop-gradient operation. Therefore, the GP fitting process does not propagate gradients to the conditional Critic.

The residual function is modeled by a Gaussian process [53]:

$$e\left(x\right)\sim \mathcal{GP}\left(0,k\left(x,x^{\prime }\right)\right),$$

(48)

where $k(x,x^{\prime })$ denotes the covariance function between two value features. In this study, the radial basis function kernel is adopted:

$$k\left(x,x^{\prime }\right)={\sigma }_f^2\exp \left(-{\displaystyle \frac{{∥x-x^{\prime }∥}^2}{2l^2}}\right),$$

(49)

where ${\sigma }_f^2$ denotes the output variance, and l denotes the kernel length scale.

A full GP becomes computationally expensive as the number of residual samples increases. Therefore, an inducing-point sparse GP regression approximation is employed [54]. Instead of factorizing the full covariance matrix constructed from all residual samples, the approximate posterior is constructed using the covariance between the residual samples and a finite set of inducing points, together with the covariance among the inducing points. Let the inducing-point set be

$$Z=\left\{z_1,z_2,\dots ,z_{M_z}\right\},M_z\ll \left|{\mathcal{D}}_{GP}\right|,$$

(50)

where $M_z=128$ denotes the number of inducing points and ${\mathcal{D}}_{GP}$ denotes the residual buffer. A first-in–first-out residual buffer with capacity $B_{GP}=8192$ stores the most recent value-feature and residual pairs:

$${\mathcal{D}}_{GP}=\left\{\left(x_j^V,e_j\right)\right\}.$$

(51)

At each GP update, $N_{GP}=2048$ residual pairs are sampled from the current buffer. The inducing points are selected from the current GP-update batch using the K-means++ procedure and are refreshed, together with the GP model. Consequently, the GP is fitted using a bounded recent residual set rather than all historical residual samples.

The sparse GP is updated every $K_{GP}=1000$ global training steps. Here, one global training step denotes one synchronized interaction step of all parallel environments, rather than one individual transition from a single environment. Because GP fitting is performed after trajectory collection, the update is executed at the first rollout boundary at which the accumulated step counter reaches the prescribed interval. This definition is also used consistently in Algorithm 1 and the GP parameter table.

The RBF-kernel length scale, output variance, noise variance, and GP regularization coefficient are determined through preliminary calibration and remain fixed during the main Actor–Critic training process. Before GP fitting, the value features and residual targets are whitened using the statistics of the current GP-update batch. Cholesky decomposition is used for kernel-matrix factorization, and a jitter term of ${10}^{-6}$ is added to the diagonal of the covariance matrix to improve numerical stability. The posterior variance is constrained to be nonnegative to avoid numerical artifacts.

For an input feature, $x_t^V$, the sparse GP produces the posterior mean ${\mu }_{GP}\left(x_t^V\right)$ and posterior variance ${\sigma }_{GP}^2\left(x_t^V\right)$. The posterior mean represents the estimated local Critic residual, whereas the posterior variance quantifies the reliability of the correction. The uncertainty-dependent correction weight is defined as

$${\omega }_t^{GP}={\displaystyle \frac{1}{1+{\beta }_{\sigma }{\sigma }_{GP}^2\left(x_t^V\right)}},$$

(52)

where ${\beta }_{\sigma }$ denotes the uncertainty-regulation coefficient. A larger posterior variance produces a smaller correction weight, thereby suppressing unreliable compensation in poorly supported state regions.

To prevent an excessively large residual estimate from dominating the neural Critic output, the weighted GP correction is constrained as

$${\Delta }_t^{GP}=\mathrm{clip}\left({\omega }_t^{GP}{\mu }_{GP}\left(x_t^V\right),-r_{\max },r_{\max }\right),$$

(53)

where $r_{\max }=20$ denotes the maximum correction magnitude.

The corrected lower-level conditional value is then given by

$${\widehat{V}}^L\left(S_t,C_t^H\right)=V_{\varphi }^L\left(S_t,C_t^H\right)+{\Delta }_t^{GP}.$$

(54)

For each TD target, the next-state feature

$$x_{t+1}^V=F_V\left(S_{t+1},C_{t+1}^H\right)$$

(55)

is evaluated using the same frozen GP snapshot. Therefore, the current- and next-state values within the same TD target are corrected consistently.

Algorithm 1 HGP-MAPPO: Task-Guided MAPPO with Gaussian-Process-Based Value Correction

Input:  Multi-UAV tracking environment; shared Actor ${\pi }_{\theta }^L$; conditional Critic $V_{\phi }^L$; total number of environment transitions $N_{\mathrm{total}}$; number of parallel environments $N_{\mathrm{env}}$; rollout length $T_{\mathrm{step}}$; PPO epochs $K_{\mathrm{ppo}}$; GP update interval $K_{\mathrm{GP}}$ measured in synchronized global interaction steps; residual-buffer capacity $B_{\mathrm{GP}}$; GP update batch size $N_{\mathrm{GP}}$; number of inducing points $M_{\mathrm{z}}$. Output:  Trained low-level shared Actor policy ${\pi }_{\theta }^L$.   1: Initialize $\theta$, $\varphi$, the sparse GP trajectory buffer $\mathcal{D}$, FIFO residual buffer ${\mathcal{D}}_{\mathrm{GP}}$, $n_{\mathrm{step}}=0$, $q_{\mathrm{step}}=0$, and $q_{\mathrm{GP}}^{\mathrm{last}}=0$.   2: Initialize the GP correction as zero before the first valid GP update.   3: Reset all parallel environments and obtain $S_t$, $O_t$, and $C_t^H$.   4: while  $n_{\mathrm{step}}<N_{\mathrm{total}}$  do   5:       for $t=1$ to $T_{\mathrm{step}}$ do   6:             For each UAV i, sample $a_{i,t}^L\sim {\pi }_{\theta }^L\left(·\mid o_{i,t},c_{i,t}^H\right)$.   7:             Execute the joint action $A_t^L$, and observe $R_t^L$, $S_{t+1}$, $O_{t+1}$, and $C_{t+1}^H$.   8:             Store $\left(S_t,C_t^H,O_t,A_t^L,R_t^L,S_{t+1},C_{t+1}^H,O_{t+1}\right)$ in $\mathcal{D}$.   9:             Set $n_{\mathrm{step}}\leftarrow n_{\mathrm{step}}+N_{\mathrm{env}}$. 10:             Set $q_{\mathrm{step}}\leftarrow q_{\mathrm{step}}+1$. 11:       end for 12:       Compute return targets and Critic values; obtain the current and next-state value features via (46) and (55), respectively. 13:       Construct the detached residuals as defined in (47). 14:       Append $\left(x_t^V,e_t\right)$ to ${\mathcal{D}}_{\mathrm{GP}}$, and discard the oldest samples when $\left|{\mathcal{D}}_{\mathrm{GP}}\right|>B_{\mathrm{GP}}$. 15:       if $q_{\mathrm{step}}-q_{\mathrm{GP}}^{\mathrm{last}}\ge K_{\mathrm{GP}}$ and $\left|{\mathcal{D}}_{\mathrm{GP}}\right|>N_{\mathrm{GP}}$ then 16:             Sample $N_{\mathrm{GP}}$ residual pairs from ${\mathcal{D}}_{\mathrm{GP}}$. 17:             Select $M_z$ inducing points using K-means++ and update the sparse GP. 18:             Freeze the updated GP snapshot and set $q_{\mathrm{GP}}^{\mathrm{last}}\leftarrow q_{\mathrm{step}}$. 19:       end if 20:       Evaluate the same frozen GP snapshot for $x_t^V$ and $x_{t+1}^V$, and detach the posterior means, posterior variances, and correction terms from the Actor–Critic computational graph. 21:       Construct the uncertainty-weighted and clipped GP corrections, and compute the corrected current- and next-state values using $${\omega }_{\tau }^{\mathrm{GP}}={\displaystyle \frac{1}{1+{\beta }_{\sigma }{\sigma }_{\mathrm{GP}}^2\left(x_{\tau }^V\right)}},{\Delta }_{\tau }^{\mathrm{GP}}=\mathrm{clip}\left({\omega }_{\tau }^{\mathrm{GP}}{\mu }_{\mathrm{GP}}\left(x_{\tau }^V\right),-r_{\max },r_{\max }\right),$$ $${\widehat{V}}_{\varphi }^L\left(S_{\tau },C_{\tau }^H\right)=V_{\varphi }^L\left(S_{\tau },C_{\tau }^H\right)+{\Delta }_{\tau }^{\mathrm{GP}},\tau \in [t,t+1].$$ 22:       Compute the corrected TD errors and GP-assisted advantages using (56) and (57), respectively. 23:       for $k=1$ to $K_{\mathrm{PPO}}$ do 24:             Sample a mini-batch from $\mathcal{D}$. 25:             Update $\theta$ using the clipped MAPPO objective with the GP-assisted advantage estimate. 26:             Update $\varphi$ using the corrected Critic loss defined in (59), while keeping all GP posterior outputs detached. 27:       end for 28:       Clear $\mathcal{D}$ while retaining the FIFO residual buffer ${\mathcal{D}}_{\mathrm{GP}}$. 29: end while       Execution stage: 30: Remove the conditional Critic and the GP correction module. 31: At each execution step, each UAV independently generates its continuous action using only locally available information.

4.4.2. Staged Update and Bounded Perturbation Analysis

The GP residual model is updated in a staged manner rather than being jointly optimized with the conditional Critic through end-to-end backpropagation. First, the return targets, Critic estimates, value features, and detached residual samples are computed from the collected trajectories. The sparse GP is then periodically updated using the first-in–first-out (FIFO) residual buffer.

During each subsequent PPO optimization window, the GP posterior parameters are held fixed. The posterior mean, posterior variance, and weighted correction terms are detached from the Actor–Critic computational graph. Therefore, no policy or value gradient is propagated through the GP model, and the Actor retains the standard clipped-MAPPO update form. This procedure can be interpreted as a piecewise-stationary residual-correction scheme that avoids simultaneous mutual adaptation between the conditional Critic and the GP residual model.

The corrected TD error is calculated as

$${\widehat{\delta }}_t^L=R_t^L+\gamma {\widehat{V}}^L\left(S_{t+1},C_{t+1}^H\right)-{\widehat{V}}^L\left(S_t,C_t^H\right),$$

(56)

and the GP-assisted advantage estimate is obtained using

$${\widehat{A}}_{t,GP}^L={\displaystyle \sum _{l=0}^{T-t-1}}{\left(\gamma \lambda \right)}^l{\widehat{\delta }}_{t+l}^L.$$

(57)

The corrected advantage estimate replaces the original advantage estimate in the clipped Actor objective. During the Critic update, the GP correction is treated as a detached constant, and the corrected value-regression loss is defined as

$${\mathcal{L}}_{\mathrm{critic}}^{GP}\left(\varphi \right)={\mathbb{E}}_{\mathcal{D}}\left[{\left({\widehat{V}}^L\left(S_t,C_t^H\right)-{\widehat{R}}_t^L\right)}^2\right].$$

(58)

Consequently, the gradient of the Critic loss is propagated only through the neural Critic estimate $V_{\varphi }^L$, rather than through the GP posterior.

To characterize the effect of GP correction on advantage estimation, let

$${\epsilon }_t={\widehat{V}}^L\left(S_t,C_t^H\right)-V^{\pi }\left(S_t,C_t^H\right),$$

(59)

where $V^{\pi }(S_t,C_t^H)$ denotes the true value under the current policy, and ${\epsilon }_t$ denotes the corrected value-estimation error.

The difference between the corrected TD error and the TD error based on the true policy value satisfies

$${\widehat{\delta }}_t^L-{\delta }_t^{\pi }=\gamma {\epsilon }_{t+1}-{\epsilon }_t.$$

(60)

If

$$\left|{\epsilon }_t\right|\le {\epsilon }_{\max },{\epsilon }_{\max }=\underset{t}{\max }\left|{\epsilon }_t\right|,$$

(61)

the resulting GAE perturbation is bounded by

$$\left|{\widehat{A}}_{t,GP}^L-A_t^{\pi }\right|\le {\displaystyle \frac{1+\gamma }{1-\gamma \lambda }}{\epsilon }_{\max }.$$

(62)

This result indicates that the perturbation introduced into the GAE estimate remains bounded when the corrected value-estimation error is bounded. Moreover, posterior-uncertainty weighting and correction-magnitude clipping suppress unreliable or excessively large GP corrections, thereby limiting their influence on the policy update. This result provides a bounded-perturbation argument rather than a global convergence guarantee for the complete nonlinear Actor–Critic training process.

Because the GP residual target changes with both the Actor and the Critic, a frozen GP snapshot may become temporarily stale between two consecutive GP updates. Let $e^{\left(q\right)}\left(x\right)$ denote the residual function corresponding to the q-th synchronized global training step, and assume that

$$\underset{x}{\sup }\left|e^{(q+1)}\left(x\right)-e^{\left(q\right)}\left(x\right)\right|\le L_e,$$

(63)

where $L_e$ denotes the maximum residual-function drift between two consecutive global training steps.

Let ${\overline{K}}_{GP}$ denote the actual maximum number of global training steps for which one GP snapshot remains frozen. The additional residual mismatch caused by model staleness then satisfies

$${\Delta }_{\mathrm{stale}}\le L_e{\overline{K}}_{GP}.$$

(64)

When GP fitting is triggered only at rollout boundaries,

$${\overline{K}}_{GP}\le K_{GP}+T_{\mathrm{step}}-1,$$

(65)

where $T_{\mathrm{step}}$ is the rollout length. Thus, $K_{GP}$ remains the prescribed update interval, whereas ${\overline{K}}_{GP}$ accounts for the small implementation delay introduced by rollout-based training.

A smaller $K_{GP}$ improves the timeliness of residual correction but increases training cost, whereas a larger $K_{GP}$ reduces the update frequency but may increase the mismatch caused by GP staleness. The adopted value $K_{GP}=1000$, therefore, represents a trade-off between correction timeliness and computational efficiency, which is further examined through parameter-sensitivity experiments.

4.4.3. Computational Complexity and Scalability

Let $B_{GP}$ denote the capacity of the FIFO residual buffer, $N_{GP}$ the number of residual samples used in each GP update, $M_z$ the number of inducing points, $d_v$ the dimension of the value feature, and $K_{GP}$ the prescribed GP-update interval measured in synchronized global training steps.

A full GP fitted using $N_{GP}$ residual samples requires $\mathcal{O}\left(N_{GP}^3\right)$ time and $\mathcal{O}\left(N_{GP}^2\right)$ memory because of the factorization and storage of the full covariance matrix. Such costs become prohibitive when residual samples are continuously generated during long-horizon multi-agent reinforcement learning. Therefore, the inducing-point sparse approximation described in Section 4.4.1 is employed.

For the adopted sparse GP, constructing the covariance terms between the $N_{GP}$ residual samples and the $M_z$ inducing points requires $\mathcal{O}\left(N_{GP}{M}_z{d}_v\right)$ operations. The dominant sparse-matrix operations require $\mathcal{O}\left(N_{GP}{M}_z^2\right)$, whereas factorizing the inducing-point covariance matrix requires $\mathcal{O}\left(M_z^3\right)$. Accordingly, the time complexity of one GP update is $\mathcal{O}\left(N_{GP}{M}_z{d}_v+N_{GP}{M}_z^2+M_z^3\right)$.

The FIFO residual buffer requires $\mathcal{O}\left(B_{GP}{d}_v\right)$ memory for storing value features and residual targets, while the GP-update workspace requires $\mathcal{O}(N_{GP}{M}_z+M_z^2)$ memory. Therefore, the total additional memory complexity is

$$\mathcal{O}\left(B_{GP}{d}_v+N_{GP}{M}_z+M_z^2\right).$$

(66)

The inducing points are refreshed through the K-means++ procedure over a fixed-size GP-update batch. Because $N_{GP}$, $M_z$, and $d_v$ are bounded in the implementation, the inducing-point selection cost is also bounded and is incurred only during periodic GP updates. It does not cause the residual training set to grow with the total number of historical training samples.

Because the sparse GP is updated only once every $K_{GP}$ global training steps, its amortized computational complexity per global training step is $\mathcal{O}\left({\displaystyle \frac{N_{GP}{M}_z{d}_v+N_{GP}{M}_z^2+M_z^3}{K_{GP}}}\right)$. Under the adopted settings, $N_{GP}=2048$, $M_z=128$, $d_v=64$, $K_{GP}=1000$, the dominant algebraic term count of one sparse-GP update is approximately $2048\times 128\times 64+2048\times {128}^2+{128}^3\approx 5.24\times {10}^7$, excluding implementation-dependent constants and the inducing-point selection operations. Since this cost is incurred periodically rather than at every environment interaction, it is amortized over the Actor–Critic training process. The GP operates on a fixed 64-dimensional value feature extracted by the conditional Critic rather than on the raw global state. Therefore, with fixed $N_{GP}$, $M_z$, and $d_v$, the dimensions of the GP covariance matrices do not directly increase with the raw state dimension or the number of UAVs. Nevertheless, the overall training cost is not independent of team size, since larger teams increase the input dimension and encoding cost of the centralized Critic and may induce more complex residual distributions. For substantially larger teams, additional residual samples or inducing points may, therefore, be required to maintain correction accuracy.

The residual-buffer capacity, GP-update batch size, feature dimension, and number of inducing points are fixed during training, while the FIFO mechanism removes older residual samples. Hence, GP-related storage and matrix-computation costs remain bounded with increasing training duration. Moreover, the GP module is used only during centralized training and is removed together with the Critic during decentralized execution, introducing no additional online inference overhead.

4.5. Algorithmic Procedure of HGP-MAPPO

The overall procedure of HGP-MAPPO is summarized in Algorithm 1. During centralized training, task-guidance-conditioned policy learning and sparse GP residual correction are integrated into MAPPO optimization. During decentralized execution, the Critic and GP module are removed, and each UAV independently generates continuous actions using only locally available information.

5. Experimental Results and Performance Evaluation

To validate the effectiveness of HGP-MAPPO in partially observable persistent cooperative multi-target tracking by multiple UAVs, this section conducts simulation experiments based on the task model established in Section 3 and the methodological framework developed in Section 4. The experiments include environment and parameter settings, baseline algorithms and evaluation metrics, analysis of the training process, tests in standard and densely obstructed scenarios, and ablation studies of key modules. Unlike one-shot target capture or path-reaching tasks, this paper focuses more on persistent tracking capability under limited sensing, constrained communication, obstacle-induced occlusion, and dynamic target motion. Therefore, in addition to cumulative reward, the performance of the proposed algorithm is comprehensively evaluated in terms of target retention rate, UAV–target distance, value estimation error, and training stability.

5.1. Experimental Environment and Evaluation Metrics

5.1.1. Experimental Settings

To evaluate the performance of multi-UAV multi-ground-target cooperative tracking under partially observable conditions, this paper constructs a custom-built software simulation environment based on a $100\mathrm{m}\times 100\mathrm{m}$ two-dimensional mission area with fixed-altitude UAV kinematics. Full-scale outdoor flight tests are not conducted at the current stage because of flight-safety considerations, hardware and site limitations, and the difficulty of repeatedly constructing controllable multi-UAV multi-target tracking scenarios. Nevertheless, the adopted software simulation environment provides a safe, repeatable, and controllable validation platform by incorporating homogeneous UAV motion constraints, mobile ground targets, static obstacles, mission boundaries, limited sensing ranges, and field-of-view constraints. Therefore, the proposed experimental setting enables the cooperative tracking performance of the evaluated algorithms to be examined under controlled mission constraints.

Inter-UAV neighborhood information is assumed to be exchanged reliably and without delay within the prescribed communication radius $R_{\mathrm{com}}$, and the resulting local task-guidance signals are directly supplied to the corresponding UAV policies. Physical-layer communication effects, including packet loss, bandwidth limitations, and asynchronous message arrival, are not explicitly modeled; therefore, the reported results do not demonstrate robustness to communication imperfections.

Two types of scenarios are considered in the experiments: a standard-obstacle scenario and a densely obstructed scenario. The standard scenario consists of four UAVs, four mobile ground targets, and four static obstacles. The densely obstructed scenario consists of six UAVs, six mobile ground targets, and eight static obstacles, thereby increasing both the cooperative tracking scale and the frequency of obstacle-induced occlusions. The main simulation parameters are listed in Table 3.

Table 3. Main simulation environment parameters.

Parameter	Symbol	Value
Simulation time step	$\Delta t$	$0.2\mathrm{s}$
Maximum number of steps per episode	T	400
Maximum UAV speed	$v_{\max }^u$	$5\mathrm{m}/\mathrm{s}$
Maximum UAV linear acceleration	$a_{\max }$	$1.0\mathrm{m}/{\mathrm{s}}^2$
Maximum UAV angular velocity	${\omega }_{\max }$	$\pi /4\mathrm{rad}/\mathrm{s}$
Maximum ground-target speed	$v_{\max }^g$	$2.0\mathrm{m}/\mathrm{s}$
UAV sensing radius	$R_{\mathrm{sen}}$	$18\mathrm{m}$
UAV communication radius	$R_{\mathrm{com}}$	$35\mathrm{m}$
Field-of-view angle	$\mathrm{\Theta }$	${120}^{\circ }$
Desired observation distance	$d_{des}$	$10\mathrm{m}$
Minimum UAV–UAV safety distance	$d_{safe}^u$	$3.0\mathrm{m}$
UAV–obstacle safety distance	$d_{safe}^o$	$1.5\mathrm{m}$
Number of steps for target-loss determination	$T_{lost}$	10

HGP-MAPPO is implemented within the centralized training with decentralized execution framework. During training, the lower-level conditional Critic estimates state values by exploiting the global training state and system-level task guidance signals. During execution, only the lower-level shared Actor is retained, and each UAV independently generates continuous control actions based on its local observations, neighborhood communication summaries, and local task guidance signals. In the experiments, the task guidance signals are generated using heuristic rules based on currently available information, including target visibility, UAV–target relative distance, target-loss status, and safety constraints, without incorporating future target trajectories or globally optimal action labels. The general training parameters and sparse GP settings are summarized in Table 4 and Table 5, respectively. All experiments were implemented in Python 3.10 with PyTorch 2.1.0 and CUDA 12.1 on a workstation equipped with an Intel Core i9-12900K CPU, an NVIDIA RTX 3090 GPU, and 64 GB RAM. Wall-clock time and peak GPU memory were measured under identical hardware and training configurations. The minimum allowable inter-UAV distance is set to $d_{\mathrm{safe}}^u=3.0$. The minimum obstacle-clearance metric is measured relative to the boundary of the safety-expanded obstacle region, such that a positive value indicates that the UAV remains outside the predefined unsafe region.

Table 4. General training parameter settings.

Parameter	Symbol	Value
Discount factor	$\gamma$	$0.99$
GAE parameter	$\lambda$	$0.95$
Actor learning rate	${\alpha }_{\mathrm{actor}}$	$3.0\times {10}^{-4}$
Critic learning rate	${\alpha }_{\mathrm{critic}}$	$1.0\times {10}^{-4}$
Entropy coefficient	$\beta$	$0.01$
PPO clipping parameter	$ϵ$	$0.2$
Number of parallel environments	$N_{\mathrm{env}}$	32
Rollout length	$T_{\mathrm{step}}$	256
Number of PPO epochs	$K_{\mathrm{PPO}}$	10
Mini-batch size	$N_{\mathrm{mb}}$	1024
Total number of environment transitions	$N_{\mathrm{total}}$	$2.0\times {10}^6$

Table 5. Sparse GP residual-correction parameter settings.

Parameter	Symbol	Value
Value-feature dimension	$d_v$	64
Residual-buffer capacity	$B_{\mathrm{GP}}$	8192, FIFO
GP-update batch size	$N_{\mathrm{GP}}$	2048
Number of inducing points	$M_z$	128
Inducing-point selection	—	K-means++
Inducing-point refresh frequency	—	Every GP update
Kernel function	—	RBF
Kernel length scale	l	$1.0$
Kernel output variance	${\sigma }_f^2$	$1.0$
Noise variance	${\sigma }_n^2$	$1.0\times {10}^{-3}$
GP regularization coefficient	${\lambda }_{\mathrm{GP}}$	$1.0\times {10}^{-6}$
Uncertainty-regulation coefficient	${\beta }_{\sigma }$	$1.0$
GP-update interval	$K_{\mathrm{GP}}$	1000 synchronized global interaction steps
Hyperparameter mode	—	Fixed during Actor–Critic training
GP snapshot during PPO optimization	—	Frozen
Gradient through GP outputs	—	Stop-gradient
Matrix factorization	—	Cholesky decomposition
Jitter term	$\delta$	$1.0\times {10}^{-6}$
Feature and target whitening	—	Yes
Maximum correction magnitude	$r_{\max }$	20
Correction scope	—	Current- and next-state values

To validate the effectiveness of the proposed method, DDPG-MHSA [55], MAPPO [56], MATD3 [57], and MADDPG [16] are selected as baseline algorithms. DDPG-MHSA denotes a DDPG-based baseline equipped with a multi-head self-attention (MHSA) module to model inter-agent interactions. For a fair comparison, all algorithms use the same total number of environment steps, simulation settings, observation and action spaces, reward function, network scale, evaluation protocol, and random seeds.

5.1.2. Evaluation Metrics

In accordance with the objectives of the cooperative tracking task, the average episodic reward, target retention rate, mean UAV–target distance error, mean absolute TD error, and safety-violation rate are selected as the core test-stage evaluation metrics. These metrics are used to evaluate training convergence, persistent tracking capability, observation-distance maintenance, and value estimation stability, respectively.

(1) Average Episodic Reward

The average episodic reward measures the cumulative team return within a single episode and is defined as:

$$R_{ep}={\displaystyle \sum _{t=1}^T}{R}_t^L,$$

(67)

where $R_t^L$ denotes the team reward at the t-th time step, and T denotes the maximum number of time steps in a single episode. A higher value of this metric indicates better overall task performance of the algorithm.

(2) Target Retention Rate

The target retention rate evaluates the ability of the multi-UAV system to maintain targets in an effectively observed state throughout the mission, and is defined as follows:

$$R_{keep}={\displaystyle \frac{1}{MT}}{\displaystyle \sum _{t=1}^T}{\displaystyle \sum _{j=1}^M}\mathbb{I}\left({\displaystyle \sum _{i=1}^N}{z}_{i,j,t}\ge 1\right).$$

(68)

A higher target retention rate indicates that the system demonstrates stronger persistent tracking capability under conditions of target motion, obstacle occlusion, and changes in the field of view.

(3) Mean UAV–Target Distance Error

The UAV–target distance metric reflects the observation geometry between the UAV and the target, thereby allowing analysis of the tracking stability of different algorithms. Since this paper adopts a two-dimensional kinematic model at a fixed flight altitude, the UAV–target distance here refers to the horizontal Euclidean distance between the projected UAV position on the ground plane and the ground target position. The mean UAV–target distance error is defined as:

$$E_d={\displaystyle \frac{{\sum }_{t=1}^T{\sum }_{i=1}^N{\sum }_{j=1}^M{z}_{i,j,t}\left|d_{i,j,t}^{ug}-d_{des}\right|}{{\sum }_{t=1}^T{\sum }_{i=1}^N{\sum }_{j=1}^M{z}_{i,j,t}+\mu }}.$$

(69)

where $d_{\mathrm{des}}$ represents the desired observation distance, and $\mu$ is a small positive constant used to avoid division by zero. Only the distance error between the i-th UAV and the j-th target is considered in the average calculation. The distance between them at time t is defined as

$$d_{i,j,t}^{ug}={∥p_{i,t}^u-p_{j,t}^g∥}_2.$$

(70)

A smaller value of this metric indicates that the UAVs can maintain the desired observation distance from the targets more stably.

(4) Mean Absolute TD Error

The mean absolute TD error is defined as:

$$E_{TD}={\displaystyle \frac{1}{T}}{\displaystyle \sum _{t=1}^T}\left|{\delta }_t\right|.$$

(71)

The mean absolute TD error is computed offline using the trained centralized Critic for diagnostic evaluation only, with a smaller value indicating more stable value estimation. Execution safety is evaluated using the safety-violation rate, defined as the percentage of test time steps with at least one obstacle collision, boundary violation, or inter-UAV safety-distance violation, where simultaneous violations are counted as one violated time step. Three event-based metrics are additionally recorded as the corresponding numbers of obstacle-collision, boundary-violation, and inter-UAV violation events per episode, with a continuous violation counted only at its onset. The minimum obstacle clearance and minimum inter-UAV distance are used to quantify geometric safety margins, and an episode with none of the three violation types is defined as a safe episode.

5.2. Training-Stage Performance Analysis

This section compares the training performance of HGP-MAPPO with that of the baseline algorithms, focusing on the evolution of average episodic reward, target retention rate, and mean absolute TD error. All curves report the average results over multiple random seeds, with the shaded regions indicating the standard deviations. A moving-average operation is further applied to smooth the curves.

5.2.1. Training Curves of Average Episodic Reward

Figure 2a presents the training-phase average episode reward curves of HGP-MAPPO, DDPG-MHSA, MAPPO, MATD3, and MADDPG. The solid curves and shaded regions represent the means and standard deviations over five independent runs, respectively. Although the rewards of all algorithms show an overall increasing trend, their convergence levels and training stability differ considerably. HGP-MAPPO gradually achieves the highest reward and maintains a relatively stable upward trend during the middle and later stages of training, reaching approximately 850 near the end of training. DDPG-MHSA exhibits the second-best performance and converges to a reward level slightly below that of HGP-MAPPO, whereas MAPPO reaches a lower final reward of approximately 750. In contrast, MATD3 and MADDPG exhibit substantially larger fluctuations. In particular, MATD3 undergoes repeated performance degradation during the later training stage, while MADDPG shows the lowest reward level and the most pronounced instability, including several substantial drops. These results indicate that HGP-MAPPO provides better final performance and training stability in the partially observable cooperative multi-target tracking task.

Figure 2. Training performance comparison of different algorithms: (a) average episodic reward and (b) target retention rate.

5.2.2. Training Curves of Target Retention Rate

As shown in Figure 2b, the target retention rates of all algorithms exhibit an overall increasing trend as training progresses, but their convergence levels and stability differ considerably. HGP-MAPPO achieves the highest retention rate during most of the middle and later training stages and reaches approximately 0.92 near the end of training, indicating that it can learn persistent multi-target observation strategies more effectively. DDPG-MHSA obtains the second-best performance, with a final retention rate close to 0.90, although it remains slightly lower than HGP-MAPPO. MAPPO converges to approximately 0.84 and exhibits more noticeable fluctuations, particularly during the middle stage of training. In contrast, MATD3 and MADDPG improve more slowly and show substantially lower final retention rates. MATD3 displays repeated oscillations after approximately $8\times {10}^5$ training steps, whereas MADDPG presents the slowest improvement and the largest performance degradation during the intermediate training stage. These results demonstrate that HGP-MAPPO provides better target-maintenance performance and more stable cooperative learning under target association, task handover, and partial-observability conditions.

5.2.3. Training Curves of TD Error

To evaluate the effect of GP residual correction on Critic value estimation, Figure 3 presents the mean absolute TD error curves of the evaluated algorithms. The solid curves and shaded regions denote the means and standard deviations over five independent runs, respectively. Although the TD errors of all algorithms generally decrease during training, HGP-MAPPO achieves the lowest final error and exhibits relatively small fluctuations in the middle and later stages. DDPG-MHSA obtains the second-lowest error, followed by MAPPO. In contrast, MATD3 and MADDPG maintain higher error levels and show more pronounced oscillations, particularly during the later training stage. These results indicate that HGP-MAPPO provides more stable Critic value estimation than the baselines, while the contribution of GP residual correction is further verified by the ablation study in Section 5.4.

Figure 3. The train curves of TD error for different algorithms.

Overall, as observed in Figure 2 and Figure 3, HGP-MAPPO outperforms the baseline algorithms across the three training metrics, namely average episodic reward, target retention rate, and mean absolute TD error. These results demonstrate that the proposed method achieves better training performance in terms of policy learning efficiency, persistent observation capability, and value estimation stability.

5.3. Test-Stage Performance Analysis

To evaluate the final execution performance under the main test-stage protocol, each trained policy is tested in both the standard-obstacle and densely obstructed scenarios with all network parameters fixed and no further policy updates. Following the evaluation protocol in Section 5.1.1, each algorithm is independently trained using five random seeds, $\{0, 1, 2, 3, 4\}$, and each trained policy is evaluated over 50 test episodes per scenario using the same evaluation seeds. The metrics defined in Section 5.1.2 are first averaged over the 50 episodes for each training seed, and the resulting five seed-level values are reported as the mean ± standard deviation. The results in this section serve as the main test-stage benchmark, and the quantitative results are summarized in Table 6.

Table 6. Test-stage performance in the standard-obstacle and densely obstructed scenarios. The results are reported as the mean ± standard deviation over five independent training runs.

(a) Standard-Obstacle Scenario
Method	Episodic Reward	Retention Rate (%)	Distance Error (m)	Mean Absolute TD Error	Safety-Violation Rate (%)
HGP-MAPPO	$\mathbf{674.29}\pm \mathbf{12.84}$	$\mathbf{93.62}\pm \mathbf{1.20}$	$\mathbf{0.72}\pm \mathbf{0.08}$	$\mathbf{0.10}\pm \mathbf{0.01}$	$\mathbf{1.8}\pm \mathbf{0.4}$
DDPG-MHSA	$649.97\pm 14.37$	$91.28\pm 1.41$	$0.91\pm 0.10$	$0.13\pm 0.02$	$2.6\pm 0.5$
MAPPO	$602.77\pm 18.62$	$86.74\pm 1.68$	$1.24\pm 0.14$	$0.17\pm 0.02$	$3.9\pm 0.7$
MATD3	$541.02\pm 22.15$	$80.31\pm 1.95$	$1.57\pm 0.17$	$0.22\pm 0.03$	$5.6\pm 0.9$
MADDPG	$487.17\pm 25.48$	$73.85\pm 2.27$	$1.92\pm 0.21$	$0.28\pm 0.04$	$7.4\pm 1.1$
(b) Densely Obstructed Scenario
Method	Episodic Reward	Retention Rate (%)	Distance Error (m)	Mean Absolute TD Error	Safety-Violation Rate (%)
HGP-MAPPO	$\mathbf{820.00}\pm \mathbf{18.72}$	$\mathbf{92.00}\pm \mathbf{1.50}$	$\mathbf{1.04}\pm \mathbf{0.12}$	$\mathbf{0.12}\pm \mathbf{0.02}$	$\mathbf{3.1}\pm \mathbf{0.6}$
DDPG-MHSA	$760.54\pm 21.36$	$87.45\pm 1.73$	$1.28\pm 0.14$	$0.16\pm 0.02$	$4.3\pm 0.7$
MAPPO	$700.00\pm 25.48$	$82.00\pm 2.10$	$1.62\pm 0.18$	$0.21\pm 0.03$	$6.1\pm 0.9$
MATD3	$540.00\pm 30.67$	$68.00\pm 2.60$	$2.07\pm 0.23$	$0.28\pm 0.04$	$8.6\pm 1.2$
MADDPG	$360.00\pm 36.15$	$56.00\pm 3.10$	$2.51\pm 0.28$	$0.35\pm 0.05$	$11.5\pm 1.5$

Note: The mean absolute TD error is computed offline using the trained centralized Critic for diagnostic evaluation only. Each reported statistic is calculated from five seed-level results, with each seed-level result being first averaged over 50 test episodes.

5.3.1. Analysis of Cooperative Tracking Performance in the Standard-Obstacle Scenario

The standard-obstacle scenario contains four UAVs, four mobile ground targets, and four static obstacles and is used to evaluate cooperative tracking under moderate obstacle constraints. As shown in Figure 4 and Figure 5, HGP-MAPPO maintains smoother UAV–target trajectories and more stable target associations than the baseline methods. After the initial adjustment stage, its UAV–target distances rapidly approach the desired observation distance and remain within a relatively narrow range. DDPG-MHSA also achieves stable tracking but exhibits larger local deviations, whereas MAPPO, MATD3, and MADDPG show progressively greater trajectory and distance fluctuations.

Figure 4. UAV–target cooperative tracking trajectories of five algorithms in the standard scenario.

Figure 5. UAV–target distance variation curves of the evaluated algorithms in the standard-obstacle scenario.

The quantitative results in Table 6a and Figure 6 further confirm these observations. HGP-MAPPO achieves the highest average episodic reward of $674.29\pm 12.84$ and target retention rate of $93.62\pm 1.20\%$, compared with $649.97\pm 14.37$ and $91.28\pm 1.41\%$, respectively, for DDPG-MHSA, the strongest baseline. HGP-MAPPO also obtains the lowest mean UAV–target distance error of $0.72\pm 0.08$ m and mean absolute TD error of $0.10\pm 0.01$, indicating improved distance regulation and more stable Critic value estimation. The remaining baselines exhibit progressively lower tracking performance and larger estimation errors.

Figure 6. Test-stage performance in the standard-obstacle scenario: (a) average episodic reward and (b) target retention rate.

HGP-MAPPO also demonstrates favorable safety performance. Its aggregate safety-violation rate is $1.8\pm 0.4\%$, lower than that of all baselines. As reported in Table 7a, it maintains a minimum obstacle clearance of $1.42\pm 0.12$ m and a minimum inter-UAV distance of $3.68\pm 0.19$ m, while achieving a safe-episode rate of $92.8\pm 2.4\%$. These results show that the improvements in tracking performance are accompanied by fewer safety-critical events and larger safety margins rather than being achieved at the expense of operational safety.

Table 7. Detailed test-stage safety performance in the standard-obstacle and densely obstructed scenarios (mean ± standard deviation over five independent runs).

(a) Standard-Obstacle Scenario
Method	Obstacle Collisions/Episode	Boundary Violations/Episode	Inter-UAV Violations/Episode	Min. Obstacle Clearance (m)	Min. Inter-UAV Distance (m)	Safe Episodes (%)
HGP-MAPPO	$\mathbf{0.03}\pm \mathbf{0.01}$	$\mathbf{0.01}\pm \mathbf{0.01}$	$\mathbf{0.04}\pm \mathbf{0.02}$	$\mathbf{1.42}\pm \mathbf{0.12}$	$\mathbf{3.68}\pm \mathbf{0.19}$	$\mathbf{92.8}\pm \mathbf{2.4}$
DDPG-MHSA	$0.05\pm 0.02$	$0.02\pm 0.01$	$0.07\pm 0.03$	$1.18\pm 0.14$	$3.44\pm 0.21$	$87.6\pm 2.8$
MAPPO	$0.09\pm 0.03$	$0.03\pm 0.02$	$0.12\pm 0.04$	$0.91\pm 0.15$	$3.17\pm 0.24$	$78.8\pm 3.2$
MATD3	$0.15\pm 0.04$	$0.06\pm 0.02$	$0.20\pm 0.05$	$0.63\pm 0.17$	$2.92\pm 0.26$	$68.4\pm 3.6$
MADDPG	$0.22\pm 0.05$	$0.09\pm 0.03$	$0.30\pm 0.06$	$0.38\pm 0.19$	$2.68\pm 0.29$	$56.8\pm 4.1$
Method	Obstacle Collisions/Episode	Boundary Violations/Episode	Inter-UAV Violations/Episode	Min. Obstacle Clearance (m)	Min. Inter-UAV Distance (m)	Safe Episodes (%)
HGP-MAPPO	$\mathbf{0.06}\pm \mathbf{0.02}$	$\mathbf{0.02}\pm \mathbf{0.01}$	$\mathbf{0.08}\pm \mathbf{0.03}$	$\mathbf{0.96}\pm \mathbf{0.14}$	$\mathbf{3.46}\pm \mathbf{0.21}$	$\mathbf{85.6}\pm \mathbf{2.9}$
DDPG-MHSA	$0.09\pm 0.03$	$0.04\pm 0.02$	$0.12\pm 0.04$	$0.77\pm 0.15$	$3.25\pm 0.23$	$78.0\pm 3.3$
MAPPO	$0.15\pm 0.04$	$0.06\pm 0.02$	$0.19\pm 0.05$	$0.54\pm 0.17$	$3.02\pm 0.25$	$66.8\pm 3.8$
MATD3	$0.25\pm 0.05$	$0.11\pm 0.03$	$0.31\pm 0.07$	$0.28\pm 0.18$	$2.75\pm 0.28$	$51.6\pm 4.4$
MADDPG	$0.38\pm 0.07$	$0.17\pm 0.04$	$0.46\pm 0.09$	$0.10\pm 0.20$	$2.49\pm 0.31$	$37.2\pm 4.9$

Note: Obstacle collisions, boundary violations, and inter-UAV violations are reported as the average number of distinct events per episode. A violation persisting over consecutive time steps is counted once at its onset. The minimum obstacle clearance is measured relative to the boundary of the safetyexpanded obstacle region; a positive value indicates that the UAV remains outside the predefined unsafe region. The minimum allowable inter-UAV distance is $d_{\mathrm{safe}}^u=3.0$ m. A safe episode contains none of the three types of safety violations.

5.3.2. Analysis of Cooperative Tracking Performance in the Densely Obstructed Scenario

The densely obstructed scenario contains six UAVs, six mobile ground targets, and eight static obstacles, resulting in a more restricted feasible motion space and more frequent obstacle-induced occlusions than in the standard-obstacle scenario. As shown in Table 6b, HGP-MAPPO achieves the highest average episodic reward of $820.00\pm 18.72$ and the highest target retention rate of $92.00\pm 1.50\%$, compared with $760.54\pm 21.36$ and $87.45\pm 1.73\%$, respectively, for DDPG-MHSA, the strongest baseline. It also obtains the lowest mean UAV–target distance error of $1.04\pm 0.12$ m and the lowest mean absolute TD error of $0.12\pm 0.02$, indicating more stable tracking and value estimation under frequent occlusion and cooperative-role changes.

HGP-MAPPO also maintains favorable safety performance under the more challenging conditions. Its aggregate safety-violation rate is $3.1\pm 0.6\%$, lower than that of all baselines. As reported in Table 7b, the minimum obstacle clearance and minimum inter-UAV distance are $0.96\pm 0.14$ m and $3.46\pm 0.21$ m, respectively, while the safe-episode rate reaches $85.6\pm 2.9\%$. These results show that the improved tracking performance is accompanied by fewer safety-critical events and adequate geometric safety margins.

5.4. Ablation Study

To evaluate the contributions of the key components in HGP-MAPPO, three ablated variants are compared with the complete framework. Specifically, w/o Guidance removes the task-guidance conditions from both the decentralized Actor and centralized Critic; w/o GP retains the task-guidance-conditioned Actor–Critic architecture but removes the GP residual correction module; and w/o Uncertainty Weight retains the GP posterior-mean correction while removing the posterior-variance-based adaptive weighting mechanism. The ablation study is conducted as an independent controlled experiment in the densely obstructed scenario. Within this study, all variants use identical observation and action spaces, reward functions, network architectures, training durations, hyperparameters, random-seed settings, and evaluation procedures, with only the corresponding component being removed.

5.4.1. Comprehensive Performance Comparison of Ablation Variants

As shown in Figure 7, the complete HGP-MAPPO achieves the best overall performance across all four metrics, with an average episodic reward of 820, a target retention rate of 0.92, a mean UAV–target distance error of 1.18 m, and a mean absolute TD error of 0.10. Among the ablated variants, w/o Guidance exhibits the largest performance degradation, with the reward and retention rate decreasing to 655 and 0.79 and the distance error increasing to 2.45 m, highlighting the importance of task guidance for cooperative tracking and distance regulation. Removing the GP correction increases the mean absolute TD error to 0.28, whereas w/o Uncertainty Weight achieves intermediate performance between the complete method and the other ablated variants.

Figure 7. Component-level ablation results of HGP-MAPPO. “w/o” denotes “without”.

Overall, the results indicate complementary roles of the three components: task guidance improves cooperative-role coordination and tracking performance, GP residual correction enhances Critic value-estimation accuracy, and uncertainty-aware weighting improves the reliability of residual correction. Their joint use therefore contributes to the overall performance of HGP-MAPPO. Because the ablation study is independently trained and evaluated, its absolute metric values are used for within-study component comparison rather than direct numerical comparison with the main test results in Table 6.

5.4.2. Training-Process Analysis of Ablation Variants

Figure 8 compares the training processes of HGP-MAPPO and its three ablated variants in terms of average episodic reward and mean absolute TD error. As shown in Figure 8a, the complete HGP-MAPPO converges faster and maintains the highest reward level, whereas w/o Guidance shows the slowest improvement and the lowest final return. The w/o GP and w/o Uncertainty Weight variants achieve intermediate performance, with the latter exhibiting more noticeable fluctuations during training. These results indicate that task guidance facilitates policy learning, while GP correction and uncertainty weighting further improve training stability.

Figure 8. Training curves of HGP-MAPPO and its component-level ablation variants: (a) average episode reward and (b) mean absolute TD error. The curves and shaded regions represent the means and standard deviations over five independent runs, respectively. “w/o” denotes “without”.

As shown in Figure 8b, HGP-MAPPO achieves the fastest reduction and lowest final mean absolute TD error. Removing the GP correction results in larger TD errors, while w/o Uncertainty Weight remains less stable than the complete method. The results are consistent with the testing performance in Figure 7, confirming that task guidance, GP residual correction, and uncertainty-aware weighting provide complementary benefits to policy learning and value-estimation stability.

5.5. Sensitivity Analysis of Reward Weights

To evaluate the sensitivity to the reward coefficients, each coefficient was independently scaled to $0.8$, $1.0$, and $1.2$ times its nominal value, while the remaining six coefficients were fixed. The reward-weight sensitivity study follows the same independent component-analysis setting used in Section 5.4, with the complete HGP-MAPPO configuration taken as the nominal reference. The target retention rate and mean UAV–target distance error were adopted as the evaluation metrics.

As shown in Table 8, moderate variations in the reward coefficients lead to limited performance changes. The tracking and obstacle-related coefficients, particularly ${\alpha }_1$ and ${\alpha }_4$, have relatively stronger effects, whereas the smoothness and energy coefficients exhibit lower sensitivity. Overall, HGP-MAPPO maintains stable tracking performance within a $\pm 20\%$ variation range.

Table 8. Sensitivity analysis of the reward-weight coefficients.

Coefficient	Reward Component	Nominal Value	${\mathit{R}}_{\mathbf{keep}}$(%)/ ${\mathit{E}}_{\mathit{d}}$ (m)
			$\mathbf{0.8}$×	$\mathbf{1.0}$×	$\mathbf{1.2}$×
${\alpha }_1$	Tracking reward	1.00	$89.6/1.43$	$\mathbf{92.0}/\mathbf{1.18}$	$91.2/1.24$
${\alpha }_2$	FOV maintenance reward	0.80	$90.4/1.33$	$\mathbf{92.0}/\mathbf{1.18}$	$91.5/1.22$
${\alpha }_3$	Target recovery reward	0.60	$90.8/1.29$	$\mathbf{92.0}/\mathbf{1.18}$	$91.6/1.21$
${\alpha }_4$	Obstacle/boundary penalty	1.20	$89.9/1.37$	$\mathbf{92.0}/\mathbf{1.18}$	$90.9/1.31$
${\alpha }_5$	Inter-UAV collision penalty	1.20	$90.3/1.34$	$\mathbf{92.0}/\mathbf{1.18}$	$91.3/1.25$
${\alpha }_6$	Control smoothness penalty	0.10	$91.5/1.22$	$\mathbf{92.0}/\mathbf{1.18}$	$91.7/1.20$
${\alpha }_7$	Energy consumption penalty	0.05	$91.8/1.20$	$\mathbf{92.0}/\mathbf{1.18}$	$91.6/1.22$

Note: Each entry reports $R_{\mathrm{keep}}(\%)/E_d\left(\mathrm{m}\right)$, where $R_{\mathrm{keep}}$ denotes the target retention rate, and $E_d$ denotes the mean UAV–target distance error. Each coefficient was independently varied, while the remaining coefficients were fixed at their nominal values.

5.6. Computational Overhead, Scalability, and Sensitivity of the GP Module

To further evaluate the practical feasibility of the proposed sparse GP correction module, its computational overhead, scalability with increasing team size, and sensitivity to the main implementation parameters are examined. Unless otherwise stated, all experiments follow the hardware configuration, training protocol, and five-random-seed setting described in Section 5.1.1, and the results are reported as the mean ± standard deviation.

Computational overhead: MAPPO, HGP-MAPPO without GP correction, and the complete HGP-MAPPO are compared under identical hardware, parallel-environment, total-training-step, and random-seed settings. MAPPO provides the standard training-cost reference, whereas HGP-MAPPO without GP correction preserves the task-guided Actor–Critic architecture and is used to isolate the additional cost of the sparse GP module. Table 9a reports the wall-clock time per ${10}^6$ environment steps, relative training overhead, peak GPU memory, average GP-update time, and test-stage action latency.

Table 9. Computational overhead and scalability of the sparse GP module.

(a) Computational Overhead
Method		Time/${\mathbf{10}}^{\mathbf{6}}$Steps (Min)	OH-M (%)	OH-GP (%)	Peak Memory (GB)	GP-Update Time (ms)	Test Latency (ms/Step)
MAPPO		$16.8\pm 0.4$	—	—	$6.4\pm 0.1$	—	$0.42\pm 0.02$
HGP-MAPPO (w/o GP)		$18.6\pm 0.5$	10.7	—	$6.9\pm 0.1$	—	$0.45\pm 0.02$
HGP-MAPPO		$20.1\pm 0.6$	19.6	8.1	$7.5\pm 0.1$	$41.8\pm 2.3$	$0.45\pm 0.02$
(b) Scalability from the 4/4 Configuration to the 6/6 Configuration
UAVs/Targets	${\mathit{d}}_{\mathit{s}}$	${\mathit{d}}_{\mathit{v}}$	Time/${\mathbf{10}}^{\mathbf{6}}$Steps (min)	GP Overhead (%)	Peak Memory (GB)	Target Maintenance Rate (%)	Mean Absolute TD Error
4/4	146	64	$20.1\pm 0.6$	8.1	$7.5\pm 0.1$	$92.0\pm 1.2$	$0.10\pm 0.01$
6/6	226	64	$23.7\pm 0.7$	8.7	$8.9\pm 0.2$	$89.1\pm 1.5$	$0.13\pm 0.02$

Note: OH-M denotes the training overhead relative to MAPPO, calculated as OH-M $=(T-T_{\mathrm{MAPPO}})/T_{\mathrm{MAPPO}}\times 100\%$. OH-GP denotes the GP-specific training overhead, calculated as OH-GP $=(T_{\mathrm{HGP}}-T_{\mathrm{w}/\mathrm{o}\mathrm{GP}})/T_{\mathrm{w}/\mathrm{o}\mathrm{GP}}\times 100\%$. $d_s$ denotes the centralized-Critic input dimension, whereas $d_v$ denotes the fixed GP value-feature dimension. GP overhead is calculated relative to the corresponding HGP-MAPPO without-GP configuration at the same team size. The reported values are, therefore, used to evaluate the relative effect of an increasing team size within this scalability study rather than to reproduce the main test-stage results in Table 6.

Scalability: A separate scalability experiment is conducted under the 4-UAV/4-target and 6-UAV/6-target configurations. Within this scalability study, the mission area, obstacle-density setting, reward function, total number of training steps, and GP hyperparameters are kept unchanged, while each team-size configuration is independently trained and evaluated. As the team size increases, the input dimension $d_s$ of the centralized Critic increases, whereas the GP value feature dimension remains fixed at $d_v=64$. Table 9b reports both computational and tracking metrics to examine whether the sparse GP remains tractable and effective under the enlarged team configuration.

Sensitivity and staged-update stability: Using the 4-UAV/4-target configuration from the scalability study as the reference setting, a one-factor-at-a-time sensitivity analysis is conducted for the number of inducing points $M_z$, the GP update interval $K_{\mathrm{GP}}$, the value-feature dimension $d_v$, the RBF-kernel length scale l, and the uncertainty-regulation coefficient ${\beta }_{\sigma }$. The default configuration is $M_z=128$, $K_{\mathrm{GP}}=1000$, $d_v=64$, $l=1.0$, ${\beta }_{\sigma }=1.0$. When one parameter is varied, all other GP and Actor–Critic settings remain fixed. The examined values are $M_z\in \{64,128,256\}$, $K_{\mathrm{GP}}\in \{500,1000,2000\}$, $d_v\in \{32,64,128\}$, $l\in \{0.5,1.0,2.0\}$, ${\beta }_{\sigma }\in \{0.5,1.0,2.0\}$.

Table 10 reports the target retention rate, mean absolute TD error, and GP-specific training overhead under different parameter settings. Figure 9 further compares the TD-error curves under different GP update intervals.

Table 10. Sensitivity to key sparse-GP parameters.

Parameter	Value	Target Retention Rate ${\mathit{R}}_{\mathbf{keep}}$ (%)	Mean Absolute TD Error	GP Overhead (%)
$M_z$	64	$89.8\pm 1.4$	$0.14\pm 0.02$	5.4
	128	$\mathbf{92.0}\pm \mathbf{1.2}$	$\mathbf{0.10}\pm \mathbf{0.01}$	8.1
	256	$92.4\pm 1.1$	$0.09\pm 0.01$	14.7
$K_{\mathrm{GP}}$	500	$92.3\pm 1.1$	$0.09\pm 0.01$	12.9
	1000	$\mathbf{92.0}\pm \mathbf{1.2}$	$\mathbf{0.10}\pm \mathbf{0.01}$	8.1
	2000	$89.2\pm 1.5$	$0.15\pm 0.02$	5.0
$d_v$	32	$90.1\pm 1.3$	$0.13\pm 0.02$	6.7
	64	$\mathbf{92.0}\pm \mathbf{1.2}$	$\mathbf{0.10}\pm \mathbf{0.01}$	8.1
	128	$92.2\pm 1.2$	$0.09\pm 0.01$	10.8
l	0.5	$90.8\pm 1.3$	$0.12\pm 0.02$	8.1
	1.0	$\mathbf{92.0}\pm \mathbf{1.2}$	$\mathbf{0.10}\pm \mathbf{0.01}$	8.1
	2.0	$90.5\pm 1.4$	$0.13\pm 0.02$	8.1
${\beta }_{\sigma }$	0.5	$91.1\pm 1.3$	$0.12\pm 0.02$	8.1
	1.0	$\mathbf{92.0}\pm \mathbf{1.2}$	$\mathbf{0.10}\pm \mathbf{0.01}$	8.1
	2.0	$90.7\pm 1.4$	$0.13\pm 0.02$	8.1

Figure 9. Meanabsolute TD error under different GP update intervals.

The kernel length scale l controls the smoothness and effective correlation range of the GP residual model. A smaller value produces more localized corrections but may make the GP sensitive to local feature variations, whereas an excessively large value may oversmooth abrupt Critic residuals. The coefficient ${\beta }_{\sigma }$ regulates the suppression effect of the GP posterior uncertainty through ${\omega }_t^{\mathrm{GP}}=1/\left[1+{\beta }_{\sigma }{\sigma }_{\mathrm{GP}}^2\left(x_t^V\right)\right]$. A smaller ${\beta }_{\sigma }$ permits stronger correction in uncertain regions, whereas a larger value results in more conservative residual compensation. Nevertheless, the current validation is limited to two-dimensional fixed-altitude software simulation and does not explicitly account for three-dimensional flight dynamics, environmental disturbances, sensing uncertainties, or communication imperfections; therefore, the reported results should not be interpreted as direct evidence of real-world flight performance.

6. Conclusions

This paper addresses the multi-UAV multi-target cooperative tracking problem under partial observability, limited communication, and obstacle constraints, and proposes HGP-MAPPO, a task-guided method with Gaussian-process-based value correction. The proposed method reduces the decision coupling in lower-level continuous control through task guidance conditions, while enhancing the stability of Critic value estimation via Gaussian-process-based residual correction. Experimental results in both the standard obstacle scenario and the densely obstructed scenario demonstrate that HGP-MAPPO outperforms the baseline methods in terms of training convergence, target retention, distance control, and TD-error stability. Ablation studies further verify the effectiveness of task guidance signals, GP residual correction, and uncertainty weighting. In a practical implementation, HGP-MAPPO would operate at the cooperative decision-making layer and generate reference velocity, heading, or waypoint commands for a conventional low-level flight controller. Future work will extend the framework to three-dimensional motion and terrain-aware visibility modeling and will incorporate domain-randomized sensor, communication, aerodynamic, and actuator disturbances. A staged validation process involving software-in-the-loop simulation, hardware-in-the-loop testing, controlled flight experiments, and outdoor field trials will be conducted to assess its practical effectiveness.