Distributed Pursuit–Evasion Game Decision-Making Based on Multi-Agent Deep Reinforcement Learning

Abstract

Pursuit–evasion games are a fundamental framework for advancing autonomous decision-making and cooperative control in multi-UAV systems. However, the application of reinforcement learning to pursuit–evasion games involving fixed-wing UAVs remains challenging due to constraints, such as minimum velocity, limited turning radius, and high-dimensional continuous action spaces. To address these issues, this paper proposes a method that integrates automatic curriculum learning with multi-agent proximal policy optimization. A self-play mechanism is introduced to simultaneously train both pursuers and evaders, enabling dynamic and adaptive encirclement strategies. In addition, a reward structure specifically tailored for the encirclement task was designed to guide the pursuers in gradually achieving the encirclement of the evader while ensuring their own safety. To further improve training efficiency and convergence, this paper develops a subgame curriculum learning framework that progressively exposes agents to increasingly complex scenarios, facilitating experience accumulation and skill transfer. The simulation results demonstrate that the proposed approach improves learning efficiency and cooperative pursuit performance under realistic fixed-wing UAV dynamics. This work provides a practical and scalable solution for multiple fixed-wing UAV pursuit–evasion missions in complex environments.

1. Introduction

With the continuous advancement of unmanned systems, unmanned aerial vehicles (UAVs) have garnered significant attention from researchers due to their low cost, high operational flexibility, and reduced risk. Compared to a single UAV, multi-UAV systems exhibit enhanced cooperative capabilities, improved task execution efficiency, and broader operational coverage, making them highly promising for applications such as search and rescue [1], target tracking [2], and border defense [3]. Among these applications, pursuit–evasion games (PEGs) represent a crucial research domain within multi-UAV systems, as they involve adversarial interactions among intelligent agents and serve as a fundamental framework for studying autonomous decision-making and cooperative control in UAV swarms. Typically, a PEG consists of two factions: the pursuer and the evader. The goal of the pursuer is to capture the evader as quickly as possible, while the evader aims to evade capture for as long as possible.

Traditionally, PEGs have been studied using differential game theory [4], with control strategies derived via optimal control [5], Model Predictive Control (MPC) [6], and other advanced control techniques. For instance, in [5], an algorithm integrating geometric methods and differential game theory was proposed to derive the optimal strategies for PEGs of players with damped double-integrator dynamics. In [6], a cooperative pursuit problem involving multiple pursuers and a single evader in an unbounded two-dimensional environment was studied. A robust MPC framework was proposed to guarantee that the evader was consistently constrained within the convex hull formed by the pursuers. In [7], an open-loop two-on-one PEG was investigated, where the evader was chosen as the leader, and a numerical solution for the time-optimal constrained trajectory was obtained using the open-loop Stackelberg approach. While these methods offer strong theoretical guarantees, they typically require accurate system models and intensive computation, posing limitations in real-time applications involving uncertain or dynamic environments. Furthermore, their scalability is severely constrained when handling large-scale multi-agent systems.

In recent years, the emergence of reinforcement learning (RL) has provided a novel solution for control and decision-making in nonlinear systems. Unlike traditional methods that rely on precise environmental models, RL algorithms directly optimize policies based on feedback obtained from interactions with the environment. Through continuous learning, agents gradually acquire the ability to take optimal actions in different states, thereby maximizing cumulative rewards. Furthermore, RL enables an offline training and onboard execution paradigm, effectively bypassing the computational complexity associated with real-time decision-making, which makes it well suited for real-time guidance in lightweight UAVs. Extensive and in-depth research has been conducted by scholars worldwide on the application of RL techniques to PEGs [8,9,10,11,12,13]. For instance, in [8], a reinforcement learning-based distributed cooperative pursuit strategy that enables agents to autonomously learn cooperative control and communication policies was designed. An online motion planning algorithm based on a target prediction network and the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm was proposed to address the UAV PEG in [9]. Xiao et al. [12] introduced an asynchronous multi-stage deep reinforcement learning framework to tackle the challenge of safe evader navigation in the presence of physical attacks from multiple pursuers. It should be noted that while extensive research has been conducted on UAV pursuit–evasion games using RL methods, most studies have focused on dynamically simpler models, such as quadcopter UAVs or particle-based models, which are not well suited for fixed-wing UAVs [14].

In certain complex environments, fixed-wing UAVs demonstrate greater endurance and resilience compared to rotary-wing UAVs. However, the design of autonomous decision-making systems for fixed-wing platforms presents significant challenges. Due to the necessity of maintaining a minimum forward speed to generate lift, fixed-wing UAVs are incapable of hovering or executing arbitrary directional turns. Moreover, they require simultaneous control over multiple degrees of freedom and parameters such as roll, yaw, and velocity, which leads to high system dimensionality and strong coupling in dynamic game scenarios [15]. These characteristics substantially increase the difficulty of applying pursuit–evasion strategies to fixed-wing UAVs. Research on PEGs involving fixed-wing UAVs has primarily focused on the air combat domain [16,17,18,19,20,21,22]. In [16], a short-range air combat autonomous operation decision-making model for UAVs was developed, leveraging reinforcement learning to enable autonomous decision-making in air combat scenarios. Building on this foundation, a communication mechanism was integrated into reinforcement learning in [19], enabling multi-UAV information sharing based on self-learning communication. In [20], a strategy combining line-of-sight (LOS) angle rate correction techniques with DRL was employed to learn high-speed UAV pursuit–evasion strategies. Nevertheless, RL is also subject to the curse of dimensionality, which poses substantial challenges in complex tasks involving numerous agents, high-dimensional state spaces, and continuous action domains. In such scenarios, RL training often suffers from convergence issues, low sample efficiency, and limited policy exploration, all of which significantly hinder its practical applicability. In [21], an algorithm combining hierarchical reinforcement learning and behavior cloning was proposed to improve the stability and learning efficiency of autonomous decision-making training in within-visual-range air combat scenarios. However, the heavy reliance on expert knowledge limits the algorithm’s adaptability and generalization capability. In [22], a training framework combining Particle Swarm Optimization (PSO) with the Minimax Multi-Agent Deep Deterministic Policy Gradient (M3DDPG) algorithm was proposed, where PSO is used to optimize the sample data set, thereby improving sample efficiency and convergence speed. However, the training is conducted solely against opponents with fixed maneuvering strategies, which limits the robustness and adaptability of the learned policy when facing more diverse or adaptive adversaries in dynamic and uncertain environments.

More recently, the emergence of curriculum learning [23] has provided a novel approach to address the aforementioned challenges. Within the curriculum learning framework, agents are initially trained in lower-complexity environments to master fundamental strategies. As training progresses, the complexity of the tasks is gradually increased. By leveraging this approach, the convergence performance and sample efficiency of the policy training process can be improved [24,25,26]. In [24,25], the task difficulty was adjusted by setting parameters such as motion speed and encirclement radius, allowing the agent to undergo training from easy to more complex tasks, thereby enhancing the robustness of the entire training process. In [26], task-specific curriculum learning was proposed to accelerate the training of collision avoidance and clustering strategies for fixed-wing UAVs. Additionally, curriculum learning has been introduced in the domain of fixed-wing UAV pursuit–evasion games to expedite the training of MARL [27,28,29]. In [27], the task was simplified by gradually weakening the strategy of the opponent to design an appropriate curriculum sequence, and a curriculum evaluation method was proposed to verify the effectiveness of the generated curriculum. In [28], the target task was divided into multiple subtasks with distinct reward structures, allowing the agent to achieve progressive learning by sequentially mastering each subtask. It is worth noting that the aforementioned curriculum learning approaches typically rely on heuristics or prior domain knowledge for curriculum design, with task sequences manually crafted. Such methods are often subjective and inflexible, lacking the adaptability to dynamically respond to changing environments or evolving task complexities.

To bridge these gaps, this paper proposes a novel framework that integrates automatic subgame curriculum learning with multi-agent proximal policy optimization (MAPPO) to address the PEG problem among multiple fixed-wing UAVs. Specifically, we introduce a self-play training mechanism to co-evolve strategies for both pursuers and evaders, enhancing policy robustness against adaptive opponents. Additionally, a reward structure tailored for encirclement is designed to guide cooperative strategies while ensuring agent safety. To improve learning efficiency, we develop a subgame curriculum learning module that adaptively samples learning scenarios based on difficulty estimation, facilitating faster and more stable policy training. The primary contributions of this paper are as follows.

• This paper investigates the dynamic PEG problem for multiple fixed-wing UAVs and proposes a distributed control method based on automatic subgame curriculum learning and MAPPO. In contrast to previous studies that typically assume fixed or simplistic evader strategies, our method employs a self-play mechanism to jointly train both pursuer and evader policies. This enables adaptive and dynamic encirclement of more capable and intelligent evaders.
• A reward structure tailored for encirclement missions is introduced in this paper. Unlike methods that rely solely on distance or UAV orientation to determine task success, our approach takes into account the dynamic surrounding of the evader while ensuring the safety of the pursuer, guiding the pursuer to perform effective encirclement without compromising its own security.
• To address the challenges of self-play solution difficulties and low sample efficiency, a training framework based on subgame curriculum learning and MAPPO is proposed. Compared to manually designed curricula or empirically determined subgame sequences, the proposed framework strategically plans the learning order by integrating self-guided subgame sampling. This enhances both sampling and learning efficiency while reducing reliance on human expertise and subjective bias.

This paper is organized as follows: Section 2 introduces the preliminaries required for this paper, including the fixed-wing UAV model, Markov games, and proximal policy optimization. In Section 3, the multiple fixed-wing UAVs PE scenario considered in this paper is described in detail, along with a comprehensive introduction to the proposed curriculum-based MARL algorithm. Section 4 covers the presentation and analysis of the simulation results, while Section 5 offers the conclusions.

2. Preliminaries

2.1. UAV Dynamics Model

To address the autonomous decision-making challenges in multi-UAV pursuit–evasion scenarios, a simplified yet representative dynamics model is essential for computational efficiency and real-time maneuver planning. In the real world, the motion kinematics and dynamics of a fixed-wing UAV can be described using a six-degrees-of-freedom (DOF) model. However, in this study, the primary focus is on the cooperative encirclement control of the pursuer UAVs against the evader UAV in the horizontal plane, where altitude variations have a limited impact on the overall encirclement performance. Therefore, it is assumed that each UAV maintains a constant flight altitude, and a simplified 4-DOF model [14,26] is employed to describe their motion.

The fixed-wing UAV model is established within a two-dimensional inertial coordinate system, as shown in Figure 1. In this system, the x-axis points toward the east, and the y-axis points north. Then, the state of the UAV can be defined as ${[x,y,v,\psi ,\phi ]}^{\mathrm{T}}$, where $(x,y)$ represents the position of the UAV in the two-dimensional inertial coordinate system; v is a scalar indicating the velocity of the UAV; and $\psi$ and $\phi$ denote the yaw angle and roll angle of the UAV, respectively. With the above five variables, the state of the UAV at a certain moment can be determined. The centroid kinematic equations for the UAV can be defined as follows.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill \dot{x}& =vcos\psi \hfill \\ \hfill \dot{y}& =vsin\psi \hfill \end{array}\right..\end{array}$$

(1)

Disregarding the attitude control of UAVs, the control variables of UAVs can be represented by two parameters, $n_x, \phi$, where $n_x$ represents the overload in the direction of the velocity of the UAV. Thus, the dynamic model of the UAV is defined as follows.

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill \dot{v}& =g{n}_x\hfill \\ \hfill \dot{\psi }& ={\displaystyle \frac{gtan\phi }{v}}\hfill \end{array}\right.,\end{array}$$

(2)

where g is the acceleration of gravity. From the dynamic model (2), it can be seen that among the three control variables, $n_x$ controls the magnitude of the velocity, while $\phi$ controls the change in the direction of the velocity collectively.

2.2. Markov Game

The Markov game is a widely used framework for addressing multi-agent decision-making and control problems. It can be viewed as an extension of the Markov Decision Process (MDP) to a multi-agent environment, which allows for multiple participants to interact, compete, or collaborate within a shared environment. Generally, a Markov game is defined as a tuple $\mathcal{MG}=<\mathcal{N},\mathcal{S},\mathcal{A},P,\mathbf{R},\gamma ,\rho >$, where $\mathcal{N}=\{1,\dots ,N\}$ is the set of N agents; $\mathcal{S}$ is the state space, which is the set of all possible states of the agents; $\mathcal{A}={\prod }_{i=1}^N{\mathcal{A}}_i$ is the joint action space with ${\mathcal{A}}_i$ being the action space of agent i; $P:\mathcal{S}\times \mathcal{A}\to \Delta \left(\mathcal{S}\right)$ denotes the function of the state transition probability; $\mathbf{R}=(R_1,\dots ,R_N):\mathcal{S}\times \mathcal{A}\to {\mathbb{R}}^N$ represents the joint reward function with $R_i$ being the reward function for agent i; $\gamma \in [0,1]$ is the discount factor of rewards; and $\rho$ is the distribution of initial states. For the given state s and joint action $\mathbf{a}=(a_1,\dots ,a_N)$, the game moves to the next state $s^{\prime }$ with probability $P\left(s^{\prime }\right|s,\mathbf{a})$ and agent i receives a reward $R_i(s,\mathbf{a})$.

For the Markov game $\mathcal{MG}$, a subgame $\mathcal{MG}\left(s\right)$ refers to the game initiated from a specific state s [30], meaning the initial state distribution satisfies $\rho \left(s\right)=1$. Thus, selecting a subgame essentially corresponds to determining the starting state of the overall Markov game.

In a zero-sum game, the interests of the participating agents are completely opposed, meaning that for all state–action pairs, the total reward for all agents sums to zero, i.e., $R_1(s,\mathbf{a})+\dots +R_N(s,\mathbf{a})=0,\forall (s,\mathbf{a})\in (\mathcal{S},\mathcal{A})$. It is assumed that agent i follows a strategy ${\pi }_i:\mathcal{S}\to {\mathcal{A}}_i$ to generate actions and maximize its own accumulated reward. To achieve this objective, the value function and Q-function are introduced.

$$V_i^{\pi }\left(s\right)=\mathbb{E}\left[\sum _t{\gamma }^t{R}_i(s^t,{\mathbf{a}}^t)|s^0=s\right],$$

(3)

$$Q_i^{\pi }(s,\mathbf{a})=\mathbb{E}\left[\sum _t{\gamma }^t{R}_i(s^t,{\mathbf{a}}^t)|s^0=s,{\mathbf{a}}^0=\mathbf{a}\right],$$

(4)

where $V_i^{\pi }\left(s\right)$ represents the value function, which measures the expected return in the current state s under the policy $\pi$, and $Q_i^{\pi }(s,\mathbf{a})$ denotes the Q-function, which evaluates the expected return for a given state–action pair $(s,\mathbf{a})$ under the policy $\pi$.

The solution to a zero-sum game is defined as a Nash equilibrium (NE). A joint policy ${\pi }^{*}=({\pi }_1^{*},\dots ,{\pi }_N^{*})$ is a Nash equilibrium of a zero-sum game if for all initial states $s^0$ with $\rho \left(s^0\right)>0$, the following condition holds.

$${\pi }_i^{*}=\underset{{\pi }_i}{argmax}{V}_i^{({\pi }_i,{\pi }_{-i}^{*})}\left(s^0\right),i=1,\dots ,N,$$

(5)

where subscript $-i$ represents the variables of all other agents excluding agent i.

Then, the minimax nature of zero-sum games can be inferred.

$$V_i^{*}\left(s\right)=\underset{{\pi }_i}{max}\underset{{\pi }_{-i}}{min}{\mathbb{E}}_{\mathbf{a}\sim \mathbf{\pi }(·|s)}\left[Q_i^{*}(s,\mathbf{a})\right],$$

(6)

$$Q_i^{*}(s,\mathbf{a})=R_i(s,\mathbf{a})+\gamma {\mathbb{E}}_{s^{\prime }\sim P(·|s,\mathbf{a})}\left[V_i^{*}\left(s^{\prime }\right)\right].$$

(7)

2.3. Proximal Policy Optimization

In the previously established Markov game environment, RL aims to obtain optimal policies by continuously interacting with the environment to maximize the expected cumulative reward gained by the agent. In DRL, the policy of the agent is typically represented by an actor network, denoted as ${\pi }_{\theta }$ with $\theta$ being the trainable parameters of the neural network. Thus, the task of finding the optimal policy ${\pi }^{*}$ of the agent can be formulated as an optimization problem over the parameters of the actor network, that is,

$${\theta }^{*}=\underset{\theta }{argmax}J\left(\theta \right)=\underset{\theta }{argmax}\left(\underset{\tau \sim {\pi }_{\theta }}{\mathbb{E}}\left[\sum _{k=0}^{N-1}{\gamma }^k{R}_k\right]\right),$$

(8)

where $J\left(\theta \right)$ represents the objective function of the agent. N denotes the total number of timesteps. $\gamma \in [0,1]$ is the discount factor of the rewards.

Building upon the aforementioned optimization objective, proximal policy optimization (PPO) [31] is a widely used policy gradient method that enhances training stability and sample efficiency by constraining the magnitude of policy updates. Specifically, in PPO, the objective defined in (8) can be rewritten as follows.

$${\theta }^{*}=\underset{\theta }{argmax}\left(J^{\mathrm{clip}}\left(\theta \right)+cS\left(\theta \right)\right),$$

(9)

where $J^{\mathrm{clip}}\left(\theta \right)$ represents the objective function for the policy update, which is obtained by modifying the data sampled from the demonstration policies through importance weights and the clipping operations. S represents the entropy term, which is incorporated to encourage the agent to explore the environment fully and to prevent premature convergence to a suboptimal solution. c is the hyperparameters that control the relative importance between items.

In addition, PPO employs the actor–critic architecture, where the actor network is responsible for policy execution, while the critic network evaluates the value of states to guide policy updates. This structure helps improve learning efficiency by reducing variance in policy gradient estimation, leading to more stable and effective training. The objective of the critic network can be defined as follows.

$${\varphi }^{*}=\underset{\varphi }{argmax}(-H\left(\varphi \right)),$$

(10)

where $\varphi$ is the parameter of the critic network. H denotes the objective function of the value network and is defined by the mean square error of the value function.

Taking the actor network as an example, stochastic gradient ascent (SGA) is employed to update the parameters $\theta$. Based on Equation (9), the update law of $\theta$ is defined as $\theta \leftarrow \theta +{\eta }_{\theta }{\Delta }_{\theta }J\left(\theta \right)$ with ${\eta }_{\theta }$ being the learning rate. The gradient ${\Delta }_{\theta }J\left(\theta \right)$ is reformulated in a manner related to the policy, defined as follows:

$${\Delta }_{\theta }J\left(\theta \right)=\underset{\tau \sim {\pi }_{\theta }}{\mathbb{E}}\left[\sum _{k=0}^{N-1}{\Delta }_{\theta }log{\pi }_{\theta }\left({\mathbf{a}}_k\right|{\mathbf{s}}_k)\left(J^{\mathrm{clip}}\left(\theta \right)+cS\left(\theta \right)\right)\right].$$

(11)

Updating the parameter $\theta$ based on Equation (11) increases the probability of selecting actions that yield positive feedback while decreasing the probability of choosing actions that result in negative feedback. This iterative process gradually optimizes the policy toward the optimal policy ${\pi }^{*}$.

3. Problem Formulation and Methods

This paper addresses the problem of PEG involving multiple fixed-wing UAVs and proposes an autonomous decision-making method based on subgame curriculum learning and multi-agent reinforcement learning. First, a Markov game model for the PEG scenario is established, derived from the kinematics and dynamics models of the fixed-wing UAVs. Next, the overall framework of the algorithm is introduced, followed by a detailed explanation of the subgame curriculum learning mechanism and the multi-agent proximal policy optimization within the CTDE framework.

3.1. Problem Statement

We consider a PEG involving multiple fixed-wing UAVs in a two-dimensional continuous space, where a group of pursuers aims to cooperatively capture an evader. The game is modeled as a Markov game, defined by the tuple: $\mathcal{MG}=<\mathcal{N},\mathcal{S},\mathcal{O},\mathcal{A},P,\mathbf{R},\gamma ,\rho >$.

The objective of each agent i is to maximize its expected discounted return:

$$J_i\left({\pi }_i\right)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{R}_i(s_t,a_t^i)\right],$$

(12)

where ${\pi }_i$ is the policy of agent i.

To improve training efficiency in this multi-agent setting, a subgame curriculum learning framework is introduced. Let s denotes the initial state of the subgame. A subgame curriculum sampler $\psi$ is defined to select subgames over time:

$$s\sim \Psi \left(D_{\pi }\right),$$

(13)

where $D_{\pi }$ evaluates the importance of each subgame based on the current policy $\pi$. The curriculum policy seeks to order subgames to facilitate progressive learning and faster convergence.

The training objective becomes the following:

$$\underset{\pi }{max}{\mathbb{E}}_{s_k\sim \Psi \left(D_{\pi }\right)}\left[\sum _{i\in \mathcal{I}}{J}_i\left({\pi }_i\right|s)\right].$$

(14)

This formulation integrates curriculum learning into multi-agent reinforcement learning (MARL), where the learning sequence is guided by estimated subgame importance.

3.2. Pursuit–Evasion Scene

To construct the Markov game environment for the multiple UAV pursuit–evasion scenario, it is necessary to specifically define the tuple $\mathcal{MG}=<\mathcal{N},\mathcal{S},\mathcal{O},\mathcal{A},P,\mathbf{R},\gamma ,\rho >$ of the Markov game. This includes defining the state representation, action space, reward structure, and other relevant aspects of the environment. The established Markov game environment is illustrated in Figure 2.

The Markov game environment established in this study models a PEG of multiple fixed-wing UAVs, where multiple pursuers coordinate to capture a single evader. As illustrated in Figure 2, the pursuers aim to move toward the evader from arbitrary initial positions, leveraging local observations while ensuring collision avoidance and cooperative encirclement. The success criterion for the pursuers is defined as achieving a uniform encirclement around the evader within a specified proximity. Conversely, the evader’s objective is to maximize its distance from the pursuers and avoid being encircled. The environment is formulated under a zero-sum game framework to model the adversarial nature of the pursuit–evasion interactions.

3.2.1. State Representation and Observation Model

In the PE game scenario, the state space of the environment provides a complete representation of the environment, where the situation at any given moment can be fully described by the environmental state. It is assumed that there are N drones in the environment, consisting of $n_{\mathrm{adv}}$ pursuers and $(N-n_{\mathrm{adv}})$ evaders, denoted by i, where $i<n_{\mathrm{adv}}$ represents a pursuer, and the remaining drones are evaders. Based on the previously established UAV kinematics model (1) and dynamics model (2), the state representation of the UAVs is selected as $s_i={[x,y,v,\psi ]}^{\mathrm{T}}$. Ultimately, by incorporating the current timestep of the environment, the state representation is defined as follows.

$$S={[s_1,\dots ,s_n,n_{\mathit{env}}]}^{\mathrm{T}},$$

(15)

where N represents the number of UAVs in the environment; $n_{env}$ denotes the current step count in the environment.

From the kinetic model (1) and dynamics model (2) of UAVs, the state transition equation for the fixed-wing UAV i is derived as follows.

$$s_i^{t+1}=s_i^t+\left[\begin{array}{c}{v}^{t}cos{\psi }^t\\ v^{t}sin{\psi }^t\\ g{n}_x^t\\ {\displaystyle \frac{gtan{\phi }^t}{v^t}}\end{array}\right]dt,$$

(16)

where the superscript t denotes the timestep of the environment, and $dt$ represents the time interval between consecutive environment steps.

Next, an observation model is established, consisting of the combined observations from each agent. In the PE game scenario, agents need to perceive the current game situation based on the observation information and make maneuvering decisions accordingly. It is assumed that each agent can access its own state as well as the relative states of other agents with respect to itself. To improve training efficiency and avoid the difficulties neural networks face when learning complex trigonometric functions, quantities related to angles in the observation information are directly represented by their trigonometric values rather than by the original angle values. Therefore, the observation gained by agent i is defined as $o_i=[{\overline{s}}_{ij}^{\mathrm{T}},n_{\mathit{env}},s_i^{\mathrm{T}}],j=0,\dots ,N,j\ne i;k=0,\dots ,m$ with ${\overline{s}}_{ij}={[x_j-x_i,y_j-y_i,v_j,cos{\psi }_j,sin{\psi }_j]}^{\mathrm{T}}$ being the relative state of the agent j.

3.2.2. Action Space

In the scenario studied in this paper, it is essential for each UAV to make maneuvering decisions in the pursuit–evasion game based on its own state and the relative states of other UAVs. The evader aims to escape, while the pursuers aim to surround and intercept the evader. Within this context, the focus is on position control of the UAVs, without considering the attitude control. Based on the dynamic model of the fixed-wing UAVs defined in Equation (2), the control variables for the UAVs in this paper are set as $n_x,\phi$, which include the lateral overload and the roll angle. Consequently, the environmental actions are defined as follows.

$$a={[n_x,\phi ]}^{\mathrm{T}}.$$

(17)

3.2.3. Reward Structure

The reward function serves as the cornerstone of the environment, directly shaping agent behavior through its incentive mechanism. A well-designed reward function not only accelerates convergence and enhances learning stability but also ensures that the agent’s behavior aligns with the underlying task objectives. In the multi-UAV pursuit–evasion game studied in this paper, the objective of the pursuers is to form and maintain a uniform encirclement around the evader, while the evader aims to avoid being surrounded through agile and evasive maneuvers. To achieve this control objective, the reward is decomposed into three intuitive components: a relative distance-related reward, an encirclement formation-related reward, and a mission success-related reward. Each component is crafted based on domain-specific considerations to effectively guide the agents’ behavior toward the desired outcomes.

• Relative distance-related reward. To encourage the pursuer to approach the evader and the evader to distance themselves from the pursuer, rewards and penalties are assigned based on the distance between the two parties. In addition, a stepwise reward and penalty scheme based on relative distance is designed. This metric is defined as follows.

  $$R_{\mathrm{d},\mathrm{p}}=-c_{\mathrm{d}}\sum _{i=0}^{n_{\mathrm{adv}}}|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}|+\left\{\begin{array}{cc}-50,\hfill & \mathrm{if}|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}| >5\mathrm{km}\hfill \\ 10,\hfill & \mathrm{if}|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}| \le 2\mathrm{km}\hfill \\ 0,\hfill & \mathrm{else}\hfill \end{array}\right.,$$

  (18)

  $$R_{\mathrm{d},\mathrm{e}}=c_{\mathrm{d}}\sum _{i=0}^{n_{\mathrm{adv}}}|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}|+\left\{\begin{array}{cc}50,\hfill & \mathrm{if}|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}| >5\mathrm{km}\hfill \\ -10,\hfill & \mathrm{if}|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}| \le 2\mathrm{km}\hfill \\ 0,\hfill & \mathrm{else}\hfill \end{array}\right.,$$

  (19)

  where $R_{\mathrm{d},\mathrm{p}}$ and $R_{\mathrm{d},\mathrm{e}}$ represent the relative distance-related reward of pursuers and evaders, respectively. ${\mathbf{r}}_i$ and ${\mathbf{r}}_{\mathrm{e}}$ are the position vectors of pursuers i and the evader, respectively. $c_{\mathrm{d}}$ denotes the weight coefficient of the relative distance-related reward term. When the relative distance between the pursuer and the evader is larger than the threshold $5\mathrm{km}$, the evader is considered to have achieved victory. A significant reward is assigned to the evader, while a substantial penalty is imposed on the pursuer to encourage it to close the distance. When the relative distance falls below $2\mathrm{km}$, the pursuer is deemed to have partially achieved its objective and is accordingly granted a reward. The thresholds $2\mathrm{km}$ and $5\mathrm{km}$ are determined by the farthest attack range and field of view of UAVs, respectively. Under this reward design, pursuers are encouraged to approach the evader while avoiding excessive proximity, thereby maintaining their own safety.
• Encirclement formation-related reward. To guide pursuers toward forming an encircling configuration around the evader, we employ a cosine-based metric to measure angular uniformity. This geometric formulation penalizes colinear configurations and encourages dispersion around the target. The reward term is defined as follows.

  $$R_{\mathrm{e},\mathrm{p}}=-c_{\mathrm{e}}\sum _{i\ne j}{\displaystyle \frac{({\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}})·({\mathbf{r}}_j-{\mathbf{r}}_{\mathrm{e}})}{|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}\left|\right|{\mathbf{r}}_j-{\mathbf{r}}_{\mathrm{e}}|}},$$

  (20)

  $$R_{\mathrm{e},\mathrm{e}}=c_{\mathrm{e}}\sum _{i\ne j}{\displaystyle \frac{({\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}})·({\mathbf{r}}_j-{\mathbf{r}}_{\mathrm{e}})}{|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}\left|\right|{\mathbf{r}}_j-{\mathbf{r}}_{\mathrm{e}}|}},$$

  (21)

  where $R_{\mathrm{e},\mathrm{p}}$ and $R_{\mathrm{e},\mathrm{e}}$ represent the encirclement formation-related reward of pursuers and evaders, respectively. $c_{\mathrm{e}}$ denotes the weight coefficient of the encirclement formation-related reward term. This formulation promotes pursuers spreading out in opposing directions around the evader, increasing the probability of successful encirclement.
• Mission success-related reward. A reward is provided for successful task completion. In this study, the pursuers succeed if they form a uniform encirclement around the evader within a certain distance; otherwise, the evader is considered successful. Therefore, task success is determined in conjunction with the encirclement-related rewards defined in (20) and (21). The specific reward for task completion is defined as follows.

  $$R_{\mathrm{m},\mathrm{p}}=\left\{\begin{array}{cc}50,\hfill & \mathrm{if}({\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}})·({\mathbf{r}}_j-{\mathbf{r}}_{\mathrm{e}})<0,\forall i\ne j\mathrm{AND}max|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}|<d_{\min }\hfill \\ 0,\hfill & \mathrm{else}\hfill \end{array}\right.,$$

  (22)

  $$R_{\mathrm{m},\mathrm{e}}=\left\{\begin{array}{cc}-50,\hfill & \mathrm{if}({\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}})·({\mathbf{r}}_j-{\mathbf{r}}_{\mathrm{e}})<0,\forall i\ne j\mathrm{AND}max|{\mathbf{r}}_i-{\mathbf{r}}_{\mathrm{e}}|<d_{\min }\hfill \\ 0,\hfill & \mathrm{else}\hfill \end{array}\right.,$$

  (23)

  where $R_{\mathrm{m},\mathrm{p}}$ and $R_{\mathrm{m},\mathrm{e}}$ represent the mission success-related reward of pursuers and evaders, respectively. The criteria for task success include constraints on both the capture formation and the relative distances. Specifically, the task is considered successfully completed if the angles between the direction vectors from the evader to each pursuer are all obtuse, and the maximum distance between the evader and any pursuer does not exceed $d_{\min }=2\mathrm{km}$. With this task success reward, the pursuers are guided to form an encirclement near the farthest attack range of the evader, effectively constraining the escape space of the evader while maximizing their own safety.

Taking into account the relative distance between the two parties, the encirclement formation, mission success, and the penalty of collisions, the overall reward function is designed as follows:

$$R_{\mathrm{p}}=R_{\mathrm{d},\mathrm{p}}+R_{\mathrm{e},\mathrm{p}}+R_{\mathrm{m},\mathrm{p}},$$

(24)

$$R_{\mathrm{e}}=-R_{\mathrm{p}}=R_{\mathrm{d},\mathrm{e}}+R_{\mathrm{e},\mathrm{e}}+R_{\mathrm{m},\mathrm{e}}.$$

(25)

3.3. Framework Design

The proposed algorithm framework is designed to address the PEG involving multiple fixed-wing UAVs through a self-play training scheme, in which both the pursuers and the evader simultaneously learn their strategies. The algorithm integrates automatic curriculum learning and MARL to enhance learning efficiency and robustness in complex, adversarial environments. A framework of the proposed algorithm is shown in Figure 3.

In the training stage, both the pursuers and the evaders are trained using a curriculum-based MARL framework. Through self-play, each side continuously adapts its strategy in response to the behavior of the opponents, encouraging the emergence of robust and dynamic policies that can generalize to strong adversaries. To improve sample efficiency and training stability, a subgame curriculum learning mechanism is employed, based on a subgame sampling metric and a subgame sampler. Specifically, the agents evaluate the training demand of different subgame initial states using the subgame sampling matrix, which quantifies the learning value of each state. Based on this evaluation, a subgame sampler autonomously determines the learning sequence, thereby enabling self-organized curriculum scheduling that guides the training process from simpler to more complex tasks. Finally, the policy training is conducted using the MAPPO algorithm under the CTDE framework, which allows the system to leverage centralized information for learning while maintaining decentralized decision-making during execution.

3.4. Subgame Curriculum Learning

As defined earlier, a subgame $\mathcal{MG}\left(s\right)$ refers to a Markov game $\mathcal{MG}$ that begins from a fixed state s. In the training of MARL, the underlying environment $\mathcal{MG}$ typically has a fixed initial state distribution $\rho$, from which each trajectory can be viewed as a subgame starting from a state sampled from $\rho$. However, not all subgames have the same training demand. For some subgames, the policies of agents may already be close to a Nash equilibrium, and further training yields limited improvement. In contrast, other subgames remain under-trained and exhibit a significant gap from equilibrium strategies; training in these regions can lead to substantial policy improvement. Relying solely on sampling from a fixed initial distribution may require numerous samples before encountering subgames that are critical for improving the policy, leading to poor sample efficiency. In contrast, prioritizing subgames that better improve the current policy can significantly boost sampling efficiency. To this end, a subgame curriculum learning method is proposed. This approach evaluates and records the training demand of visited states, and uses this information to guide the sampling of subgames during training, thereby allocating more learning effort to the most impactful regions of the state space.

3.4.1. Subgame Sampling Metric

To properly determine the learning sequence of subgames, it is crucial to evaluate their importance and prioritize them accordingly. As previously discussed, the training demand of a subgame is positively correlated with the gap between the current policy and the Nash equilibrium policy in that subgame. In zero-sum games, this gap is commonly measured by exploitability [32], which is defined as follows.

$${\delta }_i\left(\pi \right)={\widehat{r}}_i({\pi }_i^{*},{\pi }_{-i})-{\widehat{r}}_i\left(\pi \right),$$

(26)

where ${\delta }_i\left(\pi \right)$ represents the exploitability of agent i with policy ${\pi }_i$. $\widehat{r}$ represents the expected return of agent i. ${\pi }_i^{*}$ denotes the best response strategy to the opponent’s policy ${\pi }_{-i}$. This metric reflects the performance loss of agent i when using the current strategy instead of the optimal one against the best possible opponents.

For a given state s, the expected return of agent i is represented by the value function $V_i^{{\pi }_i}\left(s\right)$. Therefore, the exploitability at state s can be reformulated as follows:

$${\delta }_i(\pi ,s)=V_i^{{\pi }_i^{*},{\pi }_{-i}}\left(s\right)-V_i^{\pi }\left(s\right)=V_i^{*}\left(s\right)-V_i^{\pi }\left(s\right).$$

(27)

Accordingly, the training demand of a given state s is quantified as the average of the squared exploitability values across all agents, that is,

$$D\left(s\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^2{(V_i^{*}-V_i)}^2$$

(28)

Then, the probability weight of state s is defined as follows.

$$\begin{array}{cc}\hfill w\left(s\right)=& D\left(s\right)={\displaystyle \frac{1}{2}}\sum _{i=1}^2{\left(V_i^{*}\left(s\right)-V_i\left(s\right)\right)}^2\hfill \\ \hfill =& {\mathbb{E}}_i^2\left[V_1^{*}\left(s\right)-{\tilde{V}}_i\left(s\right)\right]+{\mathrm{Var}}_i\left[V_1^{*}\left(s\right)-{\tilde{V}}_i\left(s\right)\right],\hfill \end{array}$$

(29)

where ${\mathbb{E}}_i$ denotes the expectation with respect to subscripts i, which in this context refers to the average value over subscripts i, and ${\mathrm{Var}}_i$ represents the variance with respect to subscripts i. ${\tilde{V}}_i\left(s\right)={(-1)}^{i-1}{V}_i\left(s\right)$ is a standardized notation based on the zero-sum nature of the established PE game scenario, i.e., $V_1^{*}\left(s\right)=-V_2^{*}\left(s\right)$. This unifies the indices in a consistent format for easier interpretation and representation.

Since the NE value of a certain state is a constant, it does not affect the variance, that is, ${\mathrm{Var}}_i\left[V_1^{*}\left(s\right)-{\tilde{V}}_i\left(s\right)\right]={\mathrm{Var}}_i\left[{\tilde{V}}_i\left(s\right)\right]$. Additionally, the NE value of a state is generally difficult to obtain, which makes Equation (29) hard to solve. Considering that the convergence process of the policy training tends to fluctuate less near the optimal value and more when there is a larger deviation from the optimal value, the mean squared error of value changes between two consecutive policy optimizations can be used to approximate the mean squared error between the value and the NE value. Alternatively, Equation (29) can be replaced by the following form.

$$\tilde{w}\left(s\right)=\alpha {\mathbb{E}}_i^2\left[{\tilde{V}}_i^{{\pi }_i^t}\left(s\right)-{\tilde{V}}_i^{{\pi }_i^{t-1}}\left(s\right)\right]+{\mathrm{Var}}_i\left[{\tilde{V}}_i\left(s\right)\right],$$

(30)

where $\alpha$ is a hyperparameter controlling the relative importance of the two terms in Equation (30). ${\pi }_i^t$ and ${\pi }_i^{t-1}$ represent the current policy and the policy before the last update, respectively.

3.4.2. Subgame Sampler

After calculating the weight of each state, the initial state sampling is performed based on these weights, which helps plan the sequence of subsequent subgames. However, in the multi-UAV PE game scenario, the continuity of the state and action spaces leads to an infinite number of subgame processes. It is unpractical to directly calculate and sample weights over the whole state space. Alternatively, a state buffer $\mathcal{B}$ with maximum length K is established to store the visited states with their weights during the training process of the policy. In this setup, the effectiveness of curriculum selection largely depends on how well the state buffer represents the state space. The states in the buffer should ideally cover the entire state space and maintain a uniform distribution, which means that the states stored in the state buffer should be as far apart from each other as possible. Thus, the objective of the established state buffer can be expressed as follows.

$$\underset{\mathcal{B}\subset \mathcal{S}}{max}\sum _{s\in \mathcal{B}}\underset{s^{\prime }\in \mathcal{B}}{min}|s-s^{\prime }|,$$

(31)

where $\mathcal{S}$ is the state space. The distance between states is represented using the Euclidean distance.

Due to the continuous nature of UAV motion, the states collected during a single run in the environment tend to be concentrated and similar to each other. To address this issue, farthest-point sampling (FPS) [33] is used when the state buffer overflows, retaining the states with greater differences in the buffer, thereby improving the performance of the state buffer in representing the overall state space. As training progresses and more states are visited, the states in the buffer will gradually converge to the ideal distribution that covers the whole state space as broadly and uniformly as possible.

Then, the sampling probability for a certain state can be given as follows.

$$P\left(s\right)={\displaystyle \frac{\tilde{w}\left(s\right)}{{\sum }_{s^{\prime }\in \mathcal{B}}\tilde{w}\left(s^{\prime }\right)}}.$$

(32)

Combining the subgame sampling metric and the subgame sampler, the subgame curriculum learning framework is summarized in Algorithm 1.

It is worth noting that when the policies trained via subgame curriculum learning converge, the resulting policies correspond to the NE of all the subgames involved in training. In Algorithm 1, subgame sampling is not solely dependent on the constructed state buffer; instead, initial states are sampled from the state buffer $\mathcal{B}$ with probability p, and are sampled directly from the original initial state distribution $\rho$ with probability $1-p$. As a result, the training process still covers all subgames initiated from initial states $s_0$ where $\rho \left(s_0\right)>0$. Therefore, the NE obtained from training over these subgames remains a NE of the original Markov game. In this way, the proposed subgame curriculum learning strategy preserves the convergence property of the original MARL algorithm.

Algorithm 1 Subgame Curriculum Learning

Require: State buffers $\mathcal{B}$ with capacity K, probability p for sampling initial state in the state buffers, experience replay buffer $\mathcal{D}$, and Markov game environment $\mathcal{MG}$.   1: Initialize sampling probability $P\left(s\right)$, policy ${\pi }_i$ and value function $V_i$ for player $i=1,2$;   2: repeat   3:    $V_i^{\prime }\leftarrow V_i,i=1,2$;   4:    for each parallel environment do   5:      Sample $s^0\sim P\left(s\right)$ (sample initial state from the state buffer) with probability p, else $s^0\sim \rho (·)$ (sample initial state from the original distribution);   6:      Rollout in $\mathcal{MG}\left(s^0\right)$ and collect samples into $\mathcal{D}$;   7:    end for   8:    Train ${{\pi }_i,V_i}_{i=1}^2$ via MAPPO;   9:    ${\tilde{w}}^t\leftarrow \alpha {\mathbb{E}}_i^2\left[{\tilde{V}}_i^{{\pi }_i^t}\left(s\right)-{\tilde{V}}_i^{{\pi }_i^{t-1}}\left(s\right)\right]+{\mathrm{Var}}_i\left[{\tilde{V}}_i\left(s\right)\right],t=0,\dots ,T$; 10:    $\mathcal{B}\leftarrow \mathcal{B}\cup {(s^t,{\tilde{w}}^t)}_{t=0}^T$; 11:    if $\parallel \mathcal{B}\parallel >K$ then 12:      $\mathcal{B}\leftarrow FPS(\mathcal{B},K)$; 13:    end if 14:    Update sampling probability $P\left(s\right)$ with new weights ${\tilde{w}}^t$ and new state buffer $\mathcal{B}$; 15: until $({\pi }_1,{\pi }_2)$ converges.

3.5. Multi-Agent Reinforcement Learning

Building on the foundation of subgame curriculum learning, the multi-agent proximal policy optimization (MAPPO) algorithm is used for multi-agent policy learning. MAPPO is an extended version of the PPO algorithm within the CTDE framework. The core idea of MAPPO is to jointly optimize the learning process of multiple agents by using a centralized training approach while allowing for decentralized execution.

As a multi-agent decision-making framework that lies between centralized and decentralized decision-making, the CTDE framework allows agents to share global information during training to simplify the training process, while maintaining decentralized and independent decision-making during execution. PPO is based on the actor–critic (AC) framework, requiring both a critic network to estimate value and an actor network to make decisions. Since the critic network is not needed during decision-making, in the training phase, the critic network is built using joint observation ${\mathbf{o}}^t=\{o_1^t,\dots ,o_N^t\}$ and joint actions $\mathbf{a}=\{a_1^t,\dots ,a_N^t\}$. The actor network, on the other hand, is built based on local observations to maintain the distributed nature of execution. In this setup, the value optimization objective $L\left(\varphi \right)$ is defined as follows.

$$L\left(\varphi \right)={\displaystyle \frac{1}{BN}}\sum _{t=1}^B\sum _{i=1}^N\left(max[{(V_{\varphi }\left(s^t\right)-{\widehat{R}}^t)}^2,{(\mathrm{clip}(V_{\varphi }\left(s^t\right),V_{{\varphi }_{old}}\left(s^t\right)-\epsilon ,V_{{\varphi }_{old}}\left(s^t\right)+\epsilon )-{\widehat{R}}^t)}^2]\right),$$

(33)

where B and N denote the batch size and the number of agents, respectively. ${\widehat{R}}^t$ is the discounted sum of the rewards after step t, which is the true value at state $s^t$. $\mathrm{clip}(V_{\varphi }\left(s^t\right),V_{{\varphi }_{old}}\left(s^t\right)-\epsilon ,V_{{\varphi }_{old}}\left(s^t\right)+\epsilon )$ is the function that constrains the value of $V_{\varphi }\left(s^t\right)$ within the range $[V_{{\varphi }_{old}}\left(s^t\right)-\epsilon ,V_{{\varphi }_{old}}\left(s^t\right)+\epsilon ]$ with $\epsilon$ being the hyperparameter that controls the update magnitude of the parameter optimization.

And the objective of the actor network $L\left(\theta \right)$ can be defined as follows.

$$L\left(\theta \right)={\displaystyle \frac{1}{BN}}\sum _{t=1}^B\sum _{i=1}^N\left(min[r_{\theta ,i}^t{A}_i^t,\mathrm{clip}(r_{\theta ,i}^t,1-ϵ,1+ϵ)A_i^t]\right)+\sigma {\displaystyle \frac{1}{BN}}\sum _{t=1}^B\sum _{i=1}^{N}S\left({\pi }_{\theta }\left(o_i^t\right)\right),$$

(34)

where $r_{i,t}^{\theta }={\displaystyle \frac{{\pi }_{\theta }\left(a_i^t\right|o_i^t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_i^t\right|o_i^t)}}$ is the importance ratio. S is the policy entropy, and $\sigma$ is the entropy coefficient hyperparameter. $A_i^t$ is the advantage function. The generalized advantage estimate (GAE) is applied to estimate the value of the advantage function unbiasedly with the following form.

$${\widehat{A}}_i^t=\sum _{t^{\prime }=t}^{n-1}{\left(\gamma \lambda \right)}^{t^{\prime }-t}(R_i^t+\gamma V_{\varphi }^{t+1}\left(s_i^{t+1}\right)-V_{\varphi }^t\left(s_i^t\right)),$$

(35)

where $\lambda$ is the tunable hyperparameter in the GAE.

Algorithm 2 summarizes the training framework. First, the actor and critic networks are initialized, and an empty experience pool $\mathcal{D}$ is created. The iterative training process then begins. In each iteration, each agent generates actions based on its local observations and interacts with the environment. The environment transitions to the next state and generates a reward R. These data are stored in the experience pool $\mathcal{D}$, and the advantage function is estimated using Equation (35). Once sufficient data have been collected, the parameters are updated, and the experience pool is cleared before repeating the process.

Algorithm 2 MAPPO

Require: Markov game environment $\mathcal{MG}$, empty experience replay buffer $\mathcal{D}$, learning rate ${\eta }_{\theta }$, ${\eta }_{\varphi }$ for the actor network and critic network.   1: Initialize $\theta$, the parameters for actor $\pi$ and $\varphi$, the parameters for critic V, by Orthogonal initialization;   2: while $step\le ste{p}_{max}$ do   3:    $\mathcal{D}\leftarrow \left\{\right\}$;   4:    for each iteration do   5:      $\tau \leftarrow \left[\right]$;   6:      Reset the environment, get initial observation ${\mathbf{o}}^t$   7:      for each environment step t do   8:         for each agent i do   9:           $a_i^t\sim {\pi }_{\theta ,i}\left(o_i^t\right)$; 10:         end for 11:         $V^t\sim V_{\varphi }\left({\mathbf{o}}^t\right)$; 12:         Execute action ${\mathbf{a}}^t$, get rewards ${\mathbf{R}}^t$, state $s^{t+1}$ and observation ${\mathbf{o}}^{t+1}$; 13:         $\tau \leftarrow \tau +[s^t,{\mathbf{o}}^t,{\mathbf{a}}^t,{\mathbf{R}}^t,s^{t+1},{\mathbf{o}}^{t+1}]$ 14:      end for 15:      Compute advantage estimate $\widehat{A}$ via Equation (35) on trajectory $\tau$; 16:      Compute reward-to-go $\widehat{R}$ on $\tau$ and normalize it; 17:      $\mathcal{D}\leftarrow \mathcal{D}\cup \{\tau ,\widehat{A},\widehat{R}\}$; 18:    end for 19:    for each PPO epoch do 20:      $\varphi \leftarrow \varphi -{\eta }_{\varphi }{\nabla }_{\varphi }L\left(\varphi \right)$ update the critic network by Equation (33) 21:      $\theta \leftarrow \theta -{\eta }_{\theta }{\nabla }_{\theta }L\left(\theta \right)$ update the actor network by Equation (34) 22:    end for 23: end while

4. Results

In this section, the multiple fixed-wing UAV PE environment is implemented based on the Gym framework to validate the effectiveness of the proposed method.

4.1. Simulation Setup

4.1.1. Environmental Parameters

In this work, a multi-UAV PE training environment was developed based on the OpenAI Gym interface, and the pursuing and evading strategies were trained using PyTorch (Python 3.9). Specifically, the simulation scenario consists of three pursuer UAVs (UAV0, UAV1, and UAV2) and one evader UAV (UAV3). As illustrated in Figure 4, the evader is randomly initialized within the light blue region of the environment, i.e., $\left\{\right(x,y\left)\right|-3\le x\le -1,-3\le y\le -1\}$, while the pursuers are randomly initialized within the light red region, i.e., $\left\{\right(x,y\left)\right|1\le x\le 3,1\le y\le 3\}$. The initial yaw and roll angles of all the UAVs are set to zero, that is, ${\psi }^0={\phi }^0=0$. The initial speed of all the UAVs is set to the minimum speed of itself, i.e., $v^0=v_{\min }$. The maximum number of environment steps is limited to 200, with a time interval of $0.5$ s between consecutive steps. To enhance the accuracy of the discretized model, each environment step is further divided into five computational sub-steps to perform kinematic calculations, thereby approximating continuous-time integration. For the PE environment, the objective of the pursuers is to collaboratively encircle the evader within the maximum number of steps, maintaining a close and uniform formation around the evader to prevent escape. Meanwhile, the evader aims to maximize its distance from the pursuers and avoid being encircled. To ensure that the reward for task success is comparable in magnitude to the penalty for task failure, and that the reward for achieving proximity is similar to that for forming a uniform surrounding in terms of angular distribution, the reward weights are set as follows: $c_{\mathrm{d}}=1,c_{\mathrm{e}}=10$.

To ensure a meaningful and solvable pursuit–evasion interaction, the evader is designed to have slightly inferior maneuverability compared to the pursuers. This design choice addresses a key issue arising from the absence of environmental boundaries: if the evader has equal or superior maximum speed, it could trivially evade capture by continuously moving away from the pursuers, rendering the game unsolvable and uninformative. For example, in the initial configuration shown in Figure 4, the evader is initialized at the bottom-left corner of the map, while the pursuers are placed in the top-right corner. That is, all pursuers are initially located at a relative position to the upper right of the evader. In this setting, the evader’s optimal escape strategy is to move steadily toward the bottom left, maintaining maximum distance from the pursuers. Since there is a non-zero initial distance between the two sides, if the evader has equal or greater speed than the pursuers, this distance would remain constant or even increase over time, making it impossible for the pursuers to ever approach, let alone encircle or capture the evader. By limiting the evader’s speed, we encourage strategic behavior from both sides and facilitate effective training of pursuit policies.

To mitigate this, the maximum speed of the pursuer UAVs and the evader UAV are $v_{\max ,\mathrm{p}}=400\mathrm{m}/\mathrm{s}$ and $v_{\max ,\mathrm{e}}=300\mathrm{m}/\mathrm{s}$, respectively. The minimum speed is $v_{\min ,\mathrm{p}}=v_{\min ,\mathrm{e}}=100\mathrm{m}/\mathrm{s}$. For control space, set $n_x\in [-1,2]$, and $\phi \in [-arccos(1/8),$$arccos(1/8)]$.

4.1.2. Hyperparameters of Algorithm

The hyperparameters involved in the proposed algorithm are summarized in

Table 1. Hyperparameters related to the proposed algorithm.

Hyperparameter	Value
Learning rate of actor network (${\eta }_{\theta }$)	$5\times {10}^{-4}$
Discount rate ($\gamma$)	$0.99$
GAE parameter (${\lambda }_{\mathrm{GAE}}$)	$0.95$
Gradient clipping	$10.0$
Adam stepsize	$1\times {10}^{-5}$
Entropy coefficient ($\sigma$)	$0.01$
Parallel threads	100
PPO clipping range ($ϵ$)	$0.2$
PPO epochs	5
Probability of subgame sampling (p)	$0.7$
Capacity of the state buffer (K)	10,000
Weight of the value difference ($\alpha$)	$0.7$

, including those related to MAPPO [34] as well as the subgame curriculum learning method proposed in this paper. The hyperparameters related to MAPPO were set based on those used in similar encirclement tasks [28]. Only minor adjustments were made to the number of parallel threads and PPO epochs according to hardware constraints. Additionally, the hyperparameters related to the subgame curriculum learning were determined empirically through multiple trial experiments.

It is worth noting that, due to the high-dimensional nature of the observation space where each agent perceives not only its own state but also the relative states of all other agents, this study employs a Transformer-based architecture for constructing both the actor and critic networks. Specifically, the input observations are first processed through a LayerNorm layer, and then the normalized observations are partitioned into different agents, including the agent itself, collaborators, and opponents. Each agent is then passed through a fully connected layer with 32 units to obtain its embedding, with the weights shared among entities of the same type. Subsequently, the embedding of each agent is concatenated with its corresponding observation and processed by a self-attention network. The outputs of the attention module are normalized and concatenated with the self-embedding to obtain the final representations. These representations are then passed through two fully connected hidden layers, each with 64 units, followed by separate network heads: one for producing the value estimation (critic) and another for generating the action distribution (actor).

For subgame curriculum learning, a state buffer $\mathcal{B}$ is established to store visited states and to sample from these states for subgame training. Since the environment developed in this study does not impose any spatial boundaries on the UAVs, the corresponding state space is theoretically infinite. Constructing a buffer that covers the entire state space would thus be impractical. Alternatively, a localized state buffer is built around the initial state region, specifically including only those states with the positions within the range of $\left\{\right(x,y\left)\right|-3\le x\le 3,-3\le y\le 3\}$. A weighting mechanism is applied to these stored states to guide subgame sampling. Compared to global state-space sampling, this localized approach focuses subgame curriculum learning around the initial phase of the episode. This not only maintains sufficient curriculum learning performance but also significantly improves computational efficiency and the representational capacity of the state buffer with fixed capacity.

4.2. Results and Discussion

Simulation validation was conducted based on the environment and algorithm settings described above. Under the subgame curriculum framework, training was performed for a total of 250 million environment steps. The obtained results are presented as follows. Figure 5 shows the single-step average rewards obtained by both the pursuers and the evader during the training process. Figure 6 illustrates the evolution of the encirclement success rate of the pursuers over the course of training.

As shown in Figure 5, after training for 250 million environment steps, the single-step average reward of the pursuers approaches 0. Since the pursuers receive penalties when they are far from the evader and the initial distance between the pursuers and evaders in testing environments is set to be greater than the penalty threshold distance, the pursuers initially accumulate substantial penalties. The single-step average reward is calculated by averaging the total rewards obtained throughout the episode over the total number of environment steps. Therefore, when the single-step average reward approaches zero, it indicates that the pursuers consistently achieve successful captures in the later stages of the task, with the rewards obtained compensating for the penalties incurred earlier. Due to the self-play training framework employed between the pursuers and the evader, the reward curves are not strictly monotonically increasing or decreasing. Overall, the evolution of the rewards can be divided into three stages: $0\sim 80\mathrm{M}$, $80\mathrm{M}\sim 160\mathrm{M}$, and $160\mathrm{M}\sim 250\mathrm{M}$ steps. In the first stage, the pursuers learn to approach the evader in order to reduce penalties and gain rewards, resulting in a rapid increase in their average reward. In the second stage, although the pursuers develop effective approaching policies, learning the more complex coordinated encirclement policies proves to be challenging, whereas the training of the policy of the evader is comparatively easier. As a result, the rewards of the pursuers slightly decline. In the third stage, the pursuers gradually master effective encirclement policies, preventing the evaders from breaking out of the encirclement and thereby earning higher rewards. As a result, the average reward of the pursuers rises again. However, due to the improved evading policy at this stage, the growth rate of the reward is relatively moderate.

As shown in Figure 6, the trend of the encirclement success rate curve corresponds closely to the reward curve and can also be approximately divided into the three aforementioned stages. In the first stage, due to the limited decision-making capability of the evader, the capture success rate rises rapidly. However, the success rate only increases to about $60\%$, indicating that the pursuers have not yet learned effective encirclement policies; the rapid rise in rewards is largely attributed to the achievement of secondary objectives (reducing the distance to the evader). In the second stage, as the decision-making ability of the evader improves, the encirclement success rate declines. In the third stage, the pursuers progressively master effective encirclement policies, resulting in a significant increase in the encirclement success rate to over $80\%$.

To validate the effectiveness of the subgame curriculum learning method, the initial positions of the evader in the sampled subgames were recorded during the training process, as shown in Figure 7. In the figure, both the horizontal and vertical axes are in units of kilometers (km). As illustrated in Figure 7, as training progresses, the initial positions of the evader gradually shift from an initial uniform distribution to being concentrated near the edges of the state buffer and eventually toward the lower-left corner. This evolution indicates that, during the training process, the initial positions of the evaders gradually move farther away from the pursuers, meaning that the selection of subgames progressively shifts from simpler scenarios to more challenging ones.

To better demonstrate the effectiveness of the pursuers in capturing the evader, the trained model was tested, and the UAV trajectories at various stages of task execution are illustrated in Figure 8. In the figure, both the horizontal and vertical axes are in units of km. The starting positions of the agents are indicated by dots, while the final positions are marked by crosses. As shown, at the beginning of the task, the average initial distance between the pursuers and the evader exceeds $5\mathrm{km}$. Since both the pursuers and the evader initially move in the positive x-axis direction, the pursuers experience significant penalties during the early phase of the task. This also explains why the final per-step average reward of the pursuers remains around 0. During the first 65 steps, the evader adjusts its heading to move away from the pursuers, while the pursuers modify their trajectories to close the gap with the evader. By step 130, it can be observed that the pursuers have successfully encircled the evader. Due to the relatively even distribution of the pursuers around the evader, it becomes difficult for the evader to break through the encirclement. Constrained by its minimum speed and turning radius, the evader is forced into circular motion to maintain its current reward and avoid further penalties. In the later stages of the task, the pursuers consistently maintain the encirclement until the task concludes. From the above process, it can be observed that the pursuers successfully carried out an effective encirclement of the evader, forming a uniform surrounding formation at a close distance and effectively preventing the evader from escaping further.

Figure 9 illustrates the comparison between the proposed algorithm with two baseline algorithms—MAPPO [34] and PPO [31]—in terms of the pursuers’ reward curves and encirclement success rates after the same number of training steps. In the figure, the proposed method is abbreviated as SCL-MAPPO. Due to the zero-sum nature of the game, the reward curve of the evader is simply the mirrored version of the pursuers’ reward curve across the x-axis. For clarity, only the pursuers’ rewards are shown. The hyperparameter settings for MAPPO and PPO are consistent with those listed in Table 1. As shown in Figure 9a, after 250 million training steps, the training curves of MAPPO and PPO can be roughly divided into two phases: $0\sim 170\mathrm{M}$ and $170\mathrm{M}\sim 250\mathrm{M}$ for MAPPO, and $0\sim 210\mathrm{M}$ and $210\mathrm{M}\sim 250\mathrm{M}$ for PPO. These phases approximately correspond to the first two training stages of the proposed method: $0\sim 80\mathrm{M}$, where the pursuers’ rewards increase, and $80\mathrm{M}\sim 160\mathrm{M}$, where the rewards slightly decrease. Figure 9b presents the encirclement success rates throughout training. The proposed method consistently outperforms PPO and only slightly underperforms MAPPO during a brief period in the middle of training. By the end of training, SCL-MAPPO achieves significantly higher success rates compared to both baselines. Taken together, these results demonstrate that the proposed subgame curriculum learning framework effectively improves both sampling efficiency and convergence speed.

To evaluate the robustness of the proposed algorithm, we tested its performance under different random seeds. A total of 1000 trials were conducted, and the reward distribution statistics are presented in Figure 10. In the figure, the blue box represents the interquartile range, capturing the middle $50\%$ of the data. The red line inside the box indicates the median, the green arrow marks the mean, and the hollow circles denote outliers that fall outside the typical data range. As shown in the figure, the interquartile range of the box lies between $-10$ and 10 with few outliers, indicating a dense reward distribution. This suggests that the proposed algorithm maintains relatively stable policy performance across different random seeds, with a high median reward, demonstrating good generalization and robustness. In addition, the algorithm achieved a capture success rate of $94.1\%$ over 1000 trials, providing further evidence of its effectiveness.

Considering that the constructed model involves several simplifications, we introduce Gaussian noise to compensate for unmodeled dynamics and various external disturbances encountered during UAV flight in order to reduce the gap between simulation and real-world scenarios. Noise ${[{\eta }_x,{\eta }_y,{\eta }_v,{\eta }_{\psi }]}^T$ is independently injected into the state variables $s={[x,y,v,\psi ]}^T$ of the fixed-wing UAVs. All noise terms are sampled from zero-mean Gaussian distributions. In this study, the standard deviations for each state variable are set as follows: ${\sigma }_x={\sigma }_y={\sigma }_v=0.02\mathrm{km},{\sigma }_{\psi }=0.2\mathrm{rad}$, which were derived by proportionally scaling the uncertainty parameters used in [26]. We retrain our algorithm in this uncertainty-augmented environment. The resulting average rewards and encirclement success rates are shown in Figure 11. As shown in the figure, during the early stages of training, the reward curves and encirclement success rates for agents trained in both deterministic and uncertain environments are quite similar, indicating comparable performance. However, in the later stages of training, agents trained under uncertainty exhibit faster reward growth and higher capture success rates. This improvement is possibly due to the presence of uncertainty disrupting the local “stalemate” between the pursuers and evaders, enabling the agents to escape local optima and discover more effective strategies. Overall, the introduction of uncertainty does not degrade the performance of the algorithm; on the contrary, it leads to noticeable improvements. This further demonstrates the robustness of the proposed method and provides a preliminary attempt toward sim-to-real transfer.

Furthermore, regarding the generalizability of the proposed method, we believe it demonstrates a certain degree of robustness. The reasons are as follows: First, the training environment is configured with challenging initial conditions, where all pursuers are initialized on the same side of the evader and at a disadvantageous distance. Learning under such difficult scenarios encourages the development of robust coordination strategies. Second, both pursuers and the evader rely solely on relative observations, including the positions and headings of other agents relative to themselves. This design ensures that the learned policy is invariant to global position shifts or rotations, allowing it to generalize naturally to environments with different initial placements that preserve the underlying task structure. Additionally, we find that the learned policy transfers effectively to environments with different numbers of agents in multi-pursuer single-evader settings. Since the core objective—achieving coordinated encirclement—remains unchanged, the trained agents can adapt to varying team sizes with minimal performance degradation. However, in multi-evader scenarios, the nature of the task changes significantly, requiring more complex decision-making such as target assignment and dynamic regrouping, which are not addressed by the current model. Extending the framework to handle such cases constitutes an important direction for future research.

5. Conclusions

This paper focuses on the problem of distributed autonomous decision-making for fixed-wing UAVs in PEGs based on multi-agent reinforcement learning. A Markov PEG model is established, considering the dynamics and relevant constraints of fixed-wing UAVs. Based on a subgame curriculum learning mechanism and a self-play framework, systematic training and validation are conducted using the MAPPO algorithm within the CTDE framework. Simulation results demonstrate that the proposed reward structure effectively guides the pursuers to learn efficient strategies within the designed pursuit–evasion scenarios, enabling them to achieve a high capture success rate and prevent the evaders from further escaping. In addition, the introduced subgame curriculum learning mechanism further facilitates the training process by progressively guiding the agents from simple to more complex tasks. Finally, testing and analysis confirm the control effectiveness of the trained model in practical pursuit–evasion missions, highlighting the feasibility and potential of the proposed method.

While the proposed framework demonstrates promising performance in simulated 2D pursuit–evasion scenarios with fixed-wing UAVs, several limitations remain in the current work. First, all experiments are conducted in a two-dimensional simulation environment, which may not fully reflect the dynamics and challenges of real-world three-dimensional UAV deployments. Second, this study is restricted to many-to-one pursuit scenarios under zero-sum game assumptions. This limits the generality of the proposed approach, especially in scenarios requiring target assignment strategies or cooperative behaviors among agents. Additionally, practical issues such as sensing noise, communication constraints, and real-time control delays are not considered and may affect real-world deployment.

To address the above limitations, potential future research directions include the following: (1) extending the framework to three-dimensional environments to better reflect realistic UAV motion and control challenges; (2) adapting the method to handle multi-target cooperative pursuit tasks, which will require integrating target allocation and coordination mechanisms; (3) exploring non-zero-sum settings to model more general interaction dynamics beyond strictly competitive games; and (4) investigating practical deployment challenges by incorporating hardware-in-the-loop simulation, sensor noise modeling, and communication constraints to evaluate the real-world feasibility of the proposed approach.