SA-DSM-MADDPG for Multi-UAV Cooperative Encirclement in Obstacle-Rich Pursuit–Evasion Scenarios

Highlights

What are the main findings?

• We propose SA-DSM-MADDPG for 3v1 multi-UAV cooperative encirclement, integrating a self-attention critic, double-screened experience replay (PER + relevance screening), and curriculum learning.
• Experimental results show that SA-DSM-MADDPG achieves a higher cooperative capture success rate and more stable convergence than the MADDPG baseline in obstacle-rich environments.

What are the implications of the main finding?

• The improved success rate indicates that combining interaction-aware coordination modeling (attention), phase-relevant sample selection (DSM), and staged reward shaping (curriculum) effectively mitigates sparse-feedback learning issues in pursuit–evasion tasks.
• The proposed design provides a practical guideline for developing more reliable multi-drone interception/containment decision policies in cluttered environments under the CTDE paradigm.

Abstract

Multi-UAV cooperative encirclement in pursuit–evasion scenarios requires effective coordination under dynamic inter-agent interactions, sparse task feedback, and obstacle-constrained motion. While MADDPG offers a practical CTDE framework for multi-agent continuous control, its direct application to cooperative encirclement still faces challenges in modeling time-varying teammate dependencies, selecting informative replay samples, and maintaining stable learning under delayed rewards. To address these challenges, we propose SA-DSM-MADDPG, an enhanced multi-agent deep deterministic policy gradient method that integrates the following: (i) a self-attention critic to model dynamic inter-agent relevance, (ii) a double-screened experience replay strategy combining prioritized sampling and relevance screening to improve replay quality, and (iii) curriculum learning with staged reward shaping to provide denser and more stable training signals. We evaluate the proposed method in 3v1 cooperative encirclement environments with static obstacles and varying initial conditions. Experimental results show that SA-DSM-MADDPG improves the success rate by approximately 22 percentage points over MADDPG and 35 percentage points over MAPPO, while also exhibiting faster convergence and better training stability.

1. Introduction

Multi-UAV cooperative encirclement is a representative class of pursuit–evasion problems, in which multiple pursuers coordinate to constrain and eventually capture an evader [1,2,3]. This capability is relevant to a wide range of drone applications, including cooperative security patrol, intruder interception, and multi-robot target containment in cluttered environments [4,5]. Compared with single-agent pursuit, cooperative encirclement requires not only effective individual motion control but also team-level coordination to block escape routes, preserve formation coherence, and avoid collisions with teammates and obstacles [1,2].

Reinforcement learning (RL), especially multi-agent reinforcement learning (MARL), provides a promising data-driven paradigm for learning such cooperative behaviors without explicitly prescribing all interaction rules [6,7]. Among MARL frameworks, centralized training with decentralized execution (CTDE) is particularly attractive for UAV swarms, because each UAV can execute a local policy using onboard observations while leveraging richer joint information during training [6,7].

MADDPG is a representative CTDE baseline for continuous-control multi-agent tasks [6], and recent MADDPG-based variants have also been explored for multi-UAV pursuit–evasion games [8,9]; however, its direct application to cooperative encirclement still faces several challenges.

First, cooperative encirclement involves dynamic inter-agent dependencies. Each pursuer must continuously adapt its motion according to the evader’s maneuver and the behaviors of teammates, while the relevance of neighboring agents changes over time. Under such conditions, simple feature concatenation is often insufficient to represent structured teammate interactions, especially when coordination quality depends on which teammates are most informative at a given moment [10,11]. Second, the encirclement objective usually provides sparse and delayed supervisory signals, because high rewards are mainly obtained when the team successfully constrains or captures the evader. This weak feedback reduces exploration efficiency and often leads to unstable early-stage learning [12,13,14]. Third, off-policy replay in MARL may contain transitions with highly uneven learning value. In cooperative pursuit–evasion tasks, replay buffers often mix informative coordination samples with weakly relevant or low-quality transitions, which can reduce sample efficiency and aggravate training instability in non-stationary multi-agent settings [15].

Although these issues have been partially discussed in prior studies, existing MADDPG-based methods for pursuit–evasion and encirclement have not sufficiently explored how dynamic interaction modeling, replay refinement, and staged reward guidance can be integrated within a unified framework for cooperative encirclement. In many cases, existing approaches enhance only one aspect of the learning process, while the coupling among coordination modeling, experience selection, and reward scheduling remains insufficiently studied. Therefore, the literature gap is not simply the absence of an additional module, but the lack of an integrated training framework tailored to the coupled challenges of cooperative encirclement in obstacle-rich environments.

To address this gap, we propose SA-DSM-MADDPG (Self-Attention and Double-Screening Mechanism enhanced MADDPG), an integrated framework that improves MADDPG from three complementary aspects. First, we introduce a self-attention mechanism into the centralized critic so that each agent can adaptively focus on the most relevant teammate information, thereby enhancing interaction modeling and coordination learning under dynamic multi-agent dependencies [10,11,16]. Second, we design a double-screened experience replay (DSM) mechanism that combines prioritized experience replay (PER) with relevance screening. PER increases the reuse frequency of high-value samples [15], while relevance screening suppresses weakly informative or poorly matched transitions, thereby improving replay quality and reducing interference from heterogeneous experiences. Third, we introduce curriculum learning with staged reward shaping, which decomposes the encirclement task into progressively structured learning phases and provides denser supervision in early training. This design improves exploration efficiency and stabilizes policy optimization before strict encirclement objectives dominate learning [12,13,14].

More importantly, these three components are not independent add-ons. The self-attention critic improves the representation of dynamic coordination relationships, DSM improves the informativeness of sampled training data, and curriculum learning improves the temporal structure of the optimization process. Together, they form a coordinated learning framework for multi-UAV encirclement, in which better interaction modeling, better replay quality, and better reward guidance jointly contribute to improved convergence and cooperative performance.

We evaluate SA-DSM-MADDPG in 3v1 cooperative encirclement scenarios with static obstacles and varying initial conditions. Experimental results show that the proposed method achieves faster convergence, higher training stability, and higher success rates than strong baselines. These results indicate that integrating attention-based coordination modeling, replay refinement, and staged reward guidance is effective for cooperative encirclement in obstacle-rich pursuit–evasion environments.

Contributions

The main contributions of this work are summarized as follows:

• We propose SA-DSM-MADDPG, an integrated CTDE framework for multi-UAV cooperative encirclement that addresses three coupled challenges in MADDPG-based training: dynamic inter-agent dependency modeling, uneven replay quality, and sparse stage-dependent supervision.
• We develop a self-attention-enhanced centralized critic and a double-screened experience replay mechanism to improve coordination learning and replay efficiency. The self-attention module enables each agent to emphasize more relevant teammate information under time-varying interactions, while the DSM mechanism combines prioritized sampling and relevance screening to suppress weakly informative transitions and improve training stability.
• We design a curriculum learning strategy with staged reward shaping for pursuit–evasion encirclement, which provides denser guidance in early training and progressively strengthens task objectives, thereby improving convergence behavior and cooperative performance in obstacle-rich environments.
• Extensive experiments in 3v1 cooperative encirclement scenarios demonstrate that the proposed method achieves faster convergence, improved training stability, and higher success rates than strong baselines, including approximate success-rate gains of 22 percentage points over MADDPG and 35 percentage points over MAPPO.

Figure 1 illustrates the multi-UAV cooperative encirclement task considered in this work, and Figure 2 presents the overall architecture of the proposed SA-DSM-MADDPG framework.

2. Problem Formulation

2.1. Task Description

We consider a cooperative pursuit–evasion task in which $N_{\mathrm{pursuit}}$ pursuer UAVs cooperate to encircle and capture a single evader UAV in a bounded 2D environment with static obstacles [1,2,3]. The pursuers aim to prevent the evader from escaping by forming an enclosing formation while maintaining safety constraints (collision avoidance among UAVs, obstacle avoidance, and boundary constraints).

Let $\mathcal{P}=\{1,2,\dots ,N_{\mathrm{pursuit}}\}$ denote the set of pursuers and e denote the evader. The arena is a rectangle

$$\Omega =[0,W]\times [0,L],$$

(1)

with $W=200\mathrm{m}$ and $L=200\mathrm{m}$, containing circular obstacles with radius $R_{\mathrm{obstacle}}$.

2.2. UAV Dynamics and Control Constraints

We adopt a simplified planar point-mass model, which is commonly used in reinforcement-learning-based pursuit–evasion and encirclement studies to balance environmental fidelity and training efficiency [1,2,13]. This abstraction allows the learning algorithm to focus on cooperative decision-making and formation evolution under obstacle constraints, while avoiding the additional complexity introduced by higher-order flight dynamics, attitude control, and aerodynamic effects. Therefore, the present model should be regarded as a controlled benchmark for algorithm validation rather than a full representation of real UAV motion. The limitations of this simplification and its implications for real-world deployment are further discussed in the Conclusions section.

2.3. Geometry, Safety Constraints, and Terminal Conditions

We model each UAV as a disc. Pursuer and evader radii are

$$R_{\mathrm{pursuit}}=R_{\mathrm{evader}}=15\mathrm{m}.$$

(2)

2.3.1. UAV–UAV Safety Constraint

Let ${\delta }_{\mathrm{safety}}$ be the minimum required clearance between UAV bodies. We define a UAV–UAV safety violation (or collision) if the distance between any two pursuers $i,j\in \mathcal{P}$ satisfies

$$\parallel {\mathbf{p}}_i\left(t\right)-{\mathbf{p}}_j\left(t\right)\parallel \le 2R_{\mathrm{pursuit}}+{\delta }_{\mathrm{safety}},$$

(3)

with ${\delta }_{\mathrm{safety}}=10\mathrm{m}$.

2.3.2. UAV–Obstacle Safety Constraint

Obstacles are modeled as discs with radius $R_{\mathrm{obstacle}}=2\mathrm{m}$ and center ${\mathbf{c}}_m$. A pursuer–obstacle violation occurs if

$$\parallel {\mathbf{p}}_i\left(t\right)-{\mathbf{c}}_m\parallel \le R_{\mathrm{pursuit}}+R_{\mathrm{obstacle}},\exists m.$$

(4)

A similar condition applies to the evader using $R_{\mathrm{evader}}$.

2.3.3. Boundary Constraint

A boundary violation occurs if a UAV leaves the feasible region considering its body radius:

$$x_i\left(t\right)\notin [R_i,W-R_i]\mathrm{or}{y}_i\left(t\right)\notin [R_i,L-R_i],$$

(5)

where $R_i=R_{\mathrm{pursuit}}$ for pursuers and $R_i=R_{\mathrm{evader}}$ for the evader.

2.3.4. Capture Criterion

We define capture as a distance-based contact condition between any pursuer and the evader:

$$\exists i\in \mathcal{P}\mathrm{s}.\mathrm{t}.d_{i,e}\left(t\right)=\parallel {\mathbf{p}}_i\left(t\right)-{\mathbf{p}}_e\left(t\right)\parallel \le R_c,R_c=15\mathrm{m}.$$

(6)

2.3.5. Encirclement Criterion

To prevent trivial “touch-and-go” outcomes, we additionally require that the evader be enclosed by the pursuers’ formation [1,2]. Let $\mathcal{H}\left(t\right)$ denote the convex hull of the pursuers’ positions ${\left\{{\mathbf{p}}_i\left(t\right)\right\}}_{i\in \mathcal{P}}$. The evader is considered enclosed if

$${\mathbf{p}}_e\left(t\right)\in \mathcal{H}\left(t\right).$$

(7)

We adopt this convex-hull-based criterion because it provides a simple and interpretable geometric test of whether the evader lies inside the pursuer formation, thereby aligning the success condition with the notion of formation-based containment rather than mere momentary contact. In this sense, the convex hull serves as an operational approximation of cooperative enclosure in the studied 3v1 setting.

2.4. Observation and Action Spaces

Observation

During decentralized execution, each pursuer i selects actions based on its local observation ${\mathbf{o}}_i\left(t\right)$. To improve reproducibility, we explicitly specify the observation vector used in the experiments. For each pursuer, the local observation is constructed by concatenating four parts:

$${\mathbf{o}}_i\left(t\right)=\left[{\mathbf{o}}_i^S\left(t\right),{\mathbf{o}}_i^P\left(t\right),{\mathbf{o}}_i^B\left(t\right),{\mathbf{o}}_i^E\left(t\right)\right],$$

(8)

where ${\mathbf{o}}_i^S\left(t\right)$ denotes the self-state observation, ${\mathbf{o}}_i^P\left(t\right)$ denotes teammate-related observation, ${\mathbf{o}}_i^B\left(t\right)$ denotes obstacle/boundary sensing information, and ${\mathbf{o}}_i^E\left(t\right)$ denotes evader-related observation.

The self-state observation contains the normalized position and velocity of pursuer i:

$${\mathbf{o}}_i^S\left(t\right)=\left[{\displaystyle \frac{x_i}{W}},{\displaystyle \frac{y_i}{L}},{\displaystyle \frac{v_{x,i}}{v_{max}^p}},{\displaystyle \frac{v_{y,i}}{v_{max}^p}}\right],$$

(9)

which has 4 dimensions.

The teammate-related observation contains the normalized relative positions of the other two pursuers:

$${\mathbf{o}}_i^P\left(t\right)={\left[{\displaystyle \frac{x_j-x_i}{W}},{\displaystyle \frac{y_j-y_i}{L}}\right]}_{j\in \mathcal{P},j\ne i}.$$

(10)

Since $N_{\mathrm{pursuit}}=3$, this part has $2(N_{\mathrm{pursuit}}-1)=4$ dimensions. Relative velocities of teammates are not included in this observation term.

The obstacle/boundary-related observation is encoded by 16 laser-like range measurements:

$${\mathbf{o}}_i^B\left(t\right)=\left[{\displaystyle \frac{l_{i,1}}{L_{\mathrm{sensor}}}},{\displaystyle \frac{l_{i,2}}{L_{\mathrm{sensor}}}},\dots ,{\displaystyle \frac{l_{i,16}}{L_{\mathrm{sensor}}}}\right],$$

(11)

where $l_{i,k}$ denotes the distance measured by the k-th sensing ray of pursuer i, and $L_{\mathrm{sensor}}$ is the maximum sensing range. Each sensing ray returns the distance to the nearest obstacle or boundary along its direction. Therefore, obstacle information is not represented by explicitly listing all obstacle centers; instead, it is encoded through local range-sensor readings. This part has 16 dimensions.

The evader-related observation contains the normalized distance and relative bearing angle from pursuer i to the evader:

$${\mathbf{o}}_i^E\left(t\right)=\left[{\displaystyle \frac{d_{i,e}}{2L}},{\displaystyle \frac{{\theta }_{i,e}}{\pi }}\right],$$

(12)

where $d_{i,e}$ is the distance between pursuer i and the evader, and ${\theta }_{i,e}$ is the relative bearing angle. This part has 2 dimensions.

Accordingly, the complete observation dimension for each pursuer is

$$d_o=4+4+16+2=26.$$

(13)

All position- and distance-related quantities are normalized by the arena size or the maximum sensing range, and velocity-related quantities are normalized by the maximum pursuer speed. This normalization keeps different feature groups on comparable numerical scales and improves training stability.

The assumption of direct evader observability is adopted here to focus on the cooperative encirclement problem itself. However, this assumption simplifies the sensing difficulty of realistic UAV pursuit scenarios and should be regarded as a modeling limitation. Extension to noisy, partial, or intermittent target observation will be considered in future work.

2.5. Environment Parameters

The main geometric and dynamic parameters are selected to define a tractable yet representative benchmark for 3v1 cooperative encirclement in an obstacle-rich bounded environment. In particular, the arena size, speed limits, acceleration limits, and safety distances are chosen such that the evader remains maneuverable while successful capture still requires effective pursuer coordination rather than trivial interception. These settings are intended for algorithmic evaluation and comparative analysis, rather than direct replication of a specific UAV platform. Key environment and dynamic constraints used in our simulations are summarized in

Table 1. Key environment and task parameters for the 3v1 multi-UAV cooperative encirclement scenario.

Parameter	Value
Environment dimension D	$2\mathrm{D}$
Number of pursuers $N_{\mathrm{pursuit}}$	3
Number of evaders $N_{\mathrm{evader}}$	1
Pursuer radius $R_{\mathrm{pursuit}}$	$15\mathrm{m}$
Evader radius $R_{\mathrm{evader}}$	$15\mathrm{m}$
Capture radius $R_c$	$15\mathrm{m}$
UAV safety clearance ${\delta }_{\mathrm{safety}}$	$10\mathrm{m}$
Pursuer max speed $v_{max}^p$	$10\mathrm{m}/\mathrm{s}$
Evader max speed $v_{max}^e$	$15\mathrm{m}/\mathrm{s}$
Pursuer max acceleration $a_{max}^p$	$4\mathrm{m}/{\mathrm{s}}^2$
Evader max acceleration $a_{max}^e$	$5\mathrm{m}/{\mathrm{s}}^2$
Obstacle radius $R_{\mathrm{obstacle}}$	$2\mathrm{m}$
Max episode length $T_{max}$	200 steps
Evader observability	Directly observable (perfect detection)

.

3. Method: SA-DSM-MADDPG

3.1. Overview

We propose SA-DSM-MADDPG (Self-Attention and Double-Screening Mechanism enhanced MADDPG) for the 3v1 multi-UAV cooperative encirclement task. The method follows the centralized training with decentralized execution (CTDE) paradigm: each pursuer executes a decentralized actor using its local observation, while training employs centralized critics to mitigate non-stationarity in multi-agent learning [6,7]. SA-DSM-MADDPG contains three key components aligned with the challenges of cooperative encirclement: (i) an attention-enhanced critic to model dynamic inter-agent interactions, (ii) a double-screened experience replay (DSM) mechanism combining prioritized sampling and relevance screening, and (iii) a curriculum learning reward schedule to stabilize exploration and accelerate convergence.

3.2. MADDPG Backbone Under CTDE

For each pursuer $i\in \mathcal{P}$, the decentralized actor outputs a continuous action:

$${\mathbf{u}}_i\left(t\right)={\mu }_{{\theta }_i}\left({\mathbf{o}}_i\left(t\right)\right),$$

(14)

where ${\mu }_{{\theta }_i}$ is parameterized by ${\theta }_i$. During training, a centralized critic $Q_{{\varphi }_i}$ estimates the action-value using joint information:

$$Q_{{\varphi }_i}\left(\mathbf{x}\left(t\right),{\mathbf{u}}_1\left(t\right),\dots ,{\mathbf{u}}_{N_{\mathrm{pursuit}}}\left(t\right)\right),$$

(15)

where $\mathbf{x}\left(t\right)$ denotes the global training input (e.g., global state or concatenated observations). The critic is optimized by minimizing the temporal-difference (TD) error:

$$\mathcal{L}\left({\varphi }_i\right)={\mathbb{E}}_{(·)\sim \mathcal{D}}\left[{\left(Q_{{\varphi }_i}(\mathbf{x},\mathbf{u})-y_i\right)}^2\right],$$

(16)

with target

$$y_i=r_i+\gamma Q_{{\varphi }_i^{-}}\left({\mathbf{x}}^{\prime },{\mathbf{u}}_1^{\prime },\dots ,{\mathbf{u}}_{N_{\mathrm{pursuit}}}^{\prime }\right),{\mathbf{u}}_j^{\prime }={\mu }_{{\theta }_j^{-}}\left({\mathbf{o}}_j^{\prime }\right).$$

(17)

Here, $\gamma$ is the discount factor, and $({\varphi }_i^{-},{\theta }_i^{-})$ are target-network parameters updated softly [6].

The actor is updated by maximizing the critic’s estimate:

$${\nabla }_{{\theta }_i}J\left({\theta }_i\right)={\mathbb{E}}_{(·)\sim \mathcal{D}}\left[{\nabla }_{{\theta }_i}{\mu }_{{\theta }_i}\left({\mathbf{o}}_i\right){\nabla }_{{\mathbf{u}}_i}{Q}_{{\varphi }_i}(\mathbf{x},\mathbf{u})\right].$$

(18)

3.3. Self-Attention Enhanced Critic (SA)

In cooperative encirclement, each pursuer must coordinate with teammates whose influence varies over time according to the current formation geometry, relative distance, relative bearing, and task phase. A fixed concatenation of teammate features cannot explicitly distinguish which teammates are more relevant to the current decision. To improve interaction modeling in the centralized critic, we incorporate a self-attention mechanism that adaptively aggregates teammate-related features with learned importance weights [10,11,16].

For pursuer i, let ${\mathcal{N}}_i=\mathcal{P}\setminus \left\{i\right\}$ denote the teammate set. For each teammate $j\in {\mathcal{N}}_i$, we construct a relative feature vector

$${\mathbf{f}}_{i,j}=\left[{\mathbf{p}}_j-{\mathbf{p}}_i,{\mathbf{v}}_j-{\mathbf{v}}_i,{\mathbf{p}}_e-{\mathbf{p}}_i,{\mathbf{p}}_j-{\mathbf{p}}_e\right],$$

(19)

where ${\mathbf{p}}_i$ and ${\mathbf{v}}_i$ are the position and velocity of pursuer i, and ${\mathbf{p}}_e$ is the evader position. The relative feature ${\mathbf{f}}_{i,j}$ is encoded by a multilayer perceptron (MLP) into an embedding:

$${\mathbf{h}}_{i,j}=\psi \left({\mathbf{f}}_{i,j}\right),$$

(20)

where $\psi (·)$ denotes the embedding network.

We further compute a self-state context vector for pursuer i,

$${\mathbf{c}}_i=\left[{\mathbf{p}}_i,{\mathbf{v}}_i,{\mathbf{p}}_e-{\mathbf{p}}_i,{\mathbf{o}}_i^{\mathrm{obs}}\right],$$

(21)

where ${\mathbf{o}}_i^{\mathrm{obs}}$ represents obstacle/boundary-related features. The query vector is generated from the self-state context, while the key and value vectors are generated from teammate embeddings:

$${\mathbf{q}}_i=W_q{\mathbf{c}}_i,{\mathbf{k}}_{i,j}=W_k{\mathbf{h}}_{i,j},{\mathbf{v}}_{i,j}=W_v{\mathbf{h}}_{i,j},$$

(22)

where $W_q$, $W_k$, and $W_v$ are learnable projection matrices.

The attention weight assigned to teammate j is computed as

$${\alpha }_{i,j}={\displaystyle \frac{exp\left({\mathbf{q}}_i^{\top }{\mathbf{k}}_{i,j}/\sqrt{d}\right)}{{\displaystyle \sum _{m\in {\mathcal{N}}_i}}exp\left({\mathbf{q}}_i^{\top }{\mathbf{k}}_{i,m}/\sqrt{d}\right)}},$$

(23)

where d is the key dimension. The aggregated teammate-interaction representation is then

$${\mathbf{z}}_i=\sum _{j\in {\mathcal{N}}_i}{\alpha }_{i,j}{\mathbf{v}}_{i,j}.$$

(24)

Finally, the critic input for pursuer i is formed by concatenating the self-state context, the attention-aggregated teammate feature, and the joint action information:

$${\tilde{\mathbf{x}}}_i=\left[{\mathbf{c}}_i,{\mathbf{z}}_i,{\mathbf{u}}_1,\dots ,{\mathbf{u}}_{N_{\mathrm{pursuit}}}\right].$$

(25)

The centralized critic is thus written as

$$Q_{{\varphi }_i}=Q_{{\varphi }_i}\left({\tilde{\mathbf{x}}}_i\right).$$

(26)

This design enables the critic to adaptively emphasize the most relevant teammates under different encirclement configurations, instead of treating all teammate information equally. As a result, the critic can better capture dynamic coordination structure and provide more informative gradients for decentralized actor learning, thereby improving cooperative behavior in time-varying multi-agent interactions [10,11].

3.4. Double-Screened Experience Replay (DSM)

Off-policy training in cooperative encirclement relies heavily on replayed transitions, but the replay buffer typically contains samples with highly uneven learning value. Many transitions correspond to distant pursuit states or weakly informative behaviors, which may reduce sample efficiency and destabilize multi-agent learning. To address this issue, we propose a double-screened experience replay (DSM) mechanism, which combines prioritized experience replay (PER) with relevant experience learning (REL).

3.4.1. First Screening: Prioritized Experience Replay

For each transition i stored in the replay buffer $\mathcal{D}$, we compute its temporal-difference (TD) error as

$${\delta }_i=r_i+\gamma Q^{\prime }(s_{i+1},a_{i+1},o^{\prime })-Q(s_i,a_i,o),$$

(27)

where $Q^{\prime }$ denotes the target critic network. The priority of transition i is then defined as

$$p_i=\left|{\delta }_i\right|+\epsilon ,$$

(28)

where $\epsilon >0$ is a small constant to avoid zero sampling probability. Accordingly, the PER sampling probability is

$$P\left(i\right)={\displaystyle \frac{{\left(p_i\right)}^{\alpha }}{{\sum }_j{\left(p_j\right)}^{\alpha }}},$$

(29)

where $\alpha$ controls the degree of prioritization [15].

3.4.2. Second Screening: Relevant Experience Learning

To further improve replay quality, each transition is associated with a relevance descriptor and represented as

$$[s_t,a_t,r_t,f_r\left(s_t\right)],$$

(30)

where $f_r\left(s_t\right)$ denotes a state relevance function that measures the task-related correlation of state $s_t$. In this work, $f_r\left(s_t\right)$ is defined by geometric indicators of the encirclement configuration:

$$f_r\left(s_t\right)={\sigma }_1\left({\displaystyle \sum _{i=0}^{n-1}}{S}_{E{P}_i{P}_{i+1}}-S_{P_1{P}_2\cdots P_n}\right)+{\sigma }_2{\displaystyle \sum _{i=0}^{n-1}}{S}_{E{P}_i{P}_{i+1}}+{\sigma }_3{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{d_{O{P}_i}}},$$

(31)

where $S_{E{P}_i{P}_{i+1}}$ denotes the area of the triangle formed by the evader E and two adjacent pursuers, $S_{P_1{P}_2\cdots P_n}$ denotes the polygon area formed by all pursuers, $d_{O{P}_i}$ denotes the distance between pursuer $P_i$ and the encirclement center O, and ${\sigma }_1,{\sigma }_2,{\sigma }_3$ are weighting coefficients.

For the current state $s_t$, the relevance difference between the current state and a sampled historical state s is defined as

$$\Delta f_t=|f_r\left(s_t\right)-f_r\left(s\right)|.$$

(32)

After obtaining a candidate set of high-value samples through PER, we perform a second screening by selecting the samples with the smallest relevance difference to form the final mini-batch:

$$B_{\mathrm{rel}}=argmin|f_r\left(s_t\right)-f_r\left(s\right)|.$$

(33)

Therefore, DSM performs two successive screenings. The first screening emphasizes informative transitions with large TD errors, while the second screening emphasizes relevant transitions whose geometric states are more correlated with the current training state. The final mini-batch obtained after REL is then used to update the critic and actor networks under the CTDE framework. This “pre-sampling + secondary sampling” design improves replay quality and training efficiency by reducing the influence of weakly informative and weakly related samples.

3.5. Curriculum Learning with Staged Reward Shaping

Because cooperative encirclement provides sparse terminal feedback, directly training all pursuers with only final success rewards often leads to inefficient exploration and unstable convergence. To alleviate this issue, we adopt curriculum learning by decomposing the encirclement process into three progressively structured stages: Search, Transition, and Capture [12,13,14].

The current stage is determined by the geometric relationship between the evader and the pursuer formation. Specifically, let $S_{P_1{P}_2\cdots P_n}$ denote the polygon area formed by all pursuers, let $S_{E{P}_i{P}_{i+1}}$ denote the area of the triangle formed by the evader and two adjacent pursuers, and let $d_i$ denote the distance between pursuer $P_i$ and the evader. The three stages are defined as follows:

• Search stage: The evader is still outside the effective encirclement region, and the pursuers mainly need to approach the target while reducing their overall distance to the evader.
• Transition stage: The pursuers begin to form an enclosing structure around the evader, but the encirclement is not yet stable or sufficiently compact.
• Capture stage: The evader is enclosed by the pursuer formation and the pursuers further tighten the enclosure until the final capture condition is satisfied.

For pursuer i, the total reward at time t is defined as

$$r_i\left(t\right)={\omega }_1{r}_i^{\mathrm{task}}\left(t\right)+{\omega }_2{r}_i^{\mathrm{safe}}\left(t\right)+{\omega }_3{r}_i^{\mathrm{ind}}\left(t\right)+{\omega }_4{r}^{\mathrm{succ}}\left(t\right),$$

(34)

where ${\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4$ are weighting coefficients satisfying

$${\displaystyle \sum _{m=1}^4}{\omega }_m=1.$$

(35)

Here, $r_i^{\mathrm{task}}\left(t\right)$ is the stage-dependent task reward, $r_i^{\mathrm{safe}}\left(t\right)$ penalizes unsafe behaviors, $r_i^{\mathrm{ind}}\left(t\right)$ encourages effective individual pursuit behavior, and $r^{\mathrm{succ}}\left(t\right)$ is the terminal success reward.

3.5.1. Stage-Dependent Task Reward

The task reward switches according to the current curriculum stage.

In the Search stage, the objective is to drive all pursuers to approach the evader and reduce the overall pursuit distance. Thus, the reward emphasizes distance reduction and initial formation tendency:

$$r_i^{\mathrm{task}}\left(t\right)=r_i^{\mathrm{search}}\left(t\right),$$

(36)

where $r_i^{\mathrm{search}}\left(t\right)$ is designed to encourage smaller pursuer–evader distances and preparation for subsequent enclosure.

In the Transition stage, the objective is to gradually form an enclosing configuration around the evader. Therefore, the reward emphasizes geometric enclosure quality and the reduction in formation gaps:

$$r_i^{\mathrm{task}}\left(t\right)=r_i^{\mathrm{trans}}\left(t\right).$$

(37)

In the Capture stage, the objective is to maintain enclosure and complete final capture. At this stage, the reward further emphasizes compactness and coordinated closing behavior:

$$r_i^{\mathrm{task}}\left(t\right)=r_i^{\mathrm{capture}}\left(t\right).$$

(38)

3.5.2. Safety Reward

To avoid unsafe behaviors during learning, we penalize UAV–UAV collisions, obstacle collisions, and boundary violations:

$$r_i^{\mathrm{safe}}\left(t\right)=\left\{\begin{array}{cc}-10,\hfill & \mathrm{if}\mathrm{a}\mathrm{safety}\mathrm{violation}\mathrm{occurs},\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(39)

3.5.3. Individual Pursuit Reward

To maintain effective target-oriented motion for each pursuer, we additionally introduce an individual reward term that encourages each UAV to move toward the evader with favorable heading and distance reduction:

$$r_i^{\mathrm{ind}}\left(t\right)={\mu }_1{\displaystyle \frac{d_{i,t-1}-d_{i,t}}{d_{max}}}+{\mu }_{2}cos{\phi }_i,$$

(40)

where $d_{i,t}$ denotes the distance between pursuer i and the evader at time t, ${\phi }_i$ is the angle between the pursuer velocity direction and the target direction, and ${\mu }_1,{\mu }_2$ are balancing coefficients.

3.5.4. Terminal Success Reward

When the evader is successfully enclosed and the capture criterion is satisfied, all pursuers receive a positive terminal reward:

$$r^{\mathrm{succ}}\left(t\right)=\left\{\begin{array}{cc}{R}_{\mathrm{succ}},\hfill & \mathrm{if}\mathrm{the}\mathrm{success}\mathrm{condition}\mathrm{is}\mathrm{satisfied},\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(41)

where $R_{\mathrm{succ}}>0$ is a constant success bonus.

This staged reward design provides denser supervision in early training and gradually shifts the optimization focus from target approaching, to enclosure formation, and finally to capture completion. As a result, curriculum learning improves exploration efficiency and stabilizes policy optimization in cooperative encirclement tasks [2,13].

3.6. Training Procedure

Algorithm 1 summarizes the training loop of SA-DSM-MADDPG. The key difference from standard MADDPG lies in the following: (i) attention-enhanced critic construction, (ii) DSM replay sampling (PER + REL), and (iii) curriculum stage scheduling for reward computation.

Algorithm 1: Training procedure of SA-DSM-MADDPG

4. Experiments and Results

4.1. Experimental Setup

We evaluate SA-DSM-MADDPG in a 2D multi-UAV cooperative encirclement scenario with static obstacles, following the task formulation introduced in Section 2 and the MARL training setup described in Section 3 [1,2,13]. The arena size is $W\times L=200\mathrm{m}\times 200\mathrm{m}$, with $N_{\mathrm{pursuit}}=3$ pursuers and one evader. Each episode terminates upon (i) success (encirclement and capture), (ii) safety violation (UAV–UAV, UAV–obstacle, or boundary), or (iii) timeout. The maximum episode length is $T_{max}=200$ steps. Following our observation setting, the evader position is directly observable by all pursuers during execution. Other key parameters (agent radii, speed/acceleration constraints, obstacle radius, and safety clearance) are summarized in Table 1.

Obstacle Arrangement

In the experiments, circular obstacles are randomly generated at the beginning of each episode. The obstacle positions are sampled within the bounded arena while avoiding initial overlap with pursuers, the evader, and arena boundaries. For experiments with different numbers of obstacles, the same obstacle-generation rule is applied to all compared methods to ensure a fair comparison.

4.2. Baselines

To evaluate the effectiveness of the proposed framework, we compare SA-DSM-MADDPG with both representative MARL baselines and module-level variant methods.

4.2.1. Representative Baselines

• MAPPO: An on-policy multi-agent proximal policy optimization method used here as a representative MARL baseline for comparison.
• MADDPG: The standard CTDE baseline with centralized critics and decentralized actors, serving as the main off-policy comparison method [6].

4.2.2. Module-Level Variants

• SA-MADDPG: MADDPG enhanced with the self-attention critic, without DSM replay or curriculum learning [10,11].
• DSM-MADDPG: MADDPG with the proposed DSM replay mechanism, without attention or curriculum learning [15].
• CL-MADDPG: MADDPG with curriculum learning and staged reward shaping, without attention or DSM replay [12,13].

The full method, SA-DSM-MADDPG, integrates self-attention, DSM replay, and curriculum learning within the same CTDE framework. All compared methods are evaluated under the same environment settings, observation/action definitions, and training budget to ensure a fair comparison.

4.3. Evaluation Metrics

We use the following metrics to evaluate learning efficiency and final performance:

• Average episode return: Mean cumulative reward per episode, smoothed by a moving average window.
• Success rate: Percentage of episodes that terminate with success (encirclement and capture).
• Time-to-capture: Average number of steps required to achieve capture, computed over successful episodes.
• Safety violation rate: Percentage of episodes terminated by collisions or boundary violations.

These metrics are widely used in reinforcement-learning-based pursuit–evasion and cooperative control tasks to jointly assess convergence quality, task effectiveness, and safety performance [1,2,5].

4.4. Training Protocol

All methods are trained for $E=1000$ episodes under the same environment settings and computational budget. To reduce the influence of randomness, each configuration is repeated with multiple random seeds, and the reported curves show the mean performance across runs. For the MADDPG-based methods, exploration noise is added during training for continuous action exploration, while evaluation is conducted using the learned policies without training-time perturbation. For all methods, performance is assessed under the same observation/action definitions, episode horizon, and success/failure criteria to ensure a fair comparison.

4.5. Main Results

Figure 3 reports the learning curves in terms of average episode return and success rate. Compared with the MADDPG baseline, SA-DSM-MADDPG converges faster and reaches a higher stabilized return, indicating improved sample efficiency and better coordination quality. The success-rate curve further shows that the proposed method achieves more reliable cooperative capture performance during training. In addition, SA-DSM-MADDPG shows reduced oscillations in later training, suggesting enhanced stability under the non-stationary multi-agent setting. These observations are consistent with the intended effects of attention-based interaction modeling, prioritized/relevance-aware replay, and staged curriculum shaping [10,12,15].

4.6. Failure-Type Analysis

To better understand how the compared methods fail, we further analyze the composition of final outcomes, including successful encirclement, collision-related failures, non-coordination failures, and unencirclement failures, as shown in Figure 4. Here, collision-related failures include both inter-UAV collisions and collisions with obstacles.

The results indicate that SA-DSM-MADDPG achieves the highest success rate among the three methods, while substantially reducing failure cases caused by non-coordination and unsuccessful enclosure formation. This suggests that the proposed framework improves the consistency of cooperative decision-making and the quality of encirclement formation. Meanwhile, a non-negligible fraction of failures is still associated with collision-related events, indicating that safety-aware coordination can be further improved in future work.

Overall, this outcome-level comparison provides a more informative evaluation than success rate alone, since it reveals not only whether the task is completed, but also how and why the competing methods fail.

4.7. Effect of Obstacle Number

To further evaluate the robustness of the proposed method under environments of different complexity, we compare the success rates of SA-DSM-MADDPG, MADDPG, and MAPPO under varying numbers of obstacles, as shown in Figure 5.

When the number of obstacles is 0, corresponding to an obstacle-free scenario, SA-DSM-MADDPG still achieves the highest success rate among the compared methods. This indicates that the advantage of the proposed method is not limited to obstacle avoidance alone, but also reflects stronger cooperative encirclement capability in the basic pursuit–evasion setting. As the number of obstacles increases, the success rates of all methods gradually decrease, reflecting the increased difficulty of simultaneously maintaining coordination and avoiding environmental constraints.

Nevertheless, SA-DSM-MADDPG consistently maintains the best performance across all tested obstacle settings. This result demonstrates that the proposed framework is more robust to increasing environmental complexity and better preserves cooperative effectiveness when the task becomes more challenging.

4.8. Component Discussion

To further examine the contribution of each module in SA-DSM-MADDPG, we compare the complete framework with three module-level variants: SA-MADDPG, DSM-MADDPG, and CL-MADDPG. These comparisons provide a component-level analysis of the effects of self-attention, double-screened replay, and curriculum learning under the same task setting.

The results indicate that each component contributes positively to the overall training process, although their roles are different. The self-attention critic mainly improves coordination modeling by helping the critic focus on more relevant teammate information under time-varying interaction patterns. DSM replay mainly improves replay quality by emphasizing informative and task-relevant samples, which leads to better sample efficiency and more stable optimization. Curriculum learning mainly improves early-stage exploration by providing denser and progressively structured reward signals. When these three components are integrated, the complete SA-DSM-MADDPG framework achieves the best overall performance in terms of convergence behavior, training stability, and cooperative success.

These observations suggest that the performance gain of SA-DSM-MADDPG does not come from a single module alone, but from the complementary effects of interaction-aware coordination modeling, replay refinement, and staged reward guidance.

4.9. Qualitative Analysis

To illustrate learned behaviors, Figure 6 visualizes representative trajectories. The MADDPG baseline often exhibits inefficient pursuit and incomplete closure of the encirclement ring, and it may suffer from safety violations near obstacles. In contrast, SA-DSM-MADDPG learns more coordinated motion: pursuers spread to block escape directions, gradually tighten the enclosure, and achieve capture while maintaining safety margins. Such behavior is consistent with the objective of cooperative encirclement and with prior reinforcement-learning-based pursuit–evasion studies that emphasize both interception efficiency and safe coordinated motion [1,4,5].

4.10. Discussion of Results

Overall, the results demonstrate that SA-DSM-MADDPG improves both learning efficiency and final coordination performance. Self-attention helps the critic assign adaptive importance to teammates, DSM replay increases the ratio of informative and relevant samples, and curriculum learning provides stable shaped signals during early-stage exploration. These mechanisms jointly yield higher capture success and better stability than standard MADDPG in the studied encirclement task, indicating that the proposed integrated design is effective for obstacle-rich multi-UAV pursuit–evasion scenarios. Compared with standard MADDPG, the proposed self-attention critic and DSM replay introduce additional computational overhead during training, since attention-based feature aggregation and secondary replay screening both require extra processing. However, under the current 3v1 setting, this overhead remains manageable and is justified by the gains in convergence stability and cooperative performance. As the number of pursuers increases, both interaction complexity and replay heterogeneity are expected to grow, and further investigation of scalability to larger team sizes will be an important direction for future work.

5. Conclusions

This study examined the problem of multi-UAV cooperative encirclement in obstacle-rich pursuit–evasion scenarios and evaluated how interaction modeling, replay selection, and reward scheduling jointly affect cooperative learning performance. The results indicate that successful encirclement is not determined only by the strength of the underlying actor–critic algorithm, but also by whether the training process can provide appropriate coordination representation, informative samples, and stage-wise learning guidance.

From this perspective, the proposed SA-DSM-MADDPG framework provides several insights for multi-agent UAV decision-making. First, the self-attention critic helps capture time-varying teammate relevance, which is important when the pursuers need to adjust their roles dynamically during enclosure formation. Second, DSM replay improves the quality of off-policy updates by selecting samples that are both informative and geometrically relevant to the current encirclement process. Third, curriculum-based reward shaping reduces the difficulty of learning from sparse terminal feedback by gradually shifting the learning focus from target approach to enclosure formation and final capture. The comparative experiments, failure-type analysis, and obstacle-number analysis jointly show that these components complement each other and improve convergence stability, cooperative success, and robustness to environmental complexity.

Despite these improvements, several limitations remain. The current study is conducted in a simplified 2D planar environment with static obstacles and direct observability of the evader. These assumptions make the problem tractable for controlled algorithmic evaluation, but they do not fully capture real-world UAV sensing uncertainty, three-dimensional flight dynamics, communication constraints, or dynamic obstacles. In addition, although the 3v1 setting is useful for validating cooperative encirclement behavior, larger and more heterogeneous UAV teams may introduce more complex interaction patterns and higher computational costs.

Future work will extend the proposed framework toward more realistic and scalable scenarios. Promising directions include partial or noisy target observation, moving obstacles, larger-scale pursuer teams, higher-fidelity UAV dynamics, and safety-guaranteed policy learning. Further validation in hardware-in-the-loop simulation and real-flight experiments will also be important for evaluating the practical deployment potential of learned cooperative encirclement policies.