Active Interception for Multi-Target Encirclement by Heterogeneous UAVs: An LSTM-Enhanced Independent PPO Algorithm

Abstract

In recent years, multi-UAV systems have demonstrated broad applications in both security and civilian domains, where cooperative encirclement has emerged as a key research focus. However, existing work predominantly addresses single-target scenarios with homogeneous UAVs using passive tracking strategies, which are inadequate for handling highly maneuverable targets. To overcome these limitations, this paper proposes an active interception decision framework integrating LSTM networks with an off-policy independent actor–critic framework employing a PPO-style clipped surrogate objective, referred to as LIPPO. It aims to address the complex problem of heterogeneous UAV swarms encircling multiple continuously learning targets. The framework employs an LSTM module for real-time trajectory prediction and uses the predicted future positions as interception points, shifting the paradigm from passive tracking to proactive interception. At the decision level, LIPPO adopts a hybrid architecture where each UAV acts as an independent learner, while a shared experience pool enables efficient knowledge transfer across the swarm. Comprehensive simulations demonstrate LIPPO’s superiority. In complex scenarios, it achieves an encirclement success rate up to 10 percentage points higher than non-predictive baselines and reduces energy consumption by nearly 28% compared to centralized training multi-agent reinforcement learning algorithms. These results confirm that LIPPO’s active interception is both effective and efficient.

1. Introduction

In recent years, swarm collaboration based on multiple unmanned aerial vehicles (UAVs) has shown significant applications in public security, civil, and commercial scenarios. Among these applications, the multi-agent cooperative encirclement task has emerged as one of the core research directions. Unlike traditional pursuit–evasion problems, the objective of encirclement is not merely to minimize the distance between the hunter and the target. Instead, it requires multiple hunter agents to form an effective geometric encirclement of the target through collaborative decision-making.

With the increasing complexity of real-world scenarios, the problem often transforms into multi-hunter multi-target encirclement tasks, such as monitoring unauthorized UAV intrusions in public security or simultaneously tracking multiple suspicious targets in security applications. This not only presents high demands on the motion control of individual agents but also brings challenges to the overall system’s swarm collaboration. Efficiently accomplishing multi-target encirclement remains a key challenge in current research.

Numerous scholars have worked on this issue and have achieved significant results. Early studies mostly employed model predictive control (MPC) methods. A method combining feedback linearization (FL) and linear model predictive control (LMPC) was introduced to address the problem of multiple homogeneous UAVs collaboratively surrounding a single stationary target in 2D space [1]. Similarly, a distributed vector field control method based on an Annular Virtual Tube was introduced to solve the core problem of multiple UAVs surrounding a static target in 3D space [2]. However, these control algorithms often require highly accurate models. They struggle to handle dynamically changing environments effectively.

With the development of deep reinforcement learning, multi-agent reinforcement learning (MARL) has gradually become an important method for solving UAV encirclement problems. A MARL framework based on extrinsic–intrinsic reward mechanisms was proposed to effectively handle energy-efficient cooperative encirclement of a single moving target by heterogeneous UAVs [3]. Additionally, a curriculum experience learning-based algorithm (CEL-MADDPG) with staged task decomposition has been introduced [4]. In another study, relative distance measurements were combined with a fuzzy wavelet neural network, enabling two UAVs to dynamically encircle multiple non-cooperative moving targets [5]. Furthermore, a dynamic sector control law and a greedy algorithm-based target assignment strategy were employed, offering a new solution for cooperative encirclement of multiple static targets by multiple UAVs [6].

In recent years, trajectory prediction-based UAV pursuit evasion problems have gradually become a research hotspot. Existing studies have explored the application of temporal models such as LSTM in UAV trajectory prediction and pursuit–evasion decision-making. For example, real-time LSTM has been employed to model and predict the trajectories of UAVs for conflict detection and obstacle avoidance [7]. In pursuit–evasion tasks, offline LSTM prediction has been combined with the TD3 algorithm to improve the performance of single-UAV pursuit of a single target [8]. Additionally, offline LSTM training has been utilized to predict the future trajectories of the evader, thereby helping UAVs formulate more effective cooperative capture strategies [9].

Despite the notable advances achieved in the above studies, multiple challenges remain in multi-hunter–multi-target geometric encirclement scenarios. Existing studies predominantly focus on the encirclement or capture of a single target and assume that all UAVs have completely identical motion and energy consumption characteristics. However, real-world tasks often involve capturing multiple targets simultaneously, with UAVs differing significantly in speed and energy consumption. Traditional methods largely rely on passive pursuit strategies, which generate actions based on currently observed information. Such approaches tend to exhibit limited effectiveness when dealing with highly maneuverable and evasive targets. Although some studies have introduced trajectory prediction, its application to complex many-to-many, heterogeneous, adversarial UAV pursuit problems remains an under-explored domain.

To address the above limitations, a multi-agent cooperative interception framework named LIPPO is proposed. This framework integrates active interception with independent online learning. An online-trained LSTM network is introduced to predict the target’s future trajectory in real time. This guides hunters to target future interception points rather than the target’s current location. To overcome the high sample complexity of traditional on-policy algorithms, LIPPO adopts an off-policy PPO architecture. It introduces independent Actor–Critic networks equipped with target networks and utilizes a shared experience replay buffer to enable efficient data reuse and stable convergence.

The main contributions are as follows:

• Unlike most existing studies that focus on single-target encirclement and assume UAV homogeneity, this paper studies the many-to-many confrontation problem of heterogeneous UAVs working together to capture multiple actively escaping targets. The targets are modeled as reinforcement learning agents that can continuously learn and evolve escape strategies. This creates a dynamic environment that closely approximates real-world confrontation scenarios.
• Unlike traditional methods that rely on passive pursuit strategies, this paper proposes an active interception decision framework based on trajectory prediction. An LSTM network is introduced to learn and predict the future trajectory of the target in real time. The hunter uses the point of the predicted trajectory as the interception target, thereby achieving a strategic upgrade from passive following to active anticipation.

The structure of the rest of this paper is organized as follows: Section 2 reviews the relevant literature on UAV target encirclement; Section 3 presents the theoretical foundation; Section 4 describes the LIPPO framework in detail; Section 5 performs experimental comparisons and analyses; finally, Section 6 summarizes the conclusions of this paper and proposes future research directions.

2. Literature Review

UAV target encirclement, as a challenging problem, has garnered considerable attention from researchers. Existing research methods can be mainly divided into two categories: first is the traditional control methods based on models, and the other is the reinforcement learning methods based on learning. This section reviews the relevant literature and provides a systematic summary in Table 1 to present the current progress and limitations of the research.

Early research mostly focused on model-based control methods, particularly Model Predictive Control (MPC), which relies on known dynamic models to generate relatively precise control strategies [10]. A method combining MPC with feedback linearization was developed to address the multi-UAV cooperative encirclement of a single dynamic target in 2D space [11]. Subsequently, some studies introduced distributed guidance laws to achieve more efficient encirclement strategies. A distributed guidance law based on the reduction in the enclosed area was proposed, enabling multi-UAV systems to collaboratively capture a high-speed maneuvering target [12]. Similarly, a distributed coordinate-free control algorithm relying solely on distance measurements was developed, allowing UAVs to collaboratively surround and track a single stationary or moving target in a two-dimensional space [13]. However, such approaches rely heavily on accurate modeling and struggle to handle nonlinear characteristics and dynamically changing environments.

In recent years, with the advancement of deep reinforcement learning, multi-agent reinforcement learning (MARL) has been widely applied to encirclement problems, such as algorithms like MADDPG [14] and MAPPO [15]. MARL methods do not rely on precise environment models. Agents can achieve end-to-end learning through interaction with the environment, thereby gaining stronger adaptability and autonomous decision-making capabilities.

Table 1. Review of recent studies on UAV target encirclement.

Authors	Algorithm Category	Specific Algorithm	Targets	Obstacle Type	Energy	Agent Type	Target Behavior
[10]	Control	LMPC	1	Static	no	Homogeneous	Static
[16]	Control	LMPC + Feedback Linearization	1	None	no	Homogeneous	Mobile
[11]	Control	LMPC + Feedback Linearization	1	None	no	Homogeneous	Static
[17]	MARL	Co-DQL	1	Static and Dynamic	no	Homogeneous	Active Escape
[12]	DCC	DCC	1	None	no	Homogeneous	Mobile
[18]	Control	Cooperative Motion Path Following	1	None	no	Homogeneous	Mobile
[2]	Control	Distributed Vector Field Control	1	None	no	Homogeneous	Static
[19]	Hybrid	Hybrid Guidance + MDP + Collaborative Switching	1	Static	no	Homogeneous	Mobile
[13]	DCC	DCC + Dynamic Observer	1	None	no	Homogeneous	Static/Mobile
[20]	MARL	MADDPG	1	None	no	Homogeneous	Active Escape
[21]	Hybrid	Target Clustering + Guidance Law	2	None	no	Homogeneous	Mobile
[4]	MARL	CEL-MADDPG	1	Static	no	Homogeneous	Active Escape
[22]	MARL	GCMSA	1	Static	no	Homogeneous	Mobile
[23]	Artificial Potential Field	Artificial Potential Field	3	Static and Dynamic	no	Homogeneous	Mobile
[24]	MARL	GCMSA	1	Static	no	Homogeneous	Mobile
[6]	Hybrid	Greedy Assignment + Dynamic Sectorization	3	None	no	Homogeneous	Static
[25]	Control	Event-Triggered Distributed Control	1	None	no	Homogeneous	Mobile
[5]	Neural Network Control	FWNN + Distributed Anti-Synchronization Controller	3	None	no	Homogeneous	Mobile
[14]	MARL	MADDPG	1	Static	no	Homogeneous	Mobile
[3]	MARL	EIR-MARL	1	Static	Yes	Heterogeneous	Mobile
[26]	Game Theory + RL	SPG + TMSAC	1	Static	no	Homogeneous	Mobile
[27]	MARL	TP-MADDPG	1	Static	no	Homogeneous	Mobile
[9]	MARL	MAPPO + LSTM	1	Static	no	Homogeneous	Mobile
[15]	MARL	MAPPO	1	None	no	Homogeneous	Active Escape
[28]	Probabilistic Graphical Model	PD-PGM	1	Static and Dynamic	no	Homogeneous	Mobile
[29]	RL	SAC + LSTM	1	None	Yes	Homogeneous	Mobile
This study	RL	IPPO + LSTM	5	Static	Yes	Heterogeneous	Active Escape

Table 1. Review of recent studies on UAV target encirclement.

Authors	Algorithm Category	Specific Algorithm	Targets	Obstacle Type	Energy	Agent Type	Target Behavior
[10]	Control	LMPC	1	Static	no	Homogeneous	Static
[16]	Control	LMPC + Feedback Linearization	1	None	no	Homogeneous	Mobile
[11]	Control	LMPC + Feedback Linearization	1	None	no	Homogeneous	Static
[17]	MARL	Co-DQL	1	Static and Dynamic	no	Homogeneous	Active Escape
[12]	DCC	DCC	1	None	no	Homogeneous	Mobile
[18]	Control	Cooperative Motion Path Following	1	None	no	Homogeneous	Mobile
[2]	Control	Distributed Vector Field Control	1	None	no	Homogeneous	Static
[19]	Hybrid	Hybrid Guidance + MDP + Collaborative Switching	1	Static	no	Homogeneous	Mobile
[13]	DCC	DCC + Dynamic Observer	1	None	no	Homogeneous	Static/Mobile
[20]	MARL	MADDPG	1	None	no	Homogeneous	Active Escape
[21]	Hybrid	Target Clustering + Guidance Law	2	None	no	Homogeneous	Mobile
[4]	MARL	CEL-MADDPG	1	Static	no	Homogeneous	Active Escape
[22]	MARL	GCMSA	1	Static	no	Homogeneous	Mobile
[23]	Artificial Potential Field	Artificial Potential Field	3	Static and Dynamic	no	Homogeneous	Mobile
[24]	MARL	GCMSA	1	Static	no	Homogeneous	Mobile
[6]	Hybrid	Greedy Assignment + Dynamic Sectorization	3	None	no	Homogeneous	Static
[25]	Control	Event-Triggered Distributed Control	1	None	no	Homogeneous	Mobile
[5]	Neural Network Control	FWNN + Distributed Anti-Synchronization Controller	3	None	no	Homogeneous	Mobile
[14]	MARL	MADDPG	1	Static	no	Homogeneous	Mobile
[3]	MARL	EIR-MARL	1	Static	Yes	Heterogeneous	Mobile
[26]	Game Theory + RL	SPG + TMSAC	1	Static	no	Homogeneous	Mobile
[27]	MARL	TP-MADDPG	1	Static	no	Homogeneous	Mobile
[9]	MARL	MAPPO + LSTM	1	Static	no	Homogeneous	Mobile
[15]	MARL	MAPPO	1	None	no	Homogeneous	Active Escape
[28]	Probabilistic Graphical Model	PD-PGM	1	Static and Dynamic	no	Homogeneous	Mobile
[29]	RL	SAC + LSTM	1	None	Yes	Homogeneous	Mobile
This study	RL	IPPO + LSTM	5	Static	Yes	Heterogeneous	Active Escape

Later, a method called Co-DQL based on multi-agent reinforcement learning was proposed to address the problem of multiple homogeneous UAVs collaboratively pursuing a single moving target in 3D space [17]. In another study, a trajectory prediction network was combined with a two-stage reward function, achieving efficient collaborative encirclement of a single moving target in a two-dimensional static obstacle environment [27]. Furthermore, a hybrid guidance law combined with MDP-based policy switching has been adopted [19], while attention mechanisms and cognitive consistency have been utilized to enhance collaboration performance [22].

When dealing with highly maneuverable targets that have evasive strategies, passive pursuit strategies based solely on the current state are often inefficient. To address this, researchers have begun to introduce trajectory prediction models to build more forward-looking active interception mechanisms [30,31]. A reinforcement learning approach integrating LSTM fugitive prediction with adaptive environment generation has been proposed, enabling efficient collaborative pursuit and capture by multiple unmanned aerial vehicles within uncharted three-dimensional environments [9]. Additionally, an LSTM-based reinforcement learning path planning algorithm, RPP-LSTM, was introduced; this method significantly enhances the path planning capability and robustness of unmanned aerial vehicles in complex environments by incorporating historical information [32].

As summarized in Table 1, although significant progress has been made in UAV cooperative encirclement, three critical gaps remain in the existing literature. First, most prior studies (e.g., [10,11,12,13,14,15,16,17,18,19,20]) are limited to single-target scenarios. Methods designed for such settings often fail to scale to multi-target encirclement. Second, the majority of existing frameworks assume homogeneous UAVs. Heterogeneous agents are rarely considered. Even fewer studies explicitly model heterogeneous energy consumption constraints, although such constraints are essential for real-world endurance-aware deployment. Third, most targets follow static or predefined motion patterns. Research on intelligent targets that actively learn and adapt evasive strategies through reinforcement learning remains scarce.

More importantly, these challenges are typically addressed in isolation. To the best of our knowledge, no prior work simultaneously considers multi-target encirclement, heterogeneous UAVs with energy constraints, and actively learning evasive targets within a unified framework. As highlighted in the bottom row of Table 1, this paper proposes the LIPPO framework to bridge this gap. It integrates online LSTM-based trajectory prediction with independent PPO learning. This design enables active interception in heterogeneous multi-UAV, multi-target scenarios.

3. Theoretical Foundation

3.1. Problem Definition

This paper investigates the multi-UAV cooperative target encirclement problem within a 2D environment. This environment comprises N heterogeneous hunter UAVs and M dynamic targets with active evasion capabilities. Static obstacles are also randomly distributed throughout the environment. There are N UAVs acting as the hunters. Their heterogeneity is mainly reflected in differences in maximum speed and energy consumption, which is closer to real-world scenarios. Targets have a higher maximum speed than the hunters and use their maneuverability advantage to actively evade capture. Each hunter is equipped with onboard sensors to detect the surrounding environment. To simulate realistic sensing limitations and ensure a decentralized framework, the hunter’s observation space is strictly limited to local information. Specifically, consistent with Equation (8), the input state S_h consists of the agent’s own state, the relative positions of the k nearest neighbors (e.g., k = 2), and the specifically assigned target. Information regarding other distant agents or unassigned targets is filtered out, ensuring that the policy execution does not rely on global observability. These sensors also detect surrounding obstacles. To simulate more challenging and realistic combat scenarios, the objective in this study is not to use predefined heuristic evasion logic. Instead, each target is modeled as an independent reinforcement learning agent, which is trained in synchrony with the hunter agent.

Figure 1 illustrates the conditions for multi-UAV cooperative single-target encirclement. Black and orange UAVs denote hunters and the target, respectively. The blue circles indicate the effective encirclement range of each UAV, while black circles represent static obstacle areas. The red triangle signifies the dynamic encirclement net. Here, d_i denotes the distance between pursuing UAV i and the target. d_i,j represents the distance between UAVs i and j. d_capture is the capture radius of a single UAV. The cooperative encirclement condition requires that the target must be simultaneously within the capture range of three UAVs and located inside the triangle formed by these three UAVs.

3.2. UAV Kinematics Model

The UAV motion state is defined by position, velocity, and acceleration in Cartesian coordinates. The motion equations are governed by the following discrete time model:

$$\left\{\begin{array}{l}\left[\begin{array}{l}{p}_x^{t+1}\\ p_y^{t+1}\end{array}\right]=\left[\begin{array}{l}{p}_x^t\\ p_y^t\end{array}\right]+\Delta t\left[\begin{array}{l}{v}_x^t\\ v_y^t\end{array}\right]\\ \left[\begin{array}{l}{v}_x^{t+1}\\ v_y^{t+1}\end{array}\right]=\left[\begin{array}{l}{v}_x^t\\ v_y^t\end{array}\right]+\Delta t\left[\begin{array}{l}{a}_x^t\\ a_y^t\end{array}\right]\\ {\phi }^t=\arctan {\displaystyle \frac{v_y^t}{v_x^t}}\end{array}\right.$$

(1)

$$\left\{\begin{array}{l}\left|v^t\right|\le v_{\max }\\ \left|a^t\right|\le a_{\max }\end{array}\right.$$

(2)

$$0\le p_x\le L, 0\le p_y\le L$$

(3)

$$E_{total}={\displaystyle \sum _{t=1}^T\Delta E^t}$$

(4)

$$\Delta E_{move}=\eta \cdot \left|\left[\begin{array}{l}{p}_x^t\\ p_y^t\end{array}\right]-\left[\begin{array}{l}{p}_x^{t-1}\\ p_y^{t-1}\end{array}\right]\right|$$

(5)

$$L_{obs}=\left[l_1,l_2,\dots ,l_{16}\right]$$

(6)

The motion equation of the UAV is described in Equation (1). At time t, p^t, v^t, a^t, and φ^t denote the position, velocity, acceleration, and yaw angle of the UAV, respectively. Δt denotes the time interval, and a represents the UAV’s acceleration. v^t_x and v^t_y indicate the UAV’s velocity components along the X and Y axes at time t, while a^t_x and a^t_y correspond to the acceleration components along the X and Y axes at time t. Equation (2) indicates the maximum speed and acceleration. Equation (3) is the boundary constraints. Equations (4) and (5) show that energy consumption is modeled as a linear function of displacement distance. Equation (6) describes the UAV’s detection model, in which 16 range sensors are mounted on the UAV to detect potential obstacles.

4. The Proposed Algorithm: LIPPO

4.1. Algorithm Framework

Algorithm 1 presents the pseudocode of the LIPPO framework. This algorithm integrates PPO for decision-making with an LSTM network for active trajectory prediction. Firstly, the actor–critic networks, the LSTM predictor, and their respective replay trajectory buffers are initialized (lines 1–4). At the beginning of each episode, the environment is reset, and targets are assigned to hunters (lines 6–7). Within each step, all agents select actions based on their current policies. They interact with the environment and store the transition samples (s, a, r, s’) in their replay buffers (lines 9–12). When the update conditions are met, a mini-batch of samples are randomly selected from the buffer. The critic networks, LSTM trajectory predictors, and actor networks are updated sequentially with gradient descent. To enhance training stability, target networks are updated with the soft update strategy (lines 13–22).

Algorithm 1: Framework of LIPPO
Input: Emax (total episodes), Tmax (max steps), λ, γ, τ( RL parameters ).
Output: Trained actor μθ and critic Qϕ networks for all agents.
1:	Initialize actor networks μθ and critic Qϕ networks for hunters.
2:	Initialize target networks μ′θ and critic Q′ϕ.
3:	Initialize replay buffers RH (hunters), RT (targets).
4:	Initialize LSTM predictor μLSTM and its trajectory buffer Rtraj.
5:	For episode = 1 to Emax do
6:		Reset environment and get initial observations s.
7:		Assign targets to hunters using greedy distance-based rule (once per episode)
8:		For t = 1 to Tmax do
9:			For each agent i, select action ai
10:			Execute actions, observe next states s′, rewards r.
11:			Store transition (s, a, r, s′) in corresponding replay buffer.
12:			s ← s′
13:			If replay buffers (Size > 1024) are ready and t mod 50 == 0
14:				For each agent i do
15:					Sample a random minibatch B (BatchSize = 256).
16:					Calculate TD targets and Critic loss.
17:					Update critic parameters φ by minimizing L(φ).
18:					Calculate advantages and the importance sampling ratio.
19:					Calculate PPO’s clipped surrogate objective.
20:				End
21:				Soft-update target networks (τ = 0.01).
22:			End
23:			Train μLSTM with Equation (7).
24:		End
25:	End

Figure 2 shows the overall training process of the LIPPO. This framework is built upon a decentralized multi-agent learning paradigm. To enhance learning efficiency, an off-policy variant of the Independent Proximal Policy Optimization (IPPO) algorithm is employed. Unlike standard PPO, which discards collected samples after a single update, the proposed approach retains transition data in replay buffers and reuses them across multiple training iterations. Each agent, whether a pursuer or a target, is an independent learning unit with its own Actor–Critic network for policy approximation and state value evaluation.

During interaction with the environment, the experiences of states, actions, and rewards generated are stored in separate replay buffers for the hunter and the targets. These experiences are used for batch updates of their networks. The core of the framework is an online-trained LSTM network, which uses the target’s historical trajectory data to predict its future position. The hunters then use this predicted point as a new interception target, transitioning from passive tracking to an active interception strategy.

4.2. LSTM-Based Trajectory Prediction for Active Interception

4.2.1. Target–Hunter Assignment and LSTM Network

To address the complexity of the multi-hunter, multi-target problem, a simple yet effective greedy heuristic for task allocation is employed. While optimization-based assignment methods, such as the Hungarian algorithm, or auction-based mechanisms can theoretically achieve globally optimal allocations, their cubic computational complexity O(N³) imposes substantial overhead. Moreover, within the framework of independent reinforcement learning, highly dynamic or complex reassignment strategies tend to exacerbate environmental non-stationarity, thereby impeding stable policy convergence. Consequently, a greedy assignment strategy is adopted in this study to strike a balance between computational efficiency and learning stability. The distances between all hunters and all uncaught targets are calculated firstly. Then, for each target, the nearest hunter is greedily assigned. Each target is assigned three hunters. This greedy assignment reduces the complexity of task allocation. Each agent focuses on its assigned target, thereby speeding convergence.

4.2.2. Online Trajectory Prediction Using LSTM

The core of the predictive module is an LSTM network designed to learn and forecast a target’s motion trajectory.

An LSTM network is highly effective at capturing the temporal dependencies in sequential data, which makes it exceptionally well suited for this task. The input to the LSTM network is a sequence of historical states of the target. Each state is a four-dimensional vector with the target’s two-dimensional position and two-dimensional velocity. The model generates a complete trajectory prediction for the next time steps through a single forward pass. At each time step, the target’s current state is recorded and stored in the trajectory buffer. Once the buffer accumulates sufficient data, the LSTM model begins training. Its objective is to minimize the mean squared error (MSE) between the predicted trajectory and the actual future trajectory sampled from the buffer. The loss function is defined as follows:

$$L_{LSTM}={\displaystyle \frac{1}{BN}}{\displaystyle \sum _{\mathrm{j}=1}^B{\displaystyle \sum _{i=1}^N{‖{\stackrel{⌢}{P}}_{j,i}-P_{j,i}‖}_2^2}}$$

(7)

where B is the batch size. N is the prediction stride. ${\stackrel{⌢}{P}}_{j,i}$ is the predicted position of the j sample at the i step. P_j,i is the true position.

4.2.3. Active Interception

Traditional passive pursuit strategies involve directly tracking the target’s current location. In contrast, within the active interception framework, the hunter’s decisions are based on predictions of the target’s future position. Specifically, each hunter will obtain the LSTM-predicted trajectory of its assigned target. The hunter will use the predicted trajectory as its interception target point. This point represents the position where the target is most likely to reach in the future. By doing so, the hunter can take a more direct and advantageous interception path. This achieves active interception and significantly improves the capture efficiency.

Figure 3 illustrates the core concept of the active interception mechanism. UAV₁, UAV₂, and UAV₃ represent the hunter UAVs. P_target denotes the target’s current actual position, while P_pred represents the future position predicted by the LSTM based on the target’s historical trajectory. As shown by the black dashed lines in Figure 3, all hunter UAVs take the predicted point as their interception target rather than the target’s current position. This forward-looking decision-making approach enables the hunters to plan more direct and efficient paths. These paths prioritize interception at the predicted location and significantly improve the efficiency and success rate of the encirclement task.

4.3. LIPPO

4.3.1. State

To enable the agent to fully perceive its environment and make reasonable decisions, distinct state spaces for the hunter and the target have been designed.

The hunter h is formally defined as a 12-dimensional vector in Equation (8).

$$s_h=\left[{\displaystyle \frac{p_{n1}}{L}},{\displaystyle \frac{p_{n2}}{L}},{\displaystyle \frac{p_h}{L}},{\displaystyle \frac{v_h}{v_{\max }}},{\displaystyle \frac{{\stackrel{⌢}{p}}_h}{L}},{\displaystyle \frac{p_t}{L}}\right]$$

(8)

where p_n1 and p_n2 represent the position of the nearest neighbor. All coordinates are normalized with respect to the environment size L. ${\stackrel{⌢}{p}}_h$ denotes the optimal interception point calculated based on the LSTM model. P_h is the target’s current actual position.

The target t is formally defined as a 10-dimensional vector in Equation (9).

$$s_t=\left[{\displaystyle \frac{p_t}{L}},{\displaystyle \frac{v_t}{v_{\max }}},{\displaystyle \frac{p_{h1}}{L}},{\displaystyle \frac{p_{h2}}{L}},{\displaystyle \frac{p_{h3}}{L}}\right]$$

(9)

where p_t represents the target’s position coordinates. P_h1, P_h2, and P_h3 denote the coordinates of the three hunters closest to the target.

4.3.2. Action

The action design for both the target and the hunters is defined as shown in Equation (10).

$$a_{h,t}=\left[a_x,a_y\right] a\le a_{\max }$$

(10)

The actions of both the targets and the hunters are defined as acceleration values along the X and Y axes. Each agent learns to effectively control its acceleration in order to achieve the goal of either hunting or evading.

4.3.3. Reward

In order to effectively train a reinforcement learning model, it is crucial to design a reasonable reward function. The reward mechanisms are described as follows.

Hunter Pursuit Reward: To guide hunters in forming efficient pursuit strategies, a reward function has been designed. Its specific form is shown in Equation (11):

$$R_{pursuit}^{h_i}=\underset{Distance }{\underbrace{(1-2\cdot {\displaystyle \frac{‖p_h-{\stackrel{⌢}{p}}_h‖}{\sqrt{2}\cdot L}}}})+\underset{\mathrm{Direction}}{\underbrace{({\displaystyle \frac{v_h\cdot ({\stackrel{⌢}{p}}_h-p_h)}{‖v_h‖\cdot ‖{\stackrel{⌢}{p}}_h-p_h‖}})}}+-k\cdot \Delta s+R_{step}$$

(11)

where v_h represents the hunter’s current velocity vector. L is the boundary length of the environment. R_step is a small negative reward. It is applied at each time step to incentivize the agent to complete tasks in fewer decision steps. k is the energy penalty coefficient, and Δs is the distance moved in a single step. This term encourages the hunter to complete the task with a shorter path.

The core of the distance reward is to transform the physical distance between the hunter and the interception point into a normalized reward. Unlike traditional positive incentives, this study linearly maps this value to the interval [−1, 1], thereby providing a stronger punishment signal. In this way, the distance reward not only encourages the hunter to approach the target but also severely punishes its movement away from the target.

Directional reward is calculated based on the cosine similarity between the hunter’s velocity vector and the target’s direction vector, with a value range of [−1, 1]. When the hunter’s movement direction perfectly aligns with the target’s direction, the reward is +1; when completely opposite, it is −1; and when perpendicular, it is 0. This design provides the agent with continuous, fine-grained feedback signals, thereby guiding it to progressively optimize its action strategy and learn more efficient approach paths.

Capture Reward:

$$R_{capture}^{h_i}=r_{capture}$$

(12)

where r_capture is a predefined reward assigned for a successful capture.

Target Escape Reward:

$$R_{escape}^{t_j}=\left\{\begin{array}{ll}0.5& d_{\min }>d_{escape}\\ -0.5& d_{\min }\le d_{escape}\end{array}\right.$$

(13)

where d_min denotes the distance from the target to the nearest hunter. d_escape represents the escape distance threshold.

Hunter–Obstacle Penalty:

$$P_{obs}=C_{safe}\cdot (1-{\displaystyle \frac{L_{\min }}{L_{sensor}}})$$

(14)

where L_sensor is the maximum detection range of the sensor, and L_min is the shortest distance from the hunter to the nearest obstacle.

Boundary Penalty:

$$P_{boundary}=C_{boundary}\cdot (1-{\displaystyle \frac{L_{\min b}}{B_{boundary}}})$$

(15)

where C_boundary is the boundary penalty coefficient. B_threshold denotes the boundary distance threshold. L_minb represents the minimum distance from the agent to the boundary.

Finally, the complete reward function is formulated in Equations (16) and (17).

$$R_{total}^{h_i}=R_{pursuit}^{h_i}+R_{capture}^{h_i}+P_{obs}+P_{boundary}$$

(16)

$$R_{total}^{t_i}=R_{escape}^{t_j}+P_{obs}+P_{boundary}$$

(17)

4.3.4. Network Update

In the LIPPO framework, each agent (hunter or target) operates as an independent learner, maintaining its own Actor and Critic networks. The network parameters are updated independently using an off-policy actor–critic framework, where the policy is optimized via a PPO-style clipped surrogate objective.

The Critic network is updated to minimize MSE between the estimated value V_ϕ (s_t) and the TD target y_t. To ensure training stability, a target critic network V’_ϕ’ is employed to compute y_t, which is updated via a soft update mechanism (Polyak averaging) with a coefficient ϕ’ ← τϕ + (1 − τ) ϕ’.

The Actor network is optimized by maximizing the clipped surrogate objective function:

$$L^{CLIP}(\theta )={\stackrel{⌢}{E}}_t[\min (r_t(\theta ){\stackrel{⌢}{A}}_t,clip(r_t(\theta ),1-\epsilon ,1+\epsilon ){\stackrel{⌢}{A}}_t)]$$

(18)

where ϵ is a hyperparameter used to define the clipping range. The clip function limits the importance sampling ratio to the interval [1 − ϵ, 1 + ϵ]. This mechanism restricts the size of each update. As a result, it prevents destructive updates to the policy network and ensures the stability of the training process. The parameters θ of the Actor network are updated with gradient ascent to maximize this objective function.

To explicitly address the non-stationarity inherent in multi-agent co-learning, the proposed framework integrates three stabilization mechanisms. A shared experience replay buffer is utilized across the hunter swarm to smooth the training data distribution and attenuate variance induced by heterogeneous agent interactions. In addition, the active interception mechanism is activated only after the LSTM-based predictor has completed a warm-up training stage. This strategy mitigates policy degradation caused by unreliable predictions during early training. Furthermore, the critic networks are updated using Polyak averaging, enabling stable value function estimation in the presence of dynamically evolving opponent strategies.

5. Experiment and Analysis

5.1. Parameter Setting

In this paper, the time step is set to 0.5 s, and the environment area is a square area with a side length of 2 km. During training, the UAVs, target and obstacle locations, and the radius of the obstacles are randomly generated.

Table 2. Parameters used in environmental simulation.

Variable Name	Parameters
Boundary Length	2.0
Simulation Time Step	0.5
Number of Obstacles	3, 6, 9
Obstacle Radius	0.05~0.1
Number of Hunters	15
Initial Velocity of Hunters	0.0
Maximum Speed of Hunters	0.06~0.08
Maximum Acceleration of Hunters	0.02
Sensor Detection Range	0.2
Number of Lidar Rays	16
Energy Consumption Rate	0.1~0.3
Number of Targets	5
Initial Velocity	0.0
Maximum Speed of Targets	0.09
Maximum Acceleration of Targets	0.03
Target Escape Distance	0.2
Safety Distance Threshold	0.02

details the environmental parameters for this multi-UAV entrapment mission.

To ensure the reproducibility of the co-learning process, the training pipeline is synchronized as follows: The PPO agents update their networks every 50 steps once the replay buffer collects a minimum of 1024 samples. The LSTM predictor accumulates trajectory data (position and velocity) during a “warm-up” phase. Once the trajectory buffer exceeds 5000 points and the simulation passes 100 episodes, the LSTM begins fine-tuning at the end of each episode, and the hunters switch from reactive tracking to the active interception strategy.

All algorithms are trained independently under five mutually exclusive random seeds (0–4). Each seed trains 2000 episodes with 150 steps per episode. For testing, 100 test seeds (range 5–1000) are randomly generated and fixed to ensure that different algorithms are evaluated on identical scenarios. Crucially, to distinguish test time performance from training time co-evolution, the policies of all agents (both hunters and targets) and the LSTM predictor are frozen during the testing phase. No gradient updates or parameter changes occur during evaluation. The models used for testing are the final converged policies obtained after training. Metrics are averaged over 100 independent test episodes. Specifically, ‘Average Capture Steps’ denotes the mean mission duration required to entrap all targets. ‘Total Energy Consumption’ represents the cumulative energy expended by the entire hunter swarm per episode, reflecting the overall system efficiency. The algorithm’s training hyperparameters and neural network architecture settings are shown in

Table 3. Hyperparameter setting of neural networks.

Algorithms	Hyperparameter	Value
PPO	Learning Rate	3 × 10−4
	Gamma (γ)	0.95
	Clip Epsilon (ε)	0.2
	Update Frequency	50
	Buffer Warm-up Threshold	1024
	Batch Size	256
	Num_episodes	2000
	Max_steps	150
	Replay Buffer Size	1 × 105
	Target network soft update rate	0.01
	Actor and Critic network structure	256 × 256
	Active Strategy Trigger	100 Episodes
LSTM	Input/Hidden Size	4/64
	Output Size	2
	Sequence Length	10
	Prediction Horizon	3
	Learning Rate/Batch Size	1 × 10−4/64
	Buffer Warm-up Threshold	5000
	Update Frequency	Every 1 Episode

.

Finally, to assess real-time feasibility, computational experiments were conducted on a workstation equipped with an Intel Core i5-9300H CPU, 32 GB RAM, and an NVIDIA GeForce RTX 3060 GPU. Under this configuration, the complete training process for 2000 episodes requires approximately 2.33 h. The average inference latency per agent is approximately 2.0 ms.

5.2. Ablation Experiments

To verify the effectiveness of each component of LIPPO, two variant algorithms, XIPPO and NLIPPO, are generated. XIPPO replaces the LSTM module with a linear regression model to generate future position predictions of the target. NLIPPO completely removes the prediction and interception mechanisms. It adopts a reactive strategy that directly pursues the real-time position.

Figure 4 illustrates the average reward of all hunters and the encirclement success rate during training. LIPPO outperforms the other two variants. Its average reward rises the fastest and reaches the highest level. Regarding encirclement success rate, LIPPO displays an overwhelming advantage.

Figure 5 further illustrates the LSTM network loss of LIPPO and the average reward of all targets during training. The reward value of all targets can be observed to increase gradually with each iteration. This indicates that the target actually becomes better at escaping with each iteration. Concurrently, the loss of the LSTM network steadily decreases, which confirms its stable convergence.

In the early stages of training, the average reward of LIPPO was significantly lower than that of the other two algorithms. As illustrated in Figure 5, this is because the LSTM model had not yet been sufficiently trained at this stage. Its trajectory buffer data were sparse. The predicted points were nearly random, sometimes worse than no prediction. In contrast, NLIPPO directly used the target’s current position for pursuit from the outset, which resulted in a more stable initial reward. In the middle stage of training, the average reward curves of the algorithms tended to converge. Meanwhile, the capture success rate increased steadily.

Table 4. Performance comparison of three algorithms under different numbers of obstacles (Mean ± Standard Deviation).

Number of Obstacles	Algorithm	Success Rate (%)	Total Energy Consumption	Average Capture Steps
3	LIPPO	82.2 ± 1.92	300.63 ± 32.07	318.32 ± 19.13
	NLIPPO	76.0 ± 3.16	308.38 ± 42.18	314.44 ± 7.75
	XIPPO	70.6 ± 4.61	284.52 ± 25.78	342.30 ± 11.17
6	LIPPO	73.60 ± 5.86	242.59 ± 56.02	306.56 ± 13.02
	NLIPPO	70.8 ± 4.08	329.54 ± 44.40	334.41 ± 20.47
	XIPPO	64.4 ± 3.85	304.12 ± 61.50	339.16 ± 20.79
9	LIPPO	65.2 ± 6.06	276.06 ± 35.75	317.67 ± 19.28
	NLIPPO	56.8 ± 4.92	331.21 ± 99.80	286.43 ± 35.07
	XIPPO	55.0 ± 6.21	320.70 ± 30.11	347.81 ± 16.74

presents the test evaluation results under low, medium, and high-complexity environments with 3, 6, and 9 obstacles, respectively. Evaluation metrics include success rate, total energy, and average capture steps. In low-complexity environments, the LIPPO demonstrates the highest encirclement success rate. NLIPPO holds a slight advantage in capture steps due to its strategy of direct pursuit without prediction. XIPPO’s linear prediction model can approximately fit the target’s simple maneuvering trajectory, thereby achieving the lowest energy consumption. This indicates that in simple scenarios, the advantages of active prediction have yet to be fully realized.

As environmental complexity increases, the comprehensive advantages of LIPPO become evident. In medium complex scenarios, LIPPO surpasses both NLIPPO and XIPPO across three core metrics: success rate, total energy consumption, and capture steps. This demonstrates that the LSTM-based active interception mechanism can accurately predict complex maneuvering intentions of targets, thereby planning more efficient and energy-saving pursuit paths. In highly complex scenarios, LIPPO’s success rate remains significantly ahead. It outperforms the other two algorithms by nearly 10 percentage points while maintaining optimal energy consumption. This underscores LIPPO’s formidable adaptability within complex, dynamic adversarial environments.

Although NLIPPO achieves a lower average number of capture steps when successful, this does not equate to high efficiency. Its successful cases are highly concentrated in simple scenarios requiring no complex planning. For instance, in situations where initial conditions are favorable, the target happens not to perform complex evasive maneuvers, or obstacles do not form critical blockages. Under such circumstances, a greedy pursuit strategy will naturally complete the task swiftly.

Meanwhile, the comparison between LIPPO and XIPPO demonstrates that when confronted with agents that have the capacity for learning, simple linear prediction models are unable to capture the complexity and nonlinearity of an opponent’s strategy. It can therefore be concluded that LIPPO’s success is largely attributable to the powerful temporal modeling capabilities afforded by LSTM.

5.3. Comparative Experiments

To further validate LIPPO’s performance, comparative experiments are conducted against IDDPG, ITD3, LMAPPO, and MAPPO. Building upon the MAPPO algorithm within the standard CTDE framework, LMAPPO incorporates the same online LSTM prediction module as LIPPO and employs an active interception strategy. MAPPO contains no prediction mechanism whatsoever. IDDPG and ITD3 denote independent learning variants based on DDPG and TD3, respectively. Figure 6 presents the hunter’s average reward and success rate, while Figure 7 displays the target’s average reward. The results in Figure 6 and Figure 7 demonstrate that LIPPO achieves superior performance over the comparison algorithms.

In the early stages of training, IDDPG and ITD3 had higher reward curves, while LIPPO, LMAPPO, and MAPPO had relatively lower rewards. This was because the LSTM module had not yet accumulated sufficient trajectory data, so its predictions were severely off-target and its capture success rate remained low. MAPPO, based on the CTDE paradigm, needed to handle large-scale global states and joint action information during training. This resulted in lower initial learning efficiency and less stable performance compared to other algorithms.

As training progressed, the LSTM modules of LIPPO and LMAPPO gradually converged, enabling accurate predictions of future trajectories. At this stage, the advantages of the active interception strategy became apparent, as both the reward and success rate curves rose rapidly. MAPPO also improved steadily as the centralized critic’s value estimation matured. Its final performance surpassed that of IDDPG and ITD3 but still lagged behind LIPPO and LMAPPO. Meanwhile, as shown in Figure 7, the target’s average reward increased, indicating its improved evasiveness. However, LIPPO still maintained a high capture efficiency. This fully verified the effectiveness of the active interception mechanism in complex adversarial scenarios.

Table 5. Performance Comparison between LIPPO and Baseline Algorithms under Various Complexities (Mean ± Standard Deviation).

Number of Obstacles	Algorithm	Success Rate (%)	Total Energy Consumption	Average Capture Steps
3	LIPPO	82.2 ± 1.92	300.63 ± 32.07	318.32 ± 19.13
	LMAPPO	81.0 ± 3.16	309.27 ± 60.93	295.54 ± 13.39
	MAPPO	80.6 ± 3.58	309.27 ± 60.93	295.54 ± 13.39
	ITD3	76.8 ± 3.77	335.68 ± 116.73	244.03 ± 29.49
	IDDPG	71.4 ± 5.81	288.81 ± 61.72	252.25 ± 28.19
6	LIPPO	73.6 ± 5.86	242.59 ± 56.02	306.56 ± 13.02
	LMAPPO	75.0 ± 2.92	337.98 ± 71.51	303.14 ± 9.22
	MAPPO	73.2 ± 2.68	305.65 ± 32.43	301.24 ± 10.07
	ITD3	72.0 ± 6.28	350.76 ± 103.20	244.70 ± 24.73
	IDDPG	73.0 ± 3.94	283.98 ± 64.90	258.21 ± 17.71
9	LIPPO	65.2 ± 6.06	276.06 ± 35.75	317.67 ± 19.28
	LMAPPO	63.80 ± 4.15	309.30 ± 79.94	309.05 ± 12.53
	MAPPO	61.6 ± 3.58	281.58 ± 23.09	328.88 ± 7.45
	ITD3	56.6 ± 6.39	422.81 ± 217.20	276.59 ± 34.80
	IDDPG	57.4 ± 4.80	366.03 ± 153.20	270.75 ± 42.98

presents the test evaluation results for all comparison algorithms under low, medium, and high-complexity environments. In low-complexity scenarios, all algorithms achieved success rates exceeding 70%. LIPPO achieved the highest success rate of 82.2%, with LMAPPO (81.0%) and MAPPO (80.6%) following closely behind. The performance gap among the three is relatively small. Notably, while IDDPG and ITD3 did not match LIPPO’s success rate, they demonstrated outstanding performance in energy consumption and execution efficiency.

In the medium-complexity scenario, LMAPPO achieved a slightly higher success rate (75.0%) than LIPPO (73.6%), demonstrating the potential of the CTDE framework in specific cooperative tasks. To strictly validate the performance differences, independent Welch’s t-tests were conducted based on the data from 100 test episodes. Independent Welch’s t-tests confirmed that in the medium-complexity environment, LIPPO and LMAPPO show no statistically significant difference in success rates (t ≈ −0.47, p = 0.65 > 0.05), indicating comparable efficacy. Conversely, LIPPO demonstrates a highly significant reduction in total energy consumption (t ≈ −8.55, p < 0.001). These results objectively confirm that LIPPO maintains high encirclement performance while significantly reducing system energy costs. However, under the highest complexity, LIPPO surpassed LMAPPO with a success rate of 65.2% compared to LMAPPO’s 63.8%, showing stronger robustness. LIPPO’s most prominent advantage lies in its excellent energy efficiency. In the scenario with six obstacles, although LMAPPO had a slightly higher success rate, its energy consumption (337.98) was much higher than LIPPO’s 242.59, saving nearly 28% of energy. In the scenario with nine obstacles, LIPPO’s energy consumption was 276.06. Figure 8 shows the motion paths of 15 hunters (UAVs) surrounding five targets in environments with 3, 6, and 9 obstacles (gray circles). Quantitative analysis of the trajectories in Figure 8 reveals that LIPPO significantly improves path smoothness compared to baseline methods. By targeting a future region rather than a moving point, LIPPO reduces the ‘path tortuosity’ caused by frequent reactive adjustments to the target’s instantaneous velocity changes. Furthermore, the predictive logic implicitly distributes hunters spatially, reducing the frequency of proximity penalty triggers (collision risks) observed in reactive swarms, where agents tend to crowd directly behind the target.

The core contributions of this study are twofold. First, through comparative experiments under the two mainstream MARL paradigms of CTDE and IL, the superiority of the prediction-based active interception strategy over the traditional passive tracking strategy has been established. Second, under the fair premise that all algorithms adopt the active interception strategy, a detailed comparison was made between the two decision-making frameworks of IPPO (LIPPO) and MAPPO (LMAPPO). The results show that the two perform similarly in terms of task success rate. LIPPO, however, demonstrates an advantage in energy consumption because its independent learning framework is simpler and more efficient.

6. Conclusions

In summary, this study proposes the LIPPO framework, which introduces an active interception mechanism to address multi-target pursuit by heterogeneous UAVs. This mechanism is realized through the integration of an online-trained LSTM network for real-time trajectory prediction and an independent PPO learner for each agent. By predicting future target positions, it enables a strategic shift from passively tracking the current target location to proactively intercepting it at anticipated points. The learning process of each agent is guided by a carefully designed multi-dimensional reward function. Simulation results demonstrate that LIPPO performs excellently in complex, many-to-many adversarial scenarios, achieving efficient collaborative strategies.

It is also important to note the limitations regarding the energy model used in this study. The current linear displacement-based model serves as a first-order kinematic approximation. While sufficient for evaluating high-level strategic cooperation, it does not account for complex aerodynamic factors such as drag coefficients, banking turns, or rotor thrust dynamics.

Despite the superior performance in simulations, transitioning to real-world deployment presents distinct engineering challenges. Regarding sensing and communication, while our decentralized framework minimizes bandwidth usage by relying on local observations, future work will employ domain randomization to mitigate real-world sensor noise. In terms of scalability, the proposed task decomposition strategy ensures linear complexity growth, theoretically supporting larger swarms. Finally, while validated in 2D, the underlying active interception logic is dimension-agnostic and can be naturally extended to 3D dynamics with 6-DOF constraints to enhance realism.