Adaptive Missile Avoidance Algorithm for UAV Based on Multi-Head Attention Mechanism and Dual Population Confrontation Game

Abstract

In recent years, UAVs have faced increasingly severe and diversified missile threats. To address the challenge that reinforcement learning-based missile evasion algorithms struggle to adapt to various unknown missile types, we introduce a risk-sensitive PPO algorithm and propose a training framework incorporating multi-head attention mechanisms and dual-population adversarial training. The multi-head attention mechanism enables the policy network to extract latent features such as missile guidance laws from state sequences, while the dual-population adversarial approach ensures policy diversity and robustness. Compared to conventional self-play methods and GRU-based evasion strategies, our method demonstrates superior training efficiency and generates evasion policies with better adaptability to different missile types.

1. Introduction

Missile evasion strategies are a crucial research area in modern military operations, aimed at enhancing the survivability of UAVs and other equipment when facing missile threats [1]. The research background of missile evasion strategies stems mainly from the complexity and variability of missile systems. During missile attacks, UAVs must assess threats in real-time and respond swiftly. Ref. [2] proposed a threat-plane-based approach to improve the speed of threat alerts, enabling aircraft to initiate evasive maneuvers earlier, thereby significantly increasing their survival probability. In the context of missile evasion tasks, numerous strategies have been developed. For instance, Ref. [3] proposed a hybrid evasion strategy based on a variable-structure guidance law; Ref. [4] explored a real-time control-based pursuit-evasion game method; Ref. [5] designed a three-dimensional evasion strategy without relying on missile guidance information; and Ref. [6] investigated equilibrium strategies in three-dimensional pursuit-evasion games. These strategies leverage mathematical models and computational algorithms to continuously optimize evasion decisions, aiming to better adapt to the dynamic nature of actual defence scenarios. Despite the significant progress achieved by traditional evasion methods, they still face notable limitations in practical applications [7]. Traditional missile evasion methods face several limitations. First, they often rely on fixed threat assessment and response mechanisms, struggling to adapt to dynamic battlefield conditions—such as highly maneuverable missiles or coordinated multi-target attacks. Second, they depend on precise threat parameters, which are often unavailable in real combat. Additionally, while game-theoretic approaches offer theoretical insights, their high computational complexity hinders real-time application. To overcome these challenges, reinforcement learning (RL) has emerged as a promising alternative. RL enables dynamic strategy optimization through environmental interaction, improving adaptability and robustness in complex scenarios [7,8,9].

The combination of reinforcement learning and game theory has demonstrated tremendous potential across various domains, providing a theoretical foundation for strategic interactions in multi-agent systems and offering new perspectives and methods for solving complex decision-making problems [10,11,12]. In the field of transportation, Ref. [6] proposed a framework called SafeDriveRL, which integrates non-cooperative game theory and reinforcement learning to explore and mitigate high-level uncertainties caused by human behavior. This framework simulates the reward mechanisms of human drivers to evaluate the adaptability of autonomous vehicles in complex traffic environments. In the field of unmanned aerial vehicles (UAVs), Ref. [13] presented an improved radar tracking method based on cooperative game theory and reinforcement learning, offering innovative solutions for UAVs performing collaborative tasks in complex environments. In the domain of cyber–physical security, Ref. [14] designed a cyber–physical security framework that integrates reinforcement learning and game theory, simulating attack-defense scenarios and demonstrating its practical value in social control systems. Reinforcement learning, with its powerful learning capabilities and flexible decision-making mechanisms, has exhibited outstanding performance across various fields, particularly in addressing problems characterized by high uncertainty and dynamic complexity [15]. This suggests that reinforcement learning holds significant potential in tackling similar challenges in missile evasion tasks. By combining game theory, reinforcement learning is expected to provide smarter, real-time, and more efficient evasion strategies, opening up new directions for the research and application of missile evasion. Such an approach not only addresses the limitations of traditional strategies but also has the potential to significantly enhance the survivability of UAVs in modern battlefield environments.

The missile evasion problem presents unique challenges due to the diversity of threat characteristics (including guidance systems, speeds, and maneuverability) and the highly dynamic nature of the engagement environment [16]. These factors, combined with potential incomplete or unreliable threat information, create significant difficulties for traditional reinforcement learning approaches. Recent advances show promise but face limitations: Yan et al. [17] developed a TD3-based intelligent maneuvering strategy for hypersonic vehicles that effectively handles two interceptors in continuous action spaces, though its computational demands may hinder real-time implementation. Similarly, Jia et al. [18] proposed a model-free RL approach for pursuit-evasion games with unknown dynamics, but its generalization in complex scenarios requires further validation. These developments highlight both the progress in RL-based evasion strategies and the critical need for improved generalization capabilities to address the full spectrum of missile threats and environmental dynamics in practical applications.

Recent studies have made significant progress in enhancing the generalization capabilities of reinforcement learning (RL). Ho et al. [19] proposed a human-inspired meta-reinforcement learning (HMRL) framework that enables cross-task generalization through Bayesian knowledge analysis. Yang et al. [20] developed Ensemble Proximal Policy Optimization (EPPO), demonstrating improved robustness in policy integration. McClellan et al. [21] significantly enhanced sample efficiency using Equivariant Graph Neural Networks (EGNNs), while Prajapati et al. [22] improved temporal generalization through State Chrono Representation (SCR). Hu et al. [23] further advanced generalization by optimizing state representations. However, these methods face substantial challenges when applied to missile evasion scenarios, which are characterized by highly dynamic threats, rapidly changing environments, and incomplete information. The core assumptions of existing approaches often fail to meet the stringent requirements for real-time performance, adaptability, and partial observability inherent in missile evasion tasks. Consequently, there is an urgent need to develop specialized RL generalization enhancement methods tailored to the unique demands of missile evasion in real-world operational environments.

In order to solve the problems of poor generalization ability of the evasion strategy and information asymmetry in the missile evasion problem, this paper introduces a risk-sensitive PPO algorithm based on the multi-head attention mechanism and proposes a training framework for dual-population game confrontation. The multi-head attention mechanism is used to construct the policy network and the value network, enabling it to extract the motion characteristics of the missile from historical information. At the same time, the method of dual-population game solves the problem of overfitting the opponent in the self-play method, ensuring the diversity and generalization ability of the evasion strategy, and enabling it to effectively deal with different types of incoming missiles.

This paper is organized as follows: Section 2 introduces the mathematical model of the missile evasion problem, including the dynamics and kinematics models of the UAV, missile, and decoy flare. Section 3 introduces the method proposed in this paper, including the dual-population game framework based on the RPPO algorithm, the Markov decision model of the missile evasion problem, and the network structure design based on the multi-head attention mechanism. Section 4 verifies the effectiveness and superiority of the method proposed in this paper through simulation experiments. Conclusive remarks and a summary of findings are presented in Section 5.

2. Preliminary

2.1. UAV Dynamic Model

This paper mainly discusses the intelligent control of fixed-wing UAVs when dealing with missile threats. Therefore, the method of fixed-wing aircraft is introduced to model the kinematics and dynamics of UAVs. The external forces acting on a UAV influence its flight attitude and velocity, thereby altering its trajectory. These forces include gravity ($\mathit{W}$), thrust ($\mathit{T}$), and aerodynamic forces ($\mathit{A}$). Among them, gravity ($\mathit{W}$) acts through the UAV’s center of mass, while thrust ($\mathit{T}$) and aerodynamic forces ($\mathit{A}$) generate moments around the center of mass. These moments include the pitch moment ($\mathit{M}$), yaw moment ($\mathit{N}$), and roll moment ($\mathit{L}$), as illustrated in Figure 1.

When analyzing a UAV’s flight attitude and trajectory, it can be assumed that the deflection of control surfaces eliminates the aforementioned moments, ensuring that the total moment acting on the UAV remains balanced. This results in a spatial force system intersecting at the UAV’s center of mass. The forces acting on the UAV, excluding gravity, are collectively referred to as the controllable force ($\mathit{N}$) and can be expressed as

$$\mathit{N}=\mathit{A}+\mathit{T}$$

(1)

The resultant force ($\mathit{N}$) is decomposed along the $O_a{x}_a$ axis and the $O_a{y}_a{z}_a$ UAV as follows:

$$\left\{\begin{array}{c}{\mathit{N}}_n={\mathit{A}}_n+{\mathit{T}}_n\hfill \\ {\mathit{N}}_{\tau }={\mathit{A}}_{\tau }+{\mathit{T}}_{\tau }\hfill \end{array}\right.$$

(2)

${\mathit{N}}_n$ is referred to as the controllable normal force, which is used to change the direction of the flight velocity. ${\mathit{N}}_{\tau }$ is referred to as the controllable tangential force, which is used to change the magnitude of the flight velocity.

For a longitudinally symmetric UAV, control is primarily achieved through the manipulation of the elevator ${\delta }_e$, ailerons ${\delta }_a$, rudder ${\delta }_R$, and throttle ${\delta }_T$, as illustrated in Figure 2. Deceleration can be achieved by deploying speed brakes to increase drag $\mathit{D}$ or by using throttle and other thrust-reversing mechanisms to adjust the tangential force component ${\mathit{T}}_{\tau }$.

The elevator ${\delta }_e$, ailerons ${\delta }_a$, and rudder ${\delta }_R$ are used to alter the UAV’s flight attitude. The elevator ${\delta }_e$ changes the UAV’s flight path angle, which in turn alters the angle of attack $\alpha$, thereby affecting the magnitude and direction of the lift. The ailerons ${\delta }_a$ change the UAV’s roll attitude, which adjusts the roll angle $\varphi$, thereby altering the direction of the lift. The horizontal component of the lift force causes the UAV to change direction in the horizontal plane. The rudder ${\delta }_R$ changes the UAV’s yaw attitude, which causes the sideslip angle $\beta$ to vary, generating a side force $\mathit{C}$ that also leads to a change in the UAV’s motion in the horizontal plane. By adjusting the elevator ${\delta }_e$, ailerons ${\delta }_a$, rudder ${\delta }_R$, and throttle ${\delta }_T$, the UAV can be maneuvered in any direction in space.

Assume that the moving coordinate system ($Oxyz$) rotates with an angular velocity $\mathit{\omega }$ relative to the horizontal coordinate system ($O_l{x}_l{y}_l{z}_l$), and the absolute velocity of the center of mass is $\mathit{V}$. By projecting the velocity $\mathit{V}$ and angular velocity $\mathit{\omega }$ onto the moving coordinate system ($Oxyz$), the following equations can be derived:

$$\left\{\begin{array}{c}\mathit{V}=V_x\mathit{i}+V_y\mathit{j}+V_z\mathit{k}\hfill \\ \mathit{\omega }={\omega }_x\mathit{i}+{\omega }_y\mathit{j}+{\omega }_z\mathit{k}\hfill \end{array}\right.$$

(3)

In Equation (3), $\mathit{i}$, $\mathit{j}$, and $\mathit{k}$ represent the unit vectors in the moving coordinate system ($Oxyz$). Due to the presence of angular velocity $\mathit{\omega }$, the direction of the velocity $\mathit{V}$ continuously changes. The absolute acceleration of the center of mass is expressed as

$${\displaystyle \frac{\mathrm{d}\mathit{V}}{\mathrm{d}t}}={\displaystyle \frac{\mathrm{d}{V}_x}{\mathrm{d}t}}\mathit{i}+{\displaystyle \frac{\mathrm{d}{V}_y}{\mathrm{d}t}}\mathit{j}+{\displaystyle \frac{\mathrm{d}{V}_z}{\mathrm{d}t}}\mathit{k}+V_x{\displaystyle \frac{\mathrm{d}\mathit{i}}{\mathrm{d}t}}+V_y{\displaystyle \frac{\mathrm{d}\mathit{j}}{\mathrm{d}t}}+V_z{\displaystyle \frac{\mathrm{d}\mathit{k}}{\mathrm{d}t}}={\displaystyle \frac{\delta \mathit{V}}{\delta t}}+\mathit{\omega }\times \mathit{V}$$

(4)

In Equation (4), $\delta \mathit{V}/\delta t$ represents the acceleration when the angular velocity $\mathit{\omega }=0$, and $\mathit{\omega }\times \mathit{V}$ is the acceleration caused by the change in the direction of the velocity $\mathit{V}$ in the moving coordinate system due to the presence of angular velocity $\mathit{\omega }$.

The mechanical equation of motion for the UAV’s center of mass can be expressed as

$$\mathit{F}=m{\displaystyle \frac{\mathrm{d}\mathit{V}}{\mathrm{d}t}}=m\left({\displaystyle \frac{\delta \mathit{V}}{\delta t}}+\mathit{\omega }\times \mathit{V}\right)=F_x\mathit{i}+F_y\mathit{j}+F_z\mathit{k}$$

(5)

In Equation (5), m represents the mass of the UAV, and $\mathit{F}$ is the vector sum of all external forces acting on the UAV’s center of mass.

The scalar form of the UAV’s motion in the moving coordinate system is expressed as

$$\left\{\begin{array}{c}{F}_x=m\left({\displaystyle \frac{\mathrm{d}{V}_x}{\mathrm{d}t}}+V_z{\omega }_y-V_y{\omega }_z\right)\hfill \\ F_y=m\left({\displaystyle \frac{\mathrm{d}{V}_y}{\mathrm{d}t}}+V_x{\omega }_z-V_z{\omega }_x\right)\hfill \\ F_z=m\left({\displaystyle \frac{\mathrm{d}{V}_z}{\mathrm{d}t}}+V_y{\omega }_x-V_x{\omega }_y\right)\hfill \end{array}\right.$$

(6)

The UAV’s dynamic equation is applicable in any moving coordinate system. By substituting the flight path coordinate system into the moving coordinate system, and based on the definition of the flight path coordinate system ($O_k{x}_k{y}_k{z}_k$), we know that: ${\mathit{V}}_x=\mathit{V}$, ${\mathit{V}}_y={\mathit{V}}_z=0$. According to the relative position between the flight path coordinate system and the ground coordinate system, the flight path coordinate system first rotates with angular velocity ${\dot{\psi }}_a$ around the $O_a{z}_a$ axis, and then with angular velocity ${\dot{\theta }}_a$ around the $O_a{z}_a$ axis to form the ground coordinate system. Therefore, the angular velocity in the flight path coordinate system is represented as $\omega ={\dot{\psi }}_a+{\dot{\theta }}_a$, which can be expressed as

$$\left[\begin{array}{c}{\omega }_x\\ {\omega }_y\\ {\omega }_z\end{array}\right]={\mathit{L}}_{kl}\left[\begin{array}{c}0\\ 0\\ {\dot{\psi }}_a\end{array}\right]+\left[\begin{array}{c}0\\ {\dot{\theta }}_a\\ 0\end{array}\right]=\left[\begin{array}{c}-{\dot{\psi }}_{a}cos{\dot{\theta }}_a\\ {\dot{\theta }}_a\\ {\dot{\psi }}_{a}sin{\dot{\theta }}_a\end{array}\right]$$

(7)

By substituting the thrust $\mathit{T}$, aerodynamic force $\mathit{A}$, and gravity $\mathit{W}$ into Equation (6), and using ($\chi ,\gamma ,\mu$) to replace (${\psi }_a,{\theta }_a,{\varphi }_a$), the center of mass dynamics equation in the flight path coordinate system ($O_k{x}_k{y}_k{z}_k$) is obtained as Equation (7). Here, the lift $F_L$, drag D, and side force C represent the components of the aerodynamic force $\mathit{A}$ in the velocity coordinate system ($O_a{x}_a{y}_a{z}_a$).

$$\left\{\begin{array}{c}m{\displaystyle \frac{\mathrm{d}V}{\mathrm{d}t}}=Tcos(\alpha +\phi )cos\beta -D-mgsin\gamma \hfill \\ mVcos\gamma {\displaystyle \frac{\mathrm{d}\chi }{\mathrm{d}t}}=T[sin(\alpha +\phi )sin\mu -cos(\alpha +\phi )sin\beta cos\mu ]+Ccos\mu +F_{L}sin\mu \hfill \\ -mV{\displaystyle \frac{\mathrm{d}\gamma }{\mathrm{d}t}}=T[-sin(\alpha +\phi )cos\mu -cos(\alpha +\phi )sin\beta sin\mu ]+Csin\mu -F_{L}cos\mu +mgcos\gamma \hfill \end{array}\right.$$

(8)

The UAV’s dynamic equation describes the laws of its motion and variation, enabling effective control of the UAV. Based on relevant literature, the following assumptions are made: (1) The Earth’s rotation and revolution are neglected, treating it as a stationary plane; (2) The UAV is assumed to be a rigid body, with constant gravity; (3) The effects of wind speed are neglected, and at this point, the $O_k{x}_k$ axis, $O_a{x}_a$ axis, and $O_b{x}_b$ axis coincide; (4) The UAV is assumed to be in zero sideslip flight; (5) The thrust T is assumed to be aligned with the $O_a{x}_a$ axis. By adjusting the thrust T, the velocity deflection angle $\varphi$, and the angle of attack $\alpha$, the UAV can be controlled, simplifying the UAV’s dynamic equation as follows:

$$\left\{\begin{array}{c}m{\displaystyle \frac{\mathrm{d}V}{\mathrm{d}t}}=Tcos\alpha -D-mgsin\gamma \hfill \\ mVcos\gamma {\displaystyle \frac{\mathrm{d}\chi }{\mathrm{d}t}}=F_{L}sin\varphi \hfill \\ mV{\displaystyle \frac{\mathrm{d}\gamma }{\mathrm{d}t}}=F_{L}cos\varphi -mgcos\gamma \hfill \end{array}\right.$$

(9)

2.2. Missile Motion and Dynamics Model

In the inertial coordinate system, the missile’s kinematic equation is

$$\left\{\begin{array}{c}\dot{x}=vcos\theta cos\phi \hfill \\ \dot{y}=vcos\theta sin\phi \hfill \\ \dot{z}=vsin\theta \hfill \end{array}\right.$$

(10)

In the equation, $(x,y,z)$ represents the missile’s position in the inertial coordinate system, and $(v,\theta ,\phi )$ represents the missile’s velocity, flight path pitch angle, and flight path yaw angle, all as functions of flight time t. Based on the missile’s flight being divided into active and passive phases, in the inertial system, the forces acting on the missile during the active phase primarily include: thrust $T\left(t\right)$, gravity $G=m\left(t\right)g$, and aerodynamic drag $D\left(t\right)$ (in a non-inertial system, Coriolis forces C also need to be considered). Therefore, in the trajectory coordinate system, the missile’s particle dynamics equation is expressed as

$$\left\{\begin{array}{c}\dot{v}=g(n_x-sin\theta )\hfill \\ \dot{\phi }={\displaystyle \frac{g}{v}}{n}_{y}cos\theta \hfill \\ \dot{\theta }={\displaystyle \frac{g}{v}}(n_z-cos\theta )\hfill \end{array}\right.$$

(11)

where, $n_x\left(t\right)={\displaystyle \frac{T\left(t\right)-D\left(t\right)}{m\left(t\right)g}}$ is the overload in the velocity direction, and $n_y,n_z$ represents the missile’s lateral control overloads in the yaw and pitch directions, respectively, which are calculated using proportional guidance. $m\left(t\right)$ is the missile’s current mass, and g is the gravitational acceleration constant, taken as $g=9.81{\mathrm{m}/\mathrm{s}}^2$.

2.3. Decoy Motion Model

To effectively describe the various forces and motion states experienced by the decoy missile during flight, a three-degree-of-freedom particle model is used to model the decoy missile’s motion process. The specific model is as follows:

$$\left\{\begin{array}{c}{\ddot{x}}_d=-{\displaystyle \frac{D_d{\dot{x}}_d}{m_d{V}_d}}\hfill \\ {\ddot{y}}_d=-{\displaystyle \frac{D_d{\dot{y}}_d}{m_d{V}_d}}\hfill \\ {\ddot{z}}_d=-{\displaystyle \frac{D_d{\dot{z}}_d}{m_d{V}_d}}+g\hfill \end{array}\right.$$

(12)

In the equation, $m_d$ represents the mass of the decoy missile, $V_d$ represents the true velocity of the decoy missile relative to the incoming flow direction, and $D_a$ represents the magnitude of the decoy missile’s flight resistance in the direction opposite to the velocity axis $O_x$. Since the decoy missile cannot be controlled after being ejected, there are no control variables.

$$V_d=\sqrt{{\dot{x}}_d^2+{\dot{y}}_d^2+{\dot{z}}_d^2}$$

(13)

$$D_d={\displaystyle \frac{1}{2}}\rho V_d^2{C}_d{S}_d$$

(14)

where $\rho$ represents the air density, $C_d$ represents the aerodynamic drag coefficient, and $S_d$ represents the windward area of the decoy missile.

3. Method

3.1. Risk-Sensitive PPO

In missile evasion missions, the synergistic effect of maneuver evasion and decoy interference aims to deceive and ultimately break missile tracking. During this process, the intelligent agent must balance the timing of decoy deployment with the selection of appropriate maneuvers based on the current engagement situation.

When the missile is at a long distance from the UAV, the seeker’s ability to distinguish between decoys and the UAV is relatively weak. However, the UAV’s maneuvers have a limited impact on the line-of-sight angle, making it more difficult to escape the missile’s lock-on zone. Conversely, when the missile is at close range, the UAV’s maneuvers significantly affect the line-of-sight angle, but the decoy’s interference capability diminishes.

Additionally, the effectiveness of decoy interference exhibits a certain degree of randomness, and different missile pursuit strategies elicit varying responses to the UAV’s maneuvers. Therefore, a coordinated evasion strategy must possess sufficient diversity to adapt to various engagement scenarios. This necessitates a robust decision-making framework capable of dynamically adjusting evasion tactics based on real-time threat assessment.

We employ the Risk-Sensitive Proximal Policy Optimization (RPPO) algorithm to train our evasion strategy. This approach introduces a risk-level hyperparameter that guides the policy toward the Bellman-optimal strategy, enabling the algorithm to learn the underlying reward signal distribution of the task and ultimately produce strategies with different behavioral preferences based on varying risk levels. The incorporation of risk sensitivity modifies the Bellman operator for the value function as follows:

$$B_{\tau }V\left(s\right)=V\left(s\right)+2\alpha {\mathbb{E}}_s{\mathbb{E}}_{s^{\prime }}[\tau {\left[\delta \right]}_{+}+(1-\tau ){\left[\delta \right]}_{-}],$$

(15)

where $\delta$ is the TD error. When the discount factor is $\gamma$, $\delta (s,a,s^{\prime })=r(s,a)+\gamma V\left(s^{\prime }\right)-V\left(s\right)$. The operator ${[·]}_{+}=max(·,0)$ and ${[·]}_{-}=min(·,0)$. It can be proven that for any $\tau \in (0,1)$, $B_{\left(\tau \right)}$ is a contraction mapping under the infinity norm. This ensures that the value function with the risk level parameter can converge during the policy evaluation stage. On this basis, the advantage function can be defined as

$$A_{\tau }^{\pi }=2\alpha {\mathbb{E}}_{s^{\prime }}[\tau {\left[\delta \right]}_{+}+(1-\tau ){\left[\delta \right]}_{-}].$$

(16)

In order to effectively reduce the variance in the advantage estimation, it is necessary to introduce the generalized advantage estimation into the RPPO. To this end, the Bellman operator of the value function with the risk level needs to be rewritten in a multi-step form:

$$B_{\tau ,\lambda }V\left(s\right):=(1-\lambda )\sum _{n=1}^{\infty }{\lambda }^{n-1}{\left(B_{\tau }\right)}^{n}V\left(s\right).$$

(17)

It can be proven that the Bellman operator of the value function in the multi-step form also has the property of a contraction mapping. However, it is difficult to estimate it through sample data. Therefore, an operator in the sample form also needs to be introduced. Denote the estimated value of the value function V as $\widehat{V}$, and we have the following:

$${\widehat{B}}_{\tau }^{\widehat{V}}V\left(s\right):=V\left(s\right)+2\alpha [\tau {\left[\widehat{\delta }\right]}_{+}+(1-\tau ){\left[\widehat{\delta }\right]}_{-}],$$

(18)

Among them, $\widehat{\delta }$ is the estimated value of the TD error. When $\widehat{V}=V$ is chosen, we can obtain $B_{\tau }\left(s\right)={\mathbb{E}}_{a\sim \pi (·|s),s^{\prime }\sim p(·|s,a)}{\widehat{B}}_{\tau }^{V}V\left(s\right)$. Therefore, the Bellman operator of the multi-step value function can be expressed as

$${\widehat{V}}_{\tau ,\lambda }^{\pi }\left(s\right)=(1-\lambda )\sum _{n=1}^{\infty }{\lambda }^{n-1}{\widehat{B}}_{\tau }^{{\widehat{V}}_n}V\left(s\right),{\widehat{V}}_n\left(s\right)={\widehat{B}}_{\tau }^{{\widehat{V}}_{n-1}}V\left(s\right).$$

(19)

On this basis, the advantage estimation can be written as

$${\widehat{A}}_{\tau ,\lambda }^{\pi }(s,a)={\widehat{V}}_{\tau ,\lambda }^{\pi }\left(s\right)-V\left(s\right),$$

(20)

The multi-step value function Bellman operator in the sample form and the corresponding advantage function estimation constitute the complete RPPO algorithm. Compared with the original PPO algorithm, it only requires some modifications to the calculations of the advantage function and the value function, as shown in Algorithm 1.

Algorithm 1 Risk-sensitivePPO(RPPO)

Input: initial actor network parameter $\theta$ and critic network parameter $\varphi$, replay buffer D, update epochs K     for  $i=0,1,2,3,\dots$  do        Collect trajectories by running policy ${\pi }_{\theta }$ until the replay buffer is filled.        Compute $\lambda$-variant returns ${\widehat{V}}_{\tau ,\lambda }^{\pi }$ according to Equation (19)        Compute advantages ${\widehat{A}}_{\tau ,\lambda }^{\pi }$ according to Equation (20)        for $k=0,1,2,3,\dots ,K$ do             Update the $\theta$ by maximizing the surrogate function with ${\mathcal{L}}_{\pi }^{\mathrm{CLIP}}\left(\theta \right)={\displaystyle \frac{1}{\left|D\right|}}{\sum }_{t=0}^{\left|D\right|-1}\left[min\left({\omega }_t\left(\theta \right){\widehat{A}}_{\tau }(s_t,a_t),clip({\omega }_t\left(\theta \right),1-\epsilon ,1+\epsilon ){\widehat{A}}_{\tau }(s_t,a_t)\right)\right]$ via Adam algorithms.             Update the $\varphi$ by minimizing the mean-squared error with ${\mathcal{L}}_V\left(\varphi \right)={\displaystyle \frac{1}{2\left|D\right|}}{\sum }_{t=0}^{\left|D\right|-1}{(V_{\varphi }\left(s_u\right)-{\widehat{V}}_{\tau ,t})}^2$ via Adam algorithms.        end for     end for

3.2. Cyclic Group Game

The research of [24,25] shows that the agent trained under the framework of priority virtual self-game can achieve a strategy close to Nash equilibrium in a zero-sum game. However, the missile evasion problem is an asymmetric game, and the state transition and control amount of both missile and UAV are not the same, so it is impossible to directly apply the self-game framework, and cycle training can be conducted by reusing a single agent and its historical version to improve the strategy performance. In order to train various evasive strategies to cope with different types of incoming missiles, this paper proposes a two-population countermeasure game training framework based on RPPO algorithm.

In order to avoid the pursuit strategy overfitting an escape strategy or the escape strategy overfitting a pursuit strategy, the two-population based game consists of the UAV evading agent population and the missile pursuing agent population. Before training, each population randomly initializes several agents, each using the same network structure, but with different initialization parameters and risk coefficients. In order to avoid meaningless exploration in the early stage of training, some pursuit agents controlled by proportional guidance rate and parallel guidance law are added to the missile pursuit agent population.

In each round of training, in order to evaluate the performance of each agent in the two populations, the pursuit agent and the escape agent are selected from the two populations respectively through multiple random sampling to conduct missile avoidance confrontation. Each confrontation is conducted in several rounds. The confrontation process and outcome are recorded starting from the random initial situation in each round. The data generated in the confrontation are stored in the common experience playback pool. In order to enable RPPO algorithm to use the data generated by different agents, the recorded data include action probability density $\pi \left(a\right|s)$ in addition to $s,a,r,s^{\prime }$. The Diagram of the group game is shown as the Figure 3.

In order to avoid the phenomenon of cyclic restraint—that is, escape strategy A is superior to pursuit strategy C, pursuit strategy C is superior to escape strategy B, escape strategy B is superior to pursuit strategy D, and pursuit strategy D is superior to escape strategy A—we designed a reward and punishment correction scheme based on ELO ranking. After the completion of each cycle confrontation, each agent in the two populations is ranked by the ELO mechanism, and the reward signal in the previous game is corrected according to the ranking of each agent. When an agent performs worse against an enemy agent with a lower rank, the penalty term in the reward function is increased according to the difference in rank. If an agent wins against an enemy agent with a higher rank than himself, the reward item of all reward values in the game is amplified based on the rank difference.

$$r_{i,j}^t=\left\{\begin{array}{c}{r}_{i,j}^t\ast (1+tanh(Ran{k}_i-Ran{k}_j)),r_{i,j}^t>0\hfill \\ r_{i,j}^t\ast (1-tanh(Ran{k}_i-Ran{k}_j)),r_{i,j}^t<0\hfill \end{array}\right.$$

(21)

As shown in Equation (21), when a certain agent loses to an agent with a lower ranking than itself in the missile evasion confrontation, all the reward items in the reward values of both parties in the match will be reduced, and the penalty items will be amplified, and vice versa.

After the correction is complete, all matchup data are stored in the common experience playback pool of each population, and both trainable agents use the data in the experience playback pool to update their own strategies. In order to make full use of the better-performing agents, before each update, the parameters of the better-ranked agents in the two populations will be replaced with those of the lower-ranked agents. At the same time, in order to ensure the diversity of strategies and better exploration, the risk coefficient of the original agent will be added with noise between −0.1 and 0.1 when the parameters are overwritten.

3.3. Markov Decision Model

3.3.1. State Space

When evading incoming missiles, the drone needs to make decisions based on information such as its own position, speed, and attitude, as well as the relative situation with the missile. To ensure Markov property, it is also necessary to obtain information such as the missile’s speed, attitude, and control guidance law. However, since the missile is a non-cooperative target, this information cannot be directly acquired. In order to ensure the effectiveness of the evasion strategy, we combine several past known information sequences into a state signal, and extract hidden features from it by introducing the multi-head attention mechanism and the GRU module in the policy network and the value network. The state space is represented as

$${\mathit{S}}_{UAV}=[x_u,y_u,z_u,{\chi }_u,{\gamma }_u,{\varphi }_u,V_u,r_d,{\beta }_d,{\epsilon }_d]{|}_{i=t-n}^t,$$

(22)

The state representation includes the UAV’s own status (denoted by subscript u) such as its coordinates, flight attitude, and velocity, as well as the relative motion states (denoted by subscript d) including the relative distance and line-of-sight angle between the UAV and the missile.

The missile only needs to rely on the line-of-sight distance, the line-of-sight deflection angle, the line-of-sight inclination angle and their first-order differentials to effectively pursue the target. Therefore, the state space of the missile agent is

$${\mathit{S}}_{Missile}=[r_d,{\beta }_d,{\epsilon }_d,\dot{r_d},\dot{{\beta }_d},\dot{{\epsilon }_d}]$$

(23)

3.3.2. Action Space

According to the discussion in Section 2, the state of the UAV is controlled by the axial overload, the longitudinal overload and the roll angular velocity. In addition, whether to launch the decoy is determined by the launch probability f, which is a value between 0 and 1. After the policy network gives the value of f, whether to launch the decoy is decided by the roulette wheel selection method. Therefore, the action space of the drone is

$${\mathit{A}}_{UAV}=[T,F_L,\dot{\varphi },f]$$

(24)

Similarly, the missile is controlled by the lateral overload and the longitudinal overload, and its action space is

$${\mathit{A}}_{Missile}=[n_y,n_z]$$

(25)

3.3.3. Reward Function

The reward function is a crucial component of reinforcement learning, while the termination function is an essential part of the simulation environment. The reward function provides an immediate feedback value (reward or penalty) to the agent when taking an action $\mathit{a}$ in a given state $\mathit{s}$, guiding the agent to make the correct decisions. The termination function determines when the agent should end the current episode, marking the conclusion of a round.

3.3.4. Reward Function

In the actual process of evasion, we aim for the UAV to achieve the highest possible survival rate when defending against incoming missiles while minimizing the use of decoys. Therefore, in this study, penalties are assigned for UAV crashes, being hit by missiles, and deploying decoys, whereas successful missile defense is rewarded. The specific design of the reward function is as follows:

$$R_{UAV}(s,a)=\left\{\begin{array}{cc}-100\hfill & r_d\le r_{\exp },r_d\in s\hfill \\ -100\hfill & z_u\le 0,z_u\in s\hfill \\ -1\hfill & u\sim \mathcal{U}(0,1)\le f,f\in a\hfill \\ \sum _{n}exp(\dot{r_d}/1200)\hfill & \dot{r_d}\in s\hfill \\ +100\hfill & r_d\mathrm{Monotonically}\mathrm{increasing}\mathrm{for}5\mathrm{s}\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(26)

The pursuit-evasion game between the missile and the drone can be regarded as a zero-sum game. Therefore, the reward function of the missile is

$$R_{Missile}=-R_{UAV}$$

(27)

In the equation, $v_{\mathrm{approach}}$ represents the missile’s closing velocity. When two missiles appear simultaneously, the closing velocity for each missile must be calculated separately and then summed.

3.4. Policy and Value Network Structure

In the missile evasion mission, when a UAV makes an evasion decision, it not only needs to consider the relative situation between the UAV itself and the missile, but also needs to take into account information such as the guidance mode of the missile and the influence of decoy flares on the missile. The relative situation between the missile and the UAV can be directly obtained through observation, while characteristics such as the guidance mode of the missile and the interaction between it and the decoy flares are hidden in the missile’s response to the UAV’s maneuvering actions over a certain period of time. Even though this paper introduces the RPPO algorithm to deal with the potential randomness in the missile evasion problem, in the actual application of the algorithm, it is still necessary to process the state sequence over a certain period of time to ensure that the intelligent agent can learn a sufficiently good evasion strategy.

As shown in Figure 4 and Figure 5, when designing the Actor network and Critic network of the RPPO algorithm, we designed serialized state input signals and introduced the GRU module and the multi-head attention mechanism. The state sequence is encoded through the GRU module, and the multi-head attention mechanism is used to selectively focus on the historical information from the state sequence to better extract the hidden features [26]. This enables the intelligent agent to effectively perceive the changes in the motion laws of the missile before and after being interfered by the decoy flares and recognize the guidance laws of the missile from the motion situation sequence of the missile. As a result, the evasion strategy has a better adaptability to complex situations and has good robustness against diverse incoming missiles.

The RPPO algorithm is a stochastic policy algorithm. The Actor network first encodes the state sequence through the MLP layer, and then uses a network based on the multi-head attention mechanism to extract the temporal features in the state sequence. In addition, a feature network composed of a fully connected layer and a GRU layer is used to extract the spatial features contained in the states at different moments. After these features are concatenated, the fully connected network is used to calculate the distribution parameters of each action, and the final action instruction is obtained after sampling through these parameters.

Different from the Actor network that outputs the probability distribution of actions, the Critic network outputs an estimate of the value function, and this estimated value represents the advantages and disadvantages of the current state. As shown in Algorithm x, the RPPO algorithm guides the optimization of the policy network through the value function. The addition of the GRU layer and the multi-head attention mechanism enables it to better evaluate the situation in the missile evasion problem, allowing the intelligent agent to learn and adjust the strategy more effectively. At the same time, it endows the network with stronger representation ability, which can effectively enhance the adaptability of the intelligent agent to complex environments and diverse threats.

4. Experiments

4.1. Experiment Configuration

We trained the evasion strategy in the simulation environment constructed according to the mathematical model in Section 3.2 using the dual-population adversarial game and the RPPO algorithm based on the multi-head attention mechanism. The hyperparameters used in the algorithm are shown in the following

Table 1. Hyperparameters in training.

Parameter	Value
Actor network learning rate	$1\times {10}^{-4}$
Critic network learning rate	$3\times {10}^{-5}$
Replay buffer size	$5\times {10}^7$
Population size	30
Maximum number of steps	30
Parameter update steps	100

.

In the simulation scenario, the initial position of the UAV is set at 30 degrees north latitude, 120 degrees east longitude, and at an altitude of 8000 m. The initial heading is due east, and the initial speed is 350 m/s. The initial speed of the missile is 800 m/s, and it is generated at a distance of 10–12 km from the UAV. The initial line-of-sight deflection angle between the missile and the UAV is randomly generated within the range of $[-\pi ,\pi ]$, and the initial line-of-sight inclination angle is randomly generated within the range of $[-{\displaystyle \frac{\pi }{4}},{\displaystyle \frac{\pi }{4}}]$, both of which satisfy a uniform distribution. The detailed initial situation is shown in

Table 2. Initial state setting.

Parameter	Value
Missile’s initial velocity	800 m/s
UAV’s initial velocity	350 m/s
Initial line-of-sight deflection angle	$[-\pi ,\pi ]$
Initial line-of-sight inclination angle	$[-{\displaystyle \frac{\pi }{4}},{\displaystyle \frac{\pi }{4}}]$
Max number of decoy	30

and

Table 3. Initial software items.

Software Item	Version	Hardware Item	Model
Operating system	Window 11 10.0.22631	CPU	Intel(R) Core(TM) i9-10900K @ 3.70 GHz
Python	3.12.3 v.1938 64 bit	GPU	NVIDIA GeForce RTX 4090 *8
PyTorch	2.4.1	Internal memory	Kingston FURY DDR4 3600 16G × 2
CUDA Toolkit	11.8	Hard disk	ZhiTai PCIe 4.0 1TB SSD
Tacview	1.9.4

, as well as Figure 6.

The UAV carries 30 decoy flares. The lethal radius of the missile is 50 m. At the initial moment, the velocity vector of the missile is parallel to the line of sight and points towards the UAV. When the flight altitude of the UAV is less than 0 m or the distance between the UAV and the missile is less than the lethal radius of the missile, it is considered that the UAV has been destroyed and the simulation ends; when the distance between the UAV and the missile increases monotonically for 5 consecutive seconds, it is considered that the UAV has successfully evaded the missile and the simulation ends.

We conducted the algorithm training on 8 NVIDIA GeForce RTX 4090 graphics cards. We carried out simulated adversarial battles through 3000 parallel-running simulation environments to improve the efficiency of data collection. The detailed usage of software and hardware is shown in the following table.

The superiority of the method in this paper is demonstrated by comparing the training efficiency of the method in this paper with that of other methods, as well as indicators such as the survival rate when dealing with different types of missiles. The detailed information of the comparison schemes is as follows.

Scheme 1: Only use the basic PPO algorithm. The network structure is the same as that of the method in this paper. Training is carried out in a simulation environment with only a single type of missile. The missile is controlled by the proportional navigation guidance law, and the proportional navigation constant is 3.

Scheme 2: Use the MADDPG algorithm to train the UAV and the missile through a method similar to self-play. The network structure used in the PPO algorithm is the same as that of the method in this paper.

4.2. Training Results

As can be seen from the Figure 7, the survival rate of Scheme 1 increased rapidly in the initial stage of training. After about 200,000 steps of interaction, the survival rate had risen to 60%. In the subsequent 500,000 steps, the survival rate increased slowly and finally stabilized at about 68%. The survival rate of Scheme 2 slowly increased to 42% during the first 200,000 steps of training. Subsequently, the rate of increase in the survival rate accelerated, and it rapidly increased to 70% during the next 150,000 steps of training, exceeding that of Scheme 1. In the subsequent training, the rate of increase in the survival rate significantly slowed down and finally reached 81%. The survival rate The of method in this paper increased to about 45% during the first 150,000 steps of training, and the rate of increase was slower than that of Scheme 1. However, it rapidly increased to 76% during the subsequent 50,000 steps of training. After that, the rate of increase slowed down, and finally, the survival rate reached 94%.

The training results show that although the method in this paper has limited effect on improving the survival rate in the initial stage of training, after collecting enough data and conducting sufficient preliminary training, the method in this paper shows significant advantages in terms of the improvement rate. Moreover, due to the diversity of the missile agents in the dual-population adversarial game, the evasion strategy trained by the evasion method in this paper can adapt to different types of missiles, and the survival rate against different missiles in the test reaches a high level. In contrast, the evasion strategy trained based on the MAPPO algorithm in the comparison Scheme 2 has a relatively single opponent. In the later stage of training, the evasion strategy has a certain degree of overfitting to the opponent, which affects the diversity of the evasion measures, resulting in a lack of ability to deal with different types of missiles. The evasion strategy trained by the PPO algorithm and a specific type of missile in Scheme 1 can only ensure effective evasion of this type of missile. When the type of missile changes, the optimality of the evasion strategy cannot be guaranteed, so it is unable to effectively deal with different types of missiles.

4.3. Survival Rate Statistics

Furthermore, we statistically analyzed the performance of the evasion strategies trained by the three schemes when dealing with several different types of missiles. For each type of missile, the three schemes each carried out 5000 evasion experiments. In each experiment, the initial situation was randomly generated in the same way as during the training, and the results of the survival rate regarding the azimuth angle of the incoming missile are shown in Figure 8, Figure 9 and Figure 10.

Figure 8 shows the survival rates of the three evasion strategies when dealing with missiles controlled by the proportional navigation guidance law with different parameters. For each group of experiments, the survival rates of various strategies are relatively low when the azimuth angle of the incoming missile is near 0 degrees and 360 degrees. This is in line with the analysis in Ref. Therefore, keeping the flight direction perpendicular to the line of sight can force the missile to make a large-scale maneuver, which can consume the missile’s energy to the greatest extent. At this time, with the release of appropriate decoy flares, the pursuit of the missile can be effectively shaken off. At this time, with the release of appropriate decoy flares, the pursuit of the missile can be effectively shaken off. This is because the missile used in the experiment in Figure 8a is the same as the missile used during the training of Scheme 1, and the evasion strategy trained by Scheme 1 is completely focused on evading this missile. This is because the missile used in the experiment in Figure 8a is the same as the missile used during the training of Scheme 1, and the evasion strategy trained by Scheme 1 is completely focused on evading this missile. By comparing the performances in Figure 8b,c, it can be seen that the performance of the method in this paper is relatively stable under different missiles, while the performance of the evasion strategy of Scheme 1 drops significantly when facing missiles different from those in the training environment. By comparing the performances in Figure 8b,c, it can be seen that the performance of the method in this paper is relatively stable under different missiles, while the performance of the evasion strategy of Scheme 1 drops significantly when facing missiles different from those in the training environment.

Figure 9 and Figure 10 show the performances of the three strategies when dealing with missiles using the parallel approaching method and the pure tracking method with different parameters. In the two groups of experiments, the trends of the survival rates shown by the method in this paper and Scheme 2 are consistent with those in Figure 9, indicating that both can effectively deal with various different types of missiles. However, due to the relatively single opponent during the training process of the evasion strategy of Scheme 2, a certain degree of overfitting has occurred, which affects its approximation of the optimal evasion strategy. The survival rates of the evasion strategy trained by the method in this paper in each group of experiments are significantly higher than those of Scheme 2, showing better adaptability and diversity.

4.4. Simulation Sample

Figure 11 and Figure 12 show the specific performance of the evasion strategy trained by the method in this paper in the missile evasion experiment. As shown in Figure 11, the missile approaches from the rear of the UAV. The UAV makes a diving maneuver to convert gravitational potential energy into kinetic energy, attempting to increase its own speed and uses decoy flares to deceive the missile when the missile is at a relatively long distance. However, the effect is limited. The UAV continues to perform maneuvering actions and starts to intensively release decoy flares at the 8th second. At this time, the missile is relatively close to the UAV, and the decoy flares successfully cause the missile to deviate from the target, and the UAV successfully evades the incoming missile.

In Figure 12, the missile approaches from the rear side. The UAV deflects towards the direction of the incoming missile and dives to accelerate at the first time. After the missile gets closer, the UAV starts to release decoy flares and continues to turn left and dive, maneuvering in the direction perpendicular to the line connecting the missile and the decoy flares to maximize the influence of the decoy flares on the missile. Finally, after being deceived by the decoy flares, the missile loses the target, and the UAV successfully completes the evasion.

5. Conclusions

This study develops an adversarial gaming framework consisting of an escaping strategy population for controlled UAV and an intercepting strategy population for missiles, employing a risk-sensitive PPO algorithm to train coordinated UAV maneuver-decoy evasion strategies. To address the issue of cyclic counteractions in population-based adversarial training, we implement a RANK-based reward correction mechanism. The incorporation of multi-head attention mechanisms significantly enhances the policy network’s capability in state sequence modeling and strategy representation. Our method employs a policy network for maneuver-based evasion decision-making, which demonstrates low computational overhead during the inference phase. The decision-relevant information can be effectively acquired through onboard active radar and other sensors available on the UAV, indicating strong practical applicability.

Comparative experiments demonstrate that our method achieves faster convergence to effective evasion strategies than both self-play approaches and the original PPO algorithm. In comprehensive testing scenarios comprising various missile types, the proposed method attains a 94% successful evasion rate. Simulation results reveal that our trained strategies exhibit advanced tactical proficiency, effectively combining decoy deployment with evasive maneuvers to counter incoming missiles.

Furthermore, evasion tests against missiles with different guidance laws and parameters confirm that our approach produces strategies with superior generalization capability, consistently maintaining high performance across diverse missile types and configurations. The experimental validation suggests that our method significantly outperforms conventional approaches in both training efficiency and operational effectiveness.