UAV Spiral Maneuvering Trajectory Intelligent Generation Method Based on Virtual Trajectory

Abstract

This paper addresses the challenge of ineffective coordination between terminal maneuvering and precision strike capabilities in hypersonic unmanned aerial vehicles (UAVs). To resolve this issue, an intelligent spiral maneuver trajectory generation method utilizing a virtual trajectory framework is proposed. Initially, a relative motion model between the UAV and the virtual center of mass (VCM) is established based on the geometric principles of the Archimedean spiral. Subsequently, the interaction dynamics between the VCM and the target are formulated as a Markov decision process (MDP). A deep reinforcement learning (DRL) approach, employing the proximal policy optimization (PPO) algorithm, is implemented to train a policy network capable of end-to-end virtual trajectory generation. Ultimately, the relative spiral motion is superimposed onto the generated virtual trajectory to synthesize a composite spiral maneuvering trajectory. The simulation results demonstrate that the proposed method achieves expansive spiral maneuvering ranges while ensuring precise target strikes.

1. Introduction

As various countries around the world actively develop technologies and equipment against hypersonic unmanned aerial vehicles (UAVs), the future interception airspace will be fully covered, and the penetration channels will be closed. The penetration situation of hypersonic UAVs will become increasingly severe. Developing effective technologies and methods to counter the penetration of defense weapons has extremely important strategic significance [1,2,3].

During the terminal phase of precision ground strikes, hypersonic UAVs experience rapid decreases in altitude and velocity, rendering them more susceptible to enemy detection and interception compared to the mid-course phase [4]. The implementation of pre-programmed maneuver penetration tactics represents a prevalent countermeasure during this terminal phase [5]. This method can break the law of trajectory prediction, reduce the effectiveness of the anti-identification technology of enemy trajectory prediction, and mislead the movement of the interceptor through wrong prediction information, increase the maneuvering overload of the interceptor, and even lead to the overload and saturation of the interceptor and the loss of target tracking ability of the seeker [6,7], which interrupts the engagement link in the kill chain and improves the penetration probability.

The common maneuvering forms of programmed maneuvering penetration include snake maneuvering, pendulum maneuvering and spiral maneuvering [8]. As a complex three-dimensional maneuvering mode, spiral maneuvering has the advantages of large maneuvering amplitude, time-varying maneuvering frequency and unpredictable ballistic trajectory, which can greatly improve the penetration probability of UAVs [9,10,11]. Contemporary spiral maneuvering methodologies primarily comprise two trajectory variants: the non-convergent cylindrical spiral [12,13,14] and progressively convergent conical spiral [15,16,17]. The studies in Refs. [12,13] formulate a guidance law through variable structure control theory to track the line-of-sight (LOS) angle rate and execute spiral maneuvers, utilizing sinusoidal functions to govern the LOS angle rate dynamics. However, this approach exhibits limitations due to the distance-dependent variability in the LOS angle rate amplitude and periodicity, which introduces nonstationary trajectory characteristics. Consequently, the spiral radius and frequency cannot be prescriptively optimized, as their design remains contingent on dynamic operational constraints. Ref. [14] addresses low-speed UAV applications by proposing a guidance law that generates fixed-amplitude spiral trajectories around cylindrical virtual paths through periodic cosine-based deviations. While effective for low-velocity regimes, this method necessitates excessive control authority (i.e., high overload demands) in high-speed scenarios to maintain post-maneuver guidance accuracy, significantly degrading strike precision under kinematic constraints. Further methodologies include the integration of spiral angular velocity vectors with proportional navigation guidance in Ref. [15], where spiral commands diminish proportionally to the speed-leading angle. This results in inflexible maneuver amplitude and frequency modulation. Similarly, Ref. [16] incorporates a height-dependent sinusoidal acceleration bias into a sliding-mode guidance framework to generate spiral trajectories, though precise amplitude regulation remains unattainable. In Ref. [17], an adaptive proportional guidance law tracks logarithmic spiral trajectories for terminal dive maneuvers. While enabling spiral strikes from elevated positions, this strategy induces substantial velocity loss, compromising both penetration efficacy and kinetic energy retention during terminal engagement.

Although spiral maneuvering penetration is achieved in the mentioned references, there still exist some limitations. In contemporary UAV development, a critical focus lies in advancing intelligent perception and maneuvering capabilities to enable autonomous evasion, dynamic trajectory adjustment and precision targeting. Recent advancements in active evasive maneuvering systems for UAVs have demonstrated incremental progress toward technical viability. However, such evasion strategies—particularly during penetration scenarios—risk significant deviations from predefined trajectories [12]. To address this challenge, UAVs must integrate adaptive control mechanisms capable of executing timely directional corrections. These adjustments are essential to reconciling evasion success with mission-critical accuracy, ensuring the system avoids compromise while maintaining the capacity to deliver precise strikes post-maneuver.

Deep reinforcement learning (DRL) has emerged as a prominent research focus within the artificial intelligence domain. Owing to its advanced decision-making capabilities, DRL has facilitated significant breakthroughs in the development of control strategies for hypersonic UAVs, particularly in enhancing autonomous maneuver evasion and precision guidance systems. These advancements underscore DRL’s potential to address complex challenges in high-speed dynamic environments, where adaptive and real-time trajectory optimization is critical to mission success. In Ref. [18], an online multi-constraint re-entry guidance was achieved based on the improved hindsight experience replay method and the deep deterministic policy gradient algorithm. To address comprehensive flying object dynamics and the control mechanism, the Markov decision process (MDP) is used to solve the guidance and control problem, thereby achieving intelligent guidance laws for time control [19,20] and attack angle control [21,22] within the constraint of the field of view. In Ref. [23], to improve the survivability of the UAV in the face of multiple interceptors, the penetration process is conceptualized as a generalized three-body adversarial optimal problem, then modeled as a MDP, with a DRL scheme to solve this. In response to the issue of the UAV escape guidance, a unified intelligent control strategy synthesizing optimal guidance and meta DRL has been proposed in Ref. [4] and maneuvering strategies that combine LOS angle rate correction with DRL have been proposed in Ref. [24], which can achieve highly precise guidance and effective escape.

Building on the aforementioned insights, this study investigates a real-time spiral maneuvering trajectory generation framework integrated with DRL to reconcile the inherent tension between evasion efficacy and strike precision. The primary contributions of this work are as follows:

1.

A virtual trajectory-based spiral maneuvering trajectory design method is proposed to realize efficient coordination between maneuver penetration and precision strike;

2.

The Archimedes spiral is used in the design of the relative spiral, and the maneuvering amplitude and maneuvering frequency can be adjusted;

3.

Combined with DRL to generate virtual trajectories, the hypersonic UAV can sense the target in real time during flight, effectively adjust the maneuvering trajectory, and achieve accurate strikes on moving targets.

The organizational structure of this paper is delineated as follows: Section 2 presents the foundational mathematical frameworks, encompassing the hypersonic UAV dynamics model, spiral maneuvering trajectory formulation. Section 3 introduces a relative spiral motion paradigm grounded in Archimedes spiral geometry. A virtual reference trajectory generation methodology is rigorously derived using DRL principles, and the synthesized spiral maneuvering trajectory is realized through the kinematic superposition of relative motion dynamics onto the virtual trajectory. In order to verify the effectiveness, feasibility and real-time performance of the proposed method, simulation verification is carried out in Section 4. In Section 5, the paper’s conclusions are summarized.

2. Modeling of Spiral Maneuver Trajectory

2.1. Modeling of the UAV Motion

The three-degree-of-freedom motion equation is adopted to describe the UAV and the dynamic equation is established in the ballistic coordinating system [17]:

$$\left\{\begin{array}{c}\dot{x}=Vcos\gamma cos\psi \hfill \\ \dot{y}=Vsin\gamma \hfill \\ \dot{z}=Vcos\gamma sin\psi \hfill \\ \dot{V}=-{\displaystyle \frac{D}{m}}-gsin\gamma \hfill \\ \dot{\gamma }={\displaystyle \frac{Lcos\sigma }{mV}}-{\displaystyle \frac{g}{V}}cos\gamma \hfill \\ \dot{\psi }={\displaystyle \frac{Lsin\sigma }{mVcos\gamma }}\hfill \end{array}\right.$$

(1)

where $(x,y,z)$ is the position of the UAV in the inertial system, V is the speed, $\gamma$ is the velocity slope angle, $\psi$ is the velocity azimuth, $\sigma$ is the bank angle, m is the mass, $g=9.81$ m/s² is the gravitational acceleration, and $(D,L)$ are, respectively, drag force and lift force, whose expression is as follows:

$$\left\{\begin{array}{c}L={\displaystyle \frac{1}{2}}\rho V^2{C}_L{S}_{\mathrm{ref}}\hfill \\ D={\displaystyle \frac{1}{2}}\rho V^2{C}_D{S}_{\mathrm{ref}}\hfill \end{array}\right.$$

(2)

where $C_L$ and $C_D$ are the drag and lift coefficient, $S_{\mathrm{ref}}$ is reference area of the UAV and $\rho$ is the density of the atmosphere, using the following exponential atmospheric model:

$$\rho ={\rho }_0{\mathrm{e}}^{-H/H_s}$$

(3)

where ${\rho }_0$ is the sea level atmospheric density, $H_s=6700$ m is the equivalent density height of the earth atmosphere, $H=r-R_0$ is the height of the UAV and $R_0$ is the radius of the Earth.

2.2. Kinematic Modeling with Virtual Center of Mass

The spiral maneuvering trajectory is generated through the relative spiral motion of the UAV around a virtual reference trajectory that ensures terminal strike precision. Consequently, establishing a rigorous analytical framework to characterize the kinematic transformation and relative motion dynamics between the UAV and the virtual center of mass (VCM) is imperative to optimize evasion-strike coordination.

2.2.1. Definition and Transformation of Coordinate System

Definition 1: The VCM is a particle with only velocity and position.

Definition 2: The virtual trajectory is formed by the position of the VCM at every moment of flight to the target according to the guidance law.

The schematic diagram of the spiral maneuver trajectory is shown in Figure 1, $OXYZ$ representing the inertial system, $M_{v}T$ the virtual trajectory, $M_v$ VCM, $M_{r}T$ the spiral maneuver penetration trajectory and $M_r$ the UAV center of mass.

To enable precise state estimation of the UAV, a virtual center-of-mass-referenced coordinate system is formulated based on the methodological framework of classical flight mechanics reference frames.

1.

A virtual ballistic coordinate system, denoted as $M_v{X}_{v1}{Y}_{v1}{Z}_{v1}$, is rigorously defined with its origin $M_v$ positioned at the VCM. The $M_v{X}_{v1}$-axis is aligned with the velocity vector of the VCM, the $M_v{Y}_{v1}$-axis is orthogonally oriented upward within the vertical plane relative to $M_v{X}_{v1}$ and the $M_v{Z}_{v1}$-axis is determined via the right-hand rule to complete the orthonormal triad.

2.

A virtual LOS coordinate system, denoted as $M_v{X}_{s1}{Y}_{s1}{Z}_{s1}$, is rigorously defined with its origin $M_v$ positioned at the VCM. The $M_v{X}_{s1}$-axis is aligned with the target vector, the $M_v{Y}_{s1}$-axis is orthogonally oriented upward within the vertical plane relative to $M_v{X}_{s1}$ and the $M_v{Z}_{s1}$-axis is determined via the right-hand rule to complete the orthonormal triad.

Analogously, the ballistic coordinate system $M_r{X}_{r1}{Y}_{r1}{Z}_{r1}$ and LOS coordinate system $M_r{X}_{s2}{Y}_{s2}{Z}_{s2}$ for the UAV are constructed through the aforementioned methodological framework. The Euler angle transformations between these coordinate systems are derived through a “2–3" rotation sequence, as delineated in

Table 1. Coordinate system transformation angles.

Serial Number	Euler Angle	Definition Description
1	${\gamma }_r,{\psi }_r$	The angular relationship between the $M_r{X}_{r1}{Y}_{r1}{Z}_{r1}$ and $OXYZ$ coordinate systems is defined by the velocity slope angle and velocity azimuth of the UAV.
2	${\gamma }_v,{\psi }_v$	The angular relationship between the $M_v{X}_{v1}{Y}_{v1}{Z}_{v1}$ and $OXYZ$ coordinate systems is defined by the virtual velocity slope angle and virtual velocity azimuth of the VCM.
3	$q_{r1},q_{r2}$	The angular relationship between the $M_r{X}_{s2}{Y}_{s2}{Z}_{s2}$ and $OXYZ$ coordinate systems is defined by the LOS altitude angle and azimuth angle of the UAV.
4	$q_{v1},q_{v2}$	The angular relationship between the $M_v{X}_{s1}{Y}_{s1}{Z}_{s1}$ and $OXYZ$ coordinate systems is defined by the LOS altitude angle and azimuth angle of the VCM.

.

According to the Euler angle relationship between the coordinate systems in Table 1, the corresponding transformation matrix can be obtained.

2.2.2. Relative Motion Model

In the heading-normal plane($M_v{Y}_{v1}{Z}_{v1}$) of the VCM, the motion relation of the UAV relative to the VCM is shown in Figure 2. The red line $r_p$ is the distance of the UAV relative to the VCM; the Orange arrow $V_p$ is the velocity of the UAV relative to the VCM; the blue arrow ${\lambda }_p$ is the LOS angle of the UAV relative to the VCM, and represents the angle between the UAV and the VCM and the $M_v{Z}_{v1}$ axis; the green arrow ${\delta }_p$ is the velocity leading angle of the UAV in the virtual ballistic system $M_v{Y}_{v1}{Z}_{v1}$ plane, and represents the angle between the velocity vector of the UAV in the course normal plane of the VCM and the connection between the UAV and the VCM; the purple arrow ${\gamma }_p$ is the velocity azimuth of the UAV in the $M_v{Y}_{v1}{Z}_{v1}$ plane of the virtual ballistic coordinate system, and represents the angle between the velocity vector and the $M_v{Z}_{v1}$ axis of the UAV relative to the VCM.

The kinematic relationship between them is as follows:

$$\left\{\begin{array}{c}{z}_v=r_{p}cos\left({\lambda }_p\right)\hfill \\ y_v=r_{p}sin\left({\lambda }_p\right)\hfill \\ {\dot{z}}_v=V_{p}cos\left({\gamma }_p\right)\hfill \\ {\dot{y}}_v=V_{p}sin\left({\gamma }_p\right)\hfill \\ {\dot{\gamma }}_p={\displaystyle \frac{a_p}{V_p}}\hfill \end{array}\right.$$

(4)

where $\left(y_v,z_v\right)$ is the position vector of the UAV relative to the VCM in the $M_v{Y}_{v1}{Z}_{v1}$ plane of the virtual ballistic system.

The distance $r_p$ and LOS angle ${\lambda }_p$ of the UAV relative to the VCM meet the following relations:

$$\left\{\begin{array}{c}{\dot{r}}_p=-V_{p}cos\left({\gamma }_p-{\lambda }_p\right)=-V_{p}cos\left({\delta }_p\right)\hfill \\ r_p{\dot{\lambda }}_p=-V_{p}sin\left({\gamma }_p-{\lambda }_p\right)=-V_{p}sin\left({\delta }_p\right)\hfill \end{array}\right.$$

(5)

3. Spiral Maneuvering Trajectory Design

3.1. Plane Relative Spiral Motion Design Based on Archimedes Spiral

The plane relative spiral motion is designed with an Archimedes spiral, as shown in Figure 3.

The Archimedean spiral is canonically represented in polar coordinates by the following equation:

$$\left\{\begin{array}{c}r=b\theta \hfill \\ b={\displaystyle \frac{dr}{d\theta }}\hfill \end{array}\right.$$

(6)

where b denotes the pitch, defined as the radial displacement per unit angular increment $\theta$. Here, $\theta$ corresponds to the angular displacement of the UAV relative to the VCM. The angular parameter $\theta$, intrinsically governed by the number of spiral turns N, spans an unbounded domain $\theta \in (-\infty ,+\infty )$, with the sign of N dictating the spiral’s chirality (clockwise or counterclockwise). This relationship is formalized as follows:

$$\theta ={\lambda }_p-{\xi }_p$$

(7)

where ${\xi }_p$ represents the terminal LOS angle between the UAV and VCM, which asymptotically defines the spiral’s endpoint orientation. Like $\theta$, ${\xi }_p$ also spans $[-\infty ,+\infty ]$. Given the initial radial distance $r_{p0}$ and LOS angle ${\lambda }_{p0}$ between the UAV and VCM, the pitch b is derived as follows:

$$b={\displaystyle \frac{r_{p0}}{\theta }}={\displaystyle \frac{r_{p0}}{{\lambda }_{p0}-{\xi }_p}}$$

(8)

The total number of spiral turns $N_t$, determined by the initial and terminal LOS angles between the UAV and VCM, is expressed as follows:

$$N_t=({\lambda }_{p0}-{\xi }_p)/2\pi$$

(9)

Combining Equations (8) and (9), the pitch simplifies to the following:

$$b={\displaystyle \frac{r_{p0}}{2\pi N_t}}$$

(10)

Thus, the geometric configuration of the spiral is uniquely determined by the triad $\left(r_{p0},{\lambda }_{p0},N_t\right)$, yielding the following radial trajectory:

$$r_p=2\pi b{N}_r$$

(11)

where $N_r=({\lambda }_p-{\xi }_p)/2\pi$ quantifies the remaining spiral turns.

To ensure the terminal guidance accuracy of the UAV, the relative spiral trajectory must asymptotically converge to the reference path generated by the VCM. This necessitates precise alignment between the UAV’s terminal position post-spiral and the VCM-derived reference trajectory. By imposing a uniform deceleration profile along the Archimedean spiral, the following kinematic system governs the motion:

$$\left\{\begin{array}{c}{V}_{p0}-a_c{t}_{go}=0\hfill \\ V_{p0}{t}_{go}-{\displaystyle \frac{1}{2}}{a}_c{t}_{go}^2=S\hfill \end{array}\right.$$

(12)

where $V_{p0}$ denotes the initial relative spiral velocity, $a_c$ is the constant deceleration magnitude, $t_{go}$ represents the maneuver duration and S quantifies the cumulative arc length traversed during $t_{go}$.

The arc length $S\left({\lambda }_p\right)$ of an Archimedean spiral, as derived in [25], is expressed as follows:

$$\begin{array}{c}S\left({\lambda }_p\right)={\displaystyle \frac{1}{2}}b\left(\left({\lambda }_p-{\xi }_p\right)\sqrt{1+{\left({\lambda }_p-{\xi }_p\right)}^2}\right.\left.+{sinh}^{-1}\left({\lambda }_p-{\xi }_p\right)\right)\hfill \end{array}$$

(13)

Substituting Equation (9) into Equation (13) yields the total spiral arc length:

$$\begin{array}{c}{S}_t={\displaystyle \frac{1}{2}}b\left(2\pi N_t\sqrt{1+{\left(2\pi N_t\right)}^2}\right.\left.+{sinh}^{-1}\left(2\pi N_t\right)\right)\hfill \end{array}$$

(14)

Synthesizing the total spiral maneuver duration $t_{end}$ with Equations (12) and (14), the initial relative spiral velocity $V_{p0}$ and deceleration $a_c$ can be analytically determined.

Two critical considerations warrant emphasis:

1.

The initial LOS angle between the UAV and VCM is conventionally initialized to ${\lambda }_{p0}=0$, enabling maneuver amplitude and frequency to be modulated by adjusting the number of spiral turns $N_t$ and the initial radial distance $r_{p0}$. Thus, the geometric configuration of the spiral is determined by the triad $\left(r_{p0},N_t\right)$.

2.

The total spiral maneuver duration $t_{end}$ often deviates from the UAV’s actual flight time. To reconcile this discrepancy, $t_{end}$ is numerically optimized via the Newton–Raphson iterative method to satisfy terminal guidance precision for stationary targets. For mobile targets, the adjusted maneuver time $t_{end}-\Delta t$ is adopted, advancing convergence to the virtual trajectory to mitigate kinematic discrepancies induced by target motion, thereby ensuring terminal guidance precision.

3.2. Virtual Trajectory Generation Based on DRL

3.2.1. Reinforcement Learning Architecture

Within RL frameworks, agents iteratively engage with their operational environment through a trial-and-error learning paradigm to maximize cumulative reward signals in stochastic, high-dimensional state spaces. DRL integrates the perceptual capacities of deep neural networks with the decision-theoretic capabilities of RL, thereby enabling end-to-end autonomous decision-making systems. The interaction dynamics intrinsic to RL are formally modeled through a MDP, as illustrated in Figure 4, which establishes the foundational RL architecture. The algorithmic workflow proceeds as follows: (1) the agent observes the environmental state $S_t$; (2) the agent selects an action $a_t$ stochastically governed by the policy $\pi$; (3) the environment transitions to a successor state $S_{t+1}$ through stochastic transitions; and (4) the agent computes the expected long-term discounted return based on the scalar reward $r_t$, subsequently refining the policy $\pi$ via gradient-based optimization.

In RL, the reward function serves as a critical mechanism for facilitating policy optimization, being intrinsically linked to the environmental state ${\mathit{s}}_t$ and the agent’s selected action ${\mathit{a}}_t$, mathematically formulated as $r_t\left({\mathit{s}}_t,a_t\right)$. A trajectory $\tau$ is formally defined as the state–action sequence generated through the agent’s iterative interactions under policy $\pi$, expressed as $\tau =\left(s_0,a_0,s_1,a_1,\cdots \right)$. The return function $R\left(\tau \right)$, representing the cumulative discounted reward over the trajectory, is rigorously formalized in Equation (15), encapsulating the agent’s long-term strategic objective.

$$R\left(\tau \right)=\sum _{t=0}^T{\gamma }^t{r}_t$$

(15)

where t denotes the temporal decision index, T represents the terminal decision horizon, and $\gamma \in \left[0,1\right]$ defines the discount factor modulating the temporal weighting between immediate and deferred rewards. The principal objective of RL revolves around the maximization of the expected cumulative return, where the optimal policy ${\pi }^{*}$ is derived through the following optimization framework:

$${\pi }^{*}=\underset{\pi }{\arg \max }{E}_{\tau \sim \pi }R\left(\tau \right)$$

(16)

RL incorporates state- and action-value functions as quantitative metrics for policy evaluation and serves as the foundation for policy optimization. The state-value function $V_{\pi }\left(\mathit{s}\right)$, representing the expected cumulative discounted return when the agent executes policy $\pi$ from state $\mathit{s}$, is mathematically formalized as follows:

$$V_{\pi }\left(\mathit{s}\right)\stackrel{.}{=}{E}_{{\tau }_t\sim \pi }\left[\left.R\left({\tau }_t\right)\right|{\mathit{s}}_t=\mathit{s}\right]$$

(17)

where ${\mathit{s}}_t$ denotes the state vector at time step t, and ${\tau }_t$ represents a trajectory generated under policy $\pi$.

Similarly, the action-value function $Q_{\pi }\left(\mathit{s},a\right)$, which estimates the expected cumulative return for executing action $\mathit{a}$ in state $\mathit{s}$ under policy $\pi$, is formalized as follows:

$$Q_{\pi }\left(\mathit{s},a\right)\stackrel{.}{=}{E}_{{\tau }_t\sim \pi }\left[\left.R\left({\tau }_t\right)\right|{\mathit{s}}_t=\mathit{s},a_t=a\right]$$

(18)

where $a_t$ corresponds to the action taken at time step t.

These functions provide foundational guidance for the agent’s iterative policy improvement, directing convergence toward optimality. In DRL, parametric approximations of $V_{\pi }\left(\mathit{s}\right)$ and $Q_{\pi }\left(\mathit{s},a\right)$ are typically achieved through deep neural networks, enabling scalable representation of value functions in high-dimensional state–action spaces.

In RL, the absolute utility of an agent’s actions is often secondary to their relative efficacy compared to alternative actions. This distinction underpins the pivotal role of the advantage function, a foundational construct in RL that quantifies the relative merit of an action $\mathit{a}$ in state $\mathit{s}$ under policy $\pi$. Formally, the advantage function is defined as follows:

$$A_{\pi }\left(\mathit{s},a\right)=Q_{\pi }\left(\mathit{s},a\right)-V_{\pi }\left(\mathit{s}\right)$$

(19)

To estimate $A_{\pi }\left(\mathit{s},a\right)$, the generalized advantage estimation (GAE) method [26] serves as a prevalent estimator in policy optimization frameworks. The GAE is expressed as follows:

$$A_{\pi ,t}^{\mathrm{GAE}}\left(\mathit{s},a\right)=\sum _{l=0}^{\infty }{\left(\gamma \lambda \right)}^l{\delta }_{t+l}^V$$

(20)

where ${\delta }_t^V$ represents the temporal difference (TD) residual:

$${\delta }_t^V=\left(r_t+\gamma V_{\pi }\left({\mathit{s}}_{t+1}\right)-V_{\pi }\left({\mathit{s}}_t\right)\right)$$

(21)

where $\gamma \in [0,1]$ and $\lambda \in [0,1]$ are hyperparameters governing the discount factor and bias–variance trade-off, respectively. The GAE framework integrates multi-step returns, enabling stable and efficient credit assignment across trajectories.

3.2.2. Interactive Scene

The design of the interactive engagement environment must be predicated on the operational scenario, adversarial kinematic constraints and comprehensive inclusion of potential agent states.

(1)

Target State Initialization

In scenarios involving UAV engagement with slowly maneuvering targets, it is postulated that, during target acquisition, the target initiates rectilinear motion within the horizontal plane. The target’s velocity magnitude $V_t$ and heading angle ${\psi }_t$ are modeled as uniformly distributed variables: ${\psi }_t\sim \mathcal{U}(-\pi ,\pi ),V_t\sim \mathcal{U}(0,15)\mathrm{m}/\mathrm{s}$.

(2)

VCM State Initialization

The initial position and velocity of the VCM are derived computationally from the UAV’s relative spiral motion parameters and initial state. The kinematic relationship between the UAV and VCM is governed by the following:

$${\mathit{V}}_v={\mathit{V}}_r-{\mathit{V}}_p$$

(22)

where ${\mathit{V}}_v$, ${\mathit{V}}_r$ and ${\mathit{V}}_p$ denote the VCM’s inertial velocity, UAV’s inertial velocity and UAV’s velocity relative to the VCM, respectively.

At initialization ($t=0$),

$${\left[V_{vx0},V_{vy0},V_{vz0}\right]}^{\mathrm{T}}={\left[V_{rx0},V_{ry0},V_{rz0}\right]}^{\mathrm{T}}-{\left[V_{px0},V_{py0},V_{pz0}\right]}^{\mathrm{T}}$$

(23)

where ${\left[V_{vx0},V_{vy0},V_{vz0}\right]}^{\mathrm{T}}$ is the velocity vector of the VCM under the inertial coordinate system at the initial moment; ${\left[V_{rx0},V_{ry0},V_{rz0}\right]}^{\mathrm{T}}$ is the velocity vector of the vehicle in the inertial coordinate system at the initial moment; ${\left[V_{px0},V_{py0},V_{pz0}\right]}^{\mathrm{T}}$ is the velocity vector of the UAV relative to the VCM at the initial moment, and its specific expression is as follows:

$${\left[V_{px0},V_{py0},V_{pz0}\right]}^{\mathrm{T}}={\left[0,V_{p0}sin{\gamma }_{p0},V_{p0}cos{\gamma }_{p0}\right]}^{\mathrm{T}}$$

(24)

where ${\gamma }_{p0}$ is the velocity azimuth of the UAV in the heading-normal plane of the VCM at the initial moment.

Substituting Equations (4), (6) and (7) yields:

$$tan\left({\delta }_p\right)={\lambda }_p-{\xi }_p$$

(25)

And ${\gamma }_p$, ${\lambda }_p$ and ${\delta }_p$ have the following relationship:

$${\gamma }_p={\delta }_p+{\lambda }_p$$

(26)

By substituting ${\lambda }_{p0}$ into Equations (25) and (26), the initial velocity of the VCM under the inertial coordinate system can be calculated. Then according to Equations (23) and (24) we can calculate ${\gamma }_{v0}$ and ${\psi }_{v0}$.

The UAV’s initial position within the VCM’s heading-normal plane facilitates determination of the VCM’s inertial position via the following:

$${\left[x_{v0},y_{v0},z_{v0}\right]}^{\mathrm{T}}={\left[x_{r0},y_{r0},z_{r0}\right]}^{\mathrm{T}}-\mathit{L}\left({\psi }_{v0},{\gamma }_{v0}\right){\left[x_{r0},y_{r0},z_{r0}\right]}^{\mathrm{T}}$$

(27)

where ${[x_{v0},y_{v0},z_{v0}]}^{\mathrm{T}}$ and ${\left[x_{r0},y_{r0},z_{r0}\right]}^{\mathrm{T}}$ are the positions of the VCM and the UAV in the inertial coordinate system at the initial moment, respectively; $\mathit{L}\left({\psi }_{v0},{\gamma }_{v0}\right)$ is the transformation matrix from the inertial coordinate system to the virtual ballistic coordinate system.

(3)

VCM Velocity Compensation

The fundamental requirement for generating a stable spiral maneuvering trajectory is that the UAV maintains motion within the course-normal plane of the VCM. However, since the trajectory design methodology employs a bank-to-turn (BTT) control architecture, the UAV cannot inherently constrain its positional component along the X-axis of the virtual ballistic coordinate system to null. To address this limitation, a velocity compensation mechanism is applied to the VCM, ensuring persistent alignment of the UAV with the VCM’s course-normal plane. The compensated velocity vector is governed by the following:

$$V_{ve}=K_e\Delta x_e+V_{xd}$$

(28)

where $V_{ve}$ denotes the compensated VCM velocity; $\Delta x_e$ represents the UAV’s positional deviation along the $M_v{X}_{v1}$-axis of the virtual ballistic coordinate system; $V_{xd}$ corresponds to the UAV’s velocity component along the $M_v{X}_{v1}$-axis; $K_e$ is the feedback gain coefficient, and takes 20 in the simulation.

3.2.3. MDP for Virtual Trajectory Generation

The efficacy of RL in UAV trajectory synthesis hinges on the rigorous formulation of a MDP, encompassing the definition of a state space, action space and reward function that holistically encapsulate system dynamics and control objectives.

(1)

State Space and Action Space Design

The state space is architected to emulate the perceptual constraints of onboard sensor systems, such as radar-based seekers, which provide measurements of relative kinematics and angular geometries. To approximate real-world operational fidelity, the state vector ${\mathit{s}}_t$ integrates six degrees of freedom derived from the VCM–target engagement dynamics:

$${\mathit{s}}_t={\left[R_{\mathrm{DM}},{\dot{R}}_{\mathrm{DM}},q_{DM},{\dot{q}}_{DM},{\lambda }_{DM},{\dot{\lambda }}_{DM}\right]}^{\mathrm{T}}$$

(29)

where $R_{\mathrm{DM}}$ is the relative range between the VCM and target; ${\dot{R}}_{\mathrm{DM}}$ is the relative range rate between the VCM and the target; $q_{DM}$ and ${\dot{q}}_{DM}$ are the LOS elevation angle and LOS elevation angle rate between the VCM and the target; ${\lambda }_{DM}$ and ${\dot{\lambda }}_{DM}$ are the LOS azimuth angle and LOS azimuth angle rate between the VCM and the target.

The action space governs the VCM’s maneuverability through normalized lateral acceleration commands, bounded by operational limits to reflect actuator saturation:

$${\mathit{u}}_{\mathit{t}}=\left\{\begin{array}{c}{n}_y\in \left[-n_{max},n_{max}\right]\hfill \\ n_z\in \left[-n_{max},n_{max}\right]\hfill \end{array}\right.$$

(30)

where $n_y$ and $n_z$ denote lateral acceleration components in the pitch and yaw axes, respectively, and $n_{max}$ defines the maximum achievable g-load.

(2)

Reward Function Architecture

The reward function constitutes a pivotal mechanism in DRL, critically influencing policy convergence and optimality. Its design must holistically encode the mission’s kinematic objectives to ensure higher rewards correlate with trajectories satisfying terminal precision.

The real missile moves in a relative spiral around the virtual trajectory until it converges to the virtual trajectory. Its relative spiral motion provides powerful penetration capability, while the virtual trajectory needs to provide a high-precision strike effect. Therefore, the reward function is designed into the following two parts:

$$r_t=r_{\mathrm{s}}+r_{\mathrm{e}}$$

(31)

where $r_{\mathrm{s}}$ and $r_{\mathrm{e}}$ are the process reward and terminal reward, respectively. The specific expressions for $r_{\mathrm{s}}$ and $r_{\mathrm{e}}$ are as follows:

$$r_{\mathrm{s}}=k_{s}exp[-\sqrt{[{\left({\dot{q}}_{DM}\right)}^2+{\left({\dot{\lambda }}_{DM}\right)}^2]/0.05}]$$

(32)

$$r_{\mathrm{e}}=\left\{\begin{array}{c}{\displaystyle \frac{R_{\mathrm{miss}}-R_{\mathrm{miss}}^{\mathrm{th},1}}{R_{\mathrm{miss}}^{\mathrm{th},2}-R_{\mathrm{miss}}^{\mathrm{th},1}}}\left(r_{\mathrm{e}}^{\mathrm{th},2}-r_{\mathrm{e}}^{\mathrm{th},1}\right)+r_{\mathrm{e}}^{\mathrm{th},1},R_{\mathrm{miss}}^{\mathrm{th},1}\le R_{\mathrm{miss}}\le R_{\mathrm{miss}}^{\mathrm{th},2}\hfill \\ r_{\mathrm{e}}^{\mathrm{th},3},R_{\mathrm{miss}}\le R_{\mathrm{miss}}^{\mathrm{th},1}\hfill \\ r_{\mathrm{e}}^{\mathrm{th},4},R_{\mathrm{miss}}>R_{\mathrm{miss}}^{\mathrm{th},2}\hfill \end{array}\right.$$

(33)

where $r_{\mathrm{s}}$ is designed according to the principle of the classical guidance law to achieve a quasi-parallel approach by restraining the angular speed of the missile eye. This reward is conducive to promoting the agent to generate the instruction to fly towards the target. $r_{\mathrm{e}}$ is a reward that reflects the miss distance of the terminal, and gives a large positive reward when the interception is successful; $k_{\mathrm{s}}$ is a hyperparameter greater than 0; $R_{\mathrm{miss}}$ is the miss distance; $R_{\mathrm{miss}}^{\mathrm{th},1}$ and $R_{\mathrm{miss}}^{\mathrm{th},2}$ are hyperparameters related to the miss distance, where $R_{\mathrm{miss}}^{\mathrm{th},1}$ represents the miss distance corresponding to the successful interception of the target; $r_{\mathrm{e}}^{\mathrm{th},1}$, $r_{\mathrm{e}}^{\mathrm{th},2}$, $r_{\mathrm{e}}^{\mathrm{th},3}$ and $r_{\mathrm{e}}^{\mathrm{th},4}$ are hyperparameters related to the reward; the diagram of $r_{\mathrm{e}}$ is shown in Figure 5.

3.2.4. Model Solving Based on PPO Algorithm

The PPO algorithm demonstrates superior performance in high-dimensional continuous control tasks and has been established as the de facto standard for policy gradient methods in autonomous guidance systems, as endorsed by OpenAI. Consequently, this work adopts PPO as the foundational algorithm for virtual trajectory synthesis. A concise overview of PPO is provided below; comprehensive theoretical derivations are detailed in Ref. [27].

PPO manifests in two primary variants: (1) adaptive KL divergence-penalized policy optimization (PPO-KL) and (2) clipped surrogate objective optimization (PPO-CLIP). While both variants achieve a comparable empirical performance, PPO-CLIP is computationally advantageous, as it circumvents the necessity for iterative KL divergence calculations. This work employs the PPO-CLIP formulation for its operational efficiency.

The PPO-CLIP framework integrates dual neural networks—a policy network ${\pi }_{\theta }$ and a value network $V_{\mathit{\omega }}$—to maximize the composite objective function:

$$L^{\mathrm{PPO}}(\mathit{\theta },\mathit{\omega })=E\left[L^{\mathrm{CLIP}}\left(\mathit{\theta }\right)-c_{\mathrm{VF}}{L}^{\mathrm{VF}}\left(\mathit{\omega }\right)+c_{\mathrm{s}}{L}^{\mathcal{H}}\left(\mathit{\theta }\right)\right]$$

(34)

where $c_{\mathrm{VF}}$ and $c_{\mathrm{s}}$ are hyperparameters; $\mathit{\theta }$ and $\mathit{\omega }$ represent the parameters of policy network and value network, respectively. $L^{\mathrm{CLIP}}\left(\mathit{\theta }\right)$ is the optimization target of the policy network, which can be calculated by Equation (35). $L^{\mathrm{VF}}\left(\mathit{\omega }\right)$ is the optimization target of the value network, which can be calculated by Equation (37). $L^{\mathcal{H}}\left(\mathit{\theta }\right)=E_{a\sim {\pi }_{\mathit{\theta }}}\left[-\log {\pi }_{\mathit{\theta }}\left(\left.a\right|\mathit{s}\right)\right]$ is the entropy of the policy, reflecting the uncertainty of the policy.

$$\begin{array}{c}{L}^{\mathrm{CLIP}}\left(\mathit{\theta }\right)=E_{\left(\mathit{s},a\right)\sim {\pi }_{\mathit{\theta },\mathrm{old}}}\left[min\left(p\left(\mathit{\theta }\right)A_{{\pi }_{\mathit{\theta },\mathrm{old}}}^{\mathrm{GAE}}\left(\mathit{s},a\right),\right.\right.\left.\left.\mathrm{clip}\left(p\left(\mathit{\theta }\right),1-\epsilon ,1+\epsilon \right)A_{{\pi }_{\mathit{\theta },\mathrm{old}}}^{\mathrm{GAE}}\left(\mathit{s},a\right)\right)\right]\end{array}$$

(35)

where ${\pi }_{\mathit{\theta }}$ and ${\pi }_{\mathit{\theta },\mathrm{old}}$ represent the new policy and the old policy, respectively; $p\left(\mathit{\theta }\right)={\pi }_{\mathit{\theta }}\left(\left.a\right|\mathit{s}\right)/\phantom{{\pi }_{\mathit{\theta }}\left(\left.a\right|\mathit{s}\right){\pi }_{\mathit{\theta },\mathrm{old}}\left(\left.a\right|\mathit{s}\right)}{\pi }_{\mathit{\theta },\mathrm{old}}\left(\left.a\right|\mathit{s}\right)$ represents the probability ratio corresponding to the old and new strategies; $clip(·,·,·)$ is the clipping function, which can be expressed as follows:

$$clip(p\left(\mathit{\theta }\right),1-\epsilon ,1+\epsilon )=\left\{\begin{array}{c}1-\epsilon ,p\left(\mathit{\theta }\right)<1-\epsilon \hfill \\ p\left(\mathit{\theta }\right),1-\epsilon \le p\left(\mathit{\theta }\right)\le 1+\epsilon \hfill \\ 1+\epsilon ,p\left(\mathit{\theta }\right)>1+\epsilon \hfill \end{array}\right.$$

(36)

where $\mathcal{E}$ is a small hyperparameter, which is used to ensure that there is not much difference between the old and new policies.

$$L^{\mathrm{VF}}\left(\mathit{\omega }\right)={\left(V_{{\pi }_{\theta }}(\mathit{s};\mathit{\omega })-V_{{\pi }_{\theta ,\mathrm{old}}}^{\mathrm{targ}}(\mathit{s};\mathit{\omega })\right)}^2$$

(37)

where $V_{{\pi }_{\theta ,\mathrm{old}}}^{targ}(s;\mathit{\omega })$ is the target of the value network; the calculation formula is as follows:

$$V_{{\pi }_{\theta ,\mathrm{old}}}^{\mathrm{targ}}(\mathit{s};\mathit{\omega })=A_{{\pi }_{\theta ,\mathrm{old}}}^{\mathrm{GAE}}(\mathit{s},a)+V_{{\pi }_{\theta ,\mathrm{old}}}\left(\mathit{s};{\mathit{\omega }}_{\mathrm{old}}\right)$$

(38)

The technique of reward scaling [28] is introduced to avoid the negative impact of too large or too small rewards on the learning of the value network, which can be expressed as the goal of the scaled value network:

$${\tilde{V}}_{{\pi }_{\theta ,\mathrm{old}}}^{targ}(\mathit{s};\mathit{\omega })={\displaystyle \frac{V_{{\pi }_{\theta ,\mathrm{old}}}^{targ}(\mathit{s};\mathit{\omega })}{{\sigma }_{V^{\mathrm{targ}}}}}$$

(39)

where ${\sigma }_{V^{targg}}$ is the standard deviation of all $V_{{\pi }_{\theta ,\mathrm{old}}}^{targ}(\mathit{s};\mathit{\omega })$.

At the same time, in order to improve the stability of the training process, the gradient clipping technique [28] is introduced to solve the gradient $g_{\mathit{\theta }}$ of $\mathit{\theta }$ and the gradient $g_{\mathit{\omega }}$ of $\mathit{\omega }$:

$$\left\{\begin{array}{c}{g}_{\mathit{\theta }}=min\left\{g_{max}/∥{\nabla }_{\mathit{\theta }}{L}^{\mathrm{PPO}}(\mathit{\theta },\mathit{\omega })∥,1.0\right\}·{\nabla }_{\mathit{\theta }}{L}^{\mathrm{PPO}}(\mathit{\theta },\mathit{\omega })\hfill \\ g_{\mathit{\omega }}=min\left\{g_{max}/∥{\nabla }_{\mathit{\omega }}{L}^{\mathrm{PPO}}(\mathit{\theta },\mathit{\omega })∥,1.0\right\}·{\nabla }_{\mathit{\omega }}{L}^{\mathrm{PPO}}(\mathit{\theta },\mathit{\omega })\hfill \end{array}\right.$$

(40)

where $g_{max}$ is the gradient clipping value.

To update the parameters of the policy network and the value network, the objective function given by the Adam optimizer maximization (Equation (35)) is adopted:

$$\left\{\begin{array}{c}\mathit{\theta }\leftarrow \mathit{\theta }+{\alpha }_{1\mathrm{r}}{g}_{\mathit{\theta }}\hfill \\ \mathit{\omega }\leftarrow \mathit{\omega }+{\beta }_{\mathrm{lr}}{g}_{\mathit{\omega }}\hfill \end{array}\right.$$

(41)

where ${\alpha }_{\mathrm{lr}}$ and ${\beta }_{\mathrm{lr}}$ are the learning rates of the strategy network and value network, respectively.

Further, decaying learning rates enhance late-stage training stability:

$$\left\{\begin{array}{c}{\alpha }_{\mathrm{lr}}={\alpha }_{\mathrm{lr},0}\left(1-t_{\mathrm{e}}/N_{\mathrm{epoch}}\right)\hfill \\ {\beta }_{\mathrm{lr}}={\beta }_{\mathrm{lr},0}\left(1-t_{\mathrm{e}}/N_{\mathrm{epoch}}\right)\hfill \end{array}\right.$$

(42)

where $alph{a}_{\mathrm{lr},0}$ and ${\beta }_{\mathrm{lr},0}$ are the initial learning rates of the strategy network and value network, respectively; $t_{\mathrm{e}}$ is the current training algebra; $N_{\mathrm{epoch}}$ is the maximum training algebra.

3.2.5. Network Structure Design

The policy network architecture (Figure 6) implements a bidirectional Gaussian policy parameterization, comprising four output neurons: two-dimensional mean $\left(\mu \right)$ and standard deviation $\left({\sigma }_{std}\right)$ vectors that parameterize the stochastic action distribution. During the training phase, actions are stochastically sampled from the Gaussian distribution $\mathcal{N}\left(\mu ,{\sigma }_{std}^2\right)$, followed by a nonlinear transformation via hyperbolic tangent (tanh) activation and affine scaling to enforce bounded action spaces.

The architectural specifications of the PPO policy and value networks—comprising layer-wise neuron counts and associated activation functions—are formally delineated in

Table 2. Network structure and parameters.

Network Level	Actor Network		Critic Network
	Units	Activation Function	Units	Activation Function
Input layer	6	None	6	None
Hidden layer 1	128	Tanh	128	Tanh
Hidden layer 2	128	Tanh	128	Tanh
Hidden layer 3	128	Tanh	128	Tanh
Output layer	4	Tanh/Linear	1	Linear

to provide a comprehensive structural overview of the proposed framework.

3.2.6. Learning Process

As derived from the preceding analytical framework, the learning-driven methodology for autonomous virtual trajectory synthesis, leveraging the PPO-CLIP algorithm, is formalized in Algorithm 1.

Algorithm 1 Learning process

Initialization: Policy network with parameter $\mathit{\theta }$, behavioral policy network ${\mathit{\theta }}_{\mathrm{old}}\leftarrow \mathit{\theta }$, value network with parameter $\mathit{\omega }$, reward related hyperparameters $k_{\mathrm{s}}$, $r_{\mathrm{e}}^{\mathrm{th},1}$, $r_{\mathrm{e}}^{\mathrm{th},2}$, $r_{\mathrm{e}}^{\mathrm{th},3}$, $r_{\mathrm{e}}^{\mathrm{th},4}$, $R_{\mathrm{miss}}^{\mathrm{th},1}$ and $R_{\mathrm{miss}}^{\mathrm{th},2}$, GAE hyperparameter $\lambda$, discount factor $\gamma$, value network error weight factor $c_{\mathrm{VF}}$, entropy weight factor $c_{\mathrm{s}}$ of policy, clipping factor $\epsilon$, initial learning rate ${\alpha }_{\mathrm{lr},0}$ and ${\beta }_{\mathrm{lr},0}$ of strategy network and value network, maximum training algebra $N_{\mathrm{epoch}}$, number of network parameter updates per training $N_{\mathrm{update}}$, number of tracks rollout $N_{\mathrm{rollout}}$, experience pool size $\left|\mathcal{D}\right|$, batch transfer data size $\left|\mathcal{B}\right|$, gradient clipping threshold $g_{max}$, interactive environment boundary parameters. 1: for $t_e=1,2,\cdots ,N_{\mathrm{epoch}}$do 2:      Using rollouts to take ${\pi }_{{\mathit{\theta }}_{\mathrm{old}}}$ as the behavior strategy, the network interacts with the         environment to generate $N_{\mathrm{rollout}}$ tracks, which are stored in the experience pool         $\mathcal{D}=\left\{{\mathcal{T}}_1,{\mathcal{T}}_2,\cdots ,{\mathcal{T}}_{N_{\mathrm{collect}}}\right\}$, where $\mathcal{D}$ is a collection of tuples $\left({\mathit{s}}_t,{\mathit{a}}_{\mathit{t}},r_t,{\mathit{s}}_{t+1}\right)$. 3:      Compute the dominance function $A_{{\pi }_{\mathit{\theta },\mathrm{old}}}^{\mathrm{GAE}}$ according to Equation (20) and update the         tuple in $\mathcal{D}$ to $\left({\mathit{s}}_t,a_t,r_t,{\mathit{s}}_{t+1},A_{{\pi }_{\mathit{\theta },\mathrm{old}},t}^{\mathrm{GAE}}\right)$. 4:      Calculate the objective value function $V_{{\pi }_{\mathit{\theta },\mathrm{old}}}^{\mathrm{targ}}$ according to Equation (38) and update         the tuple in $\mathcal{D}$ to $\left({\mathit{s}}_t,a_t,r_t,{\mathit{s}}_{t+1},A_{{\pi }_{\mathit{\theta },\mathrm{old}},t}^{\mathrm{GAE}},V_{{\pi }_{\mathit{\theta },\mathrm{old}},t}^{\mathrm{targ}}\right)$. 5:      for $i=1,\cdots ,N_{\mathrm{update}}$ do 6:            Batch transfer data $\mathcal{B}$ is obtained by random sampling from experience pool $\mathcal{D}$,            and calculate PPO objective function:            $\begin{array}{cc}\hfill L^{\mathrm{PPO}}(\mathit{\theta },\mathit{\omega })={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{\mathcal{B}}& \left[min\left(p\left({\pi }_{\mathit{\theta }},{\pi }_{\mathit{\theta },\mathrm{old}}\right)A_{{\pi }_{\mathit{\theta },\mathrm{ld}}}^{\mathrm{GAE}},clip\left(p\left({\pi }_{\mathit{\theta }},{\pi }_{\mathit{\theta },\mathrm{old}}\right),1-\epsilon ,1+\epsilon \right)A_{{\pi }_{\mathit{\theta },\mathrm{old}}}^{\mathrm{GAE}}\right)\right.\hfill \\ & \left.-c_{\mathrm{VF}}{\left(V_{{\pi }_{\mathit{\theta }}}(\mathit{s};\mathit{\omega })-V_{{\pi }_{\mathit{\theta },\mathrm{ldt}}}^{\mathrm{targ}}(\mathit{s};\mathit{\omega })\right)}^2-c_{\mathrm{s}}log{\pi }_{\mathit{\theta }}\left(a_t\mid s_t\right)\right]\hfill \end{array}$ 7:            Calculate the cropped gradients x and x according to Equation (40). 8:            Update network parameters according to Equation (41). 9:      end for 10:     Clear $\mathcal{D}$. 11:     Update the behavior policy network parameter ${\mathit{\theta }}_{\mathrm{old}}\leftarrow \mathit{\theta }$. 12:     Update the learning rate according to Equation (42). 13: end for

3.3. Spiral Maneuvering Trajectory Generation

The relative spiral motion is attached to the virtual trajectory to form a spiral maneuver trajectory, as shown in Figure 7.

The position and velocity of the UAV in the virtual ballistic coordinate system are as follows:

$${\left[x_{dv},y_{dv},z_{dv}\right]}^{\mathrm{T}}={\left[0,r_{p}sin{\lambda }_p,r_{p}cos{\lambda }_p\right]}^{\mathrm{T}}$$

(43)

$${\left[{\dot{x}}_{dv},{\dot{y}}_{dv},{\dot{z}}_{dv}\right]}^{\mathrm{T}}={\left[0,V_{p}sin{\gamma }_p,V_{p}cos{\gamma }_p\right]}^{\mathrm{T}}$$

(44)

where ${[x_r,y_r,z_r]}^{\mathrm{T}}$ and ${\left[x_v,y_v,z_v\right]}^{\mathrm{T}}$ are the position and velocity of the UAV in the virtual ballistic coordinate system; $V_p$ is the velocity of the vehicle relative to the VCM.

At every moment, $r_p$ can be obtained by Equation (11), ${\lambda }_p$ can be obtained by Equation (13), and ${\gamma }_p$ can be obtained by Equations (25) and (26). By substituting $r_p$, ${\lambda }_p$ and ${\gamma }_p$ into Equations (43) and (44), the position and velocity of the UAV in the virtual ballistic coordinate system can be obtained, and the position and velocity of the UAV in the inertial coordinate system can be calculated:

$${\left[x_r,y_r,z_r\right]}^{\mathrm{T}}=\mathit{L}{\left({\psi }_v,{\gamma }_v\right)}^{\mathrm{T}}\left(\mathit{L}\left({\psi }_v,{\gamma }_v\right){\left[x_v,y_v,z_v\right]}^{\mathrm{T}}+{\left[x_{dv},y_{dv},z_{dv}\right]}^{\mathrm{T}}\right)$$

(45)

$${\left[V_{xd},V_{yd},V_{sd}\right]}^{\mathrm{T}}=\mathit{L}{\left({\psi }_v,{\gamma }_v\right)}^{\mathrm{T}}{\left[{\dot{x}}_{dv},{\dot{y}}_{dv},{\dot{z}}_{dv}\right]}^{\mathrm{T}}$$

(46)

where ${[x_r,y_r,z_r]}^{\mathrm{T}}$ and ${\left[x_v,y_v,z_v\right]}^{\mathrm{T}}$ are the positions of the UAV and the VCM in the inertial coordinate system; ${\left[V_{xd},V_{yd},V_{zd}\right]}^{\mathrm{T}}$ is the velocity of the UAV in the inertial coordinate system; $\mathit{L}{\left({\psi }_v,{\gamma }_v\right)}^{\mathrm{T}}$ is the transformation matrix from virtual ballistic coordinate system to inertial coordinate system.

Under dynamic target engagement scenarios, the empirically predetermined spiral maneuver duration is configured to be shorter than the UAV’s actual flight duration to account for adversarial motion. As prescribed by the proposed methodology, upon reaching the designated spiral maneuver interval, the UAV achieves kinematic congruence (position and velocity vector alignment) with the VCM. This synchronization enables a seamless transition to subsequent trajectory synthesis through persistent VCM-based guidance laws, thereby maintaining terminal guidance phase precision for high-fidelity target engagement.

4. Simulation Analysis

This section conducts numerical simulations to systematically evaluate the feasibility and operational efficacy of the proposed DRL-enabled framework for the online synthesis of spiral maneuvering trajectories. Initially, a robust policy network is trained through hyperparameter optimization to achieve convergence in adversarial engagement scenarios. Subsequently, statistical Monte Carlo simulations are executed to quantify terminal precision and analyze the UAV’s state-space evolution under dynamic target kinematics.

The UAV’s parametric configuration, including inertial properties and aerodynamic coefficients, is adopted from the benchmark model in Ref. [29]. The simulation parameters are shown in

Table 3. Simulation correlation parameter.

Object	Physical Quantity	Value
UAV	$x_r\left(0\right),y_r\left(0\right),z_r\left(0\right)$	−50 km, 20 km, 0 km
	$V\left(0\right),\gamma \left(0\right),\psi \left(0\right)$	2000 m/s, −20°, 0°
Target	$x_t\left(0\right),y_t\left(0\right),z_t\left(0\right)$	0 m, 0 m, 0 m

.

4.1. Training Process

The training and validation simulations presented in this study were conducted using a Python 3.11 computational framework integrated with the PyTorch 2.5.1 deep learning library. The hardware configuration comprised an Intel^® Core™ i5-12600KF central processing unit (CPU) operating at 3.70 GHz, an NVIDIA GeForce RTX 3060 graphics processing unit (GPU), 16 GB of random-access memory (RAM) and a 64-bit operating system to ensure sufficient computational throughput for policy-gradient optimization. Critical hyperparameters governing the PPO algorithm, including learning rate, discount factor, clipping threshold and the reward function architecture, are systematically tabulated in

Table 4. Hyperparameter values.

Hyperparameter	Value	Hyperparameter	Value
$k_{\mathrm{s}}$	0.01	$c_{\mathrm{VF}}$	0.25
$r_{\mathrm{e}}^{\mathrm{th},1}$	10.0	$c_{\mathrm{s}}$	0.01
$r_{\mathrm{e}}^{\mathrm{th},2}$	50.0	$g_{max}$	0.5
$r_{\mathrm{e}}^{\mathrm{th},3}$	51.0	$\epsilon$	0.2
$R_{\mathrm{miss}}^{\mathrm{th},1}$	2.0	${\alpha }_{\mathrm{lr},0}$	0.0001
$R_{\mathrm{miss}}^{\mathrm{th},2}$	100.0	${\beta }_{\mathrm{lr},0}$	0.0001
$\left|\mathcal{D}\right|$	20,000	$N_{\mathrm{epoch}}$	400
$\left|\mathcal{B}\right|$	128	$N_{\mathrm{update}}$	200
$\gamma$	0.995	$N_{\mathrm{rollout}}$	20
$\lambda$	0.95	$n_{\max }\left(g\right)$	5.0

to ensure reproducibility and methodological transparency.

The procedural workflow for synthesizing virtual trajectories through DRL training, employing the PPO algorithm, is delineated in Figure 8. This visualization quantifies the temporal evolution of the mean reward trajectory across training episodes, thereby characterizing the convergence dynamics of the DRL framework. The exponentially weighted moving average (EMA) of episodic rewards is represented by the solid line, while the shaded region quantifies the dispersion (standard deviation) of the reward distribution across policy iterations. This dispersion metric serves as a robustness indicator, reflecting the stability and stochastic variance inherent to the policy gradient optimization process.

Initial Phase (Episodes 0–50): The mean reward exhibits a rapid monotonic increase from near-zero baseline values, signifying that the agent has developed foundational policy efficacy through exploratory interactions with the environment. Concurrently, the significant stochastic variance (evidenced by the broad confidence interval) reflects high entropy in the initial policy distribution and unconstrained exploration dynamics.

Intermediate Phase (Episodes 50–250): A diminished yet sustained positive gradient in reward accumulation is observed, accompanied by progressive contraction of the reward distribution’s dispersion. This dual trend indicates stabilization of the policy’s action-value estimates and an improved exploitation–exploration trade-off. The agent transitions from stochastic exploration toward deterministic policy dominance, attenuating noise-induced perturbations in the optimization landscape.

Convergence Phase (Episodes 250–400): The reward metric asymptotically approaches steady-state values with minimal episodic fluctuations, while the standard deviation converges to a lower-bound threshold. These dynamics confirm algorithmic convergence, with policy invariance across training batches and bounded susceptibility to environmental stochasticity, thereby validating the robustness of the optimized control strategy.

4.2. Test Process

In order to verify the validity and feasibility of the method proposed in this paper, the agent trained by PPO algorithm is used to generate a virtual trajectory, and the relative spiral is attached to the virtual trajectory. Two parts of simulation test are designed. The first part verifies that the spiral trajectory in this paper is adjustable and adopts different spiral parameters to strike fixed targets. In the second part, the accuracy of attacking moving targets is verified. A Monte Carlo simulation is carried out for targets with different moving directions and different moving speeds using a set of spiral parameters.

4.2.1. Different Spiral Parameters

To empirically validate the parametric adjustability of the proposed spiral trajectory framework, three distinct parametric configurations were systematically configured for terminal engagement with stationary targets. Case 1: $r_{p0}=300$ m$,N_t=4.0$ cycles; Case 2: $r_{p0}=400$ m$,N_t=4.0$ cycles; Case 3: $r_{p0}=400$ m$,N_t=3.0$ cycles.

Within this experimental configuration, Cases 1 and 2 exhibit parametric divergence exclusively in the number of spiral turns, thereby enabling the isolation and quantitative evaluation of turn-frequency effects under a constant initial spiral radius. Conversely, Cases 2 and 3 maintain identical $N_t$ values while varying $r_{p0}$, creating a controlled basis for analyzing radius-dependent trajectory modulation.

Figure 9 delineates the three-dimensional virtual trajectory and spiral maneuvering trajectories generated under the three parametric configurations. Divergence in virtual trajectory geometry arises from the computational derivation of the VCM position and velocity vectors, which are parameterized by the UAV’s initial kinematic state (position, velocity) and the spiral trajectory coefficients. This dependency cascades into heterogeneous initial conditions for the VCM, thereby inducing case-specific trajectory variance. Furthermore, parametric variations in spiral geometry—specifically, radius($r_{p0}$) and angular frequency($N_t$)—yield distinct spatial maneuver characteristics.

Figure 10 illustrates the velocity slope angle and velocity azimuth profiles for both the UAV and the VCM. The figure reveal that the UAV’s velocity slope angle and velocity azimuth exhibit periodic oscillations around the corresponding VCM parameters, with amplitudes diminishing asymptotically until convergence is achieved. This trajectory stabilization enables high-precision strike capabilities. A comparative analysis of Cases 1–3 demonstrates that the UAV in Case 2 undergoes markedly larger angular variation amplitudes in both velocity slope angle and velocity azimuth relative to the VCM. These observations confirm a proportional relationship between spiral parameter($r_{p0},N_t$) enlargement and elevated maneuvering intensity.

Figure 11 and Figure 12 show the changes in the attack angle and the bank angle during flight. Figure 11 shows the change curve of the attack angle. The control method adopted in this paper is BTT control, and the magnitude of the attack angle will directly affect the lift force. Due to the high altitude at the initial moment, the atmosphere is relatively thin at this time, so a large attack angle is required to provide enough lift for maneuvering. As the altitude gradually decreases and the atmosphere becomes denser, and based on the maneuver design of an Archimedean spiral, the lift required for subsequent maneuvers also gradually decreases, so the required attack angle also shows a gradually decreasing trend. Figure 12 shows the change curve of the bank angle. The bank angle breaks down the lift force of the UAV into lateral and normal upward to achieve spiral maneuvering around the virtual trajectory. In addition, this paper considers that the UAV can achieve 180° rollover, and the bank angle of the UAV also presents periodic changes according to the design principle of relative spiral motion.

Figure 13 shows the variation in the UAV overload over time. As evidenced in Figure 13a,b, both lateral and normal overload exhibit oscillatory dynamics with progressive amplitude attenuation. The reason for the up and down fluctuation of the transverse normal overload is that the design of the spiral maneuvering trajectory based on the Archimedean spiral is to carry out relative spiral motion around the virtual trajectory for many revolutions; the reason for the gradual reduction and convergence of the required overload is that the UAV carries out the relative spiral motion is a uniformly decelerating motion, and the reduction of the speed leads to a gradual reduction in the overload required to achieve the change of the speed direction. A comparative analysis of Cases 1–3 reveals that the UAV in Case 2 necessitates significantly higher lateral and normal overloads. This observation suggests an enlarged spiral radius and turns requiring enhanced overload to achieve.

The temporal evolution of the UAV’s velocity profile is delineated in Figure 14. The speed of the UAV gradually decreases from the initial 2000 m/s, Case 1 to 779.91 m/s, Case 2 to 669.17 m/s and Case 3 to 878.70 m/s. Empirical analysis reveals a positive correlation between the spiral trajectory parameters and the final falling speed: the larger the spiral radius and the more spiral turns, the longer the flight time of the UAV, and the slower the final falling speed.

4.2.2. Attack Moving Target

To verify the accuracy of the proposed method in attacking moving targets, 500 Monte Carlo simulations were carried out. The spiral parameters are Case 2 in the previous section and the adjusted maneuvering time $\Delta t=3$ s. The moving speed of the target $V_t\sim U(0,15)$ m/s and the moving direction ${\psi }_t\sim U(-\pi ,\pi )$.

Figure 15 shows the 3D trajectory changes of 500 independent simulation experiments. As can be seen from Figure 15, although the target moves at different speeds and directions in each independent simulation experiment, the three-dimensional trajectory presents similar spiral characteristics due to the same set of spiral parameters.

Figure 16 illustrates the variations in the UAV states observed across 500 Monte Carlo simulations. Given the low velocity of the target, minimal discernible variation is exhibited among trajectories in aerodynamic control parameters, including attack angle, bank angle and overload. However, as depicted in Figure 16c–g, the UAV achieves convergence in its spiral trajectory, thereby ensuring full positional and velocity synchronization between the UAV and the VCM. Following this alignment, the system proceeds to generate subsequent high-precision strike trajectories. The transition between these operational phases introduces minor discontinuities in control commands, attributable to the abrupt shift from maneuver trajectory to precision-guided execution.

The distribution of terminal miss distances is delineated in Figure 17. Empirical analysis reveals a mean miss distance of $\mu =1.50\mathrm{m}$, a standard deviation of $\sigma =0.059\mathrm{m}$ and a maximum miss distance of $max\left(d_{miss}\right)=1.64\mathrm{m}$. These metrics demonstrate that the DRL-trained intelligent agent synthesizes spiral maneuver ballistic trajectories with sub-meter precision (mean error < 2$\sigma$), achieving high strike accuracy while compensating for dynamic target displacement scenarios. The results underscore the efficacy of the proposed framework in reconciling adversarial evasion with terminal precision.

4.3. Analysis of Computational Complexity

The strategy network for generating virtual trajectories online based on the PPO algorithm training is a four-layer fully connected neural network with ${\left[R_{\mathrm{DM}},{\dot{R}}_{\mathrm{DM}},q_{DM},{\dot{q}}_{DM},{\lambda }_{DM},{\dot{\lambda }}_{DM}\right]}^{\mathrm{T}}$ as the input and ${\left[n_y,n_z\right]}^{\mathrm{T}}$ as the output, containing 34,952 parameters. When applying this strategy network in practice, the computer only needs to perform four matrix multiplications to map ${\left[R_{\mathrm{DM}},{\dot{R}}_{\mathrm{DM}},q_{DM},{\dot{q}}_{DM},{\lambda }_{DM},{\dot{\lambda }}_{DM}\right]}^{\mathrm{T}}$ to ${\left[n_y,n_z\right]}^{\mathrm{T}}$. Meanwhile, storing each parameter as a 32-bit floating-point number only occupies approximately 136.5 KB of storage space.

Using the STM32F407 single-chip microcontroller (STMicroelectronics, Geneva, Switzerland) as the experimental platform and adopting the double data type, an algorithm time consumption experiment was conducted. When applying the decision algorithm in this paper, only running the strategy network is required. The time consumption of the strategy network obtained from the timer function of the single-chip microcontroller is approximately 4.3 ms, indicating good real-time performance and the possibility of application on the on-board computer.

5. Conclusions

This study resolves the inherent dichotomy between evasion maneuverability and terminal strike precision in hypersonic UAV systems through a dual-layer trajectory architecture. The proposed framework integrates a nominal virtual trajectory, optimized for terminal guidance accuracy, with a superimposed relative spiral trajectory to induce evasion. In the design of the relative spiral, the innovative use of an Archimedean spiral is adopted, which enables the gradual convergence to the virtual trajectory while conducting maneuvering penetration, providing a foundation for a precise strike. Meanwhile, this method can precisely adjust the spiral radius and frequency. In the design of the virtual trajectory, a MDP is constructed, and the virtual trajectory generation strategy is obtained through training based on the PPO algorithm, effectively enhancing the target perception ability of the UAV and improving the adjustment ability for the maneuvering trajectory.

From the simulation results, the methodology yields an average miss distance of 1.5 m, with a maximum deviation of merely 1.64 m under parameters of a 400 m spiral radius and 4 spiral turns. It can be seen that the spiral maneuvering trajectory realized in this paper achieves large-scale spiral maneuvering and high-precision strike capabilities.