Hierarchical Decision-Making for UAV Close-Range Dynamic Tracking Using a Pursuit-Strategy Action Space

Abstract

In close-range dynamic UAV tracking, the sharp decrease in relative distance and rapidly changing relative-motion conditions require UAVs to execute highly dynamic maneuvers. Traditional autonomous decision-making systems struggle with the curse of dimensionality in continuous action spaces or suffer from strategy-level rigidity when using predefined discrete maneuver primitives. This paper aims to resolve these limitations by developing a dimension-reduced yet highly continuous decision-making framework. We propose a hierarchical deep reinforcement learning architecture based on a geometric pursuit-strategy action space. The top-level Proximal Policy Optimization agent evaluates the relative-motion state to output discrete guidance-mode commands: lag pursuit, lead pursuit, or pure pursuit. A mid-level guidance translator converts these intents into continuous flight reference commands based on angular geometry and energy maneuverability. The bottom-level guidance translator utilizes a high-fidelity JSBSim fixed-wing aircraft flight-dynamics model for precise aerodynamic control. Monte Carlo simulations and comparative experiments across representative initial postures show that the proposed framework improves training convergence compared with a conventional continuous-control PPO baseline and achieves more stable high-level guidance-mode selection than a Double-DQN baseline. In simulation tests under predefined geometric tracking-success criteria, the model achieved a 91.5% success rate in initially favorable configurations and a 64.0% success rate when starting from a challenging configuration. By abstracting complex maneuvers into geometric pursuit strategies, this hierarchical framework lowers exploration dimensionality while maintaining the continuous kinematic logic of flight trajectories, providing an interpretable and simulation-validated decision-making framework for UAV close-range dynamic tracking and autonomous flight control.

1. Introduction

With the rapid development of unmanned aerial vehicle (UAV) technology, close-range dynamic tracking has become an active research topic in UAV autonomous decision-making [1]. During close-range dynamic-tracking scenarios, the dramatically reduced relative distance often compels UAVs to perform drastic maneuvers; consequently, the relative-motion process exhibits highly dynamic and strongly nonlinear characteristics [2], thereby placing stringent demands on the real-time capability and rationality of autonomous decision-making systems [3]. Recently, deep reinforcement learning (DRL) has been increasingly applied to UAV maneuvering and dynamic-tracking decision-making, primarily owing to its capacity to optimize action strategies through interaction with the environment while reducing reliance on manually designed decision rules [4]. To mitigate the curse of dimensionality associated with high-dimensional continuous decision spaces, many recent studies have adopted hierarchical reinforcement learning (HRL) frameworks, which decompose complex decision-making problems into temporally abstract subtasks or options, thereby reducing the effective exploration horizon and improving policy-learning efficiency [5]. Nevertheless, existing hierarchical UAV maneuvering decision-making frameworks still possess certain limitations in how they define the “strategy-level execution layer” and design the “top-level action space.” These limitations motivate a closer examination of the gap between high-level guidance-mode representation and low-level executable flight commands.

Beyond UAV maneuvering studies, interaction-aware intent prediction has also been extensively investigated in autonomous driving, where an autonomous agent must infer the latent preferences, aggressiveness, or semantic intentions of surrounding agents before selecting its own maneuver. For example, Hu et al. [6] formulated lane-changing for autonomous heavy vehicles as socially interactive game-theoretic agents under asymmetric driving aggressiveness, indicating that game-theoretic reasoning can support maneuver decisions in highly interactive traffic scenarios. Deng et al. [7] further modeled lane-change decision-making as an incomplete-information game and quantified social driving preferences through driver aggressiveness, thereby reducing uncertainty in the interaction process. In addition to game-theoretic mechanisms, recent multimodal decision-making studies have explored the fusion of visual, LiDAR, and task-oriented textual cues for interpretable high-level driving decisions through semantic reasoning [8]. Although close-range UAV dynamic tracking differs from highway lane-changing in vehicle dynamics, task objectives, and safety constraints, both tasks require interaction-aware high-level decision-making under rapidly changing multi-agent situations. Inspired by these studies, the high-level PPO policy in this paper is interpreted as a guidance-mode selection module that maps the current relative-motion state to one of three pursuit intents—lag pursuit, lead pursuit, or pure pursuit—while the mid-level guidance laws and low-level flight controller ensure physically feasible execution.

Some studies have attempted to define the top-level action space as continuous flight reference commands, such as reference heading, pitch angle, or velocity commands. Yuan et al. [9] developed a UAV maneuvering decision model with a continuous action space utilizing the Deep Deterministic Policy Gradient algorithm, where the action space was defined as continuous changes in throttle, angle of attack, and roll angle. Chai et al. [10] introduced a hierarchical deep reinforcement learning decision framework whose top-level outer loop utilizes the Proximal Policy Optimization (PPO) algorithm to directly yield reference roll and pitch angular rates as macroscopic continuous action commands; by integrating a self-play mechanism for policy iteration, it demonstrated complex maneuvering capabilities in simulations. To meet the real-time requirements of continuous trajectory control and decision-making, Tan et al. [11] proposed a Surrogate-Assisted Differential Evolution algorithm for online trajectory planning, utilizing continuous variables such as throttle, angle of attack, and roll angle as decision variables, and achieved high-fidelity trajectory tracking in combination with Nonlinear Dynamic Inversion control, validating its effectiveness under various initial situations through Monte Carlo simulations. By utilizing variable-scale virtual tracking points and velocities as action outputs and transforming them into continuous flight path angle commands, Wang et al. [12], in conjunction with a BP-LSTM trajectory prediction network, attained a superior task success rate in close-range maneuvering simulations relative to conventional strategies employing fixed action amplitudes. Although continuous action spaces can theoretically provide a complete flight envelope, this underlying control space is excessively large, causing the agent to face the dilemmas of low exploration efficiency and sparse rewards, ultimately making it highly susceptible to falling into local optima and difficult to converge [13].

In an effort to avoid the combinatorial explosion inherent in continuous spaces, numerous researchers have shifted towards discretization strategies, defining the middle layer as a set of predefined discrete maneuver primitives [14] or pre-established expert rules. Cao et al. [15] discretized the action space into a set of 7 NASA basic maneuvers and combined the Double Deep Q-Network with the minimax algorithm for maneuver decision-making. By extracting expert knowledge from expert maneuvering manuals, Yang et al. [16] developed an automated maneuvering decision system using a Behavior Tree model, allowing aircraft to select appropriate predefined maneuver modes under different relative-motion configurations. For long-range UAV maneuvering scenarios, Wang et al. [17] introduced a hierarchical hybrid decision architecture, defining the strategy-level execution layer’s action space as a set of predefined discrete maneuvering rules and realizing action transitions via a rule-driven state switching mechanism. To reduce the dimensional complexity of decision actions when employing hierarchical reinforcement learning for multi-UAV maneuvering decision-making, Wang et al. [18] discretized flight control actions into fundamental discrete action sets comprising heading, altitude, and speed.

While defining the action space as predefined discrete maneuver primitives or fundamental discrete commands effectively lowers the decision-making dimensions, such an approach fragments the continuous nature of close-range flight dynamics [19]. When tracking rapidly maneuvering objects, relying on the switching of predefined discrete maneuvers not only constrains the UAV’s kinematic capabilities but often results in rigid policy execution, thereby making the UAV susceptible to missing transient geometric alignment opportunities. To overcome the constraints associated with fixed-duration strategy-level execution and inflexible actions, several studies have started exploring more adaptable hierarchical frameworks. Qian et al. [20] introduced a three-layer hierarchical decision framework (H3E) embedding expert knowledge, which sought to improve decision flexibility by employing a top-level strategy selector to coordinate mid-level guidance modes for different relative-motion conditions. Li et al. [21] proposed a hierarchical reinforcement learning framework based on a maximum entropy objective, designed three guidance strategies with clear geometric and energy logic—angle, snapshot, and energy—at the bottom layer, and realized the dynamic termination and flexible selection of guidance modes at the high layer.

As indicated by the preceding research, a key aspect of strategy-level decision-making in close-range interactive tracking is to maintain relative angular advantage [22]. However, the above studies still leave a methodological gap between high-level guidance-mode representation and low-level continuous flight execution. Continuous-action methods, such as those in [9,10,11,12], directly output flight-control or reference-command variables and therefore preserve maneuver flexibility, but they do not explicitly reduce the exploration burden caused by the large continuous control space. In contrast, maneuver-template- or rule-library-based methods [15,16,17,18] reduce the action dimensionality by introducing predefined discrete maneuvers, but their fixed maneuver templates may fragment the continuous geometric evolution of close-range relative motion and limit the timely capture of transient geometric alignment opportunities. Existing hierarchical frameworks [20,21] improve strategy-level abstraction to some extent, yet they still do not explicitly construct an action space that is both compact at the decision level and continuously executable at the guidance level. To bridge this gap and address the control requirements of maintaining favorable relative geometry and anticipatory tracking in close-range dynamic flight, this paper proposes a hierarchical UAV maneuvering reinforcement learning decision-making method based on a pursuit-strategy action space. In this framework, the high-level policy selects among three interpretable guidance-mode intents—lag pursuit, lead pursuit, and pure pursuit—while the mid-level guidance laws continuously translate these intents into heading, altitude, and velocity commands for the low-level flight controller. The primary contributions of this study, which distinguish the proposed framework from existing continuous-action methods, maneuver-template-based discrete-action methods, and general hierarchical UAV maneuvering decision methods, are summarized below:

• Formulation of a dimensionally reduced but trajectory-continuous action space anchored in pursuit strategies: Different from existing methods that assign top-level outputs either to low-level continuous control commands or to fixed-duration maneuver templates, this study defines the top-level decision-making action space as three geometric pursuit strategies: lag pursuit, lead pursuit, and pure pursuit. Unlike conventional discrete maneuvers, each action corresponds to a continuously updated guidance law rather than a rigid maneuver template.
• Development of strategy-level guidance laws grounded in angular geometry: Specifically addressing the characteristics of close-range anticipatory tracking, mathematical models for lag pursuit, lead pursuit, and pure pursuit are formulated. These models serve as an intermediary layer to translate top-level guidance-mode intentions into continuous low-level flight reference parameters.
• Realization of high-fidelity hierarchical closed-loop validation: by constructing a low-level flight control model within the JSBSim flight-dynamics environment, a hierarchical closed-loop architecture is established alongside mid-level geometric guidance and top-level pursuit-strategy selection, which enables the proposed framework to be evaluated under nonlinear aerodynamic constraints.

2. Materials and Methods

2.1. Comprehensive Architectural Overview

To address the challenges posed by continuous decision spaces and highly dynamic relative-motion conditions in close-range UAV tracking, this paper presents a hierarchical decision framework designed to decouple the intricate “maneuvering-control” process into logically distinct task, guidance, and execution layers. By means of functional decoupling, this framework mitigates the training complexity associated with reinforcement learning algorithms, with its general architecture illustrated in Figure 1.

2.2. Flight Controller

The flight controller is constructed in the JSBSim flight-dynamics simulation environment using a fixed-wing aircraft model that provides nonlinear six-degree-of-freedom state feedback [23]. Therefore, the proposed guidance-mode decisions are evaluated not only at the geometric-kinematic level but also through a nonlinear flight-dynamics loop involving attitude response, velocity variation, actuator deflection, and aerodynamic constraints. Given the strongly coupled nature of the aircraft’s equations of motion, we follow the methodology of Li et al., utilizing the PPO algorithm to train the flight controller [24]. The network architecture and training hyperparameters of the low-level flight-controller PPO are provided in Appendix A,

Table A1. Network architecture and training hyperparameters of the low-level glight-controller PPO agent.

Hyperparameter	Value
Number of hidden layers in the critic network	3
Hidden-layer sizes of the critic network	{256, 128, 128}
Optimizer of the critic network	Adam
Learning rate of the critic network	3 × 10−4
Hidden-layer sizes of the actor network	{512, 256, 128}
Optimizer of the actor network	Adam
Learning rate of the actor network	3 × 10−4
Number of parallel environments	30
Discount factor γ	0.99
Buffer size	900
Entropy coefficient	1 × 10−3
GAE parameter λ	0.95
Clipping parameter	0.2

. The designated state space incorporates attitude angles, three-axis velocity components, and expected command tracking errors, whereas the output actions map directly to normalized commands for the elevator, ailerons, rudder, and throttle. To ensure smooth control commands, a first-order low-pass filter is applied prior to action output:

$$\delta (k)=\alpha {\delta }_{\mathrm{ppo}}(k)+(1-\alpha )\delta (k-1)$$

(1)

Addressing the control conflict between turn rate and flight speed, a combined reward function based on adaptive weights was designed [25]:

$$r=W_1({\varphi }_{err})e_{\mathrm{heading}}+W_2{e}_{\mathrm{speed}}+W_3{e}_{\mathrm{altitude}}$$

(2)

Here, since the aircraft cannot sustain the maximum turn rate at high velocities, dynamic speed weights are established such that the speed weight progressively increases as the heading error decreases, thereby achieving coordinated execution of turning and acceleration.

After training in a parallelized environment, this low-level model can track the reference heading, altitude, and velocity commands generated by the mid-level geometric guidance laws. Consequently, the high-level guidance-mode selection policy is tested under physically constrained command tracking rather than idealized instantaneous maneuver execution, which improves the credibility of the validation for highly dynamic level-flight, turning, and climbing maneuvers.

2.3. Definition of State Space and Relative-Motion Characteristics

To quantitatively depict the close-range relative-motion state and effectively steer top-level decision-making, the state space essential for state assessment must satisfy the Markov property while reflecting the spatial geometry and kinematic states of both aircraft [26]. In the present implementation, the MDP is formulated using ideal state observations directly provided by the JSBSim-based simulation environment; wind disturbances, sensor measurement noise, and partial observability are not explicitly modeled in the state-transition process, which limits the direct extension of the current results to real-world disturbed flight conditions. Therefore, the state space vector $S$ defined in

Table 1. Definition of state space feature vectors.

Symbol	Definition & Physical Meaning
$R$	Line-of-sight range between both aircraft, used to determine whether the tracked aircraft satisfies the predefined close-range geometric condition.
$\varphi ={\cos }^{-1}\left[{\displaystyle \frac{V_A·\rho }{\left|V_A\right|\left|\rho \right|}}\right]$	Antenna train angle (ATA), the angle between the tracking aircraft’s velocity vector and the line-of-sight vector.
$q=\pi -{\cos }^{-1}\left[{\displaystyle \frac{V_T·\rho }{\left|V_T\right|\left|\rho \right|}}\right]$	Aspect angle (AA), the angle between the tracked aircraft’s velocity vector and the line-of-sight vector.
$\Delta z$	Relative altitude, used to evaluate the margin for potential energy conversion and assess the potential energy difference between the two aircraft.
$V_{\mathrm{A}}, V_{\mathrm{T}}$	Flight airspeeds of the tracking aircraft and the tracked aircraft, which determine their respective kinetic energy levels.
$n_z, n_{z_t}$	Current normal load factors of both aircraft, reflecting the intensity of their maneuvers.
$\alpha ,\beta ,\gamma$	$\alpha$, $\beta$, $\gamma$ represent the angle of attack, sideslip angle, and flight-path angle, respectively. In the local North-East-Up inertial frame, the flight-path angle is defined by $\gamma =\arcsin (V_U/\Vert V\Vert )$, where $V_U$ is the vertical component of the velocity vector and $\Vert V\Vert$ is the flight speed. The pitch angle $\theta$ describes the attitude of the aircraft body, whereas $\gamma$ describes the direction of the velocity vector; for coordinated small-sideslip flight, $\theta \approx \gamma +\alpha$.
$\begin{array}{cc}\hfill S=& \{x_{\mathrm{A}},y_{\mathrm{A}},h_{\mathrm{A}},\hfill \\ & {\psi }_{\mathrm{A}},V_{\mathrm{A}},{\varphi }_{\mathrm{A}},q_{\mathrm{T}},\hfill \\ & \Delta V,\Delta h,R\}\hfill \end{array}$	The complete state-space vector used for relative-motion state assessment.

should be interpreted as a fully observable simulation-state representation.

Leveraging these feature vectors, the system is capable of performing real-time evaluations to determine whether the current relative geometry is favorable, neutral, or challenging, thereby providing a foundational basis for guidance-mode transitions. The geometric relative-motion configuration between the two aircraft is illustrated in Figure 2. Unless otherwise stated, all positions are expressed in a local North-East-Up inertial frame $F_{\mathrm{I}}=\{O_{\mathrm{I}},x_{\mathrm{N}},y_{\mathrm{E}},z_{\mathrm{U}}\}$, where $x_{\mathrm{N}}$ points north, $y_{\mathrm{E}}$ points east, and $z_{\mathrm{U}}$ is positive upward and is equivalent to the altitude $h$ used in the figures. The tracking-aircraft body frame $F_{\mathrm{A}}$ and tracked-aircraft body frame $F_{\mathrm{T}}$ are fixed to the corresponding aircraft, with the $x_b$ axis pointing forward along the fuselage, $y_b$ pointing to the right wing, and $z_b$ completing the right-handed frame. The line-of-sight vector is defined as $\rho ={\mathrm{Pos}}_{\mathrm{T}}-{\mathrm{Pos}}_{\mathrm{A}}$, and the line-of-sight range is $R=\Vert \rho \Vert$.

From the perspective of the tracking aircraft, four representative relative-motion configurations are depicted in Figure 3.

Since long-range interaction mechanisms are outside the scope of this study, the simulation outcome is determined by a predefined close-range geometric tracking rule. Each aircraft is assigned an initial simulation score of 100. A tracked aircraft is considered to satisfy the predefined geometric tracking condition when the range is not greater than 900 m and the tracking-aircraft-side ATA is not greater than 25°. To avoid unrealistic instantaneous success judgments caused by transient entry into the predefined geometric condition, the simulation score is updated only after the tracked aircraft remains continuously inside the specified geometric region for 0.2 s. After this dwell-time threshold is satisfied, the tracked aircraft’s simulation score decreases at a rate of 50 score units per second. The aircraft whose simulation score reaches zero first is judged to have failed to maintain the required tracking condition. If both aircraft reach zero simulation score within the same simulation step, the result is recorded as a simultaneous termination case. To reduce boundary jitter, a hysteresis mechanism is adopted: after entering the specified geometric region, the tracked aircraft is regarded as remaining inside the region until the range exceeds 950 m or the ATA exceeds 30°.

2.4. Geometric Guidance Laws and High-Level Decision-Making Model

To convert the guidance-mode intents from high-level decisions into low-level executable control commands, this paper designs three typical pursuit guidance laws based on angular-maneuver geometry. Lag pursuit is used to maintain and improve favorable relative geometry, lead pursuit is employed to predict the tracked aircraft’s future position and reserve an anticipatory tracking angle, and pure pursuit acts as a rapid angular-recovery strategy under challenging conditions [27]. These strategies are selected because they correspond to three representative guidance requirements in close-range dynamic tracking: maintaining favorable relative geometry, capturing an anticipatory tracking solution, and recovering from angular misalignment. As summarized in

Table 2. Guidance purpose, applicable relative-motion condition, and limitation of the three pursuit strategies.

Strategy	Guidance Purpose	Typical Applicable Relative-Motion Condition	Possible Limitation
Lag pursuit	Maintains or enlarges angular alignment advantage while preventing overshoot by guiding the tracking aircraft toward the rear region of the tracked aircraft.	The tracking aircraft has obtained a preliminary favorable relative position or is approaching the tracked aircraft with excessive closure speed; the main objective is to remain within a favorable tracking region while avoiding crossing the tracked aircraft’s turn circle.	It may delay geometric-alignment acquisition when the tracking aircraft needs to rapidly align the nose with the tracked aircraft; in extreme challenging situations, lag pursuit alone may not provide sufficient angular recovery.
Lead pursuit	Captures an anticipatory tracking geometry by aiming ahead of the tracked aircraft and compensating for prediction delay and vertical motion effects.	The tracking aircraft is close to satisfying the predefined geometric tracking condition and needs to reserve an anticipatory angle for tracking a maneuvering object.	The current implementation uses a guidance-level first-order prediction of the tracked aircraft position; under rapidly varying three-dimensional maneuvers, higher-order trajectory-prediction and acceleration modeling may be required.
Pure pursuit	Rapidly reduces the line-of-sight angle by pointing directly toward the tracked aircraft’s current position, serving as an angular-recovery mechanism under challenging relative geometry.	The tracking aircraft is in a challenging or neutral angular position and needs to quickly regain nose-pointing direction toward the tracked aircraft.	Pure pursuit can cause excessive closure speed and may lead to overshoot or altitude loss if not coordinated with energy and safety constraints. It is not intended to replace more complex emergency recovery maneuvers.

, the three-strategy set is not intended to exhaust all possible maneuvers; rather, it provides a compact and interpretable guidance-mode action space for validating the proposed hierarchical decision framework.

2.4.1. Relative-Motion State Assessment and High-Level Decision-Making Mechanism

The PPO algorithm is adopted at the top level, with the action space defined as $A=\{a_{lag},a_{lead},a_{pure}\}$. The three actions correspond to three guidance-mode intents: $a_{lag}$ aims to maintain angular advantage and avoid overshoot, $a_{lead}$ aims to capture an anticipatory tracking geometry by aiming ahead of the tracked aircraft, and $a_{pure}$ aims to rapidly reduce the line-of-sight angle under challenging conditions. The high-level PPO does not rely on artificially established angular or distance thresholds for policy switching; instead, through environmental interaction and long-term cumulative reward maximization, it implicitly learns the mapping $\pi (a\Vert S)$ from varying relative-motion states $S$ to optimal guidance-mode actions $a$. This process can also be interpreted as online guidance-mode selection, since interaction-aware UAV decision-making studies have shown that inferring another aircraft’s motion tendency from trajectory and relative-state features is a key prerequisite for state awareness and autonomous maneuver decision-making [28]. In other words, the above guidance-mode roles are used to define the meaning of the action space, while the final switching boundary among lag, lead, and pure pursuit is learned from data rather than manually prescribed.

Based on the variations in the azimuth angle $\varphi$ and the tracked-aircraft aspect angle $q$, and following the common practice of constructing relative-motion state and reward indicators from angular, range, and altitude-related factors [29,30], the situation value $S_{\mathrm{A}}$ of the tracking UAV and the situation value $S_{\mathrm{T}}$ of the tracked UAV are evaluated as follows. Specifically, the azimuth angle reflects the nose-pointing and alignment advantage of the tracking aircraft, whereas the tracked-aircraft aspect angle reflects whether the tracking aircraft is approaching the tracked aircraft’s rear tracking region.

$$S_{\mathrm{A}}=\left\{\begin{array}{ll}{\displaystyle \frac{k{(\pi -|{\varphi }_{\mathrm{A}}|)}^2+(1-k)|q_{\mathrm{T}}{|}^2}{{\pi }^2}},& R\le d_{\mathrm{ref}}\\ {\displaystyle \frac{k{(\pi -|{\varphi }_{\mathrm{A}}|)}^2+(1-k)|q_{\mathrm{T}}{|}^2}{{\pi }^2}}\cdot e^{{\displaystyle \frac{d_{\mathrm{ref}}-R}{d_{\mathrm{ref}}}}},& R>d_{\mathrm{ref}}\end{array}\right.$$

(3)

$$S_{\mathrm{T}}=\left\{\begin{array}{ll}{\displaystyle \frac{k{(\pi -|{\varphi }_{\mathrm{T}}|)}^2+(1-k)|q_{\mathrm{A}}{|}^2}{{\pi }^2}},& R\le d_{\mathrm{ref}}\\ {\displaystyle \frac{k{(\pi -|{\varphi }_{\mathrm{T}}|)}^2+(1-k)|q_{\mathrm{A}}{|}^2}{{\pi }^2}}\cdot e^{{\displaystyle \frac{d_{\mathrm{ref}}-R}{d_{\mathrm{ref}}}}},& R>d_{\mathrm{ref}}\end{array}\right.$$

(4)

where $d_{\mathrm{ref}}$ is the maximum reference range for the predefined tracking condition, used as a geometric range reference in the situation-value and reward-shaping functions, and $k\in [0,1]$ is the weight coefficient used to balance two guidance preferences: the azimuth-angle term associated with nose-pointing accuracy and the aspect-angle term associated with tail-chase positioning. This function accounts for the correlation among azimuth-angle alignment, tracked-aircraft-aspect geometry, and the geometric range reference $d_{ref}$. When $R\le d_{ref}$, angular alignment directly contributes to the state value; when $R>d_{ref}$, an exponential range-decay factor is introduced to penalize geometrically favorable but tracking-ineffective long-range states. The larger the value of $k$, the greater the variation of the situational value relative to the azimuth angle $\left|\varphi \right|$, meaning that the reward becomes more sensitive to nose-pointing accuracy. Conversely, a smaller $k$ assigns relatively greater importance to target-aspect superiority and tail-chase positioning. A related analysis of the situation-evaluation function is shown in Figure 4.

The guidance reward function is formulated based on the state-evaluation function as follows. Compared with using only sparse terminal rewards, this shaped process reward provides dense feedback for angular alignment and effective-range acquisition, thereby guiding the PPO agent to learn meaningful guidance-mode transitions before the terminal success/failure state is reached:

$$r_{process}=\left\{\begin{array}{ll}{\displaystyle \frac{k{(\pi -|{\varphi }_{\mathrm{A}}|)}^2+(1-k)|q_{\mathrm{T}}{|}^2}{{\pi }^2}},& R\le d_{ref}\\ {\displaystyle \frac{k{(\pi -|{\varphi }_{\mathrm{A}}|)}^2+(1-k)|q_{\mathrm{T}}{|}^2}{{\pi }^2}}\cdot e^{{\displaystyle \frac{d_{ref}-R}{d_{ref}}}},& R>d_{ref}\end{array}\right.$$

(5)

Furthermore, to reinforce the agent’s awareness of the terminal simulation states, a terminal reward function is configured as follows. The terminal reward distinguishes success, simultaneous termination, and failure outcomes, while the process reward in Equation (5) shapes the intermediate guidance behavior; thus, the final reward combines geometric guidance during close-range tracking with explicit termination-state evaluation.

$$r_{end}=\left\{\begin{array}{l}10, \mathrm{success}\\ 0, \mathrm{simultaneous} \mathrm{termination}\\ -10, \mathrm{failure}\end{array}\right.$$

(6)

Thus, the ultimate reward function is given by $r=r_{\mathrm{process}}+r_{\mathrm{end}}$.

It should be emphasized that the coefficient $k$ is not updated online during PPO training. Instead, $k$ is treated as a fixed design parameter that determines the shape of the reward landscape before training starts. The surfaces in Figure 4 are used to analyze the sensitivity of the state-assessment function to different $k$ values: a smaller $k$ assigns relatively more importance to target-aspect superiority and tail-chase positioning, whereas a larger $k$ makes the reward more sensitive to azimuth-angle reduction and nose-pointing accuracy. In the present training process, a fixed value of $k$ is used, and the PPO agent adapts its guidance-mode policy to the corresponding reward landscape rather than adaptively changing $k$ during interaction.

2.4.2. Lag Pursuit Guidance Law

When the tracking aircraft has a favorable relative position, lag pursuit involves pointing the nose towards the rear of the tracked aircraft, effectively preventing an overshoot due to excessive speed while simultaneously closing the distance. Its geometric guidance core lies in reserving the reference tracking distance $d_{\mathrm{ref}}$ and designing the position command based on the coordinate transformation matrix:

$$\left\{\begin{array}{l}Po{s}_{\mathrm{cmd}}^{\mathrm{lag}}=Po{s}_{\mathrm{T}}+C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})[d_{\mathrm{ref}},0,0]\\ e_{\mathrm{p}}^{\mathrm{lag}}=Po{s}_{\mathrm{cmd}}^{\mathrm{lag}}-Po{s}_{\mathrm{A}}\end{array}\right.$$

(7)

Here, $Po{s}_{\mathrm{T}}={[x_{\mathrm{T}},y_{\mathrm{T}},h_{\mathrm{T}}]}^T$ represents the current position vector of the tracked aircraft in the local North-East-Up inertial frame. $C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})$ denotes the transformation matrix from the tracked-aircraft body frame to the local inertial frame, where ${\psi }_{\mathrm{T}}$, ${\theta }_{\mathrm{T}}$, and ${\phi }_{\mathrm{T}}$ are the yaw, pitch, and roll angles of the tracked aircraft, respectively. The flight-path angle ${\gamma }_{\mathrm{T}}$ is computed from the tracked-aircraft velocity vector and is not used as an Euler attitude angle; under coordinated small-sideslip flight, ${\theta }_{\mathrm{T}}\approx {\gamma }_{\mathrm{T}}+{\alpha }_{\mathrm{T}}$.

Upon closing the distance, a velocity tracking error is introduced for braking to ensure that the tracking aircraft does not overshoot the tracked aircraft’s turning circle. Consequently, the velocity command under the lag pursuit strategy is designed as:

$$\left\{\begin{array}{l}{e}_{\mathrm{V}}^{\mathrm{lag}}=V_{\mathrm{T}}-V_{\mathrm{A}}\\ V_{\mathrm{cmd}}^{\mathrm{lag}}=V_{\mathrm{T}}+C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})(K_{\mathrm{VP}}{e}_{\mathrm{P}}+K_{\mathrm{VD}}{e}_V)\end{array}\right.$$

(8)

The tracking UAV must not only track the velocity of the tracked aircraft $V_{\mathrm{T}}$, but also apply dynamic corrections via the feedback from the position error $e_p$ and velocity error $e_v$, with $K_{\mathrm{VP}}$ and $K_{\mathrm{VD}}$ acting as control gain coefficients.

After determining the position command and velocity command, these commands need to be converted into heading angle commands and altitude commands. Among them, the heading command is computed using the four-quadrant arctangent function to avoid quadrant ambiguity:

$${\psi }_{\mathrm{cmd}}^{\mathrm{lag}}=atan2\left({\displaystyle \frac{y_{\mathrm{cmd}}-y_{\mathrm{A}}}{x_{\mathrm{cmd}}-x_{\mathrm{A}}}}\right)$$

(9)

To avoid excessive depletion of kinetic energy during maneuvers, an altitude compensation command is formulated based on the energy maneuverability theory [31]: $\Delta h=h-h^{\prime }={\displaystyle \frac{V^2-V_{\mathrm{cmd}}^2}{2g}}$.

Consequently, the modified altitude command is: ${h^{\prime }}_{\mathrm{cmd}}^{\mathrm{lag}}=h_{\mathrm{cmd}}+\Delta h$.

2.4.3. Lead Pursuit Guidance Law

Lead pursuit is employed to establish an anticipatory tracking geometry when the predefined close-range geometric condition is satisfied. By forecasting the future position of the tracked aircraft, the tracking aircraft is guided to proactively capture the required anticipatory angle for close-range tracking. Let the reference propagation velocity be $v_{\mathrm{ref}}$ and the tracked aircraft’s velocity be $V_{\mathrm{T}}$. For computational efficiency in the guidance layer, the prediction time offset is approximately computed as:

$$t_{\mathrm{PRE}}\approx {\displaystyle \frac{R}{v_{\mathrm{ref}}+V_{\mathrm{T}}\cos (\varphi )}}$$

(10)

The relationship between the anticipated tracking point in lead pursuit $Po{s}_{\mathrm{Lead}}$ and the tracked aircraft’s position at the subsequent time step $Po{s}_{\mathrm{T}+t_{\mathrm{PRE}}}$ is defined as:

$$\begin{array}{cc}\hfill Po{s}_{\mathrm{Lead}}& =Po{s}_{\mathrm{T}+t_{\mathrm{PRE}}}-[0,0,h_g]+[0,0,h_{\alpha }]\hfill \\ & =Po{s}_{\mathrm{T}+t_{\mathrm{PRE}}}-\left[0,0,{\displaystyle \frac{1}{2}}g{t}_{\mathrm{PRE}}^2\right]+\left[0,0,d\tan \alpha \right]\hfill \end{array}$$

(11)

In the formula, $h_g$ represents the compensation for vertical deviation during the prediction interval, and $h_{\alpha }=d\tan \alpha$ denotes the axis-elevation compensation caused by the aircraft’s angle of attack. This simplified compensation improves the consistency between the lead-pursuit command and the simulated flight state, but it does not fully account for unmodeled propagation effects, tracked-aircraft jerk, or rapidly varying three-dimensional rolling maneuvers.

By integrating the tracked aircraft’s current load factor and velocity to predict its future location, the guidance command for the anticipated tracking point is generated as follows:

$$\left\{\begin{array}{l}Po{s}_{\mathrm{D}}^{\mathrm{lead}}=Po{s}_{\mathrm{T}}+C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})[d_{\mathrm{ref}},0,0]\\ e_p=Po{s}_{\mathrm{D}}^{\mathrm{lead}}-Po{s}_{\mathrm{A}}\\ Po{s}_{\mathrm{cmd}}^{\mathrm{lead}}=Po{s}_{\mathrm{Lead}}+C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})(K_p{e}_p+[d\mathrm{ref},0,0])\end{array}\right.$$

(12)

This prediction is used as a guidance-level approximation for lead-pursuit guidance rather than as a complete high-fidelity trajectory-prediction solution.

In lead pursuit, an appropriate reference tracking range from the tracked aircraft needs to be maintained. When the two aircraft approach each other, the tracking aircraft’s speed is regulated to track that of the tracked aircraft; therefore, the velocity command under the lead pursuit strategy is formulated as:

$$\left\{\begin{array}{l}{e}_{\mathrm{V}}^{\mathrm{lead}}=V_{\mathrm{T}}-V_{\mathrm{A}}\\ V_{\mathrm{cmd}}^{\mathrm{lead}}=V_{\mathrm{T}}+C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})(K_{\mathrm{VP}}{e}_{\mathrm{P}}+K_{\mathrm{VD}}{e}_V)\end{array}\right.$$

(13)

Likewise, upon deriving the position and velocity commands, they are transformed into heading angle and altitude commands that can be processed by the flight controller:

$$\left\{\begin{array}{l}{\psi }_{\mathrm{cmd}}^{\mathrm{lead}}=atan2\left({\displaystyle \frac{y_{\mathrm{cmd}}-y_{\mathrm{A}}}{x_{\mathrm{cmd}}-x_{\mathrm{A}}}}\right)\\ {h^{\prime }}_{\mathrm{cmd}}^{\mathrm{lead}}=h_{\mathrm{cmd}}+\Delta h\end{array}\right.$$

(14)

2.4.4. Pure Pursuit Guidance Law

Pure pursuit dictates that the aircraft’s nose constantly points straight at the current position of the tracked aircraft, acting as a rapid angular recovery mechanism under challenging relative-motion conditions. The guidance heading command ${\psi }_{cmd}$ is continuously modified to align with the current line-of-sight vector direction.

$${\psi }_{\mathrm{cmd}}^{\mathrm{pure}}=atan2\left({\displaystyle \frac{y_{\mathrm{T}}-y_{\mathrm{A}}}{x_{\mathrm{T}}-x_{\mathrm{A}}}}\right)$$

(15)

Flight energy is maintained by converting the velocity difference into an altitude difference. The spatial position command for pure pursuit is directly assigned to the tracked aircraft’s present location, i.e., $Po{s}_{\mathrm{D}}^{\mathrm{pure}}=Po{s}_{\mathrm{T}}$.

When separated by a large distance, the aircraft can accelerate to close in; as the distance narrows, the velocity difference predominantly takes effect to prevent the tracking aircraft’s speed from being excessive, which would cause an overshoot of the tracked aircraft. Accordingly, the velocity command under the pure pursuit strategy is designed as:

$$\left\{\begin{array}{l}Po{s}_{\mathrm{cmd}}^{\mathrm{pure}}=Po{s}_T,e_p^{\mathrm{pure}}=Po{s}_{\mathrm{cmd}}^{\mathrm{pure}}-Po{s}_A\\ e_V^{\mathrm{pure}}=V_T-V_A\\ V_{\mathrm{cmd}}^{\mathrm{pure}}=V_T+C_b^I({\psi }_{\mathrm{T}},{\theta }_{\mathrm{T}},{\phi }_{\mathrm{T}})(K_{\mathrm{VP}}{e}_p^{\mathrm{pure}}+K_{\mathrm{VD}}{e}_V^{\mathrm{pure}})\end{array}\right.$$

(16)

The design of the altitude command for the pure pursuit strategy also needs to consider the law of conservation of energy, and the altitude command is corrected to ${h^{\prime }}_{\mathrm{cmd}}^{\mathrm{pure}}=h_{\mathrm{cmd}}+\Delta h$.

3. Results

3.1. Simulation and Validation of Pursuit Guidance Laws

To validate the effectiveness of the proposed pursuit guidance laws, simulation experiments were conducted in which the tracking UAV applied different pursuit strategies while the tracked aircraft executed various maneuvers, with the initial states of both aircraft listed in

Table 3. Initial state settings of both aircraft under different pursuit strategies.

Scenario	Tracked Aircraft Maneuver	Initial Velocity ${\mathit{V}}_{\mathbf{A}},{\mathit{V}}_{\mathbf{T}}$ (m/s)	Initial Altitude ${\mathit{h}}_{\mathbf{A}},{\mathit{h}}_{\mathbf{T}}$(km)	Initial Heading ${\mathit{\psi }}_{\mathbf{A}},{\mathit{\psi }}_{\mathbf{T}}$ Angle (∘)
Lag pursuit	Level flight	244, 183	6, 6	45, 0
Lag pursuit (Tracking aircraft inside the turning circle)	Turning	304, 183	6, 6	45, 0
Lag pursuit (Tracking aircraft outside the turning circle)	Turning	304, 183	6, 6	45, 0
Lead pursuit	Turning	244, 183	6, 6	45, 0
Pure pursuit	Lag pursuit	183, 183	6, 6	0, 0

.

The corresponding simulation results are presented in Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9.

Figure 5 illustrates that during the tracked aircraft’s level flight, the tracking aircraft maintains a stable tracking distance behind it. As indicated by the state curves, the distance converges to the predetermined reference tracking range, while the altitude is dynamically regulated to conserve energy.

Under turning maneuver conditions, the lag pursuit control law reduces the risk of overshoot by directing the aircraft’s heading toward the rear region of the tracked aircraft. Whether initiated from within or outside the tracked aircraft’s turn circle, the tracking aircraft converges to a steady tracking state, suggesting the effectiveness of the position and velocity commands derived from the coordinate transformation matrix and energy maneuverability theory.

Lead pursuit is employed when the tracking aircraft approaches a favorable anticipatory tracking geometry and needs to aim ahead of the tracked aircraft. Figure 8 demonstrates that the tracking aircraft can move toward a favorable lead-pursuit geometry by forecasting the tracked aircraft’s future location based on the prediction time offset and current maneuvering state.

Figure 7. Maneuver trajectory and states under lag pursuit with the tracking aircraft initially inside the turning circle. (a) A 3D spatial representation of the flight paths during the simulation. (b) Time-series plots of critical parameters.

Figure 7. Maneuver trajectory and states under lag pursuit with the tracking aircraft initially inside the turning circle. (a) A 3D spatial representation of the flight paths during the simulation. (b) Time-series plots of critical parameters.

Figure 8. Maneuver trajectory and states under lead-pursuit guidance. (a) A 3D spatial representation of the flight paths during the simulation. (b) Time-series plots of critical parameters.

Figure 8. Maneuver trajectory and states under lead-pursuit guidance. (a) A 3D spatial representation of the flight paths during the simulation. (b) Time-series plots of critical parameters.

Figure 9. Maneuver trajectory and states under pure-pursuit guidance. (a) A 3D spatial representation of the flight paths during the simulation. (b) Time-series plots of critical parameters.

Figure 9. Maneuver trajectory and states under pure-pursuit guidance. (a) A 3D spatial representation of the flight paths during the simulation. (b) Time-series plots of critical parameters.

When the tracking aircraft is in a challenging relative position, pure pursuit is used for angular recovery. Figure 9 indicates that the tracking aircraft is guided to aim directly at the present position of the tracked aircraft, swiftly diminishing the line-of-sight angle, while its velocity is effectively regulated upon closing in on the tracked aircraft to prevent an early overshoot.

3.2. Convergence Evaluation of the Hierarchical Decision Framework

The proposed hierarchical decision framework adopts a multi-frequency coupling mechanism, in which the interaction frequency between the underlying JSBSim environment and the control network is set to 60 Hz to ensure the continuity of nonlinear aerodynamic calculations and the stability of attitude control, whereas the top-level PPO guidance-mode decision frequency is set to 5 Hz. In this architecture, the 5 Hz decision frequency corresponds to a guidance-mode update period of 0.2 s, while the mid-level guidance laws and low-level flight controller continuously update the executable heading, altitude, velocity, and actuator commands at a higher frequency. This configuration prevents command oscillations triggered by high-frequency guidance-mode switches, allowing the UAV ample time to execute the selected pursuit intent [32]. Since lag pursuit, lead pursuit, and pure pursuit are guidance-mode intents rather than actuator-level commands, excessively high-frequency switching may lead to unstable intent alternation without improving physical maneuver execution. The representative simulation analyzed in Section 3.4 further shows that the agent can switch from pure pursuit to lag pursuit and then to lead pursuit in response to changes in angular alignment, indicating that the 5 Hz guidance-mode update can support the current simulated close-range tracking settings.

A conventional continuous-control PPO baseline, which directly outputs low-level control commands, is used for the convergence comparison in Section 3.2. To ensure a fair comparison, the proposed high-level guidance-mode PPO and the conventional PPO baseline use the same network architecture and training hyperparameters, as summarized in

Table 4. Shared network architecture and training hyperparameters of the high-level guidance-mode PPO and the conventional continuous-control PPO baseline.

Hyperparameter	Value
Number of hidden layers in the critic network	3
Hidden-layer sizes of the critic network	{512, 256, 128}
Optimizer of the critic network	Adam
Learning rate of the critic network	3 × 10−4
Hidden-layer sizes of the actor network	{256, 256, 128}
Optimizer of the actor network	Adam
Learning rate of the actor network	3 × 10−4
Number of parallel environments	32
Discount factor γ	0.99
Buffer size	512
Entropy coefficient	1 × 10−3
GAE parameter λ	0.95

. The training process was conducted over a total of 4 million steps across 32 parallel environments. All PPO algorithms were implemented on a Windows operating system using Python 3.8.1, PyTorch 2.1.2, Gym 0.20.0, and JSBSim package version 1.2.3. The hardware platform consisted of an Intel^® Core^TM i5-13600KF CPU, an NVIDIA GeForce RTX 4070 Super GPU, and 64 GB of RAM. The curve illustrating the variation in cumulative reward per episode is presented in Figure 10.

Figure 10 shows that incorporating the guidance-mode generation layer improves both the convergence rate of the training process and the final cumulative reward under the current training setting. This improvement is attributed to the reduction of the high-level exploration space from continuous low-level control commands to three interpretable pursuit-strategy intents, while the mid-level guidance laws preserve continuous trajectory-generation capability. Therefore, the high-level agent performs policy learning in a compact three-action guidance-mode space rather than in a continuous or high-dimensional maneuver-control space.

The comparison in this subsection focuses on the benefit of the pursuit-strategy action abstraction over direct low-level continuous control, whereas Section 3.3 further compares different high-level learning algorithms under the same pursuit-strategy action space.

3.3. Comparison Between PPO and Double-DQN for High-Level Guidance-Mode Decision-Making

To further evaluate whether the performance gain originates from the proposed high-level guidance-mode decision mechanism rather than from the low-level flight controller, an additional cross-algorithm comparison was conducted under the same hierarchical framework. In this comparison, the proposed PPO policy was compared with a Double-DQN baseline. Both methods used the same pursuit-strategy action space, namely lag pursuit, lead pursuit, and pure pursuit, and both shared the same pre-trained low-level flight controller. Therefore, the comparison focuses on the learning algorithm used for high-level guidance-mode selection while keeping the executable guidance laws and JSBSim flight-dynamics loop unchanged.

During evaluation, the counterpart aircraft follows a fixed rule-based pursuit-switching policy rather than a learning-based policy. This rule-based counterpart shares the same 16-dimensional high-level observation vector, the same three pursuit-strategy action space, the same mid-level guidance laws, and the same pre-trained low-level PPO flight controller as the learning agent. The only difference is that its high-level guidance mode is selected by hand-coded geometric rules instead of a neural network. Specifically, at each $5 \mathrm{Hz}$ high-level decision step, the rule-based baseline selects lead pursuit when the range satisfies $R\le d_{\mathrm{ref}}$, the absolute azimuth angle satisfies $\left|\varphi \right|\le {15}^{°}$, and the absolute tracked-aircraft aspect angle satisfies $\left|q\right|\le {60}^{°}$. It switches to pure pursuit when $\left|\varphi \right|\ge {75}^{°}$ or $\left|q\right|\ge {120}^{°}$ and defaults to lag pursuit otherwise. The selected pursuit intent is then converted into heading, altitude, and velocity commands by the same mid-level guidance laws and executed by the same low-level flight controller. Therefore, the rule-based counterpart provides a deterministic and reproducible geometric evaluation reference, while the observed performance difference between PPO and Double-DQN mainly reflects the learned high-level decision policy rather than differences in low-level control capability. To avoid mixing this cross-algorithm setting with the convergence experiment in Section 3.2, the algorithm-specific hyperparameters used for the PPO–Double-DQN comparison are listed separately in Appendix A.2,

Table A2. Algorithm-specific hyperparameters for the PPO–Double-DQN comparison.

Group	Parameter	Proposed PPO	Double-DQN
Algorithm-specific setting	Implementation	Stable-Baselines3 PPO	Custom PyTorch Double-DQN
	Learning paradigm	On-policy policy gradient	Off-policy value-based learning
	Learning rate	3 × 10−4	3 × 10−4
	Discount factor γ	0.99	0.99
	Batch size	256	256
	Network architecture	SB3 MlpPolicy, 64 × 64	Q-network, 256 × 256 with ReLU
	Exploration mechanism	Entropy regularization	ε-greedy
	Replay buffer	Not used	200,000 transitions
	Target network	Not used	Updated every 2000 steps
PPO-specific setting	Rollout length n_steps	1024	--
	Effective rollout size	16 × 1024 = 16,384	--
	Number of epochs	10	--
	GAE parameter λ	0.95	--
	Clip range	0.3	--
	Entropy coefficient	3 × 10−4	--
Double-DQN-specific setting	Learning starts	--	10,000 transitions
	Training frequency	--	Every 4 environment steps
	Gradient clipping	--	10.0
	ε schedule	--	1.0 to 0.05
	ε decay length	--	500,000 transitions

.

All evaluation episodes were conducted under the simplified close-range simulation setting and the simulation-score-based geometric termination rule defined in Section 2.3. The rule-based baseline followed the same pursuit-switching policy, and the initial simulation scenarios were sampled from the four representative posture categories used in the Monte Carlo evaluation. Each method was trained for 1 million transitions and evaluated over 200 episodes. In addition to the conventional success, failure, and simultaneous-termination rates, crash-aware metrics were reported. In particular, the non-crash success rate counts only the episodes in which the learning agent achieves success through the predefined geometric tracking condition, thereby excluding passive success cases caused by opponent crashes.

Table 5. Cross-algorithm comparison under the simulation-score-based geometric termination rule.

Method	Avg. Reward	Success (%)	Non-Crash Success (%)	Failure (%)	Simultaneous Termination (%)	Learning-Agent Crash (%)
Double-DQN	634.98	57.0	34.5	29.0	14.0	20.5
Proposed PPO	827.35	83.5	67.0	8.5	8.0	1.0

summarizes the comparison results. Compared with Double-DQN, the proposed PPO achieves a higher success rate, increasing from 57.0% to 83.5% under the same pursuit-strategy action space and simulation-score-based geometric termination rule. The non-crash success rate increases from 34.5% to 67.0%, indicating that the proposed PPO is more capable of maintaining a favorable geometric tracking condition. Meanwhile, the learning-agent crash rate decreases from 20.5% to 1.0%, suggesting that the policy-gradient-based high-level decision model learns a more stable pursuit-strategy switching policy under the nonlinear JSBSim flight-dynamics constraints.

These results indicate that, under the same hierarchical pursuit-strategy action abstraction, PPO provides more stable high-level guidance-mode decision-making than the value-based Double-DQN baseline. The improvement is particularly evident in the non-crash success rate and learning-agent crash rate, which provide additional information beyond the raw success rate by separating active geometric-condition success cases from crash-induced outcomes.

3.4. Analysis of Representative Dynamic-Tracking Trajectories

To further illustrate the decision-making mechanism of the agent in simulated close-range tracking scenarios, this section analyzes a representative simulation episode. As observed in Figure 11, starting from an initially neutral state, both aircraft perform turning maneuvers to avoid entering a challenging angular position.

As observed in Figure 11, from an initially balanced configuration, both aircraft perform turning maneuvers to avoid potentially challenging postures. During the first 10–20 s of the simulation, the rule-based aircraft temporarily obtains a slight angular-alignment advantage. In response to this relative-motion state, the high-level network issues a “pure pursuit” command to increase the angular correction rate. After the 20-s mark, the tracked aircraft loses its relative angular-alignment advantage owing to a degraded turn rate, and the network autonomously transitions to “lag pursuit” to reduce the relative distance. At approximately 34 s, the azimuth angle decreases to below 15°, and the policy switches to lead pursuit to approach a favorable anticipatory tracking geometry. Around 44 s, the predefined geometric tracking condition in Section 2.3 is satisfied, leading to a successful simulation-score-based termination of the rule-based counterpart.

3.5. Results of Monte Carlo Randomized Testing

To reduce the influence of random initial conditions and assess the algorithm’s generalization capability under the current simulation assumptions, this study employs the Monte Carlo method to conduct 200 randomized simulation tests across four categories of initial configurations. The duration of each simulation episode is capped at 512 high-level decision steps. An episode is recorded as a success or failure when one aircraft’s simulation score reaches zero under the simulation-score-based geometric termination rule, or when one aircraft violates the flight-safety termination conditions while the other aircraft remains within the valid flight envelope. If neither aircraft reaches a terminal condition before the maximum step count is reached, the episode is recorded as a maximum-step termination case. If both aircraft reach a zero simulation score or violate the termination condition within the same simulation step, the result is treated as a simultaneous termination case. The statistical results for the four representative initial configurations are summarized in Figure 12.

The statistical data show that, under the predefined simulation rules, the ideal state-observation assumption, and the JSBSim-based fixed-wing aircraft flight-dynamics model, the hierarchical decision-making method obtains stable simulated performance across the four initial configuration categories. Notably, in scenarios with an initially favorable relative configuration, the success rate reaches 91.5% under the current test settings. An analysis of several simulation trajectories indicates that some simultaneous termination outcomes are associated with both aircraft violating the low-altitude safety condition before the maximum step limit is reached. This phenomenon suggests that, in some challenging recovery situations, the current reward design may overemphasize altitude-to-speed conversion and angular recovery while insufficiently penalizing unsafe low-altitude states. This limitation does not negate the effectiveness of the proposed hierarchical action-space design, because such unsafe terminations are counted as non-success outcomes in the Monte Carlo statistics; however, it indicates that future versions of the reward function should introduce more explicit safety constraints, such as altitude-margin rewards, ground-collision penalties, and safety-aware termination conditions.

4. Discussion and Conclusions

To address the high exploration burden of continuous action spaces and the rigidity of traditional discrete maneuver primitives in close-range dynamic tracking, this paper proposes a hierarchical reinforcement-learning decision-making method based on a pursuit-strategy action space. By decoupling low-level flight-dynamics control from high-level guidance-mode selection, the proposed framework enables closed-loop intelligent maneuvering decision-making under nonlinear flight-dynamics constraints.

(1) A dimension-reduced action space for close-range dynamic-tracking decision-making is proposed. The complex six-degree-of-freedom close-range relative-motion problem is abstracted into three geometric guidance modalities: lag pursuit, lead pursuit, and pure pursuit. The simulation results indicate that, compared with a baseline that directly outputs low-level continuous control commands, the proposed method reduces the exploration burden and improves both the convergence rate and the achieved cumulative reward under the current training setting.

(2) A high-fidelity hierarchical closed-loop execution mechanism is established. Based on the geometric mechanism of close-range dynamic tracking and energy maneuverability theory, mid-level guidance laws, including velocity compensation and altitude compensation, are designed for the three pursuit strategies, and low-level error tracking is realized using the JSBSim high-fidelity flight-dynamics model. This framework mitigates the latency and limitations inherent in traditional discrete strategy-level maneuvers when tracking highly dynamic aircraft.

(3) Improved simulated tracking performance is observed across representative initial configurations. Monte Carlo randomized simulation tests demonstrate that, under the predefined simulation assumptions and simulation-score-based geometric termination criteria, the proposed decision-making model achieves a success rate of up to 91.5% in initially favorable configurations; even under initially challenging configurations, the agent obtains a 64.0% success rate through adaptive switching among pursuit strategies. These results support the effectiveness of the proposed hierarchical strategy within the current simulation scope, while broader validation under incomplete observations, environmental disturbances, multiple random seeds, high-dynamic maneuvers, and more diverse comparative policies remains necessary.

This study provides a hierarchical decision-making framework for UAV close-range dynamic tracking that balances the exploration efficiency of reinforcement learning with the physical constraints of flight control. It should be noted that the current Monte Carlo evaluation is conducted under predefined simulation assumptions, without explicitly modeling incomplete observations, wind disturbances, or sensor noise. Future work will further relax these assumptions by incorporating incomplete information, sensor noise, wind disturbances, multi-seed statistical evaluation, safety-constrained altitude penalties, ground-collision penalties, higher-order trajectory-prediction models, and additional recovery and energy-management maneuvers. In particular, maneuvers such as coordinated turn, vertical recovery, altitude-exchange maneuver, roll-based recovery, and energy-recovery guidance modes can be introduced into an expanded strategy action space to improve recovery robustness under low-altitude or strongly challenging tracking conditions. The extension of this hierarchical framework to multi-UAV cooperative and interactive dynamic-tracking scenarios will also be investigated.