Autonomous Maneuvering Decision-Making Method for Unmanned Aerial Vehicle Based on Soft Actor-Critic Algorithm

Highlights

What are the main findings?

• Proposed a continuous control strategy based on the Soft Actor-Critic (SAC) algorithm for UAV autonomous maneuvering in 1v1 tactical encounter scenarios.
• Introduced a multi-dimensional situation-coupled reward function with a Health Point (HP) system to quantitatively evaluate situational advantages and cumulative tactical performance.

What are the implications of the main findings?

• Enhanced decision-making precision and robustness under both ideal and noisy observation conditions.
• Demonstrated the effectiveness of SAC in handling high-dimensional state spaces and generating precise maneuvering commands.

Abstract

Focusing on continuous action space methods for autonomous maneuvering decision making in 1v1 unmanned aerial vehicle scenarios, this paper first establishes a UAV kinematic model and a decision-making framework under the Markov Decision Process. Second, a continuous control strategy based on the Soft Actor-Critic (SAC) reinforcement learning algorithm is developed to generate precise maneuvering commands. Then, a multi-dimensional situation-coupled reward function is designed, introducing a Health Point (HP) metric to assess situational advantages and simulate cumulative effects quantitatively. Finally, extensive simulations in a custom Gym environment validate the effectiveness of the proposed method and its robustness under both ideal and noisy observation conditions.

1. Introduction

Advances in artificial intelligence are facilitating the intelligent evolution of unmanned systems. Unmanned aerial vehicles (UAVs) are extensively employed in both offensive/defensive maneuvering and collaborative missions [1]. With significant enhancements in UAV maneuverability, close-range decision making between UAVs is characterized by high dynamics and high overload, imposing rigorous requirements on the real-time performance and flexibility of UAV autonomous maneuvering decision-making systems [2]. Currently, three primary approaches address UAV close-range maneuver decision making: game theory, optimization search, and data-driven methods [3]. Game theory methods encompass differential games and matrix games. While both models are continuous, dynamic autonomous aerial maneuvering decision processes that derive analytical solutions, they struggle with high-dimensional state spaces [4]. Optimization search methods formulate autonomous aerial maneuvering objectives as multi-objective optimization problems, employing techniques like heuristic search and expert systems [5]. However, their performance is constrained by search space limitations and rule design complexity. Data-driven methods, including Deep Learning (DL) and Deep Reinforcement Learning (DRL), generate maneuver policies autonomously by learning from UAV maneuvering data. However, DL approaches are limited by the quality of training data used in behavioral cloning. In contrast, DRL typically requires minimal external data, acquiring experience autonomously through agent–environment interaction. This capability demonstrates potential for identifying advantageous maneuvers, making DRL a prominent research focus globally.

However, current deep reinforcement learning (DRL) research on UAV close-range maneuvering decision making primarily employs discrete action space methods, such as Deep Q-Network (DQN) [6], Actor-Critic (AC) [7], or Proximal Policy Optimization (PPO) [8]. The fixed action space inherently restricts the precision of generated maneuver commands. Studies by Yang et al. [9] and Gao et al. [10] leverage continuous action space algorithms (Deep Deterministic Policy Gradient (DDPG) and Twin Delayed Deep Deterministic Policy Gradient (TD3), respectively) to enhance command accuracy. Nevertheless, both of their simulation victory conditions rely on an oversimplified premise when the friendly side enters the enemy’s attack zone and meets the weapon launch conditions. This assumption neglects battlefield uncertainties, specifically weapon hit probability fluctuations that cause damage outcome randomness, thereby compromising the robustness validation of the proposed strategies.

To address these limitations, this paper employs a multidimensional situation-coupled reward function and the Soft Actor-Critic (SAC) algorithm to generate maneuver decision commands in continuous action spaces. To mitigate algorithmic randomness, the UAV’s Health Points are introduced as an indicator of situational advantage. Furthermore, simulation experiments validate the robustness of the SAC algorithm’s decisions under fully observable inputs or conditions with observational errors.

This study investigates reinforcement learning-based autonomous maneuver decision-making methods applicable to close-range UAV pursuit and evasion scenarios. The main works are as follows:

• This paper completed the modeling of UAV kinematics and the engagement scenario, as well as the construction of a discrete maneuver command library.
• This paper implemented SAC deep reinforcement learning within a continuous action space using a multidimensional coupled reward function to generate maneuver decision commands.
• This paper established a Gym simulation environment to validate the robustness of SAC/TD3 reinforcement learning algorithms under both error-free and erroneous input state observations.

2. Problem Formulation

Figure 1 shows the decision-making framework for 1v1 UAV close-range maneuvering, comprising the environment, UAV kinematic models, and maneuvering decision models for both sides. The environment feeds real-time states and situational data to the decision models, which generate maneuver commands executed kinematically to update UAV states and environmental dynamics, closing the perception–decision–action loop.

2.1. UAV Maneuvering Model

Focusing on UAV maneuver decision making, this paper prioritizes positional/trajectory dynamics over attitude change mechanisms. Consequently, a point-particle model is adopted for kinematic and dynamic representations, as follows:

$$\left\{\begin{array}{c}{\dot{x}}_g=vcos\theta cos\psi \hfill \\ {\dot{y}}_g=vcos\theta sin\psi \hfill \\ {\dot{z}}_g=vsin\theta \hfill \\ \dot{v}=g(n_x-sin\theta )\hfill \\ \dot{\theta }={\displaystyle \frac{g}{v}}(n_{z}cos\gamma -cos\theta )\hfill \\ \dot{\psi }={\displaystyle \frac{g{n}_{z}sin\gamma }{vcos\theta }}\hfill \end{array}\right.$$

(1)

where $\theta$ denotes the pitch angle, defined as the angle between the body longitudinal axis (currently aligned with the velocity vector) and the horizontal plane $x_{g}O{y}_g$; $\psi$ represents the yaw angle, which is the angle between the projection of the body longitudinal axis onto the horizontal plane $x_{g}O{y}_g$ and the axis $O{x}_g$; and $\gamma$ is the roll angle (see Figure 2).

The variables $n_x$ and $n_z$ correspond to the tangential overload (aligned with the velocity vector) and the normal overload (perpendicular to the velocity vector within the longitudinal symmetry plane of the aircraft), respectively. The velocity components in the ground frame $O{x}_g{y}_g{z}_g$ are denoted by ($\dot{x}$, $\dot{y}$, $\dot{z}$), $\dot{v}$ is the path acceleration, and $\dot{\theta }$ and $\dot{\psi }$ are the angular rates of the pitch and yaw angles, respectively.

The gravitational acceleration is represented by g, and the dynamic model presented in Equation (1) enforces the following constraints on the state and control variables, as shown in the following Equation (2). While this simplified 3-DOF point-mass model exhibits certain limitations—such as not fully incorporating real-world elements like actuator dynamics (e.g., servo response delays), control latency, or environmental disturbances—it still holds significant value in the initial exploration of autonomous maneuvering decisions. This is particularly true in task scenarios requiring the isolation of underlying dynamic complexities to focus on high-level logic and algorithmic interpretability.

$$\left\{\begin{array}{c}{v}_{min}⩽v⩽v_{max}\hfill \\ -{\displaystyle \frac{\pi }{2}}⩽\theta ⩽{\displaystyle \frac{\pi }{2}}\hfill \\ -\pi ⩽\psi ⩽\pi \hfill \\ n_{x min}⩽n_x⩽n_{x max}\hfill \\ n_{z min}⩽n_z⩽n_{z max}\hfill \\ -\pi ⩽\gamma ⩽\pi \hfill \end{array}\right.$$

(2)

2.2. Autonomous Aerial Maneuvering Environment

The autonomous aerial maneuvering environment computes Health Points (HPs), relative states, and geometric situation angles from bilateral UAV states in (1), providing reference inputs for the maneuver decision-making part. Through situation assessment, it quantifies bilateral advantages.

Figure 3 depicts key situational information in autonomous maneuvering, where red and blue denote friendly and enemy UAVs, respectively. From the friendly UAV’s perspective, the line-of-sight ($\mathbf{LOS}$) is defined as the vector pointing from the friendly to the enemy position; $\phi \in [0,\pi ]$ represents the Antenna Train Angle (ATA), measuring the angular deviation between the friendly velocity vector ${\mathbf{v}}_r$ and the $\mathbf{LOS}$ direction; $q\in [0,\pi ]$ denotes the Aspect Angle (AA), quantifying the enemy’s escape potential as the angle between the enemy velocity vector ${\mathbf{v}}_b$ and the reverse $\mathbf{LOS}$ direction. The parameters ${\phi }_{max}$ and $q_{max}$ define the angular constraints for the Weapon Engagement Zone (WEZ) and No Escape Zone (NEZ), while $D_{min}$ and $D_{max}$ specify the radial boundaries of these engagement zones.

$$\left\{\begin{array}{cc}\hfill {\phi }_r=& arccos{\displaystyle \frac{{\mathbf{v}}_r·\mathbf{LOS}}{\parallel {\mathbf{v}}_r\parallel ·\parallel \mathbf{LOS}\parallel }}\hfill \\ \hfill q_r=& arccos{\displaystyle \frac{{\mathbf{v}}_b·\mathbf{LOS}}{\parallel {\mathbf{v}}_b\parallel ·\parallel \mathbf{LOS}\parallel }}\hfill \end{array}\right.$$

(3)

To comprehensively quantify the cumulative impact of adverse situational factors during dynamic interactions between two parties, a Health Point (HP) metric is introduced, denoted as $H_r$ for the friendly side and $H_b$ for the enemy side. Lower HP values indicate greater accumulated performance loss due to sustained exposure to threat conditions. The simulation terminates when either side’s HP drops to ≤0, exceeds airspace limits, or reaches the maximum simulation steps K.

The HP update mechanism is governed by real-time relative geometric relationships, specifically including $\phi$, q, and inter-UAV distance d. The following Equation (4) defines the HP update rule:

$$H_k(H_{k-1},\phi ,q,d)=\left\{\begin{array}{cc}{H}_{k-1}-\Delta H_{max}{\displaystyle \frac{f_{\mathrm{att}}\left(\phi \right)+f_{\mathrm{esp}}\left(q\right)+f_{\mathrm{dist}}\left(d\right)}{3}},\hfill & \phi \in [0,{\phi }_{max}]\cap d\in [d_{min},d_{max}]\hfill \\ H_{k-1},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(4)

where $H_k$ represents current health at step k, $H_{k-1}$ denotes previous health, and $\Delta H_{max}$ is the maximum possible health deduction.

To accurately model the contribution degree of a single interaction to HP attenuation, three normalized attenuation factors determine the HP reduction magnitude. The ATA angle factor $f_{\mathrm{att}}$ follows a sigmoid response curve: ${\displaystyle f_{\mathit{att}}}$ represents the threat level based on the Antenna Train Angle (ATA). Smaller ${\displaystyle \phi }$ values indicate better alignment between the velocity vector and the line of sight of our own UAV, thereby enhancing attack accuracy. The sigmoid function models the nonlinear increase in threat as ${\displaystyle \phi }$ approaches zero, mimicking real sensor sensitivity where minor angular deviations significantly affect weapon efficacy. The parameter ${\displaystyle k_{\phi }}$ controls the steepness of the curve, tuned according to empirical data from UAV engagements.

$$f_{\mathrm{att}}=1-{\displaystyle \frac{1}{1+e^{-k_{\phi }(\phi -0.5{\phi }_{max})}}}$$

(5)

The AA angle factor $f_{\mathrm{esp}}$ implements piecewise-linear attenuation: ${\displaystyle f_{\mathit{esp}}}$ quantifies the threat level from our side to the enemy’s escape potential via the Aspect Angle (AA). Smaller ${\displaystyle q}$ values (e.g., below ${\displaystyle {20}^{\circ }}$) indicate that the enemy is not fleeing directly, making it more vulnerable to attack; thus ${\displaystyle f_{\mathit{esp}}}$ attains higher values (closer to 1). The piecewise-linear design reflects tactical realism: at lower ${\displaystyle q}$, our advantage is maximized; as ${\displaystyle q}$ increases, uncertainty in enemy maneuvers increases; at higher ${\displaystyle q}$, the threat diminishes as the enemy escapes.

$$f_{\mathrm{esp}}\left(q\right)=\left\{\begin{array}{cc}1,\hfill & q\le {20}^{\circ }\hfill \\ 1-k_{q_1}(q-20)/70,\hfill & q\in [{20}^{\circ },{90}^{\circ }]\hfill \\ 0.2-k_{q_2}(q-90)/60,\hfill & q\in [{90}^{\circ },{150}^{\circ }]\hfill \\ 0.05,\hfill & q>{150}^{\circ }\hfill \end{array}\right.$$

(6)

The distance factor $f_{\mathrm{dist}}$ employs exponential decay centered at optimal engagement distance: ${\displaystyle f_{\mathit{dist}}}$ captures the effect of distance on weapon efficacy. Maximum damage occurs near ${\displaystyle d_{\mathit{opt}}}$, with rapid decay outside this range due to aerodynamic and sensor limitations. The exponential decay aligns with physical weapon characteristics, such as reduced hit probability at extreme distances. The parameters ${\displaystyle k_1}$, ${\displaystyle k_2}$, ${\displaystyle p}$, and ${\displaystyle r}$ are calibrated to fit the engagement envelope of the UAV’s cannon.

$$f_{\mathrm{dist}}\left(d\right)=\left\{\begin{array}{cc}{e}^{-k_1{\left({\displaystyle \frac{d_{\mathrm{opt}}-d}{d_{\mathrm{opt}}-d_{\min }}}\right)}^p},\hfill & d\in [d_{\min },d_{\mathrm{opt}}]\hfill \\ e^{-k_2{\left({\displaystyle \frac{d-d_{\mathrm{opt}}}{d_{\max }-d_{\mathrm{opt}}}}\right)}^r},\hfill & d\in [d_{\mathrm{opt}},d_{\max }]\hfill \end{array}\right.$$

(7)

Opponent health updates analogously using the same equations but computed from the enemy’s perspective (${\phi }_b$). Three normalized attenuation factors in Equation (4) are shown in Figure 4. Simulation termination and outcome determination are governed by a conditional logic based on real-time Health Points (HPs), step progression, and operational constraints. The simulation terminates under one of the following scenarios: (1) Timeout: if the maximum step count K is reached without a decisive outcome, resulting in a timeout; (2) Constraint violation: if airspace boundaries are breached, such as altitude limits (${\displaystyle h<h_{min}}$ or ${\displaystyle h>h_{max}}$) or distance thresholds (${\displaystyle R>D_{th}}$ or ${\displaystyle R<R_{min}}$), leading to a penalty for operational breach; (3) Crash: if Health Points of either side drop to zero or below, resolving into a win (when ${\displaystyle H_r>0}$ and ${\displaystyle H_b⩽0}$), loss (when ${\displaystyle H_r⩽0}$ and ${\displaystyle H_b>0}$), or draw (when both ${\displaystyle H_r⩽0}$ and ${\displaystyle H_b⩽0}$). Otherwise, if none of these conditions are met, the simulation continues.

2.3. Opponent Maneuver Policy

The blue team applies Minimax search [11] with discrete actions. We expanded NASA’s seven basic maneuvers [12] into 27 action elements (see

Table 1. Basic maneuver primitive instruction set.

Action ID	Maneuver Name	Maneuver Primitive Command $\left[{\mathit{n}}_{\mathit{x}},{\mathit{n}}_{\mathit{z}},\mathit{\gamma }\right]$
1	Constant Speed Forward	$\left[0,1,0\right]$
2	Accelerated Forward	$\left[n_x^{max},1,0\right]$
3	Decelerated Forward	$\left[n_x^{min},1,0\right]$
4	Constant Speed Right Turn	$\left[0,n_z^{max},-{cos}^{-1}(1/n_z^{max})\right]$
5	Accelerated Right Turn	$\left[n_x^{max},n_z^{max},-{cos}^{-1}(1/n_z^{max})\right]$
6	Decelerated Right Turn	$\left[n_x^{min},n_z^{max},-{cos}^{-1}(1/n_z^{max})\right]$
7	Constant Speed Left Turn	$\left[0,n_z^{max},{cos}^{-1}(1/n_z^{max})\right]$
8	Accelerated Left Turn	$\left[n_x^{max},n_z^{max},{cos}^{-1}(1/n_z^{max})\right]$
9	Decelerated Left Turn	$\left[n_x^{min},n_z^{max},{cos}^{-1}(1/n_z^{max})\right]$
10	Constant Speed Climb	$\left[0,n_z^{max},0\right]$
11	Accelerated Climb	$\left[n_x^{max},n_z^{max},0\right]$
12	Decelerated Climb	$\left[n_x^{min},n_z^{max},0\right]$
13	Constant Speed Dive	$\left[0,n_z^{min},0\right]$
14	Accelerated Dive	$\left[n_x^{max},n_z^{min},0\right]$
15	Decelerated Dive	$\left[n_x^{min},n_z^{min},0\right]$
16	Constant Speed Right Climb	$\left[0,n_z^{max},-{cos}^{-1}(c/n_z^{max})\right]$
17	Accelerated Right Climb	$\left[n_x^{max},n_z^{max},-{cos}^{-1}(c/n_z^{max})\right]$
18	Decelerated Right Climb	$\left[n_x^{min},n_z^{max},-{cos}^{-1}(c/n_z^{max})\right]$
19	Constant Speed Left Climb	$\left[0,n_z^{max},{cos}^{-1}(c/n_z^{max})\right]$
20	Accelerated Left Climb	$\left[n_x^{max},n_z^{max},{cos}^{-1}(c/n_z^{max})\right]$
21	Decelerated Left Climb	$\left[n_x^{min},n_z^{max},{cos}^{-1}(c/n_z^{max})\right]$
22	Constant Speed Right Dive	$\left[0,n_z^{min},-{cos}^{-1}(c/n_z^{max})\right]$
23	Accelerated Right Dive	$\left[n_x^{max},n_z^{min},-{cos}^{-1}(c/n_z^{max})\right]$
24	Decelerated Right Dive	$\left[n_x^{min},n_z^{min},-{cos}^{-1}(c/n_z^{max})\right]$
25	Constant Speed Left Dive	$\left[0,n_z^{min},{cos}^{-1}(c/n_z^{max})\right]$
26	Accelerated Left Dive	$\left[n_x^{max},n_z^{min},{cos}^{-1}(c/n_z^{max})\right]$
27	Decelerated Left Dive	$\left[n_x^{min},n_z^{min},{cos}^{-1}(c/n_z^{max})\right]$

), assigning acceleration, constant-speed, and deceleration overload commands to each of the nine principal directions in three-dimensional space, where $c>1$ denotes a constant parameter used for compound maneuvers.

The Minimax policy is an algorithmic prescription for perfect-information zero-sum games, guaranteeing the minimization of the maximum possible loss under the premise of an optimally counteracting opponent, and thus the maximization of the minimum payoff. Figure 5 illustrates the computational process of the Minimax decision-making policy executed by the adversarial blue UAV, through which it selects the current action that maximizes the projected situational advantage over a k-step time horizon.

In this single-round game-theoretic framework, both agents sequentially select elementary maneuvers from the maneuver action primitive library (Table 1) to be executed over k consecutive time steps (arrows in Figure 5). The middle column nodes represent the payoffs obtained after the blue agent executes one action choice, while the rightmost column nodes denote the payoffs acquired after the red agent (opponent) completes its decision making. When the red agent’s maneuvering action results in the most unfavorable position for the blue side at time k in the future, the algorithm selects the child node with the smallest payoff in the rightmost column for each parent node in the middle column. The blue agent’s action selection should then correspond to the node with the largest payoff in the middle column, thereby maximizing its minimum positional advantage at time k in the future.

3. Methodologies

This section details the proposed reinforcement learning maneuver decision method, encompassing state/action space design, reward function formulation, and the SAC decision-making approach.

3.1. State Space and Action Space

This paper adopts a Markov Decision Process (MDP) framework under the assumption of fully observable states. The state input s is defined as a 27-dimensional continuous variable (see Equation (8)), including internal friend states $s_{\mathrm{own}}$ from Equation (1), friend Health Point $H_{\mathrm{r}}$, and friend previous action in the last step $a_{t-1}$, alongside sensor-derived observations such as bilateral relative states $s_{\mathrm{rel}}$ and relevant geometric angles $s_{\mathrm{geo}}$.

$$\begin{array}{cc}\hfill & s=\left\{s_{\mathrm{own}},s_{\mathrm{rel}},s_{\mathrm{geo}},a_{t-1}\right\}\hfill \\ \hfill & s_{\mathrm{own}}=\left\{x_r,y_r,z_r,v_r,{\theta }_r,{\psi }_r,H_r\right\}\hfill \\ \hfill & s_{\mathrm{rel}}=\left\{\Delta x,\Delta y,\Delta z,R_h,R,v_c,\Delta \dot{x},\Delta \dot{y},\Delta \dot{z},\Delta v\right\}\hfill \\ \hfill & s_{\mathrm{geo}}=\left\{{\phi }_r,q_r,{\theta }_L,{\psi }_L,{\theta }_{VL},{\psi }_{VL},\mathrm{HCA}\right\}\hfill \\ \hfill & a_{t-1}=\left\{n_x,n_z,\gamma \right\}\hfill \end{array}$$

(8)

Here, $R_h$ denotes the $\mathbf{LOS}$ onto the horizontal plane, while R represents the magnitude of the $\mathbf{LOS}$ (i.e., the three-dimensional distance between the two parties). The variable $v_c$ refers to the closure velocity between both sides, as defined in Equation (18). The terms $\delta x$, $\delta y$, and $\delta z$ correspond to the components of the difference between the opponent’s position vector and our position vector along the $O_x$, $O_y$, and $O_z$ axes, respectively. Similarly, $\Delta \dot{x}$, $\Delta \dot{y}$, and $\Delta \dot{z}$ denote the components of the difference between the opponent’s velocity vector and our velocity vector along the same axes.

The angle ${\theta }_L$ indicates the angular deviation between the $\mathbf{LOS}$ and the horizontal plane $O_{xg}$, while ${\psi }_L$ is the azimuth angle of the projection of the $\mathbf{LOS}$ onto the $O{x}_g{y}_g$ plane relative to the $O{x}_g$ axis. Additionally, ${\theta }_{VL}$ describes the angle between the projection of the friend velocity vector and the projection of the $\mathbf{LOS}$ onto the $y_{g}O{z}_g$ plane, while ${\psi }_{VL}$ denotes the angle between the friend velocity vector and the projection of the $\mathbf{LOS}$ onto the $x_{g}O{y}_g$ plane. Finally, HCA (Heading Crossing Angle) refers to the angular separation between the velocity vectors of the opponent and our own side. All these state components are normalized to $s_{\mathrm{in}\in \left[-1,1\right]}$ before being fed into the fully connected network (shown in Equation (9)), and ${\delta }_{\mathrm{own}},{\delta }_{\mathrm{rel}},{\delta }_{\mathrm{geo}}$ is taken as the constant.

$$\begin{array}{cc}\hfill & \delta =\left\{{\delta }_{\mathrm{own}},{\delta }_{\mathrm{rel}},{\delta }_{\mathrm{geo}}\right\}\hfill \\ \hfill & {\delta }_{\mathrm{own}}=\left\{{\delta }_x,{\delta }_y,{\delta }_z,{\delta }_v,\pi ,2\pi ,{\delta }_{\mathrm{HP}}\right\}\hfill \\ \hfill & {\delta }_{\mathrm{rel}}=\left\{{\delta }_{\Delta x},{\delta }_{\Delta y},{\delta }_{\Delta z},{\delta }_{R_h},{\delta }_R,v_{cmax},{\delta }_{\Delta \dot{x}},{\delta }_{\Delta \dot{y}},{\delta }_{\Delta \dot{z}},{\delta }_{\Delta v}\right\}\hfill \\ \hfill & {\delta }_{\mathrm{geo}}=\left\{2\pi ,2\pi ,\pi ,2\pi ,2\pi ,\pi ,2\pi \right\}\hfill \\ \hfill & {\delta }_a=\left\{n_{x max},n_{z max},{\gamma }_{max}\right\}\hfill \\ \hfill & s_{\mathit{in}}=\left\{tanh\left(s_i/{\delta }_i\right)\right\},i\in \left[1,27\right]\hfill \end{array}$$

(9)

We define a continuous 3-dimensional bounded action space for the friendly agent, parameterized as $\left[n_x,n_z,{\gamma }_v\right]$ and subject to the constraint in Equation (2). This enables direct, precise control over the aircraft’s states, while the opponent’s actions are limited to a discrete maneuver library of finite commands (see Table 1).

3.2. Reward Function Design

Effective reward functions critically impact the convergence speed and quality of reinforcement learning. Traditional designs typically use isolated angle, speed, altitude, and distance metrics with minimal consideration of coupling, potentially compromising situational assessment accuracy and maneuver precision. This paper designs an expert-guided multi-dimensional coupled dense reward [13], including the relative position reward $R_{\mathit{relpos}}$, the closure speed reward $R_{\mathit{closure}}$, the altitude reward $R_{\mathit{altitude}}$, the firing rewards $R_{\mathit{ownfire}}$ and $R_{\mathit{enemyfire}}$ based on bilateral health changes, and event rewards (attack advantage reward $R_{cz}$, termination reward $R_{\mathit{end}}$).

$$\begin{array}{cc}& R_{\mathit{relpos}}\left({\overline{\phi }}_r,{\overline{q}}_r\right)=2\left[\left({\overline{\phi }}_r-1\right)·S\left({\overline{q}}_r,{\alpha }_1,{\displaystyle \frac{1}{2}}\right)-{\overline{\phi }}_r+{\displaystyle \frac{1}{2}}\right]\hfill \end{array}$$

(10)

$$R_{\mathit{closure}}\left({\overline{q}}_r,v_c,d\right)=\left\{\begin{array}{c}{\displaystyle \frac{v_c}{v_{c max}}}\left[1-S\left({\overline{q}}_r,{\alpha }_1,{\displaystyle \frac{1}{2}}\right)\right]·S\left(d,{\alpha }_2,d_0\right),d⩾d_0\hfill \\ -1,d<d_0\hfill \end{array}\right.$$

(11)

$$R_{\mathit{altitude}}\left(h_r\right)=\left\{\begin{array}{cc}S\left(h_r,{\alpha }_3,h_{min}\right)-1\hfill & \hfill ,h_r<h_0\\ -S\left(h_r,{\alpha }_4,h_{max}\right)\hfill & \hfill ,h_r⩾h_0\end{array}\right.$$

(12)

$$\begin{array}{c}\hfill R_{\mathit{ownfire}}\left({\overline{\phi }}_r,d\right)={\mathrm{\Gamma }}_R\left(d\right)·\left[1-S\left({\overline{\phi }}_r,{\alpha }_5,{\overline{\phi }}_{max}\right)\right]\end{array}$$

(13)

$$\begin{array}{c}\hfill R_{\mathit{enemyfire}}\left({\overline{q}}_r,d\right)=-{\mathrm{\Gamma }}_B\left(d\right)·S\left({\overline{q}}_r,{\alpha }_8,{\overline{q}}_{max}\right)\end{array}$$

(14)

$$R_{\mathit{end}}=\left\{\begin{array}{c}200+200\left({\displaystyle \frac{K-k}{K}}\right),\mathit{if}\mathit{result}==\mathit{win}\hfill \\ +400,\mathit{if}\mathit{result}==\mathit{lose}\mathit{or}\mathit{balance}\hfill \\ -600,\mathit{if}\mathit{result}==\mathit{timeout}\mathit{or}\mathit{beyond}\mathit{the}\mathit{boundaries}\hfill \\ 0,others\hfill \end{array}\right.$$

(15)

$$R_{\mathit{cz}}=\left\{\begin{array}{c}+25,{\phi }_r⩽{\phi }_{max},q_r⩽q_{max},D_{min}⩽d⩽D_{max}\hfill \\ 0,else\hfill \end{array}\right.$$

(16)

Formulas (10)–(14) are dense reward functions. Here, S denotes the sigmoid function (17). As illustrated in Figure 6, $\alpha$ represents the rate of change of the sigmoid function at $x=x_0$. It is obvious that a larger $\alpha$ indicates a more pronounced variation in the sigmoid curve at $x=x_0$, rendering it more sensitive to perturbations in x. Conversely, a smaller $\alpha$ results in a less sensitive response, meaning that the function exhibits a weaker dependence on changes in x. Constants include ${\alpha }_1\sim {\alpha }_{10}$, which are not arbitrarily set but are determined based on expert prior knowledge from [13]. Moreover, these parameters are not unique; they can be adjusted within a reasonable range as long as the sigmoid function effectively captures the intended trends in the relevant dimensions, such as the sensitivity to angular or distance variations.

The remaining constants include ${\overline{\phi }}_r$ and ${\overline{q}}_r$ (normalized $\phi$ and q for red), plus optimal attack distance $d_0$ and flight altitude $h_0$. $v_{\mathit{c}}$ is the inter-UAV approach speed, ${\mathrm{\Gamma }}_R\left(d\right)$ and ${\mathrm{\Gamma }}_B\left(d\right)$ are reward coefficients during own fire and enemy attacks, and $\Vert {\beta }_R\Vert$ and $\Vert {\beta }_B\Vert$ are offensive strategy aggressiveness for red and blue teams.

$$\begin{array}{c}\hfill S\left(x,\alpha ,x_0\right)={\displaystyle \frac{1}{1+e^{-\alpha \left(x-x_0\right)}}}\end{array}$$

(17)

$$\begin{array}{c}\hfill v_c={\displaystyle \frac{\left({\mathbf{v}}_{\mathbf{r}}-{\mathbf{v}}_{\mathbf{b}}\right)·\left({\mathbf{o}}_{\mathbf{b}}-{\mathbf{o}}_{\mathbf{r}}\right)}{\parallel {\mathbf{o}}_{\mathbf{b}}-{\mathbf{o}}_{\mathbf{r}}\parallel }}\end{array}$$

(18)

$${\mathrm{\Gamma }}_R\left(d\right)=\left\{\begin{array}{cc}{\beta }_R·S\left(d,{\alpha }_6,D_{min}\right)\hfill & \hfill ,d<d_0\\ {\beta }_R·\left[1-S\left(d,{\alpha }_7,D_{max}\right)\right]\hfill & \hfill ,d⩾d_0\end{array}\right.$$

(19)

$${\mathrm{\Gamma }}_B\left(d\right)=\left\{\begin{array}{cc}{\beta }_B·S\left(d,{\alpha }_9,D_{min}\right)\hfill & \hfill ,d<d_0\\ {\beta }_B·\left[1-S\left(d,{\alpha }_{10},D_{max}\right)\right]\hfill & \hfill ,d⩾d_0\end{array}\right.$$

(20)

The relative position reward function $R_{\mathit{relpos}}\left({\overline{\phi }}_r,{\overline{q}}_r\right)$ incentivizes our aircraft to maintain a tail-chase position behind the enemy, maximizing tactical advantage. The variables ${\overline{\phi }}_r$ and ${\overline{q}}_r$ are normalized versions of ${\phi }_r$ and $q_r$, respectively, defined as ${\overline{\phi }}_r={\phi }_r/\pi$ and ${\overline{q}}_r=q_r/\pi$, scaling them to the range $[0,1]$. As illustrated in Figure 7, the reward function exhibits a rapid decay at ${\overline{q}}_r=0.5$ (corresponding to the Aspect Angle $q=\pi /2$), which serves as a critical threshold: for ${\overline{q}}_r<0.5$, our aircraft is behind the enemy with a high advantage, while at ${\overline{q}}_r=0.5$, it is at the beam position; and for ${\overline{q}}_r>0.5$, the scenario transitions to a head-on engagement where enemy threat escalates and our positional superiority diminishes, often resulting in penalized rewards.

Furthermore, the function incorporates directional alignment constraints via ${\overline{\phi }}_r$; when ${\overline{\phi }}_r>0.5$ (indicating a substantial misalignment between our velocity vector and the line-of-sight, implying the aircraft is turning away from the target), the reward value decreases to penalize deviations from precise orientation and sustained pursuit, thereby indirectly incentivizing the agent to maintain optimal tracking behavior and ensuring robust multi-faceted situation assessment. This integrated approach enhances decision-making realism by simultaneously accounting for both positional and orientational factors.

The closure speed reward function $R_{\mathit{closure}\left({\overline{q}}_r,v_c,d\right)}$ incentivizes our agent to approach the enemy UAV from the rear to maximize tactical advantage, while penalizing cases where the enemy aircraft approaches from behind our own. Its magnitude scales with distance to adaptively balance aggression and safety: larger rewards at greater distances encourage a proactive approach, while diminishing rewards at closer ranges mitigate overshooting risks. When $d<d_0$, the reward remains constant negative to prevent collision or overshooting from excessive proximity—here, $d_0$ denotes the optimal engagement distance: though closer proximity may enhance attack effectiveness, it significantly heightens crash/overshoot risks due to limited maneuvering space.

The design rationale for this function is grounded in operational constraints of UAV combat dynamics. Specifically, the sigmoid function parameters, such as ${\displaystyle \alpha =0.0125}$ for ${\displaystyle S\left(d,{\alpha }_2,d_0\right)}$, are derived from the UAV’s optimal gun engagement zone, typically within ${\displaystyle \left[D_{min},D_{max}\right]}$ (e.g., 500${\displaystyle \sim }$1200 m, as illustrated in Figure 8). This parameter selection ensures a gradual decline in reward values at the optimal engagement distance ${\displaystyle d_0}$, promoting smooth transitions in UAV distance and velocity. The gradual decay prevents abrupt behavioral changes, enhancing training stability and real-world applicability by avoiding erratic maneuvers. Thus, the reward function effectively balances encouragement and penalty mechanisms, aligning with physical constraints and ensuring robust policy convergence. This approach not only reinforces safe engagement practices but also leverages expert knowledge to optimize performance in dynamic aerial environments (see Figure 9).

The altitude reward function $R_{\mathit{altitude}}\left(h_r\right)$ is designed to constrain our UAV to maneuver within a specified altitude range $h_r\in \left[h_{min},h_{max}\right]$, mitigating risks at excessively high altitudes (where thin air may cause engine aerodynamic failure and degraded control efficiency) and excessively low altitudes (where increased drag and crash hazards prevail). Here, $h_0$ denotes the optimal flight altitude. Through the synergistic design of parameters ${\alpha }_3$ and ${\alpha }_4$, the reward function ensures a smooth transition in reward values as altitude deviates from $h_0$, mitigating abrupt changes and avoiding excessive height fluctuations caused by velocity spikes. This design stabilizes the UAV’s altitude within the target range, balancing safety with operational flexibility (see Figure 10).

The firing reward function $R_{\mathit{ownfire}}\left({\overline{\phi }}_r,d\right)$ and $R_{\mathit{enemyfire}}\left({\overline{q}}_r,d\right)$ are designed to guide our agent to gain significant offensive advantages while avoiding enemy fire damage. It encourages our side to achieve the smallest possible target Aspect Angle within a certain attack range, thereby maximizing the fire damage inflicted on the enemy. Here, $d_0$ denotes the optimal weapon engagement distance, with $D_{min}$ and $D_{max}$ defining the minimum and maximum bounds of this optimal range.

${\mathrm{\Gamma }}_R\left(d\right)$ and ${\mathrm{\Gamma }}_B\left(d\right)$ model the distance-dependent variation of our and the enemy’s fire damage reward coefficients, respectively. The constants ${\beta }_R$ and ${\beta }_B$ quantify the offensive intent intensity of our and the enemy’s UAVs, where larger absolute values signify stronger aggressiveness. Notably, we assume symmetric intent strengths ($\parallel {\beta }_R\parallel =\parallel {\beta }_B\parallel$) to ensure balanced evaluation. As illustrated in Figure 11b, conservative design assigns the enemy a slightly larger engagement range (via parameters ${\alpha }_8,{\alpha }_9$) versus our UAV (via ${\alpha }_6,{\alpha }_7$), motivating our agent to devise superior maneuver strategies.

Event-based rewards comprise two fundamental components: the attack advantage reward $R_{\mathit{cz}}$ (refer to Equation (16)) and the termination reward $R_{\mathit{end}}$ (refer to Equation (15)). Specifically, $R_{\mathit{cz}}$ is designed to incentivize the agent to engage the enemy from within its no-escape zone, thereby maximizing offensive efficiency by promoting optimal positioning and tactical dominance.

Conversely, $R_{\mathit{end}}$ functions as an environmental bonus awarded at the conclusion of each episode, contingent upon the final outcome and simulation context. This reward mechanism aims to motivate our side to accomplish missions rapidly and reliably within constrained time frames. For victorious scenarios, $R_{\mathit{end}}$ allocates a base reward of +200, augmented by an additional bonus of up to +200 that is proportional to the remaining simulation time (where K denotes the maximum steps per episode and k represents the termination step). This time-sensitive incentive encourages early decisive successes by rewarding expedient victories. In cases of loss or draw, a fixed penalty of −400 is enforced to discourage adverse outcomes, whereas timeout or boundary violations incur a stricter penalty of −600 to preclude deadlock situations and rule infractions.

This hierarchically structured reward design ensures that the reward signals acquired by the agent are intrinsically aligned with the dynamic complexities of real-world aerial combat. Consequently, the derived decision-making policy maintains high operational efficiency and reliability under stringent physical constraints, including platform dynamics and airspace limitations, thereby enhancing practical combat effectiveness.

To ensure the robustness and tactical effectiveness of the reward function, we adopted an incremental constructive design strategy driven by domain expert insights [14] and simulation failure analysis. Initially, a baseline reward set ${\displaystyle \{R_{\mathrm{relpos}}, R_{\mathrm{closure}}, R_{\mathrm{altitude}}, R_{\mathrm{ownfire}}, R_{\mathrm{cz}}, }$$R_{\mathrm{end}}\}$ was established. Preliminary experiments indicated that although this configuration achieved basic convergence, it exhibited poor tactical efficiency; specifically, the agent frequently entered a mutual tail-chasing Nash equilibrium [15]. To break these deadlocks, the weights of $R_{\mathrm{relpos}}$ and $R_{\mathrm{closure}}$ were increased to 2–3 times their initial values to emphasize rapid approach and positional advantage acquisition.

However, despite improved positioning, the agent exhibited a critical vulnerability: it frequently sustained high damage from enemy fire due to a lack of defensive awareness. To address this, the $R_{\mathrm{enemyfire}}$ term was integrated, which significantly enhanced survivability. Subsequently, observations revealed a ‘delayed aiming’ phenomenon where the agent failed to lock onto the target quickly enough, allowing the enemy to fire first in head-on engagements. Consequently, to accelerate convergence and improve aiming precision, the alignment penalty term ${\displaystyle R_{{\overline{\phi }}_r}=-{\overline{\phi }}_r}$ was introduced. Concurrently, other weights were fine-tuned: ${\omega }_{\mathrm{ownfire}}$ was increased to enhance firing-driven convergence, and ${\omega }_{\mathrm{relpos}}$ was slightly reduced to avoid over-reliance on situational angular advantages, which could cause stagnation. This step-wise integration balanced tactical priorities with training stability, ultimately yielding the final robust configuration.

The overall reward function $R_{\mathrm{total}}$ with the final weights ${\omega }_1$ to ${\omega }_6$ satisfies ${\sum }_{i=1}^6{\omega }_i=1$, normalizing the dense rewards to prevent any single component from dominating. Additionally, a scaling factor $K>1$ is applied to amplify the total reward magnitude, mitigating saturation and stabilizing gradient updates during training.

$$R_{\mathit{total}}=K\left({\omega }_1{R}_{{\overline{\phi }}_r}+{\omega }_2{R}_{\mathit{relpos}}+{\omega }_3{R}_{\mathit{closure}}+{\omega }_4{R}_{\mathit{altitude}}+{\omega }_5{R}_{\mathit{ownfire}}+{\omega }_6{R}_{\mathit{enemyfire}}\right)+R_{\mathit{cz}}+R_{\mathit{end}}$$

(21)

3.3. Policy Network Training Method

Unlike traditional strategy optimization, the core innovation of SAC incorporates maximum entropy into optimization to maximize expected cumulative reward while maintaining action distribution entropy of policy $\pi$ in states, i.e.,

$$\begin{array}{c}\hfill \mathit{J}\left(\pi \right)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t\left(r_t+\alpha \mathcal{H}\left(\pi \left(·\mid s_t\right)\right)\right)\right]\end{array}$$

(22)

Among them, the temperature coefficient $\alpha$ dynamically adjusts (Equation (23)) by optimizing the target entropy ($\overline{\mathcal{H}}=-dim\left(\mathcal{A}\right)$), balancing the exploration–exploitation tradeoff of the SAC strategy, significantly enhancing training stability in uncertain environments, and mitigating the local optima risk.

$$\begin{array}{c}\hfill \mathit{J}\left(\alpha \right)={\mathbb{E}}_{{\mathbf{a}}_{\mathbf{t}}\sim {\pi }_t}\left[-\alpha log{\pi }_t\left({\mathbf{a}}_{\mathbf{t}}\mid {\mathbf{s}}_{\mathbf{t}}\right)-\alpha \overline{\mathcal{H}}\right]\end{array}$$

(23)

SAC exhibits significant advantages over other policy gradient (PG) and Actor-Critic (AC) algorithms. The maximum entropy objective facilitates more robust exploration, avoiding the premature convergence often seen in traditional PG methods. Compared with deterministic policies like those in DDPG, SAC’s stochastic policy with entropy regularization maintains a better exploration–exploitation balance, adapting to various environments without extensive hyperparameter tuning. Furthermore, the use of dual Q-networks reduces the overestimation bias common in AC methods, leading to more stable and reliable learning. These characteristics make SAC highly effective for continuous control tasks, such as UAV maneuvering, where precision and robustness are paramount.

SAC adopts the Actor-Critic architecture (Figure 12), comprising two soft Q networks (${\theta }_1,{\theta }_2$), two target Q networks (${\overline{\theta }}_1,{\overline{\theta }}_2$), and one actor network ($\varphi$). The critic optimizes the Q networks by minimizing the Bellman residual MSE loss (Equation (24)) and employs a dual Q mechanism to take the minimum Q value (Equation (25)), thereby mitigating Q-value overestimation. To handle the non-differentiability of random actions, the actor uses a reparameterization trick to convert action sampling into a deterministic linear transformation with random noise (Equation (26)), rendering the policy loss differentiable (Equation (27)).

$$\begin{array}{cc}\hfill {\mathit{J}}_Q\left({\theta }_t\right)=& {\mathbb{E}}_{\left({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}}\right)\sim \mathcal{D}}\left[{\displaystyle \frac{1}{2}}{\left(Q_{{\theta }_i}\left({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}}\right)-\left(r\left({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}}\right)+\gamma {\mathbb{E}}_{s_{t+1}\sim p}\left[V_{\overline{\theta }}\left({\mathbf{s}}_{\mathbf{t}+\mathbf{1}}\right)\right]\right)\right)}^2\right]\hfill \end{array}$$

(24)

$$\begin{array}{c}\hfill Q_{\theta }\left({\mathbf{s}}_{\mathbf{t}},{\mathbf{a}}_{\mathbf{t}}\right)=min\left\{Q_{{\overline{\theta }}_1}\left({\mathbf{s}}_{\mathbf{t}+\mathbf{1}},{\mathbf{a}}_{\mathbf{t}+\mathbf{1}}\right),Q_{{\overline{\theta }}_2}\left({\mathbf{s}}_{\mathbf{t}+\mathbf{1}},{\mathbf{a}}_{\mathbf{t}+\mathbf{1}}\right)\right\}\end{array}$$

(25)

$$\begin{array}{c}\hfill {\mathbf{a}}_{\mathbf{t}}=f_{\varphi }\left({\epsilon }_t;{\mathbf{s}}_{\mathbf{t}}\right)=\mu +\sigma ·{\epsilon }_t\end{array}$$

(26)

$$\begin{array}{c}\hfill {\mathit{J}}_{\mathcal{\beta }}\left(\varphi \right)={\mathbb{E}}_{{\mathbf{s}}_{\mathbf{t}}\sim \mathcal{D},{\epsilon }_t\sim \mathcal{N}}\left[\alpha log\left({\pi }_{\varphi }\left(f_{\varphi }\left({\epsilon }_t;{\mathbf{s}}_{\mathbf{t}}\right)\mid {\mathbf{s}}_{\mathbf{t}}\right)\right)-Q_{\theta }\left({\mathbf{s}}_{\mathbf{t}},f_{\varphi }\left({\epsilon }_t;{\mathbf{s}}_{\mathbf{t}}\right)\right)\right]\end{array}$$

(27)

4. Simulation and Results

The simulation environment was built on OpenAI Gym [16], a widely adopted benchmark for reinforcement learning research. For the training of agents, we utilized the stable-baselines3 library [17] (based on the PyTorch framework (version 2.6.0)). To ensure a comprehensive comparison and evaluation of performance, we implemented and trained agents using the Soft Actor-Critic (SAC) and Twin Delayed DDPG (TD3) algorithms, respectively. The neural network architecture for both algorithms consisted of four fully connected layers with ReLU activation functions.

Each training episode lasted 200 s with a 10 Hz decision frequency, and the total training was conducted over 50,000 episodes. For policy evaluation, pre-trained policy network weights were loaded to ensure that fixed strategies were used, and assessments were performed under randomized initial conditions (including both advantageous and parity scenarios). The evaluation was divided into two distinct phases: First, we assessed the clean performance of the trained policies under nominal conditions without any perturbations. Then, in the second phase, observational noise was injected into the state inputs to test the robustness of the policies.

4.1. Simulation Parameters and Initialization Conditions Settings

In the scenario, the blue UAV represents the opponent and is initialized with a random heading at a randomized altitude within a specified range. A local East–North–Up (END) coordinate frame is established at the blue’s initial position. Representing our own agent, the red UAV is then initialized relative to the blue UAV within a defined rear cone sector. Its initial position is constrained by a specific $\mathbf{LOS}$ geometric bearing ${\psi }_{Lr}$, $\mathbf{LOS}$ elevation angle ${\theta }_{Lr}$, and distance range R, with its heading ${\psi }_r$ facing toward the blue UAV within a certain angular tolerance. The initial states information and simulation parameters of the red and blue UAVs are listed in

Table 2. Initial state information.

State	x (km)	y (km)	z (km)	v (m/s)	$\mathit{\theta }$ (rad)	$\mathit{\psi }$ (rad)	$\mathit{HP}$	${\mathit{\theta }}_{\mathit{L}}$	${\mathit{\psi }}_{\mathit{L}}$
Blue	0	0	$[4,5]$	$[180,200]$	0	$[-\pi ,\pi ]$	100	-	-
Red	-	-	-	$[180,200]$	0	$[{\displaystyle \frac{\pi }{3}},{\displaystyle \frac{2\pi }{3}}]$	100	$[-{\displaystyle \frac{\pi }{6}},-{\displaystyle \frac{\pi }{6}}]$	$[{\displaystyle \frac{\pi }{3}},{\displaystyle \frac{2\pi }{3}}]$

and

Table 3. Simulation parameter settings.

Parameter	Value	Parameter	Value	Parameter	Value
$v_{min}$	0.25 Ma	$v_{max}$	1 Ma	${\delta }_{x,y,z}$	1 × 104
${\theta }_{min}$	$-\pi /2$	${\theta }_{max}$	$\pi /2$	${\delta }_v$	1 × 103
$n_{xmin}$	$-1$	$n_{xmax}$	2	${\delta }_{\mathrm{HP}}$	200
$n_{zmin}$	$-3$	$n_{zmax}$	6	$d_0$	0.9 km
$D_{min}$	0.5 km	$D_{max}$	1.2 km	$h_0$	5.5 km
${\phi }_{max}$	$15deg$	$q_{max}$	$30deg$	${\alpha }_{1,2}$	25
$h_{min}$	3 km	$h_{max}$	8 km	${\alpha }_{3,4}$	0.01
$v_{cmax}$	2 Ma	$z_{min}$	1 km	${\alpha }_5$	500
$z_{max}$	10 km	lr	2.5 × 10−4	${\alpha }_{6,7}$	1/10
batch	256	buffer	1 × 106	${\alpha }_8$	1 × 103
$\gamma$	0.99	${\alpha }_{9,10}$	1/15

, respectively.

4.2. Results Analysis

The training is performed on a desktop computer equipped with 12th Gen Intel(R) Core(TM) i5-12600KF and NVIDIA GeForce RTX 4070 SUPER GPU. We utilize eight encapsulated environments for parallel simulation and experience collection, thereby accelerating policy optimization and reducing training time. Two representative DRL algorithms are selected for comparison: SAC and TD3, which are state-of-the-art, single-agent DRL algorithms that support continuous control.

We conducted five independent training sessions for both the SAC and TD3 algorithms, with each session comprising 50,000 episodes. As illustrated in Figure 13, the left panel presents the training performance of SAC, where the five thin curves denote the average reward per 100 episodes (smoothed via Savitzky–Golay filtering) from each of the five runs, the thick curve represents the smoothed mean reward across these five runs, and the shaded area corresponds to the 95% confidence interval (i.e., ±1.96 standard deviations from the mean curve); similarly, the right panel presents the training performance of TD3. It can be observed that both SAC and TD3 algorithms converge within a limited number of training episodes, indicating that the reward function design is universally applicable to both methods in this scenario. However, the convergence speed of TD3 is significantly slower than that of SAC, with SAC converging at around 15,000 episodes and TD3 converging after nearly 25,000 episodes.

We evaluated the performance of the policies from the SAC and TD3 methods over 300 episodes each. During the first 100 episodes, the input states were fully observable and noise-free. In the subsequent 200 episodes, we introduced zero-mean multivariate independent Gaussian noise with a diagonal covariance matrix into the input states. Specifically, for 100 episodes, we applied low-level noise (with smaller standard deviations). For the remaining 100 episodes, we applied high-level noise (with larger standard deviations), using distinct values for each dimension. To ensure comparability across experiments, the same random seed was used throughout all evaluations to maintain consistent initial conditions, and for each policy, identical noise was applied at the corresponding episode and simulation step. The evaluation results are summarized in

Table 4. Evaluation comparison between SAC and TD3 with statistical significance.

Evaluation Comparison	Win Rate (95% CI)	Loss Rate (95% CI)	Tie Rate (95% CI)	Mean Returns	Variance of Returns	One-Sided Win Rate p-Value (SAC > TD3)
SAC with no noise input	$87\%\pm 6.6\%$	$1\%\pm 2.0\%$	$12\%\pm 6.4\%$	266.5532	380.5362	$p=0.0226$
TD3 with no noise input	$76\%\pm 8.4\%$	$8\%\pm 5.3\%$	$16\%\pm 7.2\%$	138.5664	530.2239	–
SAC with low noise input	$85\%\pm 7.0\%$	$1\%\pm 2.0\%$	$14\%\pm 6.8\%$	250.4452	420.4957	$p=0.0385$
TD3 with low noise input	$75\%\pm 8.5\%$	$8\%\pm 5.3\%$	$17\%\pm 7.4\%$	133.1989	545.2348	–
SAC with high noise input	$84\%\pm 7.2\%$	$0\%$	$16\%\pm 7.2\%$	263.4485	450.7286	$p=0.0203$
TD3 with high noise input	$72\%\pm 8.8\%$	$6\%\pm 4.7\%$	$22\%\pm 8.1\%$	72.4408	591.0407	–

.

Statistical analysis of the results in Table 4 demonstrates the significant superiority of the SAC policy over the TD3 baseline, validated by one-sided hypothesis tests ($\alpha =0.05$). The SAC agent achieved statistically significantly higher win rates across all conditions: no noise (p = 0.0226), low noise (p = 0.0385), and high noise (p = 0.0203). Multiple performance metrics support this conclusion—SAC maintained substantially higher mean returns (e.g., 266.55 vs. 138.57 under no noise) while simultaneously achieving lower loss rates ($1\%\pm 2.0\%$ vs. $8\%\pm 5.3\%$) and reduced return variance (380.54 vs. 530.22), indicating enhanced stability and robustness in noisy environments. These findings statistically validate the theoretical advantages of SAC’s maximum entropy framework in handling high-dimensional state spaces and sensor uncertainties, providing empirical evidence for its efficacy in a continuous action space for UAV autonomous maneuvering decisions.

Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, Figure 21, Figure 22, Figure 23, Figure 24, Figure 25, Figure 26, Figure 27, Figure 28, Figure 29, Figure 30, Figure 31, Figure 32, Figure 33, Figure 34, Figure 35 and Figure 36 show the simulation trajectories of the SAC and TD3 algorithms under different initial situational conditions. Among them, Figure 14, Figure 15 and Figure 16 correspond to the initial condition where our side is in a tail-chase scenario with an altitude advantage; Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21 represent the initial condition where our side is in a tail-chase scenario but at an altitude disadvantage; Figure 22, Figure 23 and Figure 24 depict the initial condition where our side is in a tail-chase scenario with altitude parity; Figure 25, Figure 26, Figure 27, Figure 28, Figure 29 and Figure 30 illustrate the head-on initial situation with our side having an altitude advantage; and Figure 31, Figure 32, Figure 33, Figure 34, Figure 35 and Figure 36 portray the head-on initial situation with our side at an altitude disadvantage. In these figures, the pentagram denotes the trajectory starting point (corresponding to $T_1$), and the solid circle indicates the endpoint.

In each scenario depicted in Figure 14 and Figure 15, both sides start from identical initial conditions. The SAC algorithm successfully hits the target regardless of the presence of observational noise, whereas the TD3 algorithm fails to fully defeat the target when subjected to higher noise levels. As illustrated in Figure 16, during the interval from T2 to T3 (i.e., simulation time 60 s to 120 s), the red side gains a significant relative positional and range advantage, thereby securing a brief firing opportunity. However, from T3 to T4, due to a sharp left-turn maneuver executed by the blue side, the noise likely introduces substantial errors into the red side’s observational inputs. This causes the TD3 policy to fail in executing appropriate maneuvers correctly, ultimately resulting in deviation from the intended trajectory and a crash beyond the minimum altitude boundary. This outcome underscores TD3’s inherent sensitivity to observational noise due to its deterministic policy design, rather than reflecting neural network overfitting; in contrast, SAC’s maximum entropy framework maintains stability by promoting exploratory actions under similar uncertain conditions.

Based on Figure 17, Figure 18 and Figure 19, it can be observed that, under the initial condition of tail-chase altitude disadvantage, the SAC method consistently maintains both positional and angular advantages in tail pursuit, thereby securing firing opportunities. In contrast, as shown in Figure 20 and Figure 21, the performance of the TD3 policy in this scenario is less favorable. From Figure 21, it can be seen that, during the period from T1 to T6 (the first 500 s of simulation), the red side generally holds a superior posture. However, starting at T6, when the blue side executes a circular maneuver toward the upper right, the TD3 policy fails to continue tracking the opponent’s movement. Instead, it adopts a relatively conservative climbing maneuver, leading to an immediate decline in its situational advantage. Although by time T8, its relative positional advantage improves, the policy still opts for damage avoidance, ultimately allowing the blue side to overtake and gain dominance.

Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21 compare the trajectory trends of the SAC and TD3 algorithms under an initial tail-chase advantage for our agent. When our agent has an initial altitude advantage, the SAC policy in Figure 14 exhibits lower average standard deviations of maneuver commands (with standard deviations of 0.947 for $n_x$, 1.196 for $n_z$, and 1.093 for $\gamma$) compared with the TD3 policy in Figure 15 (with standard deviations of 1.469 for $n_x$, 1.172 for $n_z$, and 1.516 for $\gamma$). Furthermore, in terms of Energy Consumption (measured by control effort $\sum \left(n_x^2+n_z^2\right)$ in a trajectory) and Control Smoothness (measured by the mean absolute difference of command signals), SAC achieves significantly lower values of 3486.13 and 0.2002, respectively, whereas TD3 records higher values of 8043.52 and 0.2881. Based on the quantitative analysis of these three metrics, it is evident that the SAC algorithm generates trajectories that are not only statistically more stable but also physically smoother and more energy efficient.

Similarly, when our agent is at an initial altitude disadvantage (Figure 17), the SAC policy maintains low metric values across all dimensions: standard deviations of 0.879, 1.055, and 1.397 for $n_x$, $n_z$, and $\gamma$, with an Energy Consumption of 9272.8 and a Control Smoothness of 0.4289. In contrast, the TD3 policy (Figure 20) shows degraded performance, characterized by elevated standard deviations (1.282, 4.036, 2.356 for $n_x$, $n_z$, and $\gamma$), a substantial energy cost of 18935.0633, and a poor smoothness of 1.5675.

It can be observed that, in complex environments with introduced observational noise, SAC-generated trajectories exhibit significantly smoother characteristics and a markedly higher mission success probability than TD3. SAC demonstrates consistently lower control signal variances, superior Control Smoothness, and lower Energy Consumption: its lower variances highlight robust generalization against noise-induced disturbances, and its enhanced smoothness and efficiency validate physically smoother, more energy-conservative trajectory generation. Conversely, TD3 exhibits higher variances, inferior smoothness, and reduced energy efficiency. The consistently lower values across these metrics confirm SAC’s robust generalization against noise-induced disturbances, which directly mitigates overfitting-like oscillatory behaviors observed in TD3.

This superiority stems from SAC’s maximum-entropy optimization framework, which incorporates a policy-entropy regularization term. This encourages stochastic exploration, preventing the policy from prematurely converging to a local optimum and thereby enhancing its robustness against disturbances such as sensor noise and environmental uncertainties. In contrast, TD3, as a deterministic policy algorithm, is advantageous in sample efficiency but suffers from its relatively fixed policy mode. This makes it prone to overfitting when confronted with observational noise, leading to unnecessary oscillations or deviations in its trajectory that ultimately compromise its effectiveness in the engagement. Therefore, the phenomena depicted in Figure 16 and Figure 21 should be interpreted as evidence of TD3’s algorithmic limitations in handling uncertainties, not as signs of overtraining; SAC’s consistent performance stems from its designed robustness, which effectively mitigates overfitting risks through stochastic exploration.

Figure 22, Figure 23 and Figure 24 depict the tail-chase trajectories when both sides start at similar altitudes. The results indicate that both strategies perform well, successfully defeating the opponent with relatively smooth trajectories. Figure 25, Figure 26, Figure 27, Figure 28, Figure 29 and Figure 30 illustrate the scenario where both parties start head-on, with our side having an altitude advantage. As shown by the corresponding dense reward function, the SAC strategy fails to achieve a favorable geometric angle between $T_3$ and $T_4$, allowing the blue side to fire at us first. Figure 31, Figure 32, Figure 33, Figure 34, Figure 35 and Figure 36 represent the case of a head-on start with our side at an altitude disadvantage.

Here, the comparative analysis between Figure 33 and Figure 36 indicates that the TD3 strategy generates smoother trajectories than the SAC, with quantitative metrics further supporting this observation: lower standard deviations in most maneuver commands (specifically, 1.269 for $n_x$, 0.672 for $n_z$, and 1.182 for $\gamma$ of the TD3 policy, compared with 1.047 for $n_x$, 1.350 for $n_z$, and 1.205 for $\gamma$ of the SAC policy), lower Energy Consumption (3057.75 for TD3, 3544.70 for SAC), and superior Control Smoothness (0.0713 for TD3, 0.7472 for SAC). Additionally, the TD3 policy initiates attacks earlier and demonstrates a certain level of robustness.

5. Conclusions

This study addresses the challenge of autonomous maneuvering decision making for unmanned aerial vehicles (UAVs) by proposing a continuous action space decision-making framework based on the Soft Actor-Critic (SAC) algorithm. Through the construction of a simulation environment that incorporates three-dimensional kinematic constraints and a multi-dimensional situational reward mechanism, the framework achieves the generation of high-precision control commands, offering a valuable reference for research on autonomous behavior decision making of intelligent unmanned systems in dynamic environments.

The primary innovations of this work are manifested in two key aspects: first, the design of a comprehensive multi-dimensional reward function that integrates factors such as relative position, closure speed, altitude relationship, and interactive effectiveness, coupled with the introduction of a Health Point (HP) system for quantitative assessment of situational advantages, which enhances the strategy’s capability to perceive and respond to complex dynamic conditions; second, the application of the SAC algorithm with maximum entropy optimization, which facilitates the generation of smooth and reliable control commands while maintaining an equilibrium between exploration and exploitation, thereby improving the adaptability and stability of the decision-making system.

Despite these contributions, the current research exhibits certain limitations: the three-degrees-of-freedom point-mass model employed, while ensuring basic motion feasibility through overload constraints (e.g., the ranges of $n_x$ and $n_z$), does not fully account for higher-order dynamic characteristics such as aerodynamic coupling effects and actuator response delays. Consequently, the practical feasibility of the generated trajectories under extreme maneuvers warrants further validation. Future efforts will prioritize two directions: the development of a six-degrees-of-freedom high-fidelity simulation platform that incorporates aerodynamic parameters and system dynamics to enhance model reliability and trajectory authenticity and the extension to multi-agent collaborative decision-making scenarios, focusing on cooperative maneuvering strategies for multiple UAVs in complex environments to strengthen the algorithm’s practicality and scalability.