Enhancing Automated Maneuvering Decisions in UCAV Air Combat Games Using Homotopy-Based Reinforcement Learning

Abstract

In the field of real-time autonomous decision-making for Unmanned Combat Aerial Vehicles (UCAVs), reinforcement learning is widely used to enhance their decision-making capabilities in high-dimensional spaces. These enhanced capabilities allow UCAVs to better respond to the maneuvers of various opponents, with the win rate often serving as the primary optimization metric. However, relying solely on the terminal outcome of victory or defeat as the optimization target, but without incorporating additional rewards throughout the process, poses significant challenges for reinforcement learning due to the sparse reward structure inherent in these scenarios. While algorithms enhanced with densely distributed artificial rewards show potential, they risk deviating from the primary objectives. To address these challenges, we introduce a novel approach: the homotopy-based soft actor–critic (HSAC) method. This technique gradually transitions from auxiliary tasks enriched with artificial rewards to the main task characterized by sparse rewards through homotopic paths. We demonstrate the consistent convergence of the HSAC method and its effectiveness through deployment in two distinct scenarios within a 3D air combat game simulation: attacking horizontally flying UCAVs and a combat scenario involving two UCAVs. Our experimental results reveal that HSAC significantly outperforms traditional algorithms, which rely solely on using sparse rewards or those supplemented with artificially aided rewards.

1. Introduction

The prevailing research in the field of Unmanned Combat Aerial Vehicles (UCAVs) is centered on enhancing autonomy, which necessitates advanced intelligent decision-making capabilities. To effectively navigate dynamic air combat game environments, a UCAV must execute swift and informed decisions. Traditional methods like the differential game method [1,2], influence diagram method [3,4], model predictive control [5], genetic algorithms [6], and Bayesian inference [7] have been deployed to tackle tactical decision-making challenges. These approaches analyze relationships in air combat games to derive optimal solutions but often falter in complex environments that require real-time calculations.

With the increasing complexity of air combat game scenarios, decision-making strategies must evolve. Recently, artificial intelligence (AI) has shown promise in addressing these challenges, particularly through its capability for real-time performance. Traditional AI methods include expert systems, which utilize comprehensive rule bases to generate empirical policies [8,9,10,11], and supervised learning, which employs neural networks to translate sensor data into tactical decisions based on expert demonstrations [12,13,14]. However, these methods are limited by their reliance on hard-to-obtain expert data, which may not adequately represent the complexities of real-world air combat games.

Reinforcement Learning (RL) addresses these limitations by enabling autonomous learning through environmental interactions, thus maximizing returns without prior expertise [15]. This approach has gained traction in autonomous air combat game applications, where UCAVs operate as agents [16,17,18,19,20]. Pioneers like McGrew [21] and Crumpacker [22] have applied approximate dynamic programming to develop maneuver decision models for 1 vs. 1 air combat game scenarios, with McGrew even testing these on micro-UCAVs. Their work underscores the viability of RL in autonomous tactical decision-making. Additional advancements include efforts by Xiaoteng Ma [23], who integrated speed control into the action space using Deep Q-Learning, and Zhuang Wang [24], who introduced a freeze game framework to mitigate nonstationarity in RL through a league system that adapts to varying opponent strategies. Furthermore, Qiming Yang [25] developed a motion model for a second-order UCAV and a close-range air combat game model in 3D, employing DQN and basic confrontation training techniques. Notably, Adrian’s use of Hierarchical RL [26] secured a second-place finish in DARPA’s AlphaDogfight Trials, showcasing the effectiveness of artificially aided rewards in challenging air combat game scenarios.

Despite these advancements, the sparse reward challenge in RL persists, especially in high-dimensional settings like air combat games, where agents must identify a sequence of correct actions to attain the sparse reward. Traditional random exploration often proves inadequate, which provides minimal feedback on action quality. In response, scholars have explored various strategies, including reward shaping [27,28,29], curriculum learning [30,31,32], learning from demonstrations [33,34,35], model-guided learning [36], and inverse RL [37]. However, these methods typically rely on artificial prior knowledge, potentially biasing policies toward suboptimal outcomes.

Furthermore, previous studies often oversimplify air combat game scenarios, assuming 2D movement or discretized basic flight maneuvers (BFM), thus hampering the realism and efficacy of learned policies. In contrast, this paper advances the following contributions:

(1)

We introduce the homotopy-based soft actor–critic (HSAC) algorithm to address the sparse reward problem in air combat games. We theoretically validate the feasibility and consistent convergence of this approach, offering a new perspective for RL-based methods.

(2)

We develop a more realistic air combat game environment, utilizing a refined 3D flight dynamics model and expanding the agent’s action space from discrete to continuous.

(3)

We implement the HSAC algorithm in this enhanced environment, leveraging self-play. Simulations demonstrate that HSAC significantly outperforms models trained with either sparse or artificially aided rewards.

The paper is structured as follows: Section 2 reviews existing RL-based methods and introduces the air combat game problem. Section 3 details our air combat game environment setup, incorporating a refined 3D flight dynamics model and a continuous action space. In Section 4, we describe the homotopy-based soft actor–critic algorithm and provide theoretical evidence of its convergence and effectiveness. An overview diagram of the HSAC algorithm is shown in Figure 1. Section 5 showcases the performance of the HSAC algorithm in various training and experimental setups. Links to video footage of the training process are available online (Supplementary Materials). Finally, Section 6 summarizes our contributions and discusses future research directions.

2. Preliminaries

In this section, we provide an overview of the dynamics model of a UCAV during short-range air combat game scenarios and introduce the soft actor–critic (SAC) method, a widely recognized reinforcement learning (RL) algorithm for training autonomous agents.

2.1. One-to-One Short-Range Air Combat Game Problem

Short-range air combat games, also known as dogfights, entail the objective of shooting down an enemy UCAV while avoiding return fire. To distinguish between the UCAVs of opposing sides, we use superscript b for blue-side UCAVs and superscript r for red-side UCAVs. The interaction between ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ is inherently adversarial. For simplicity, our focus will primarily be on the learning problem of ${\mathrm{UCAV}}^b$.

Figure 2 illustrates a typical one-to-one short-range air combat game scenario. In this model, four variables are crucial for defining the advantageous position of ${\mathrm{UCAV}}^b$: Aspect Angle (AA), Antenna Train Angle (ATA), Line of Sight (LOS), and Relative Distance. These metrics allow ${\mathrm{UCAV}}^b$ to determine the position $P^r$ and velocity $V^r$ of the opposing ${\mathrm{UCAV}}^r$.

For ${\mathrm{UCAV}}^b$ to successfully engage ${\mathrm{UCAV}}^r$, it must fulfill the following criteria:

(1)

The relative distance between the UCAVs must be within the maximum attack range and above the minimum collision range.

(2)

${\mathrm{UCAV}}^b$ must maintain an advantageous position, enabling the pursuit of ${\mathrm{UCAV}}^r$ while evading potential counterattacks.

These requirements are governed by the following constraints [21]:

$$\{\begin{array}{c}{d}_{min}<d_{LOS}<d_{max},\hfill \\ \left|A{A}^b\right|<{60}^{\circ },\hfill \\ \left|AT{A}^b\right|<{60}^{\circ }.\hfill \end{array}$$

(1)

If all constraints in Equation (1) are satisfied, ${\mathrm{UCAV}}^b$ can engage ${\mathrm{UCAV}}^r$ within its firing envelope [38]. The same operational constraints apply to ${\mathrm{UCAV}}^r$.

A two-target differential game model [39,40] aptly describes this air combat game scenario:

$$\begin{array}{c}\underset{u^b}{min}\underset{u^r}{max}J(u^b,u^r)=\hfill \\ f(t_f,{\mathbf{x}}^b\left(t_f\right),{\mathbf{x}}^r\left(t_f\right))+\underset{t_0}{\overset{t_f}{\int }}{\mathcal{L}}_b(t,{\mathbf{x}}^b\left(t\right),{\mathbf{x}}^r\left(t\right),u^b\left(t\right),u^r\left(t\right))dt,\hfill \\ \underset{u^r}{min}\underset{u^b}{max}J(u^r,u^b)=\hfill \\ f(t_f,{\mathbf{x}}^r\left(t_f\right),{\mathbf{x}}^b\left(t_f\right))+\underset{t_0}{\overset{t_f}{\int }}{\mathcal{L}}_r(t,{\mathbf{x}}^r\left(t\right),{\mathbf{x}}^b\left(t\right),u^r\left(t\right),u^b\left(t\right))dt.\hfill \end{array}$$

(2)

Here, $u^b$ and $u^r$ denote the control policies of each UCAV, whereas ${\mathbf{x}}^b$ and ${\mathbf{x}}^r$ represent their respective state vectors, as defined in Equation (3). In Equation (2), ${\mathcal{L}}_b(·)$ and ${\mathcal{L}}_r(·)$ are the loss functions for ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$, respectively, with $f(·)$ representing the terminal punishment function:

$$\begin{array}{cc}{\mathbf{x}}^b\left(t\right)\hfill & ={[x^b\left(t\right),y^b\left(t\right),z^b\left(t\right),v^b\left(t\right),{\gamma }^b\left(t\right),{\chi }^b\left(t\right)]}^{\mathrm{T}},\hfill \\ {\mathbf{x}}^r\left(t\right)\hfill & ={[x^r\left(t\right),y^r\left(t\right),z^r\left(t\right),v^r\left(t\right),{\gamma }^r\left(t\right),{\chi }^r\left(t\right)]}^{\mathrm{T}}.\hfill \end{array}$$

(3)

Here, v, $\gamma$, $\chi$ represents the velocity, the flight path angle, and the heading angle of UCAV.

The primary aim of this research is to enable ${\mathrm{UCAV}}^b$ to rapidly attain an advantageous position. In reinforcement learning, the concept of self-play is often utilized to simplify the adversarial game, a strategy that has been successfully implemented in AlphaZero [41]. Consequently, we have simplified the two-target differential game into an optimization problem that satisfies the UCAV dynamics constraints. More details are provided in Section 4.4.

2.2. UCAV’s Dynamics Model

The dynamics model of the UCAV is crucial for accurately simulating air combat game scenarios and is constructed using the ground coordinate system. This model is described by a three-degree-of-freedom (3-DoF) particle motion model [3]:

$$\{\begin{array}{cc}\dot{x}\hfill & =vcos\gamma cos\chi ,\hfill \\ \dot{y}\hfill & =vcos\gamma sin\chi ,\hfill \\ \dot{z}\hfill & =-vsin\gamma .\hfill \end{array}$$

(4)

In Equation (4), $\dot{x}$, $\dot{y}$, and $\dot{z}$ denote the UCAV’s rates of change in position along the X, Y, and Z axes, respectively. The variables $\gamma$, $\chi$, and v represent the flight path angle, heading angle, and velocity, respectively. Figure 3 illustrates the associated flight dynamics.

Control variables for the UCAV, including the attack angle $\alpha$, throttle setting $\eta$, and bank angle $\mu$, interplay intricately via:

$$\{\begin{array}{c}\dot{v}={\displaystyle \frac{1}{m}}\left\{\eta T_{max}cos\alpha -D(\alpha ,v)\right\}-gsin\gamma ,\hfill \\ \dot{\chi }={\displaystyle \frac{1}{mvcos\gamma }}\left\{\eta T_{max}sin\alpha +L(\alpha ,v)\right\}sin\mu ,\hfill \\ \dot{\gamma }={\displaystyle \frac{1}{mv}}\left\{\left(\eta T_{max}sin\alpha +L(\alpha ,v)\right)cos\mu -mgcos\gamma \right\}.\hfill \end{array}$$

(5)

Here, g, m, and $T_{max}$ represent the acceleration due to gravity, the mass of the UCAV, and the maximum thrust force of the engine, respectively. Lift and drag forces are given as follows:

$$\{\begin{array}{c}L(\alpha ,v)={\displaystyle \frac{1}{2}}\rho v^2{S}_w{C}_L\left(\alpha \right),\hfill \\ D(\alpha ,v)={\displaystyle \frac{1}{2}}\rho v^2{S}_w{C}_D\left(\alpha \right).\hfill \end{array}$$

(6)

Here, the reference wing area is denoted by $S_w$. The air density $\rho$ is assumed to be a constant in this study. $C_L$ and $C_D$ denote lift and drag coefficients, respectively, related through the attack angle $\alpha$:

$$\{\begin{array}{c}{C}_L\left(\alpha \right)=C_{L0}+C_{L\alpha }\alpha ,\hfill \\ C_D\left(\alpha \right)=C_{D0}+BDP·C_L{\left(\alpha \right)}^2.\hfill \end{array}$$

(7)

Variables $C_{L0}$, $C_{L\alpha }$, $C_{D0}$, and $BDP$ in Equation (7) signify the zero-lift coefficient, the derivative of the lift coefficient with respect to attack angle, zero-drag coefficient, and the drag–lift coefficient, respectively.

Constraints on both the control variables and their rate of change are defined to ensure performance and maintain UCAV inertia within operational limits:

$$\{\begin{array}{c}\alpha \in [{\alpha }_{min},{\alpha }_{max}],\hfill \\ \dot{\alpha }\in [-\Delta \alpha ,\Delta \alpha ],\hfill \\ \mu \in [-\pi ,\pi ],\hfill \\ \dot{\mu }\in [-\Delta \mu ,\Delta \mu ],\hfill \\ \eta ={\eta }_{max}=1,\hfill \\ \dot{\eta }=0.\hfill \end{array}$$

(8)

The throttle setting $\eta$ is assumed constant at 1 to maximize pursuit capabilities.

The load factor $n(·)$ and dynamic pressure $q(·)$ are defined as follows:

$$\begin{array}{cc}\hfill n(\alpha ,v)& ={\displaystyle \frac{L(\alpha ,v)}{mg}},\hfill \\ \hfill q\left(v\right)& ={\displaystyle \frac{1}{2}}\rho v^2.\hfill \end{array}$$

(9)

Constraints on these parameters prevent overloading of the UCAV:

$$\{\begin{array}{c}n(\alpha ,v)-n_{max}\le 0,\hfill \\ h_{min}-h\le 0,\hfill \\ q\left(v\right)-q_{max}\le 0.\hfill \end{array}$$

(10)

where $h_{min}$, $n_{max}$, and $q_{max}$ denote the minimum altitude, maximum load factor, and maximum dynamic pressure, respectively. Additionally, it is important to note that the relationship between altitude h and the functions $n(·)$ and $q(·)$ is not considered here, as the air density $\rho$ is assumed constant.

2.3. Soft Actor–Critic Method

The Markov Decision Process (MDP) is fundamental to RL-based methods, facilitating the modeling of sequential decision-making problems using mathematical formalism. An MDP consists of a state space S, an action space A, a transition function T, and a reward function R, forming a tuple $<S,A,T,R>$. RL-based methods aim to optimize the agent’s policy through interaction with the environment. At each time step t, the agent in state $s_t$ chooses an action $a_t$, and the environment provides feedback on the next state $s_{t+1}$ based on the transition probability $T\left(s_{t+1}\right|s_t,a_t)\in [0,1]$, along with a reward $r_{t+1}=R(s_t,a_t)$ to evaluate the quality of the action. The policy $\pi :S\to A$ directs the agent’s action selection in each state, with the probability of each action given by $\pi \left(a\right|s)$. The ultimate objective is to discover an optimal policy ${\pi }^{*}$ that maximizes the expected cumulative reward.

Finding the optimal policy ${\pi }^{*}$ can be challenging, especially if the agent has not extensively explored the state space. This challenge is addressed by employing maximum entropy reinforcement learning techniques, such as the soft actor–critic (SAC) algorithm [42]. SAC is a model-free, off-policy RL algorithm that integrates policy entropy into the objective function to encourage exploration across various states [43]. Recognized for its effectiveness, SAC has become a benchmark algorithm in RL implementations, outperforming other advanced methods like SQL [44] and TD3 [45] across multiple environments [42,46].

The optimal policy ${\pi }^{*}$ is described as follows:

$$\begin{array}{cc}{\pi }^{*}\hfill & =arg\underset{\pi }{max}{\mathbb{E}}_{\begin{array}{c}{s}_0\sim {\rho }_0\left(s_0\right)\\ a_t\sim \pi \left(a_t\right|s_t)\\ s_{t+1}\sim T\left(s_{t+1}\right|s_t,a_t)\end{array}}\left\{\sum _{t=0}^{T_{ter}}{R}_{t+1}(s_t,a_t)+{\alpha }_{\mathrm{temp}}\mathrm{H}\left[\pi (·|s_t)\right]\right\}.\hfill \end{array}$$

(11)

Here, $\mathrm{H}(·|s_t)$ denotes the entropy of the action distribution at state $s_t$, and ${\rho }_0$ is the distribution of the agent’s initial state. The discount factor ${\gamma }_{\mathrm{dis}}\in (0,1)$ balances the importance of immediate versus future rewards, and ${\alpha }_{\mathrm{temp}}$ is the temperature parameter that influences the algorithm’s convergence.

Following the Bellman Equation [15], the soft Q-function is expressed as follows:

$$\begin{array}{c}Q(s_t,a_t)=r_{t+1}+{\gamma }_{\mathrm{dis}}{\mathbb{E}}_{s_{t+1}\sim T\left(s_{t+1}\right|s_t,\pi \left(s_t\right))}\left[V\left(s_{t+1}\right)\right].\hfill \end{array}$$

(12)

The soft value function is derived from the soft Q-function:

$$\begin{array}{c}V\left(s_t\right)={\mathbb{E}}_{a_t\sim \pi \left(s_t\right)}[Q(s_t,a_t)-{\alpha }_{\mathrm{temp}}log\pi \left(a_t\right|s_t)].\hfill \end{array}$$

(13)

The parameters of the soft Q-function are trained to minimize the temporal difference (TD) error, whereas the policy parameters minimize the Kullback-Leibler divergence between the normalized soft Q-function and the policy distribution.

The critic network’s loss function is defined as follows:

$$\begin{array}{c}{J}_Q\left(\theta \right)={\mathbb{E}}_{(s_t,a_t)\sim D}\left[{\displaystyle \frac{1}{2}}{(Q_{\theta }(s_t,a_t)-(r+{\gamma }_{\mathrm{dis}}{\mathbb{E}}_{s_{t+1}\sim T}\left[V_{\tilde{\theta }}\left(s_{t+1}\right)\right]))}^2\right].\hfill \end{array}$$

(14)

The actor network’s loss function, simplifying the actor’s learning objective, is given in Equation (15):

$$\begin{array}{c}{J}_{\pi }\left(\varphi \right)={\mathbb{E}}_{s_t\sim D}\left[{\mathbb{E}}_{a_t\sim {\pi }_{\varphi }}[{\alpha }_{\mathrm{temp}}log\pi \left(a_t\right|s_t)-Q_{\theta }(s_t,a_t)]\right].\hfill \end{array}$$

(15)

To manage the sensitivity of the hyperparameter ${\alpha }_{\mathrm{temp}}$, an automatic adjustment technique is proposed, framing the problem within a maximum entropy RL paradigm with a minimum entropy constraint. This approach, outlined in Equation (16), employs a neural network to approximate ${\alpha }_{\mathrm{temp}}$, targeting a desired minimum expected entropy $\overline{H}$:

$$\begin{array}{c}J\left({\alpha }_{\mathrm{temp}}\right)={\mathbb{E}}_{a_t\sim {\pi }_t}[-{\alpha }_{\mathrm{temp}}log{\pi }_t\left(a_t\right|s_t)-{\alpha }_{\mathrm{temp}}\overline{H}].\hfill \end{array}$$

(16)

3. Configuration of One-to-One Air Combat Game in RL

This section outlines the environment setup required for applying RL-based methods to the air combat game optimization problem specified in Equation (2). The environment encapsulates the state space, action space, transition function, and reward function. We also discuss the transformation of the global state space into a relative state space, which simplifies the representation of state information for UCAVs significantly.

3.1. Action Space

The control laws for UCAVs, denoted as $u^b$ and $u^r$, are represented by the vector format in Equation (17):

$$\begin{array}{cc}\hfill u^b& ={[{\alpha }^b,{\mu }^b,{\eta }^b]}^{\mathrm{T}},\hfill \\ \hfill u^r& ={[{\alpha }^r,{\mu }^r,{\eta }^r]}^{\mathrm{T}}.\hfill \end{array}$$

(17)

Constraints on these control variables are defined in Equation (8). The action vectors, which influence the control variables, are designed as shown in Equation (18):

$$\begin{array}{c}\begin{array}{cc}{a}^b\hfill & =[\dot{{\alpha }^b},\dot{{\mu }^b}],\hfill \\ a^r\hfill & =[\dot{{\alpha }^r},\dot{{\mu }^r}].\hfill \end{array}\hfill \end{array}$$

(18)

These action vectors modify the control laws at each timestep as outlined in Equation (19), where constraints ensure the actions remain within feasible limits:

$$\{\begin{array}{cc}\dot{\alpha }\left(t\right)\hfill & \in [-\Delta \alpha ,\Delta \alpha ],\hfill \\ \alpha \left(t\right)\hfill & ={\alpha }_0+{\int }_{t_0}^t\dot{\alpha }\left(u\right)du\in [{\alpha }_{min},{\alpha }_{max}],\hfill \\ \dot{\mu }\left(t\right)\hfill & \in [-\Delta \mu ,\Delta \mu ],\hfill \\ \mu \left(t\right)\hfill & ={\mu }_0+{\int }_{t_0}^t\dot{\mu }\left(u\right)du\in [-\pi ,\pi ].\hfill \end{array}$$

(19)

${\alpha }_0$ and ${\mu }_0$ are the initial attack angle and initial bank angle of the UCAV, respectively.

3.2. State Space

A well-designed state space can facilitate faster convergence of learning algorithms. Referring to [47], the state space for each UCAV can initially be represented as follows:

$$\begin{array}{c}s={[x^b,y^b,z^b,v^b,{\gamma }^b,{\chi }^b,x^r,y^r,z^r,v^r,{\gamma }^r,{\chi }^r]}^T.\hfill \end{array}$$

(20)

To reduce the dimensionality and simplify this state information, we transform the global state into a relative state for each UCAV as follows:

$$\begin{array}{c}\begin{array}{cc}{s}^b\hfill & ={[AT{A}_{XOY}^b,AT{A}_{YOZ}^b,A{A}_{XOY}^b,A{A}_{YOZ}^b,{\mu }^b,D_{LOS},v^b,{\gamma }^b,v^r,{\alpha }^b,z^b]}^{\mathrm{T}}.\hfill \end{array}\hfill \end{array}$$

(21)

$$\begin{array}{c}\begin{array}{cc}{s}^r\hfill & ={[AT{A}_{XOY}^r,AT{A}_{YOZ}^r,A{A}_{XOY}^r,A{A}_{YOZ}^r,{\mu }^r,D_{LOS},v^r,{\gamma }^r,v^b,{\alpha }^r,z^r]}^{\mathrm{T}}.\hfill \end{array}\hfill \end{array}$$

(22)

The angle projection onto the $XOY$ and $YOZ$ planes, as illustrated in Figure 4, are denoted by the subscripts $XOY$ and $YOZ$, respectively. $D_{LOS}$ represents the line of sight distance between the UCAVs. Additionally, we normalize the state variables to facilitate neural network processing, ensuring learning efficiency:

$$\begin{array}{c}\begin{array}{cc}\hfill & s^b=[{\displaystyle \frac{AT{A}_{XOY}^b}{2\pi }},{\displaystyle \frac{AT{A}_{YOZ}^b}{2\pi }},{\displaystyle \frac{A{A}_{XOY}^b}{2\pi }},{\displaystyle \frac{A{A}_{YOZ}^b}{2\pi }},{\displaystyle \frac{{\mu }^b}{2\pi }},{\displaystyle \frac{D_{LOS}}{D_{max}}},\hfill \\ \hfill & {\displaystyle \frac{v^b}{v_{max}-v_{min}}},{\displaystyle \frac{{\gamma }^b}{2\pi }},{\displaystyle \frac{v^r}{v_{max}-v_{min}}},{\displaystyle \frac{{\alpha }^b}{{\alpha }_{max}-{\alpha }_{min}}},{\displaystyle \frac{-z^b}{h_{max}-h_{min}}}{]}^{\mathrm{T}}.\hfill \end{array}\hfill \end{array}$$

(23)

$$\begin{array}{c}\begin{array}{cc}\hfill & s^r=[{\displaystyle \frac{AT{A}_{XOY}^r}{2\pi }},{\displaystyle \frac{AT{A}_{YOZ}^r}{2\pi }},{\displaystyle \frac{A{A}_{XOY}^r}{2\pi }},{\displaystyle \frac{A{A}_{YOZ}^r}{2\pi }},{\displaystyle \frac{{\mu }^r}{2\pi }},{\displaystyle \frac{D_{LOS}}{D_{max}}},\hfill \\ \hfill & {\displaystyle \frac{v^r}{v_{max}-v_{min}}},{\displaystyle \frac{{\gamma }^r}{2\pi }},{\displaystyle \frac{v^b}{v_{max}-v_{min}}},{\displaystyle \frac{{\alpha }^r}{{\alpha }_{max}-{\alpha }_{min}}},{\displaystyle \frac{-z^r}{h_{max}-h_{min}}}{]}^{\mathrm{T}}.\hfill \end{array}\hfill \end{array}$$

(24)

3.3. Transition Function

To improve the accuracy of our simulation results, we employ the fourth-order Runge-Kutta method [48] for solving the differential equation described in Section 2.2. The specifics of this method are presented in Equation (25), ensuring precise integration of the UCAV dynamics:

$$\begin{array}{c}\begin{array}{cc}\hfill k_1& =f\left(\mathbf{x}\right(t),t,a(t\left)\right),\hfill \\ \hfill k_2& =f(\mathbf{x}\left(t\right)+k_1{\displaystyle \frac{\Delta t}{2}},t+{\displaystyle \frac{\Delta t}{2}},a\left(t\right)),\hfill \\ \hfill k_3& =f(\mathbf{x}\left(t\right)+k_2{\displaystyle \frac{\Delta t}{2}},t+{\displaystyle \frac{\Delta t}{2}},a\left(t\right)),\hfill \\ \hfill k_4& =f(\mathbf{x}\left(t\right)+k_3\Delta t,t+\Delta t,a\left(t\right)),\hfill \\ \hfill \mathbf{x}(t+\Delta t)& =\mathbf{x}\left(t\right)+{\displaystyle \frac{1}{6}}(k_1+2k_2+2k_3+k_4)\Delta t.\hfill \end{array}\hfill \end{array}$$

(25)

where we consider the system of equations ${\displaystyle \frac{d\mathbf{x}(t,a)}{dt}}=f(\mathbf{x},t,a)$, with initial condition $\mathbf{x}\left(t_0\right)={\mathbf{x}}_0$. In this context, the function $f(·)$ denotes the rate of change of $\mathbf{x}$ when the agent chooses the action a at the time step t.

This method accounts for the nonlinear dynamics of the UCAVs, providing a robust framework for accurate state predictions over the simulation timeframe.

3.4. Reward

In this study, we categorize scenarios involving UCAVs based on Algorithm 1. This classification serves as the foundation for constructing the reward function for ${\mathrm{UCAV}}^b$, as detailed in Equation (26).

$$\begin{array}{c}R=\left\{\begin{array}{cc}{r}_1,\hfill & \mathbf{if}{C}^b=\mathrm{win},\hfill \\ r_2,\hfill & \mathbf{if}{C}^b=\mathrm{survival}\mathbf{and}{C}^r=\mathrm{overloaded},\hfill \\ r_3,\hfill & \mathbf{if}{C}^b=\mathrm{overloaded}\mathbf{or}{C}^b=\mathrm{killed},\hfill \\ r_4,\hfill & \mathbf{if}{C}^b=\mathrm{survival}\mathbf{and}{C}^r=\mathrm{survival}.\hfill \end{array}\right.\hfill \end{array}$$

(26)

This reward structure incentivizes ${\mathrm{UCAV}}^b$ to achieve a tactically advantageous position, granting a win reward of $r_1$. Should ${\mathrm{UCAV}}^r$ become overloaded while ${\mathrm{UCAV}}^b$ remains operational, ${\mathrm{UCAV}}^b$ is deemed the winner and receives a reward of $r_2$. Conversely, if a UCAV is either killed or becomes overloaded, it incurs a penalty of $r_3$. Additionally, to encourage expeditious resolution of engagements, a time penalty of $r_4$ is applied to each UCAV until the end of the game. Figure 5 visually represents the logical framework of the environment.

Algorithm 1 Air combat game criterion of ${\mathrm{UCAV}}^b$

Input: State: $s^b$ Output: Situation of UCAV in air combat game: $C^b$   1: $D_{LOS}$: Euclidean distance between UCAV   2: if    $h^b\notin [h_{min},h_{max}]\mathrm{or}n(\alpha ,v)>10g\mathrm{or}{v}^b\notin [v_{min},v_{max}]$ then   3:      $C^b=overloaded$   4: else if   $|A{A}^r| <{60}^{\circ }$ and $|AT{A}^r| <{60}^{\circ }$ and $D_{LOS}\in [D_{min},D_{max}]$ then   5:      $C^b=killed$   6: else if   $|A{A}^b| <{60}^{\circ }$ and $|AT{A}^b| <{60}^{\circ }$ and $D_{LOS}\in [D_{min},D_{max}]$ then   7:      $C^b=win$   8: else   9:      $C^b=survival$ 10: end if

4. Homotopy-Based Soft Actor–Critic Method

Learning a policy for achieving an advantageous posture using only sparse rewards and random exploration is a formidable challenge for any agent. Alternatively, relying solely on artificial prior knowledge may prevent the discovery of the optimal policy. To navigate these challenges, we propose a novel homotopy-based soft actor–critic (HSAC) algorithm that leverages the benefits of both artificial prior knowledge and a sparse reward formulation. Initially, the algorithm employs artificial prior knowledge to establish a suboptimal policy. Subsequently, it follows a homotopic path in the solution space, guiding a sequence of suboptimal policies to converge toward the optimal policy. This method effectively balances exploration and exploitation, leading to a robust policy capable of securing an advantageous posture.

4.1. Homotopy-Based Reinforcement Learning

The challenge of reinforcement learning with sparse rewards, denoted as R, is modeled as a nonlinear programming (NLP) problem, referred to as NLP1, following the framework proposed by Bertsekas [49].

• NLP1 (Original Problem)

$$\begin{array}{c}\begin{array}{cc}\underset{\varphi }{max}{F}_1\left(\varphi \right)=\hfill & {\mathbb{E}}_{s_0\sim {\rho }_0\left(s_0\right)}\left[\sum _{t=0}^T{\gamma }_{\mathrm{dis}}^t\left[R(s_t,a_t)\right]\right],\hfill \\ \hfill s.t.& h_j\left(\varphi \right)=0,j=1,\dots ,m,\hfill \\ \hfill & g_j\left(\varphi \right)\ge 0,j=1,\dots ,s.\hfill \end{array}\hfill \end{array}$$

Here, $a_t\sim \pi (·|s_t)$, and $\varphi$ denotes the parameters of the actor network. The functions $h(·)=0$ and $g(·)\ge 0$ correspond to the environment’s equality and inequality constraints, respectively.

To address the sparse reward challenge in reinforcement learning, researchers often introduce artificial priors that generate an additional reward signal, $R^{extra}$, to aid agents in policy discovery [23,24,25]. This enhancement improves the feedback on the agent’s actions at each step, facilitating learning. Thus, we define the total reward as the sum of R and $R^{extra}$, and formulate the enhanced reinforcement learning problem as NLP2.

• NLP2 (Auxiliary Problem)

$$\begin{array}{c}\begin{array}{cc}\underset{\varphi }{max}{F}_2\left(\varphi \right)={\mathbb{E}}_{s_0\sim {\rho }_0\left(s_0\right)}\hfill & \left[\sum _{t=0}^T{\gamma }_{\mathrm{dis}}^t[R(s_t,a_t)+R^{extra}(s_t,a_t)]\right],\hfill \\ \hfill s.t.& h_j\left(\varphi \right)=0,j=1,···,m\hfill \\ \hfill & g_j\left(\varphi \right)\ge 0,j=1,···,s\hfill \end{array}\hfill \end{array}$$

The feasible policy derived from NLP2 is not the optimal solution to NLP1 due to distortions caused by the artificial priors. To achieve convergence to the optimal solution of the original problem, we recommend a gradual transition using homotopy. We introduce a function operator $F_3(\varphi ,q)$:

$$\begin{array}{c}{F}_3(\varphi ,q)=(1-q)F_2\left(\varphi \right)+q{F}_1\left(\varphi \right),\hfill \end{array}$$

(27)

where $q\in [0,1]$. This function operator facilitates the formulation of a homotopy NLP problem as depicted in NLP3.

• NLP3 (Homotopy Problem)

$$\begin{array}{c}\begin{array}{cc}\underset{\varphi }{max}{F}_3(\varphi ,q)=\hfill & (1-q)F_2\left(\varphi \right)+q{F}_1\left(\varphi \right),\hfill \\ s.t.h_j\left(\varphi \right)=\hfill & 0,j=1,···,m\hfill \\ \hfill g_j\left(\varphi \right)\ge & 0,j=1,···,s\hfill \end{array}\hfill \end{array}$$

It is evident that

$$\begin{array}{c}\begin{array}{cc}{F}_3[\varphi ,q]{|}_{q=0}\hfill & =F_2\left[\varphi \right],\hfill \\ F_3[\varphi ,q]{|}_{q=1}\hfill & =F_1\left[\varphi \right].\hfill \end{array}\hfill \end{array}$$

(28)

By adjusting q from 0 to 1, we can derive a continuous set of solutions transitioning from the solution to NLP2 to the solution to NLP1.

We propose a homotopic reward function, $R^{\mathrm{H}}(s_t,a_t,q)$, that integrates both sparse and extra rewards proportionally based on q:

$$\begin{array}{c}\begin{array}{c}{R}^{\mathrm{H}}(s_t,a_t,q)=q[R(s_t,a_t)+R^{extra}(s_t,a_t)]+(1-q)R(s_t,a_t).\hfill \end{array}\hfill \end{array}$$

(29)

The equivalence of maximizing the expected accumulated homotopic reward $R^{\mathrm{H}}$ to solving the NLP3 problem is demonstrated in Theorem 1, with proof provided in Appendix A.

Theorem 1. 

$F_3(·,·)$ equals to the expectation of accumulated homotopic reward as shown in Equation (30).

$$\begin{array}{c}\begin{array}{c}{F}_3(\varphi ,q)={\mathbb{E}}_{s_0\sim {\rho }_0\left(s_0\right)}\left[\sum _{t=0}^T{\gamma }_{\mathit{dis}}^t[R^{\mathrm{H}}(s_t,a_t,q)]\right].\hfill \end{array}\hfill \end{array}$$

(30)

4.2. Homotopic Path Following with Predictor–Corrector Method

The solution to the auxiliary problem (NLP2) can be seamlessly transformed into the solution for the original problem (NLP1) using the function $F_3(·)$, as assured by Theorem 2.

Theorem 2. 

Under Assumption A1, a continuous homotopic path necessarily exists between the solutions to NLP2 and NLP1.

Here, a “path” denotes a piecewise differentiable curve within the solution space. The proof for the existence of this homotopic path is provided in Appendix B.

This section utilizes a classical method known as the horizontal corrector and elevator predictor [50] to navigate from a feasible solution toward the optimal solution for the original problem (NLP1), tracing the homotopic path in the solution space.

Initially, we identify the auxiliary problem (NLP2) as ${\mathcal{M}}_0$ and label the original problem (NLP1) as ${\mathcal{M}}_T$. The goal of ${\mathcal{M}}_0$ is to determine the optimal policy ${\pi }_{{\mathcal{M}}_0}^{*}\left(\varphi \right)$, maximizing cumulative reward. Theoretically, by following a feasible homotopic path, we can transition to obtaining the optimal policy ${\pi }_{{\mathcal{M}}_T}^{*}\left(\varphi \right)$ for the original task.

Additionally, we introduce a parameter N, representing the number of steps required for q to complete the transition from the auxiliary to the original solution. This sequence is defined by $q_0,q_1,q_2,\dots ,q_N$, where $q_0=0<q_1<q_2<\dots <q_N=1$. This definition allows for a sequence of corresponding sub-tasks ${\mathcal{M}}_0,{\mathcal{M}}_1,\dots ,{\mathcal{M}}_N$, where ${\mathcal{M}}_N={\mathcal{M}}_T$. The pair consisting of the parameter of the actor network $\varphi$ and the value of q at step n is denoted as $y_n=({\varphi }_n,q_n)$.

4.2.1. Predictor

If the optimal policy ${\pi }_{{\mathcal{M}}_n}^{*}\left(\varphi \right)$ has been determined for task ${\mathcal{M}}_n$ at step n, we can predict the parameter ${\varphi }_{n+1}^{*}$ for the subsequent optimal policy ${\pi }_{{\mathcal{M}}_{n+1}}^{*}\left(\varphi \right)$ on the homotopic path using the elevator predictor approach [50], as shown in Equation (31).

$$\begin{array}{c}{\widehat{y}}_{n+1}=({\varphi }_n^{*},q_{n+1}).\hfill \end{array}$$

(31)

4.2.2. Corrector

After predicting for the new task ${\mathcal{M}}_{n+1}$, the predicted parameter tuple ${\widehat{y}}_{n+1}$ may not exactly align with the homotopic path. To re-align, the corrector step adjusts the predicted tuple ${\widehat{y}}_{n+1}$ back to the true solution tuple $y_{n+1}$ using a technique known as the horizontal corrector. During this correction, $q_{n+1}$ remains constant.

The solution’s accuracy is the sole focus during this process, as RL-based methods inherently manage correction through gradient descent. The convergence criteria are established by the transformation of the homotopy problem (NLP3) into the condition $H\left(y_n\right)=0$, with the precise conditions detailed in Appendix B.

$$\begin{array}{c}\parallel H\left(y_n\right)\parallel \le \epsilon .\hfill \end{array}$$

(32)

Here, $\epsilon$ denotes the convergence threshold for the criterion.

4.3. Algorithm

The convergence proof for the predictor–corrector path-following method is delineated in Appendix B, and Theorem 3 posits the following principle:

Theorem 3. 

As the parameter $q_n$ transitions from 0 to 1 using the horizontal corrector and elevator predictor methods, the solution ${\pi }_{{\mathcal{M}}_n}^{*}\left(\varphi \right)$ for task ${\mathcal{M}}_n$ may converge to the optimal solution ${\pi }_{{\mathcal{M}}_T}^{*}\left(\varphi \right)$ for task ${\mathcal{M}}_T$.

Inspired by this theorem, we have developed the homotopy-based soft actor–critic (HSAC) algorithm, which is encapsulated in Algorithm 2. This approach utilizes the variance trend of the parameter $\varphi$ within the actor network of the SAC algorithm, denoted as $k_{\nabla \pi }$, to assess policy convergence. The slope $k_{\nabla \pi }$ of the policy gradient’s changes is computed by fitting the policy gradient data to a first-order function using the least squares method, subsequently utilizing this slope to evaluate the quality of convergence within task ${\mathcal{M}}_n$.

Algorithm 2 includes a buffer, $\mathcal{P}$, to store the gradient of the policy $\nabla \pi$ for computing $k_{\nabla \pi }$. M denotes the sample size for the least squares method, while N represents the total number of iterations required for the homotopy method.

During the horizontal corrector step, the policy parameter $\varphi$ within ${\pi }_{{\mathcal{M}}_n}\left(\varphi \right)$ is iteratively adjusted to approximate the optimal policy ${\pi }_{{\mathcal{M}}_n}^{*}\left(\varphi \right)$ for task ${\mathcal{M}}_n$. If the slope $k_{\nabla \pi }$ meets the criterion $|k_{\nabla \pi }| <\epsilon$, the $n^{th}$ corrector step is completed. Otherwise, the iteration of the policy parameter $\varphi$ continues.

The elevator predictor step begins by clearing the policy gradient buffer $\mathcal{P}$ to accommodate new data for the subsequent task ${\mathcal{M}}_{n+1}$. The parameter $\varphi$ from the optimal policy ${\pi }_{{\mathcal{M}}_n}^{*}\left(\varphi \right)$ of the preceding task is directly transferred to the policy for the new task ${\mathcal{M}}_{n+1}$.

This structured approach ensures rapid convergence to a feasible policy for the initial task ${\mathcal{M}}_0$ using the additional reward $R^{extra}$ at the training’s commencement. The undesirable effects of this extra reward are progressively mitigated through iterations using the corrector-predictor method, leading to the attainment of the optimal policy for the target task ${\mathcal{M}}_T$ as the weight q transitions from 0 to 1.

Algorithm 2 Homotopy-based soft actor–critic

Input: Initialized Parameters: {${\theta }_1$, ${\theta }_2$, $\varphi$, M, N} Output: Optimized parameters: {${\theta }_1$, ${\theta }_2$, $\varphi$}   1: Initialize target network weights:   2: $\overline{{\theta }_1}\leftarrow {\theta }_1$, $\overline{{\theta }_2}\leftarrow {\theta }_2$, $\epsilon >0$   3: Initialize the auxiliary weight:   4: $q_0\leftarrow 1$, $n\leftarrow 0$   5: Empty the Replay Buffer:   6: $\mathcal{D}\leftarrow \varnothing$   7: Empty the gradient of policy $\nabla \pi$ Buffer:   8: $\mathcal{P}\leftarrow \varnothing$, $\mathcal{X}\leftarrow {[1,2,3,\dots ,M]}^{\mathrm{T}}$   9: $\mathcal{X}\leftarrow \mathrm{Add}\mathrm{A}\mathrm{Column}\mathrm{With}\mathrm{Ones}\left(\mathcal{X}\right)$ 10: for each episode do 11:       for each step in the episode do 12:              $a_t\sim {\pi }_{\varphi }(·|s_t)$ 13:              $s_{t+1},R_{t+1}^{\mathrm{H}}\left(k_n\right)\sim T(·|s_t,a_t)$ 14:              $\mathcal{D}\leftarrow \mathcal{D}\cup (s_t,a_t,r_{t+1},s_{t+1})$ 15:       end for 16:       for each update step do 17:              Horizontal Corrector: 18:              ${\theta }_i\leftarrow {\theta }_i-{\lambda }_Q{\widehat{\nabla }}_{{\theta }_i}{J}_Q\left({\theta }_i\right)$ for $i\in \{1,2\}$ 19:              $\varphi \leftarrow \varphi -{\lambda }_{\pi }{\widehat{\nabla }}_{\varphi }{J}_{\pi }(\varphi ,{argmin}_{{\theta }_i}{Q}_{{\theta }_i})$ 20:              ${\alpha }_{\mathrm{temp}}\leftarrow {\alpha }_{\mathrm{temp}}-{\lambda }_{{\alpha }_{\mathrm{temp}}}{\widehat{\nabla }}_{{\alpha }_{\mathrm{temp}}}{J}_{{\alpha }_{\mathrm{temp}}}\left({\alpha }_{\mathrm{temp}}\right)$ 21:              $\overline{{\theta }_i}\leftarrow \tau {\theta }_i+(1-\tau )\overline{{\theta }_i}$ for $i\in \{1,2\}$ 22:              $\mathcal{P}\leftarrow \mathcal{P}\cup {\widehat{\nabla }}_{\varphi }{J}_{\pi }$ 23:              Elevator Predictor: 24:              if $\mathrm{len}\left(\mathcal{P}\right)\ge M$ then 25:                    $k_{\nabla \pi }\leftarrow {\left({\mathcal{X}}^{\mathrm{T}}\mathcal{X}\right)}^{-1}{\mathcal{X}}^{\mathrm{T}}\mathcal{P}$ 26:                    if $|k_{\nabla \pi }|<\epsilon$ and $q_n>0$ then 27:                          $q_{n+1}\leftarrow q_n-{\displaystyle \frac{1}{N}}$ 28:                          $n\leftarrow n+1$ 29:                          $\mathcal{P}\leftarrow \varnothing$ 30:                    end if 31:              end if 32:       end for 33: end for

4.4. The Application of HSAC in Air Combat Game

In air combat game scenarios, developing a high-quality policy involves finding an equilibrium for the two-target differential game described in Equation (2). To streamline this complex game, we adopt the concept of self-play, where both ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ employ the same policy throughout the two-UCAV game task. This method aims to maximize the cumulative reward, aligning with the framework of NLP1, thereby transforming a complicated two-target differential game into a more manageable self-play reinforcement learning problem. This setup facilitates the application of the HSAC method within a self-play environment.

The HSAC approach unfolds in several phases. Initially, a simple reward function R is specified as shown in Equation (26), and the foundational problem is structured according to NLP1. The equality constraints from Equations (4)–(6) are reduced to $h(·)=0$. Meanwhile, the inequality constraints listed in Equations (8) and (10) are simplified to $g(·)\ge 0$.

Next, the additional reward, $R^{extra}$, is configured in accordance with Equation (33) to develop the auxiliary function $F_2(·)$. This preparation makes it straightforward to construct the homotopic reward $R^{\mathrm{H}}$ using Equation (29), aligning with the execution of the HSAC algorithm to address the original problem.

$$\begin{array}{c}{R}^{extra}=\left\{\begin{array}{cc}-{\Phi }^{\mathrm{T}}Q\Phi ,\hfill & \mathrm{if}{D}_{LOS}>D_{max},\hfill \\ -{\Phi }^{\mathrm{T}}Q\Phi -k\left({\left({\displaystyle \frac{2D_{LOS}}{D_{max}+D_{min}}}-1\right)}^2+1\right),\hfill & \mathrm{elif}{D}_{LOS}\le D_{max}.\hfill \end{array}\right.\hfill \end{array}$$

(33)

In this model, the vector $\Phi$ represents the relative angle, including $AT{A}^b$ and $A{A}^b$. The matrix $Q\in {\mathbb{R}}^{3\times 3}$ is a diagonal matrix filled with positive weights, and k is the penalty coefficient related to relative distance. The design of $R^{extra}$ specifically accounts for the influence of relative distance only when it is less than the maximum attack range $D_{max}$.

5. Simulation

This section demonstrates the effectiveness of our HSAC method in addressing sparse reward challenges in reinforcement learning, particularly in air combat game scenarios, through two distinct experiments.

In the first experiment, we focus on a simplified task—the attack on a horizontally flying UCAV—to highlight the advantages of the HSAC method. The second experiment extends the application of HSAC combined with self-play in a complex two-UCAVs game task, illustrating its potential in more dynamic and intricate scenarios.

We use two variants of the SAC algorithm for comparison: SAC-s, which utilizes only the original reward R, and SAC-r, which employs a combined reward of $R+R^{extra}$.

5.1. Simulation Platform

The simulation environment is built on the Gym framework [51], programmed in Python, and visualized through Unity3D. The data interaction between Gym and Unity3D is facilitated using socket technology. The platform interface is shown in Figure 6.

Training leverages an NVIDIA GeForce GTX 2080TI graphics card with PyTorch acceleration. The framework for the HSAC algorithm utilized by the agent is depicted in Figure 7. The model parameters for the air combat game, as discussed in Section 2.1 and Section 2.2, along with the hyperparameters for the HSAC algorithm, are detailed in

Table A1. Air combat game simulation parameters.

Single UCAV Parameter	Value
Mass of UCAV	150 kg
Max Load Factor ($n_{max})$	10 g
Velocity Band	$[80,400]$ m/s
Height Range	$[2000,8000]$ m
Max Thrust ($T_{max})$	100 kg
Range of $\dot{\alpha }$	$[-5,5]$ deg/s
Range of $\dot{\mu }$	$[-50,50]$ deg/s
Range of $\alpha$	$[-15,15]$ deg
Game Parameter	Value
Optimum Attack Range $(d_{min},d_{max})$	(200, 2000) m
Maximum Iteration Step	2000

and

Table A2. Hyperparameters of HSAC.

Parameter	Value
Optimizer	Adam
Learning Rate	$3\times {10}^{-4}$
Discount (${\gamma }_{\mathrm{dis}}$)	0.996
Number of Hidden Layers (All Networks)	3
Number of Hidden Units per Layer	256
Number of Samples per Minibatch	256
Nonlinearity	ReLU
Replay Buffer size	${10}^6$
Entropy Target	−dim(A)
Target Smoothing Coefficient ($\tau$)	0.005
Policy Gradient Buffer (M)	${10}^4$
Total Number of Homotopy Iterations (N)	100
Threshold of $k_{\nabla \pi }$ ($\epsilon$)	${10}^{-5}$

, respectively.

5.2. Attacking Horizontal-Flight UCAV

5.2.1. Task Setting

This experiment is designed to compare the convergence rates of different methods in a straightforward task setup. In this task, ${\mathrm{UCAV}}^r$ maintains a horizontal flight path at a random starting position and heading, while ${\mathrm{UCAV}}^b$, controlled by the trained agent, attempts to execute an attack.

The starting point for ${\mathrm{UCAV}}^b$ is designated as point o in Figure 8, positioned at a height of 5 km, and aligned with the X axis with an initial heading angle of zero. The initial positioning for ${\mathrm{UCAV}}^r$ is within a predefined area, as depicted in Figure 8, with the initial relative distance ranging from 4 km to 6 km and the relative height from −1 km to 1 km. The initial heading angle for the red UCAV varies randomly from $-{180}^{\circ }$ to ${180}^{\circ }$.

5.2.2. Training Process

To test the convergence of the methods and the performance in the original problem, We analyze from two perspectives: the training process and the evaluation process. During the training process, the episode reward comprises both the original reward R and additional rewards such as $R^{extra}$ and $R^{\mathrm{H}}$. In contrast, during the evaluation process, only the original reward R is considered. We periodically evaluate the quality of the policy every ten training episodes.

The models are trained for $5\times {10}^4$ episodes as illustrated in Figure 9. Changes in episode rewards for different methods during the training process are displayed in Figure 9a, and those during the evaluation process are presented in Figure 9b.

As shown in Figure 9a, the episode rewards during the training process generally rise with iteration, indicating that agents trained by different methods are progressively mastering the task. Notably, the agent trained with SAC-r records the lowest episode rewards, which is attributable to the negative design of $R^{extra}$.

To gauge the training quality of each agent, attention must be directed to Figure 9b. Here, it is evident that the agent employing HSAC achieves the highest episode rewards compared to those trained by SAC-s and SAC-r.

Although $R^{extra}$ in SAC-r is crafted using artificial priors, the agent trained with SAC-r still attains lower episode rewards than the agent trained with SAC-s. Conversely, our HSAC method, using the same artificial priors, outperforms both SAC-s and SAC-r. This demonstrates that HSAC is not adversely affected by the negative aspects of artificial priors and can effectively leverage them to address the challenges posed by sparse reward RL problems with the aid of $R^{\mathrm{H}}$.

Subsequent analyses will compare the performance of agents trained by these methods in various air combat game scenarios post-training.

5.2.3. Simulation Results

The initial state space of air combat game geometry can be divided into four categories [24] from the perspective of ${\mathrm{UCAV}}^b$: head-on, neutral, disadvantageous, and advantageous, as illustrated in Figure 10. Depending on these initial conditions, the win probability for ${\mathrm{UCAV}}^b$ varies significantly. For instance, ${\mathrm{UCAV}}^b$ is more likely to win in an advantageous position, while the odds are reversed in a disadvantageous scenario. In neutral and head-on scenarios, the win probabilities for both sides are similar. We evaluate the performance of each well-trained agent across these diverse initial scenarios.

We selected four representative initial states from each category, detailed in

Table A3. Initial state settings for evaluating the quality of agents.

Initial State		$\mathit{X}\left(\mathbf{m}\right)$	$\mathit{Y}\left(\mathbf{m}\right)$	$\mathit{H}\left(\mathbf{m}\right)$	$\mathit{V}\left(\mathbf{m}/\mathbf{s}\right)$	$\mathit{\chi }{(}^{\circ })$
Advantageous	Blue	0	0	5000	150	45
	Red	5000	5000	5000	150	45
Disadvantageous	Blue	0	0	5000	150	−45
	Red	−5000	5000	5000	150	−45
Head-On	Blue	0	0	5000	150	45
	Red	5000	5000	5000	150	−135
Neutral	Blue	0	0	5000	150	45
	Red	5000	−5000	5000	150	−135

. The initial bank angle $\mu$ and path angle $\gamma$ for both ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ are set to zero. The performance metrics, including the win rates and average time cost per episode for the agents trained by different methods, are captured in Figure 11 and

Table 1. Performance of agents trained by different methods in attacking horizontal-flight tasks with different initial states after ${10}^4$ episode evaluations.

	Method	SAC-s	SAC-r	HSAC
Initial State
Advantageous	Winning Rate	100.00%	100.00 %	100.00%
	Average Time Cost	42.07 s	37.90 s	39.40 s
Disadvantageous	Winning Rate	$6.61\%$	$0.00\%$	98.38%
	Average Time Cost	44.66 s	26.23 s	35.91 s
Head-On	Winning Rate	$73.32\%$	$7.43\%$	99.94%
	Average Time Cost	39.15 s	31.02 s	24.73 s
Neutral	Winning Rate	100.00%	$3.66\%$	100.00%
	Average Time Cost	40.05 s	22.57 s	33.20 s

.

As illustrated in Table 1, agents trained by SAC-s, SAC-r, and HSAC can successfully complete the task when starting from an advantageous position. The agent using SAC-r achieves this task faster than others in advantageous situations. However, in disadvantageous and head-on initial scenarios, the agent trained by HSAC significantly outperforms the others, with the highest success rates. In a disadvantageous scenario, the agent trained by SAC-s has only a $6.61\%$ success rate, while the agent trained by SAC-r fails to succeed at all. In contrast, in head-on scenarios, while the agent trained by SAC-s has a $73.32\%$ success rate, the one trained by SAC-r only achieves $7.43\%$. Although both the SAC-s and HSAC-trained agents show equal winning rates in neutral scenarios, the HSAC-trained agent completes the task more quickly.

In summary, while SAC-s fairly represents the original task, it struggles with sparse reward issues. SAC-r performs well only in advantageous scenarios due to the bias introduced by the extra rewards. However, HSAC excels in all tested scenarios, outperforming SAC-s and SAC-r by a substantial margin. HSAC effectively harnesses artificial priors without being negatively impacted by them, demonstrating its robustness across varying initial air combat game scenarios.

5.3. Self-Play Training and Two-UCAVs Game Task

5.3.1. Two-UCAVs Game Task Setting

This experiment aims to demonstrate the effectiveness of the HSAC method combined with self-play in the two-UCAVs game task. Integrating self-play with HSAC, as described in Section 4.4, allows for the development of a more sophisticated UCAV policy under air combat game conditions, optimizing the policy through competitive interactions with itself. Figure 12 illustrates the self-play training setup, where both ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ share the same actor and critic networks. Furthermore, they share the experience replay buffer and the gradient of the policy buffer, which is essential for HSAC. The initial setup follows the parameters of the attacking horizontal-flight UCAV task outlined in Section 5.2.1. To enhance the realism of radar detection, we set the initial line-of-sight angle component for ${\mathrm{UCAV}}^r$ in the $XOY$ plane between $-{60}^{\circ }$ and ${60}^{\circ }$. While this setup gives ${\mathrm{UCAV}}^r$ an initial advantage, it serves to test if the policy can converge to an equilibrium point within the policy space, ideally demonstrating balanced offensive and defensive capabilities from any starting position.

Following the training, the performance of agents trained by SAC-s, SAC-r, and HSAC will be compared in direct confrontations.

5.3.2. Training Process

The training processes for SAC-s, SAC-r, and HSAC are depicted in Figure 13 after ${10}^4$ training episodes. Variations in episode rewards for ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ during the self-play training are illustrated in Figure 13a. The difference in episode rewards between ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ throughout the training process is shown in Figure 13b. Additionally, we record the number of episodes required for method convergence, the rewards upon convergence of both ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$, and the difference in these rewards, as summarized in

Table 2. The convergence episode cost for three different methods, the episode reward of the converged policy, and the difference value of converged episode reward between ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ in the self-play training process.

		SAC-r		SAC-s		HSAC
		Blue	Red	Blue	Red	Blue	Red
Quality of Policy	Episode Reward	−217.64	111.72	−1469.09	−1339.13	−387.52	−382.57
	Difference Value	329.36		129.96		4.95
Convergence Episode Cost (episodes)		8250		6675		4455

, using the criteria set forth in Algorithm 2 for convergence determination.

The data in Table 2 and Figure 13 highlight the minimal difference in rewards upon convergence, suggesting that the policy developed through HSAC aligns closely with an equilibrium point, in accordance with Nash’s theories [52], particularly when this difference approaches zero. This outcome underscores the efficacy of HSAC in rapidly achieving the policy that performs robustly in both offensive and defensive scenarios, as evidenced by the lowest convergence episode cost and the smallest reward discrepancies.

Post-training analysis reveals that HSAC not only converges to an equilibrium point more effectively than the other methods but also does so with superior performance consistency between the competing agents. We will next examine whether the policy honed through HSAC training outperforms those trained via other methods in subsequent head-to-head tests.

5.3.3. Simulation Results

We assess the performance of HSAC through 1 vs. 1 games played among agents trained by SAC-s, SAC-r, and HSAC. Each agent alternates between playing on the red side and the blue side over ${10}^5$ games. The winning rates are detailed in

Table 3. The relative winning rates of agents trained by different methods after ${10}^5$ episodes’ evaluation in two-UCAVs game task, where DN means the number of draws.

	Red	SAC-s	SAC-r	HSAC
Blue
SAC-s	Blue Win	$34.9\%$	$42.1\%$	$18.7\%$
	Red Win	$\mathbf{65}.\mathbf{1}\%$	$\mathbf{57}.\mathbf{9}\%$	$\mathbf{81}.\mathbf{3}\%$
	DN	43,785	372	49,939
SAC-r	Blue Win	$\mathbf{61}.\mathbf{0}\%$	$30.5\%$	$39.4\%$
	Red Win	$39.0\%$	$\mathbf{69}.\mathbf{5}\%$	$\mathbf{60}.\mathbf{6}\%$
	DN	328	1	619
HSAC	Blue Win	$\mathbf{66}.\mathbf{8}\%$	$\mathbf{60}.\mathbf{8}\%$	$45.4\%$
	Red Win	$33.2\%$	$39.2\%$	$\mathbf{54}.\mathbf{6}\%$
	DN	35,448	843	75,228

, with DN representing the number of draws.

Table 3 shows that when ${\mathrm{UCAV}}^b$ and ${\mathrm{UCAV}}^r$ operate under the same policy, players trained by SAC-r seldom achieve draws, likely due to the disproportionate reward structures which favor aggressive tactics over balanced strategic play. This undermines their ability to perform defensively, rendering SAC-r less suited for self-play training.

In contrast, the HSAC-trained agents demonstrate almost equal win rates in head-to-head matchups, reflecting robust performance across both aggressive and defensive scenarios. Notably, HSAC consistently outperforms the other methods, showcasing superior adaptability and strategic depth.

Figure 14 depicts these interactions, demonstrating how HSAC-trained agents often reach equilibrium states, effectively balancing aggressive and defensive strategies.

In summary, while SAC-s often struggles with the sparse reward problem, leading to protracted exploratory phases and suboptimal local convergence, SAC-r tends to skew learning due to its reward design. HSAC, on the other hand, utilizes homotopic techniques to smoothly transition from exploratory to optimal strategies, effectively navigating the sparse reward landscape without deviating from the task’s objectives.

6. Conclusions

In this study, we introduced the HSAC method, a novel reinforcement learning approach specifically designed to tackle the challenges associated with sparse rewards in complex RL tasks such as those found in air combat game simulations. The HSAC method begins by integrating artificial reward priors to enhance exploration efficiency, allowing for the rapid development of an initial feasible policy. As training progresses, these artificial rewards are gradually reduced to ensure that the learning process remains focused on achieving the primary objectives, thus following a homotopic path toward the optimal policy.

The results of our experiments, conducted within a sophisticated 3D simulation environment that accurately models both offensive and defensive maneuvers, demonstrate that agents trained with HSAC significantly outperform those trained using the SAC-s and SAC-r methods. Specifically, HSAC-trained agents exhibit superior combat skills and achieve higher success rates, highlighting the algorithm’s ability to effectively balance exploration and exploitation in sparse reward settings.

HSAC not only accelerates the convergence process by efficiently navigating the sparse reward landscape but also maintains robustness across various initial conditions and scenarios. This adaptability is particularly evident in our self-play experiments, where HSAC consistently achieves near-equilibrium strategies, showcasing its potential for both competitive and cooperative multi-agent environments.

Looking forward, we plan to extend the application of HSAC to multi-agent systems, further enhancing its capabilities in complex, dynamic environments. By doing so, we aim to improve its proficiency in both competitive and cooperative contexts, paving the way for broader applications in real-world scenarios.