Hierarchical Reinforcement Learning with Automatic Curriculum Generation for Unmanned Combat Aerial Vehicle Tactical Decision-Making in Autonomous Air Combat

Abstract

This study proposes an unmanned combat aerial vehicle (UCAV)-oriented hierarchical reinforcement learning framework to address the temporal abstraction challenge in autonomous within-visual-range air combat (WVRAC) for UCAVs. The incorporation of maximum-entropy objectives within the MEOL framework facilitates the optimization of both autonomous low-level tactical discovery and high-level option selection. At the low level, three tactical policies (angle, snapshot, and energy tactics) are designed with reward functions informed by expert knowledge, while the high-level policy dynamically terminates current tactics and selects new ones through sparse reward learning, thus overcoming the limitations of fixed-duration tactical execution. Furthermore, a novel automatic curriculum generation mechanism based on Wasserstein Generative Adversarial Networks (WGANs) is introduced to enhance training efficiency and adaptability to diverse initial combat conditions. Extensive experiments conducted in UCAV air combat simulations have shown that MEOL not only achieves significantly better win rates than other policies when training against rule-based opponents, but also that MEOC achieves superior results in tests against tactical intra-option policies as well as other option learning policies. The framework facilitates dynamic termination and switching of tactics, thereby addressing the limitations of fixed-duration hierarchical methods. Ablation studies confirm the effectiveness of WGAN-based curricula in accelerating policy convergence. Additionally, the visual analysis of UCAVs’ flight logs validates the learned hierarchical decision-making process, showcasing the interplay between tactical selection and manoeuvring execution. This research provides novel methodologies combining hierarchical reinforcement learning with tactical domain knowledge for the autonomous decision-making of UCAVs in complex air combat scenarios.

1. Introduction

The rapid evolution of UCAV platforms has intensified the complexity of autonomous air combat decision-making, where real-time tactical generation and agile manoeuvring control are critical for UCAVs to dominate WVRAC engagements. Autonomous and intelligent decisions are expected as an auxiliary or even a substitution for human fighter pilots, which are of great significance for prospective air combat situations.

Accordingly, a plethora of studies have concentrated on the autonomous and intelligent decisions of UCAVs in air combat scenarios. The rule-based expert system [1] firstly merges prior knowledge into scripts for situation orientation and the manoeuvring procedure. The method used in [2] firstly converts the decisions on air combat into optimization problems based on key geometry metrics in air combat scenario, which lays the groundwork for optimization methods, including greedy algorithms [3], genetic algorithms [4], and approximate dynamic programming (ADP) [5,6,7,8]. However, the above methods are subjected to the complexity of modelling air combat dynamics. More recently, model-free reinforcement learning (RL) is capable of solving model-agnostic decisions and control problems through the trial-and-error mechanism. Based on the above optimization objective, there has been significant progress on applying RL [9,10] to the high-fidelity air combat environment. The proposed methods learn to manoeuvre and control the fighter aircraft with a six-degree-of-freedom (6-DoF) nonlinear dynamic model.

Despite the advances made in the field of reinforcement learning for UCAVs in addressing air combat challenges, decisions on tactical selection and manoeuvring control in the intrinsically hierarchical structure still remains to be explored. Inherently, a human fighter pilot continuously repeats the following procedure: (1) the pilot selects tactics through observing situations at a given moment and then (2) executes the selected tactics by performing basic fighter manoeuvring (BFM) during a period of time [11]. It can be concluded that the different temporal granularities at a hierarchical decision level exists in the process of tactic decision-making and manoeuvring execution. This is known as temporal abstraction [12,13,14,15,16], and hierarchical reinforcement learning (HRL) offers a promising solution to address it.

Furthermore, semi-Markov decision processes (SMDPs) [17,18], where the decision condition and timescale are generalized to varying cases, theoretically formalize the temporal abstraction for hierarchical reinforcement learning (HRL). Moreover, options, defined as temporally extended actions, combine options over MDPs with primitive actions selected among those options [15], which have been theoretically proven by a case of SMDPs. Following the paradigm of RL, option learning integrally optimizes both option discovery (low-level policy for actions or skills) and option utilization (high-level policy for options) for brevity.

Likewise, attempts of applying option learning to the challenges faced by UCAVs in air combat problems have been thriving. Such research has been discussed surrounding three questions in air combat decision: (1) when to terminate the present tactic; (2) which discrete, new tactic to select; and (3) how to execute the selected tactic with continuous manoeuvring control for a period of time. Pope et al. [19] applied the soft actor–critic (SAC) [20] to train both levels, where the options switch subpolicies at a frequency of a fixed batch of timesteps and set three subpolicies as the conservative, aggressive, and control zones. Likewise, Kong et al. [21] designed low-level polices including offensive, aggressive, and defensive subtasks, and the termination condition of options was handcrafted based on some key geometric situations. Selmonaj et al. [22] trained the fight and escape subpolicies with the proximal policy optimization (PPO), and the structure of the policy was heterogeneous. For the generation of specific tactics, the above research consistently separates the option discovery from the hierarchical optimization of the option framework with the help of baselines in RL. Moreover, the termination condition in these proposed methods is predefined, which excludes the concept of when to terminate the current option from hierarchical optimization. With regard to the random and unpredictable initial engagement situation that consists of range and geometric angles, predefined curriculum learning [10,19] is utilized to ensure adaptability and robustness with the assumption that the difficulty of combat is linearly increased with the initial condition. Nevertheless, this assumption restricts skill discovery naturally. From the above, further exploration is required into the questions concerning hierarchical optimization for UCAV air combat needs.

In this study, we extend our proposed method, the maximum-entropy option learning framework (MEOL) [23], to address UCAVs’ temporal abstraction in tactical decision-making and manoeuvring control for within-visual-range air combat (WVRAC) scenarios. The efficacy of this approach is substantiated by its convergence, which governs the discovery and utilization of options within a regulated entropy framework of hierarchical policies. Within this framework, the option discovery intends to characterize the low-level policy as the three typical tactics [11]: angle, snapshot, and energy tactics. Specifically, the angle tactic is committed to achieving a geometric advantage against an adversary fighter, which is featured by manoeuvring into its rear quarter to “bite the bandit”. The snapshot tactic prefers to snatch every opportunity as long as the bandit is targeted in the weapon engagement zone (WEZ). The energy tactic aims to strike a balance between the geometric advantages and energy advantages. In turn, we design the reward functions for angle, snapshot, and energy tactics based on expert knowledge [11] to guide the optimization of option discovery. To adapt to the prevailing initial conditions, we propose a novel automatic curriculum generation method to enhance the learning efficiency of option discovery, through which we introduce the WGANs [24] to learn and generate the intermediate-difficulty curricula (initial condition). An ablation study of the process of automatic curriculum generation has demonstrated that such methods have the capacity to accelerate the rate at which model training converges on a stable success rate. Subsequent to the training of the three low-level policies, the utilization of options is learned in order to both terminate the current tactic and determine a new tactic at high-level optimization with sparse rewards. The efficacy of MEOL in utilizing three types of low-level tactics to achieve superior win rates has been demonstrated by two comparative experiments. The contributions of this study are as follows:

• The training of UCAV agents is conducted in accordance with a hierarchical structure within the MEOL framework. The option discovery enables the manoeuvring control of fighters with nonlinear flight dynamic models in a high-fidelity air combat simulation. The utilization of options extends beyond the selection of the most optimal option, but also extends the fixed duration of tactics [19,21,25] to a varied term.
• Propose an end-to-end automatic curriculum generation method that enhances learning efficiency in option discovery and adaptability to random initial situations. The WGANs is trained to propose feasible initial conditions for option discovery.
• The reward functions for the three types of tactical intra-options (angle, snapshot, and energy tactics) are shaped based on expert knowledge, and the tactical option-selection reward function is further designed. The final win rate against the rule-based adversary fighter is approximately 0.8, and the comparison test proves to be significantly better than the three types of tactical intra-option policies.
• The UCAVs’ flight logs are analysed in order to provide a visual representation of combat with a variety of adversaries, including an explanation of the skills that have been acquired and the manner in which these skills have been employed.

The remainder of this paper is organized as follows: In Section 2, the fundamentals of UCAV air combat and the preliminaries of reinforcement learning are introduced. In Section 3, the automatic curriculum generation algorithm based on WGNs and the MOEL algorithm are presented. In Section 4, the application of MOEL to WVRAC is presented. The experiments are described in detail in Section 5. Finally, Section 6 offers a conclusion to this paper.

2. Preliminaries and Problem Formulation

In this section, we provide a theory basis of SMDPs with options with the basic approximate optimization algorithm, and delineate the basic geometric of air combat situations as well as a simulation of weaponry.

2.1. Semi-Markov Decision Process with Options

According to Theorem 1 in [15], any MDP with a fixed set of Markovian options is proven to be SMDP. The option framework is defined as a tuple $(\mathcal{S},\mathrm{\Omega },\mathcal{A},p,r)$ where $\forall s\in \mathcal{S}$, $\forall \omega \in \mathrm{\Omega }$, $\forall a\in \mathcal{A}$, reward function $r:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$, and transition function $p:S\times A\times S\times {\mathbb{R}}^{+}\to [0,1]$. The temporal abstraction mechanism consists of an option-selection policy ${\pi }_{\mu }:\mathcal{S}\times \mathrm{\Omega }\to \mathcal{P}\left(\mathrm{\Omega }\right)$, intra-option policy ${\pi }_U:\mathcal{S}\times \mathrm{\Omega }\to \mathcal{P}\left(\mathcal{A}\right)$, and termination function $\beta :\mathcal{S}\times \mathrm{\Omega }\to [0,1]$, where $\mathcal{P}\left(\mathrm{\Omega }\right)$ and $\mathcal{P}\left(\mathcal{A}\right)$ represent the set of probability distributions for the set of options $\mathrm{\Omega }$ and the set of actions $\mathcal{A}$, respectively, which are depicted by the Bellman equation as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill U^{*}(s,\omega )=(1-\beta (s,\omega ))Q_{\mathrm{\Omega }}^{*}(s,\omega )+\beta (s,\omega )\underset{\omega \in \mathrm{\Omega }}{max}{Q}_{\mathrm{\Omega }}^{*}(s,{\omega }^{\prime }),\end{array}\end{array}$$

(1)

where

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{\mathrm{\Omega }}^{*}(s,\omega )={\mathbb{E}}_{a\sim \pi (·|s,\omega )}\left[Q_U^{*}(s,\omega ,a)\right],\end{array}\end{array}$$

(2)

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_U^{*}(s,\omega ,a)=r(s,a)+\gamma {\mathbb{E}}_{s^{\prime }\sim p(·|s,a)}\left[U^{*}(s^{\prime },\omega )\right],\end{array}\end{array}$$

(3)

where $U:\mathcal{S}\times \mathrm{\Omega }\to \mathbb{R}$ is the option-value function upon arrival, $Q_{\mathrm{\Omega }}:\mathcal{S}\times \mathrm{\Omega }\to \mathbb{R}$ is the option-value function and $Q_U:\mathcal{S}\times \mathrm{\Omega }\times \mathcal{A}\to \mathbb{R}$ is the value function for executing an action in the context of a state-option pair called the state-option-action value function hereafter. The termination function $\beta$ enables us to interrupt the executed option, which achieves the varied timesteps for operating a subpolicy. Furthermore, it includes the gradient of a decision moment over the overall optimality. In summary, this study integrates the option termination function $\beta$ and option-selection policy ${\pi }_{\mu }$ of option learning and redefines the option strategy ${\pi }_{\mathrm{\Omega }}:\mathrm{\Omega }\times \mathcal{S}\times \mathrm{\Omega }\to [0,1]$ as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\pi }_{\mathrm{\Omega }}({\omega }_t\mid s_t,{\omega }_{t-1})=(1-\beta (s_t,{\omega }_{t-1})){\mathbf{1}}_{{\omega }_t={\omega }_{t-1}}+\beta (s_t,{\omega }_{t-1}){\pi }_{\mu }({\omega }_t\mid s_t),\end{array}\end{array}$$

(4)

where ${\mathbf{1}}_{{\omega }_t={\omega }_{t-1}}$ is an indicator function that takes the value of 1 when ${\omega }_t={\omega }_{t-1}$.

2.2. Intra-Option Q-Learning

In intra-option Q-learning, the optimal option value function upon arrival can be solved iteratively by the Bellman Equation (1). It is approximated by applying an off-policy one-step temporal difference, where the option-value function $Q_{\mathrm{\Omega }}^{{\varphi }_{\mathrm{\Omega }}}$ is modelled with the parameter ${\varphi }_{\mathrm{\Omega }}$. The $Q_{\mathrm{\Omega }}^{{\varphi }_{\mathrm{\Omega }}}$ is optimized by minimizing the squared residual error for all states and options. As a result, ${\varphi }_{\mathrm{\Omega }}$ is updated as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\varphi }_{\mathrm{\Omega }}\leftarrow {\varphi }_{\mathrm{\Omega }}+{\xi }_{{\theta }_{\omega }}{\widehat{\nabla }}_{{\varphi }_{\mathrm{\Omega }}}{Q}^{{\varphi }_{\mathrm{\Omega }}}(Q_{\mathrm{\Omega }}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_t,{\omega }_t)-Q_{\mathrm{\Omega }}^{{\varphi }_{\mathrm{\Omega }}}(s_t,{\omega }_t)),\end{array}\end{array}$$

(5)

where

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{\mathrm{\Omega }}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_t,{\omega }_t)=r(s_t,a_t)+\gamma U^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }_t),\end{array}\end{array}$$

(6)

and $U^{{\overline{\varphi }}_{\mathrm{\Omega }}}$ is approximated as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill U^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }_t)=(1-\beta (s_{t+1},{\omega }_t))Q_{\mathrm{\Omega }}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }_t)+\beta (s_{t+1},{\omega }_t)\underset{\omega \in \mathrm{\Omega }}{max}{Q}_{\mathrm{\Omega }}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }^{\prime }).\end{array}\end{array}$$

(7)

2.3. Within-Visual-Range Air Combat

WVRAC, otherwise referred to as dogfighting, denotes close-range tactical engagements between aircraft within visual contact. The tactical objective of WVRAC is to establish an effective gun firing envelope. In contradistinction to beyond-visual-range air combat, which relies more on radar and weapon performance, WVRAC imposes more stringent requirements on the manoeuvring capabilities of UCVAs. This study aims to explore the generation mechanism of autonomous combat strategies for UCAV agents in WVRAC based on gun weapons.

In the context of geometric representation, a three-dimensional spatial geometric relationship model for two UCAVs is established, as illustrated in Figure 1. The ground coordinate system is defined as the reference frame, wherein the adversary fighter and self-fighter are designated by subscripts a and s, respectively. The position vector in the local coordinate system is represented by the vector $\overrightarrow{p}$, while the velocity vector in the local coordinate system is represented by the vector $\overrightarrow{v}$. The vector $\overrightarrow{e}$ is the normalized vector of the X-axis in the body coordinate system. The Line of Sight $\overrightarrow{LOS}$ is the vector from the centroid of the self-fighter to the centroid of the adversary fighter, with its magnitude $\parallel \mathrm{LOS}\parallel$ being the distance $Range$ between the two aircraft. The rate of change of $Range$ is defined as the proximity rate $Proximity$. The Aspect Angle ($AA$) is defined as the angle between the heading vector of the adversary fighter and the $\overrightarrow{LOS}$, while the Antenna Train Angle ($ATA$) is the angle between the heading vector of the self-fighter and the $\overrightarrow{LOS}$. The Head Cross Angle ($HCA$) is the angle between the heading vectors of the self-fighter and the adversary fighter. It is important to note that the ranges of $AA$, $ATA$, and $HCA$ are all $[0,\pi ]$. The following formulas are used to calculate these parameters:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \left\{\begin{array}{c}\overrightarrow{\mathrm{LOS}}={\overrightarrow{p}}_a-{\overrightarrow{p}}_s\hfill \\ \mathrm{Range}=\Vert \overrightarrow{\mathrm{LOS}}\Vert \hfill \\ \mathrm{Proximity}=(\overrightarrow{v_a}-\overrightarrow{v_s})·{\displaystyle \frac{\overrightarrow{\mathrm{LOS}}}{\Vert \overrightarrow{\mathrm{LOS}}\Vert }}\hfill \\ \mathrm{ATA}=arccos\overrightarrow{e_s}·{\displaystyle \frac{\overrightarrow{\mathrm{LOS}}}{\Vert \overrightarrow{\mathrm{LOS}}\Vert }}\hfill \\ \mathrm{AA}=arccos\overrightarrow{e_a}·{\displaystyle \frac{\overrightarrow{\mathrm{LOS}}}{\Vert \overrightarrow{\mathrm{LOS}}\Vert }}\hfill \\ \mathrm{HCA}=arccos\overrightarrow{e_a}·\overrightarrow{e_s}\hfill \end{array}\right.,\end{array}\end{array}$$

(8)

2.4. Weapon Engagement Zone

The weapon engagement zone (WEZ) is predicated on the principle of WVRAC and the ballistic characteristics of the aerial gun. When the target distance exceeds the maximum effective striking distance set by the gun, the bullet’s kinetic energy will be attenuated to the extent that it will not be able to produce a lethal effect on the adversary fighter. Conversely, when the distance is less than the minimum safe firing distance set by the gun, the weapon system will trigger the prohibited logic to avoid the secondary damage caused by the splashing back of the bullet fragments to the UCAV. The maximum effective striking distance of the machine gun was defined as 1000 $\mathrm{m}$, while the minimum safe firing distance was set at 150 $\mathrm{m}$. The effective range of the aerial gun was simulated through the construction of a three-dimensional conical weapon attack area (illustrated in Figure 2). The targetable angle range is constrained to the three-dimensional angular domain of ±${15}^{\circ }$ on both sides of the nose axis. In the event that an adversary fighter enters the WEZ, the system will initiate the virtual fire projection process.

3. Methods

According to [23], MEOL applies maximum-entropy objective to both levels in the option framework and achieves efficient automatic skill discovery and selection in complex continuous tasks. In this section, we decouple MEOL for option discovery and option utilization to generate particular tactics and selection strategies. With reward shaping, the former learns tactical manoeuvring (angle, snapshot, and energy tactics). Additionally, the WGAN method was employed to facilitate automated curriculum generation, thereby assisting MEOL in option discovery. The latter learns to terminate the current tactic and select a new one.

3.1. Automatic Curriculum Generation with WGANs for Option Discovery

We build upon the concepts outlined in [26,27] by implementing Wasserstein Generative Adversarial Networks (WGANs) [24] to identify a sequence of learnable tasks for policy training. To address the critical challenge of mode collapse inherent in traditional GAN frameworks [28], we leverage the theoretical advantages of Wasserstein distance optimization with gradient penalty regularization [29]. This approach ensures stable training dynamics while maintaining high sample diversity through Lipschitz-constrained adversarial learning.

3.1.1. Labelling Positive Initial Condition

The initial engagement situation can be continuously parameterized as an initial condition $x\in \mathcal{I}\subseteq {\mathbb{R}}^d$. And the continuously parameterized tasks of universe initial situation are $\mathcal{T}\left(x\right)$. We define the intermediate-difficulty set ${\mathcal{I}}_{feasible,i}$ as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathcal{I}}_{feasible,i}=\{x:l_{win,min}\le l_{win,{\pi }_{U,i}},l_{win,{\pi }_{U,i}}\le l_{win,max}\}\subseteq \mathcal{I},\end{array}\end{array}$$

(9)

where $l_{\mathrm{win},min}$ and $l_{\mathrm{win},max}$ are the boundary values defining the win rate interval for moderately difficult tasks, and $l_{\mathrm{win},{\pi }_{U,i}}$ represents the average win rate of strategy ${\pi }_{U,i}$ under repeated confrontations on this course.

The above method is referenced in [27]. We screen the curriculum samples in the air combat task by presetting the win rate interval, and train the generative model to generate curricula with moderate difficulty, thus reducing the training difficulty of reinforcement learning.

3.1.2. Wasserstein Generative Adversarial Networks

According to [24], the adversarial training framework consists of a generator $G_{\psi }$ and a critic $D_{\varphi }$ that optimize the Wasserstein distance through a minimax game. The objective function is defined as

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \underset{\psi }{min}\underset{\varphi }{max}{\mathbb{E}}_{x\sim {\mathbb{P}}_r}\left[D_{\varphi }\left(x\right)\right]-{\mathbb{E}}_{z\sim p\left(z\right)}\left[D_{\varphi }\left(G_{\psi }\left(z\right)\right)\right],\end{array}\end{array}$$

(10)

where z is sampled from latent space $p\left(z\right)$, and ${\mathbb{P}}_r$ represents the real data distribution. The critic is constrained to be 1-Lipschitz through gradient penalty [29]:

$${\mathcal{L}}_{GP}=\lambda {\mathbb{E}}_{\widehat{x}\sim {\mathbb{P}}_{\widehat{x}}}\left[\right(\left|\right|{\nabla }_{\widehat{x}}{D}_{\varphi }\left(\widehat{x}\right){\left|\right|}_2-1{)}^2].$$

(11)

For curriculum generation, we designed the generator to produce environment parameters $x_0=G_{\psi }\left(z\right)$ with latent codes $z\sim \mathcal{N}(0,\mathbf{I})$. The quality assessment of generated environments was performed through policy optimization:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{env}}={\mathbb{E}}_{z\sim p\left(z\right)}[-\mathcal{R}({\pi }_{\theta }^{*},G_{\psi }\left(z\right))],\end{array}$$

(12)

where $\mathcal{R}$ represents the winning rate of optimized policy ${\pi }_{\theta }^{*}$ from Algorithm 1. The complete optimization objective combines adversarial training with curriculum learning:

$${\mathcal{L}}_{\mathrm{total}}=\mathbb{E}\left[D_{\varphi }\left(x_{\mathrm{real}}\right)\right]-\mathbb{E}\left[D_{\varphi }\left(G_{\psi }\left(z\right)\right)\right]+\gamma {\mathcal{L}}_{\mathrm{env}}+{\mathcal{L}}_{GP}.$$

(13)

The implementation of the algorithm for automatic curriculum generation for option discovery with WGANs is shown in Algorithm 2.

Algorithm 1 Option discovery within MEOL framework for subpolicy ${\omega }_i$

1: Initialize the parameterized air combat environment with initial condition p   2: Initialize parameter vectors ${\theta }_{U,{\omega }_i}$, ${\varphi }_{U,{\omega }_i}$, ${\alpha }_{U,{\omega }_i}$   3: Assign target network ${\overline{\varphi }}_{U,{\omega }_i}\leftarrow {\varphi }_{U,{\omega }_i}$   4: Initialize learning rate ${\xi }_{{\varphi }_{U,{\omega }_i}}$, ${\xi }_{{\theta }_{U,{\omega }_i}}$, ${\xi }_{{\alpha }_{U,{\omega }_i}}$,${\sigma }_{U,{\omega }_i}$   5: for each batch do   6:     Select an action $a_t\sim {\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_t\right|s_t,{\omega }_i)$   7:     Observe reward $r_{{\omega }_i,t+1}$ and $s_{t+1}$   8:     Store transition in $\mathcal{D}\cup (s_t,a_t,r_{{\omega }_i,t+1},s_{t+1})$   9:     Sample a training batch of transitions from $\mathcal{D}$ 10:     Update ${\varphi }_{U,{\omega }_i}$ according to (14) 11:     Update ${\theta }_{U,{\omega }_i}$ according to (17) 12:     Update ${\overline{\varphi }}_{U,{\omega }_i}$, ${\overline{\varphi }}_{U,{\omega }_i}\leftarrow {\sigma }_{U,{\omega }_i}{\varphi }_{U,{\omega }_i}+(1-{\sigma }_{U,{\omega }_i}{\overline{\varphi }}_{U,{\omega }_i})$ 13:     Update ${\alpha }_{U,{\omega }_i}$ according to (19) 14: end for

Algorithm 2 Wasserstein Curriculum Generation for Option Discovery

1: Initialize generator $G_{\psi }$ and critic $D_{\varphi }$   2: Initialize latent dimension $d_z$, gradient penalty coefficient $\lambda$   3: Initialize learning rates ${\xi }_{\psi },{\xi }_{\varphi }$   4: for each training epoch do   5:     for k critic steps do   6:         Sample real environments ${\left\{x^{\left(i\right)}\right\}}_{i=1}^m\sim {\mathbb{P}}_r$   7:         Sample latent codes ${\left\{z^{\left(i\right)}\right\}}_{i=1}^m\sim p\left(z\right)$   8:         Generate synthetic environments ${\tilde{x}}^{\left(i\right)}=G_{\psi }\left(z^{\left(i\right)}\right)$   9:         Compute gradient penalty ${\mathcal{L}}_{GP}$ 10:         Update critic: $\varphi \leftarrow \varphi -{\xi }_{\varphi }{\nabla }_{\varphi }{\mathcal{L}}_{\mathrm{total}}$ 11:     end for 12:     Sample new latent codes ${\left\{z^{\left(j\right)}\right\}}_{j=1}^n\sim p\left(z\right)$ 13:     for each generated environment $G_{\psi }\left(z^{\left(j\right)}\right)$ do 14:         Initialize parameterized environment with $x_0^{\left(j\right)}$ 15:         Execute Algorithm 1 for intra-option policy optimization 16:         Evaluate winning rate $\mathcal{R}({\pi }_{\theta }^{*},x_0^{\left(j\right)})$ 17:     end for 18:     Compute environment quality reward ${\mathcal{L}}_{\mathrm{env}}$ 19:     Update generator: $\psi \leftarrow \psi -{\xi }_{\psi }{\nabla }_{\psi }{\mathcal{L}}_{\mathrm{total}}$ 20: end for

3.2. Option Discovery Within MEOL Framework

When training the low-level policy, the options $\omega$ are fixed as an identification of different tactical intra-option policy. For each option or tactic ${\omega }_i$, the parameters ${\varphi }_{U,{\omega }_i}$ of the soft option-action-value function $Q_{soft,U}^{{\varphi }_{U,{\omega }_i}}$ are updated as

$$\begin{array}{c}\hfill {\varphi }_{U,{\omega }_i}\leftarrow {\varphi }_{U,{\omega }_i}-{\xi }_{{\varphi }_{U,{\omega }_i}}{\nabla }_{{\varphi }_{U,{\omega }_i}}{Q}_{soft,U}^{{\varphi }_{U,{\omega }_i}}(s_t,{\omega }_i,a_t)(Q_{soft,U}^{{\overline{\varphi }}_{U,{\omega }_i}}(s_t,{\omega }_i,a_t)-Q_{soft,U}^{{\varphi }_{U,{\omega }_i}}(s_t,{\omega }_i,a_t)),\end{array}$$

(14)

with the target parameters ${\overline{\varphi }}_{U,{\omega }_i}$

$$\begin{array}{c}\hfill Q_{soft,U}^{{\overline{\varphi }}_{U,{\omega }_i}}(s_t,{\omega }_i,a_t)=r_{{\omega }_i}(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}\sim p\left(s_{t+1}\right|s_t,a_t)}\left[U_{soft}^{{\overline{\varphi }}_{U,{\omega }_i}}(s_{t+1},{\omega }_i)\right],\end{array}$$

(15)

where

$$\begin{array}{cc}\hfill & U_{soft}^{{\overline{\varphi }}_{U,{\omega }_i}}(s_{t+1},{\omega }_i)={\mathbb{E}}_{a_{t+1}\sim {\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_{t+1}\right|s_{t+1},{\omega }_i)}[\hfill \\ \hfill & Q_{soft,U}^{{\overline{\varphi }}_{U,{\omega }_i}}(s_{t+1},{\omega }_i,a_{t+1})-{\alpha }_{U,{\omega }_i}log{\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_{t+1}\right|s_{t+1},{\omega }_i)].\hfill \end{array}$$

(16)

The parameters for ${\theta }_{U,{\omega }_i}$ of intra-option policy ${\pi }_U^{{\theta }_{U,{\omega }_i}}$ are updated as

$${\theta }_{U,{\omega }_i}\leftarrow {\theta }_{U,{\omega }_i}-{\xi }_{{\theta }_{U,{\omega }_i}}{\widehat{\nabla }}_{{\theta }_{U,{\omega }_i}}{J}_{{\pi }_U}\left({\theta }_{U,{\omega }_i}\right),$$

(17)

where

$$\begin{array}{cc}\hfill & {\widehat{\nabla }}_{{\theta }_{U,{\omega }_i}}{J}_{{\pi }_U}\left({\theta }_{U,{\omega }_i}\right)={\alpha }_{U,{\omega }_i}{\nabla }_{{\theta }_{U,{\omega }_i}}log{\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_t\right|s_t,{\omega }_i)\hfill \\ \hfill & +{\nabla }_{a_t}({\alpha }_{U,{\omega }_i}log{\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_t\right|s_t,{\omega }_i)-Q_{soft,U}^{{\varphi }_{U,{\omega }_i}}(s_t,{\omega }_i,a_t)){\nabla }_{{\theta }_{U,{\omega }_i}}{f}_{{\theta }_{U,{\omega }_i}}({ϵ}_t;s_t,{\omega }_i).\hfill \end{array}$$

(18)

And the coefficient ${\alpha }_{U,{\omega }_i}$ for regulating the entropy of intra-option policy is adjusted as

$${\alpha }_{U,{\omega }_i}\leftarrow {\alpha }_{U,{\omega }_i}-{\xi }_{{\alpha }_{U,{\omega }_i}}{\widehat{\nabla }}_{{\alpha }_{U,{\omega }_i}}{J}_{\overline{\mathcal{H}}}\left({\alpha }_{U,{\omega }_i}\right),$$

(19)

in which

$$J_{{\mathcal{H}}_U}\left({\alpha }_{U,{\omega }_i}\right)={\mathbb{E}}_{a_t\sim {\pi }_U^{{\theta }_{U,{\omega }_i}}}[-log{\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_t\right|s_t,{\omega }_i)]-{\overline{\mathcal{H}}}_{U,{\omega }_i}.$$

(20)

3.3. Option Utilization Within MEOL Framework

For a higher level, the parameters ${\varphi }_{\mathrm{\Omega }}$ of the soft option-value function $Q_{soft,\mathrm{\Omega }}^{{\varphi }_{\mathrm{\Omega }}}$ are updated as

$$\begin{array}{cc}\hfill {\varphi }_{\mathrm{\Omega }}\leftarrow & {\varphi }_{\mathrm{\Omega }}-{\xi }_{{\varphi }_{\mathrm{\Omega }}}{\nabla }_{{\varphi }_{\mathrm{\Omega }}}{Q}_{soft,\mathrm{\Omega }}^{{\varphi }_{\mathrm{\Omega }}}(s_t,{\omega }_t)({\widehat{Q}}_{soft,\mathrm{\Omega }}^{{\overline{\varphi }}_U}(s_t,{\omega }_t)-Q_{soft,\mathrm{\Omega }}^{{\varphi }_{\mathrm{\Omega }}}(s_t,{\omega }_t)),\hfill \end{array}$$

(21)

where the target parameters ${\overline{\varphi }}_{\mathrm{\Omega }}$

$$\begin{array}{cc}\hfill {\widehat{Q}}_{soft,\mathrm{\Omega }}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_t,{\omega }_t)=& r(s_t,a_t)+{\widehat{U}}_{soft}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }_t),\hfill \end{array}$$

(22)

in which

$$\begin{array}{cc}\hfill & U_{soft}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }_t)={\mathbb{E}}_{{\omega }_{t+1}\sim {\pi }_{\mathrm{\Omega },\theta }\left({\omega }_{t+1}\right|s_{t+1},{\omega }_t)}[\hfill \\ \hfill & Q_{soft,\mathrm{\Omega }}^{{\overline{\varphi }}_{\mathrm{\Omega }}}(s_{t+1},{\omega }_{t+1})-{\alpha }_{\mathrm{\Omega }}log{\pi }_{\mathrm{\Omega }}^{{\theta }_{\mathrm{\Omega }}}\left({\omega }_{t+1}\right|s_{t+1},{\omega }_t)].\hfill \end{array}$$

(23)

The parameters ${\theta }_{\mathrm{\Omega }}$ of the option-selection policy ${\pi }_{\mathrm{\Omega }}^{{\theta }_{\mathrm{\Omega }}}$ are updated as

$${\theta }_{\mathrm{\Omega }}\leftarrow {\theta }_{\mathrm{\Omega }}-{\xi }_{{\theta }_{\mathrm{\Omega }}}{\widehat{\nabla }}_{{\theta }_{\mathrm{\Omega }}}{J}_{{\pi }_{\mathrm{\Omega }}}\left({\theta }_{\mathrm{\Omega }}\right),$$

(24)

in which

$$J_{{\mathcal{H}}_{\mathrm{\Omega }}}\left({\alpha }_{\mathrm{\Omega }}\right)={\mathbb{E}}_{{\omega }_t\sim {\pi }_{\mathrm{\Omega }}^{{\theta }_{\mathrm{\Omega }}}}[-log{\pi }_{\mathrm{\Omega }}^{{\theta }_{\mathrm{\Omega }}}\left({\omega }_t\right|s_t,{\omega }_{t-1})]-{\overline{\mathcal{H}}}_{\mathrm{\Omega }}.$$

(25)

The process of option utilization within the MEOL framework is governed as in Algorithm 3.

Algorithm 3 Option utilization within MEOL framework

1: Initialize parameter vectors ${\theta }_{\mathrm{\Omega }}$, ${\varphi }_{\mathrm{\Omega }}$, ${\alpha }_{\mathrm{\Omega }}$   2: Assign target network ${\overline{\varphi }}_{\mathrm{\Omega }}\leftarrow {\varphi }_{\mathrm{\Omega }}$   3: Initialize learning rate ${\xi }_{{\varphi }_{\mathrm{\Omega }}}$, ${\xi }_{{\theta }_{\mathrm{\Omega }}}$, ${\xi }_{{\alpha }_{\mathrm{\Omega }}}$,${\sigma }_{\mathrm{\Omega }}$   4: Assign each ${\pi }_{U,{\omega }_i}$ with trained ${\theta }_{U,{\omega }_i}$   5: for each epoch do   6:     for each batch do   7:         Select an option ${\omega }_t\sim {\pi }_{\mathrm{\Omega }}^{{\theta }_{\mathrm{\Omega }}}\left({\omega }_t\right|s_t,{\omega }_{t-1})$   8:         Select an action $a_t\sim {\pi }_U^{{\theta }_{U,{\omega }_i}}\left(a_t\right|s_t,{\omega }_t)$   9:         Observe reward $r_{t+1}$ and $s_{t+1}$ 10:         Store transition in $\mathcal{D}\cup ({\omega }_{t-1},s_t,{\omega }_t,a_t,r_{t+1},s_{t+1})$ 11:         Sample a training batch of transitions from $\mathcal{D}$ 12:         Update ${\varphi }_{\mathrm{\Omega }}$ according to (21) 13:         Update ${\theta }_{\mathrm{\Omega }}$ according to (24) 14:         Update ${\overline{\varphi }}_{\mathrm{\Omega }}$, ${\overline{\varphi }}_{\mathrm{\Omega }}\leftarrow {\sigma }_{\mathrm{\Omega }}{\varphi }_{\mathrm{\Omega }}+(1-{\sigma }_{\mathrm{\Omega }}{\overline{\varphi }}_{\mathrm{\Omega }})$ 15:         Update ${\alpha }_{\mathrm{\Omega }}$ according to (25) 16:     end for 17:     end for

4. Hierarchical Training Architecture

In this section, we propose the utilization of parameterized air combat scenarios for the purpose of automatic curriculum reinforcement learning. Secondly, the reward shaping is designed for intra-option policies for tactical manoeuvring and control. Finally, the sparse reward of combat results is retained for the option-selection policy, with the objective of enhancing the success rate of UCAV air combat. The complete structure is delineated in Figure 3.

4.1. Parameterized Environment of Air Combat Scenario

According to [11], the nature of the initial engagement situation dictates the subsequent manoeuvres, a principle that remains applicable in the context of the UCAVs’ WVRAC. For instance, if the initial engagement situation is defined as ‘offensive’, signifying that the self-fighter maintains a position at the rear quarter of the adversary, the self fighter is expected to execute a pursuit manoeuvre in order to engage the target. Conversely, in an initial engagement situation that is head-on, characterized by both fighters occupying the front quarter of each other’s respective positions, the employment of either a one-circle or two-circle manoeuvre is to be considered. It is important to note that a wide range of initial engagement scenarios are interwoven with the curriculum, as elaborated in [10]. In this paper, the parameters of initial condition are included as follows:

$$(\mathrm{Range},\mathrm{AA},\mathrm{ATA})$$

(26)

where the value domain is listed in

Table 1. Initial condition of parameterized air combat environment.

Initial Parameter	Value Domain
Range	$[1900\mathrm{m},5100\mathrm{m}]$
AA	$[-{180}^{\circ },{180}^{\circ }]$
ATA	$[-{180}^{\circ },{180}^{\circ }]$

.

4.2. Construction of Tactical Intra-Option Policy

To facilitate the training of tactical intra-option policy, the learning objectives are aligned with the core principles of WVRAC [11]. Each intra-option policy is specialized for one of the three canonical tactics: angle, energy, and snapshot.

Faced with the sparse feedback and the problem of the long sequence of air combat, reward shaping [30,31] has proven to be efficient in training complex tasks by merging with the domain knowledge. Consequently, the basic term of the reward function for training tactical intra-option policies is composed of the sparse reward of air combat results $R_{goal}$, the shaping reward $R_{shaping}$, and the regularization term to constrain behaviour $R_{regularization}$, which is given below:

$$R=R_{goal}+R_{shaping}+R_{regularization}.$$

(27)

Mathematically, the reward shaping is designed based on the potential situation assessment, which measures the advantages in air combat geometry, including range distance, orientation, and energy advantages.

The potential function of the range distance ${\mathrm{\Phi }}_{range}$ is generally an exponential term, which is formalized as

$${\mathrm{\Phi }}_{range}=exp(-{\displaystyle \frac{\left|\mathrm{Range}-d_r\right|}{c_r}})$$

(28)

The potential functions of orientation, ${\mathrm{\Phi }}_{AA}$ and ${\mathrm{\Phi }}_{ATA}$, are the exponential forms of the angles AA and ATA, which are given as

$${\mathrm{\Phi }}_{AA}=exp(-{\displaystyle \frac{\left|\mathrm{AA}\right|}{c_{AA}}}),$$

(29)

$${\mathrm{\Phi }}_{ATA}=exp(-{\displaystyle \frac{\left|\mathrm{ATA}\right|}{c_{ATA}}}),$$

(30)

The potential function of energy ${\mathrm{\Phi }}_{orientation}$ is the ratio of the self-fighter and adversary fighter’s energy:

$${\mathrm{\Phi }}_{energy}={\displaystyle \frac{E_s}{E_a}},$$

(31)

where $E_s$ and $E_a$ are the energies of the self- and adversary fighters, respectively, which are jointly defined as

$$E=altitude+{\displaystyle \frac{V_{IAS}^2}{2g}},$$

(32)

The regularization terms play a vital role in improving flight performance and penalizing dangerous flight manoeuvring, which is an implicit way to avoid suboptimality.

The $P_{deck}$ penalizes the fighter agent for reaching lower than the minimum altitude, which prevents the agent from approaching too close to the horizon:

$$P_{deck}=\left\{\begin{array}{c}-1.0,altitude\le altitud{e}_{deck}\hfill \\ 0.0,otherwise\hfill \end{array}\right..$$

(33)

The $P_{V_{IAS}}$ penalizes the fighter agent for a low indicated airspeed $V_{IAS}$, as this would make it difficult to provide enough lift for the fighter:

$$P_{V_{IAS}}=\left\{\begin{array}{c}-{((V_{IAS}-V_{corner})/c_{V_{IAS}})}^2,V_{IAS}\le V_{corner}\hfill \\ 0.0,otherwise\hfill \end{array}\right..$$

(34)

The $P_{{\alpha }_{AoA}}$ penalizes the fighter agent for violating the stall of angle of attack to avoid stall, which is

$$P_{{\alpha }_{AoA}}=\left\{\begin{array}{c}-1,\left|{\alpha }_{AoA}\right|\le {\alpha }_{AoA,stall}\hfill \\ 0,otherwise\hfill \end{array}\right..$$

(35)

To maximize manoeuvrability alongside the flight envelope, $P_{deck}$, $P_{V_{IAS}}$, and $P_{{\alpha }_{AoA}}$ only take effect when the corresponding parameters are out of bounds.

The $P_{\beta }$ penalizes the fighter agent for the sideslip, which causes energy inefficiency and aircraft deviation from the expected flight status:

$$P_{\beta }=\left\{\begin{array}{c}-{(\beta /c_{\beta })}^2,\left|\beta \right|\le {\beta }_{max}\hfill \\ -1.0,otherwise\hfill \end{array}\right..$$

(36)

The ideal status is when there is no sideslip, so the penalty is set to be dense.

The cost of action $C_{action}$ smooths the control output [32], which stabilizes the response process:

$$C_{action}=-0.1{\left(\left|action\right|\right)}_2.$$

(37)

4.2.1. Reward Function for Angle Tactic

The fundamental objective of the angle tactic is to swiftly modify the UCAVs’ flight paths to ensure a nose-pointing advantage, a strategy that is particularly advantageous in WVRAC. In such situations, it is advantageous for the self-fighter to position itself in the 6 o’clock direction relative to the adversary fighter, thereby establishing a trailing position. In comparison to the other two tactics, the angle tactic does not involve the separation of the AA angle as a separate term. Rather, it integrates the potential functions of AA, ATA, and range, with the objective of achieving an angular superiority over the adversary aircraft at the optimal distance. It is evident that both $c_0$ and $c_1$ can be readily modified as hyperparameters. The angle tactic’s reward function is defined as follows:

$$R_{angle}=R_{goal}+\underset{\mathrm{shaping}}{\underbrace{c_0{\mathrm{\Phi }}_{ATA}{\mathrm{\Phi }}_{AA}{\mathrm{\Phi }}_{range}+c_1{\mathrm{\Phi }}_{energy}+R_{proximity}}}+\underset{\mathrm{regularization}}{\underbrace{P_{deck}+P_V+P_{\alpha }+P_{\beta }+C_{action}}}$$

(38)

The rest of the terms in the reward functions are shared with each other. Cooperating with ${\mathrm{\Phi }}_{range}$, the $R_{proximity}$ is proposed to accelerate the decreasing process of the range distance, which is

$$R_{proximity}=-\mathrm{Proximity}/1000.$$

(39)

When the range distance is decreased, proximity is negative with $R_{proximity}$ set as positive. Otherwise, proximity is positive with $R_{proximity}$ set as negative. This reward takes dominant effect in closing the adversary when the range distance is large and the influence of ${\mathrm{\Phi }}_{range}$ is feeble.

4.2.2. Reward Function for Snapshot Tactic

The snapshot tactic is indicative of a more aggressive strategy, with an emphasis on the capture of the shooting window. This is intuitively reflected in the reward function, which places a higher value on the AA. This approach suggests a certain degree of disregard for potential attacks by adversary fighters. The design of the reward function for the snapshot tactic is as follows:

$$R_{snapshot}=R_{goal}+\underset{\mathrm{shaping}}{\underbrace{c_0{\mathrm{\Phi }}_{ATA}{\mathrm{\Phi }}_{range}+{\mathrm{\Phi }}_{AA}+c_1{\mathrm{\Phi }}_{energy}+R_{proximity}}}+\underset{\mathrm{regularization}}{\underbrace{P_{deck}+P_V+P_{\alpha }+P_{\beta }+C_{action}}}$$

(40)

4.2.3. Reward Function for Energy Tactic

Energy tactics are centred on the accumulation and utilization of energy, a critical aspect of the WVRAC. Large-angle manoeuvres necessitate substantial energy reserves to support them, and the exchange of energy for angular and positional advantages constitutes a form of attack strategy in WVRAC. The design of the reward function for the energy tactic is as follows:

$$R_{energy}=R_{goal}+\underset{\mathrm{shaping}}{\underbrace{c_0{\mathrm{\Phi }}_{ATA}{\mathrm{\Phi }}_{range}{\mathrm{\Phi }}_{energy}+c_1{\mathrm{\Phi }}_{AA}+R_{proximity}}}+\underset{\mathrm{regularization}}{\underbrace{P_{deck}+P_V+P_{\alpha }+P_{\beta }+C_{action}}}$$

(41)

4.3. Construction of Tactical Option-Selection Policy

The option-selection policy serves as a high-level decision-maker that dynamically chooses tactical options based on real-time combat situations, which is designed to choose the most appropriate tactic among the three intra-option tactics (angle, snapshot, and energy) based on the current situation of air combat. Inspired by the maximum-entropy framework in MEOL, the policy balances the exploitation of learned tactics and exploration of option specialization through entropy regularization. The reward function extends the sparse combat outcome with entropy incentives:

$$\begin{array}{c}\hfill R_{tactic}=R_{goal}\end{array}$$

(42)

4.4. Observation Space and Action Space

In the real WVRAC, the self-fighter’s onboard sensors limit the acquisition of adversary fighter information. However, in modern air warfare, the intervention of ground radar, early warning aircraft, and other systems enables more accurate access to the adversary fighter position, speed, trajectory, and other information. Consequently, the observation space for each UCAV agent encompasses both raw data from the environment and processed information representing the air combat geometry. The raw data, including indicated airspeed ($V_{IAS}$), angle of attack (${\alpha }_{AoA}$), sideslip angle ($\beta$), and altitude, are used to reflect flight safety and the quality of manoeuvring. The incorporation of processed data, such as range, proximity rate, AA, and ATA, serves to depict the positional relationship with the adversary, aligning with the pilot’s perspective. These states are subsequently fed into the network after undergoing a process of normalization:

$$(\mathrm{Range},\mathrm{Proximity},\mathrm{AA},\mathrm{ATA},V_{IAS},altitude,{\alpha }_{AoA},\beta )$$

(43)

To comprehensively analyse and optimize the tactics and manoeuvring performance of UCAVs, this study employs a continuous action space, with control commands for ailerons, rudder, elevator, and throttle as outputs:

$$({\delta }_a,{\delta }_r,{\delta }_e,{\delta }_t)$$

(44)

5. Experiment

The objective of this study is to investigate the role of MEOL in facilitating the development of tactical and manoeuvring skills in UCAV agents. To this end, a series of experiments have been conducted to address the following research questions: (1) Does MEOL enable the discovery of manoeuvring skills that are necessary for tactics at a low level? (2) Does the option framework effectively integrate tactics through high-level training? (3) How do trained UCAV agents cooperate with tactics and manoeuvre hierarchically?

Accordingly, the initial phase of the training process involves the training of the angle, snapshot, and energy tactics, along with the corresponding reward designs, to establish tactical intra-option policies. Subsequently, the option-selection policy for tactic selection was trained using this method. Furthermore, a pair of comparative experiments were employed to illustrate the MEOL framework’s superior performance. An ablated study demonstrates the effectiveness of automatic curriculum generation techniques based on WGANs. In the context of the adversary fighter, we designed the rule-based expert adversary fighter from the doctrines and instructions in [11]. Finally, we visualized tactical manoeuvring as well as options for tactical selection in the air combat process. The overall hyperparameters are in

Table 2. Initial condition of parameterized air combat environment.

Hyperparameters	Value
Number of hidden layers	2
Dimensions of hidden layers	256
Nonlinearity of hidden layers	ReLU
Sample batch size	5000
Mini-batch size	256
Reward scale	1
Discount factor ($\gamma$)	0.99
Target update interval	1
Gradient steps	100
Total environment steps	$1.5\times {10}^7$
Optimizer	Adam
Reward function parameter (${\mathbf{c}}_0$)	10
WGAN latent dimension ($d_z$)	128
WGAN gradient penalty coefficient ($\lambda$)	10
Target action entropy (${\overline{\mathcal{H}}}_{U,{\omega }_i}$)	−4
Initial action entropy (${\alpha }_{U,{\omega }_i}$)	0.1
Target option entropy (${\overline{\mathcal{H}}}_{\mathrm{\Omega }}$)	1
Initial option entropy (${\alpha }_{\varphi }$)	0.1
Learning rate of WGAN generator (${\xi }_{\psi }$)	$1\times {10}^{-4}$
Learning rate of WGAN critic (${\xi }_{\theta }$)	$5\times {10}^{-5}$
Target Q-network soft update factor (${\tau }_{U,{\omega }_i}$)	$5\times {10}^{-3}$
Learning rate of option selection (${\xi }_{{\theta }_{\mathrm{\Omega }}}$)	$3\times {10}^{-5}$
Learning rate of intra-option selection (${\xi }_{{\theta }_{U,{\omega }_i}}$)	$3\times {10}^{-4}$
Learning rate of Q function (${\xi }_{{\varphi }_{\mathrm{\Omega }}}$)	$3\times {10}^{-4}$
Learning rate of Sub-Q function (${\sigma }_{U,{\omega }_i}$)	$3\times {10}^{-4}$
Learning rate of option alpha (${\xi }_{{\alpha }_{\mathrm{\Omega }}}$)	$3\times {10}^{-5}$
Learning rate of action entropy (${\xi }_{{\alpha }_{U,{\omega }_i}}$)	$3\times {10}^{-4}$

.

5.1. Air Combat Simulation Platform for HRL Training

In the implementation stage, the high-fidelity flight dynamic model (FDM) of the F-16 Fighting Falcon, derived from the JSBsim 1.1.6 [33], was utilized as the UCAV agent. Subsequently, the WEZ, observation assessment, and gaming judgment systems were constructed to facilitate the air combat simulation. The data, encompassing the state of the FDM and the control commands from the environment, were transmitted at a frequency of 50 $\mathrm{Hz}$. The API (application interference) was provided for the purpose of training. The implementation of hierarchical reinforcement learning was carried out by Pytorch 1.8.0+cu111 and supported by Garage [34]. Additionally, Tacview 1.9.4 was utilized for the purpose of the UCAVs’ flight log analysis.

5.2. Decoupled Training for Tactical Intra-Option Policies

In this subsection, the three types of intra-option policies—angle, snapshot, and energy—that were trained individually based on the previous construction are presented. The characteristics of each of the three types of intra-option policies were analysed to demonstrate the generative power of the underlying tactics in the method.

5.2.1. Angle Tactic

The angle tactic was trained in accordance with the defined reward function. The success rate during the training phase was calculated based on the most recent 100 assessment results. The same calculation method was applied to the snapshot and energy tactics. The success rate curve of the angle tactic training process is shown in Figure 4. After the success rate reached 0.6 for the first time, a precipitous decline was observed, which may have been due to the increase in exploration rate caused by the overfitting of the policy. However, this ‘regression’ facilitated the transcendence of the limitations imposed by local optima. The subsequent recovery of the win rate curve also proves the success of the exploration of the better strategy path, and the final win rate stabilized at about 0.6.

The ensuing engagement between the self-fighter and the adversary fighter is illustrated in Figure 5. The initial situation was characterized by a neutral engagement. Between time markers 0 and 1, both UCAVs sought smaller turning radii through energy-to-kinetic conversion to enter each other’s loops. Subsequent to this, during the WVRAC single-loop dogfight from time markers 1 to 2, both parties minimized their turning radii to gain angular advantages. Between time markers 2 and 3, the self-fighter attained positional and angular superiority through two consecutive barrel rolls. Subsequently, from time markers 3 to 4, while the adversary fighter performed an S-turn, the self-fighter moved inside its turning radius and successfully shot it down. Figure 6 provides a visual representation of the UCAVs’ angular positions, the distance between them, the proximity rate, and the energy changes during the engagement.

5.2.2. Snapshot Tactic

As demonstrated in Figure 7, the win rate of the snapshot tactic exhibited stability within the 0.4 to 0.5 range following the training phase. The underlying reason for this relative deficiency in comparison to the other two tactics is that the fundamental design concept of the snapshot tactic is to prioritize the maintenance of the AA angle advantage. This approach, however, comes at the expense of responsiveness to potential threats from an adversary fighter.

A detailed examination of a typical engagement of the snapshot tactic (Figure 8) reveals that at time marker 0, the adversary fighter is located at the 11 o’clock position, and the AA angle remains stable at approximately 100°. From time markers 0 to 1, the self-fighter executed a rapid turn manoeuvre, successfully positioning itself behind the adversary’s 3/9 line. In response, the adversary fighter, perceiving the threat, adopted a strategy of potential and kinetic energy conversion, and endeavoured to extricate itself from the unfavourable situation by descending and turning sharply. At time markers 1 and 2, both sides employed descending turn manoeuvres in a coincidental manner. As illustrated in Figure 9f, the self-fighter encountered an energy deficit during this phase, which hindered our ability to lock on to the adversary fighter and launch an effective attack. Prior to time marker 3, a strategic realignment was executed, involving the adoption of a more assertive descending turn tactic. Utilizing precise manoeuvring capabilities, the self-fighter successfully infiltrated the inner ring of the enemy’s turn, thereby seizing a valuable window of opportunity to initiate offensive action.

A comparison of the snapshot tactic with the other two tactics at the macro level (Figure 9) reveals distinctive features attributable to the difference in the reward function. As illustrated in Figure 9a, the snapshot tactic demonstrates a particular strength in AA angle advantage retention, which is its core strength. However, as demonstrated in Figure 9f, the snapshot tactic’s relative paucity of emphasis on energy resulted in the self-fighter being at a long-term disadvantage in terms of energy posture, which also limited its win rate to a certain extent.

5.2.3. Energy Tactic

As demonstrated in Figure 10, the success rate of the energy tactic in the training process was eventually stabilized between 0.5 and 0.6. A refined analysis of the single confrontation process (Figure 11) demonstrates that at the initial moment of the confrontation (time marker 0), the self-fighter was in the 1 o’clock direction of the adversary fighter, a position that placed the self-fighter at a disadvantage in the initial posture. However, from time markers 0 to 1, the self-fighter successfully shifted both sides into an anisotropic position by rapidly dropping altitude, thereby simultaneously accruing a significant energy advantage. This in turn ensured strong energy support for subsequent manoeuvres. From time markers 1 to 2, the self fighter adroitly entered the rear of the adversary’s 3/9 line through a Low Yo-Yo manoeuvre. This pivotal manoeuvre ensured the self-fighter’s advantageous positioning within the tactical arrangement, thereby establishing the conditions for the subsequent attack. Subsequently, from time markers 2 to 3, the self-fighter executed forward tracking manoeuvres, successfully cutting into the adversary fighter’s turning radius and neutralizing the enemy. The angle, range, closure, and energy ratios during the engagement are shown in Figure 12.

5.3. Comparable Evaluation for Tactical Option-Selection Policies

In order to validate the MEOL framework’s ability to integrate tactics within options through coupled training and to reasonably use tactics during confrontation with adversary fighters, this study carried out a systematic validation from the dimensions of tactical coupled training effectiveness and tactical hierarchical application capability in the confrontation environment. In terms of comparison method selection, based on the theoretical framework of ref. [35], this study selected two advanced option learning algorithms: the double actor–critic architecture with Proximal Policy Optimization (DAC+PPO) and the double actor–critic architecture with advantage actor–critic (DAC+A2C), as the baseline methods. This selection accounts for the algorithms’ tactical optimization characteristics in dynamic confrontation environments and their comparability with the MEOL framework. Specifically, the decision was motivated by their proven efficiency in hierarchical decision-making tasks and their structural compatibility with the MEOL framework’s dual-layer optimization mechanism. Meanwhile, two comparative experiments were designed in this study: the first evaluates the MEOL framework’s training success rate against two baseline methods to demonstrate its superior ability in deploying tactical intra-options against rule-based expert opponents, while the second conducts post-training confrontation tests involving both baseline algorithms and three tactical intra-options to validate the framework’s operational mechanism for integrating tactical hierarchies and executing graded manoeuvres.

5.3.1. Coupling Training for Option-Selection Policy

In order to maintain the consistency of the exploration strategies during the training process, the two types of baseline methods contain the same three types of tactical options, and entropy regularization is used for both their option and movement hierarchies. As demonstrated in Figure 13, the success rates of the MEOL, DAC+A2C, and DAC+PPO option-selection policies in the training process of an expert rule-based UCAV opponent are illustrated; the success rate of the training process is also calculated by the last 100 evaluation results, and the total environment steps is set to $1.5\times {10}^7$. It is evident that the final win rate of the MEOL framework is stabilized at around 0.8, which is significantly higher than the 0.6 of DAC+PPO and the 0.5 of DAC+A2C. Furthermore, it has been observed to improve by approximately 0.2, 0.35, and 0.25 compared to the three categories of tactical intra-option policies, respectively. This experimental result provides substantial evidence that the MEOL option framework can effectively utilize intra-option tactics to achieve a substantial increase in the success rate.

5.3.2. Confrontation Assessment

Following the conclusion of the training phase, the MEOL was subjected to 1000 confrontation experiments with DAC+PPO, DAC+A2C, and three types of tactical intra-option policies under random initial conditions. The results of these experiments are shown in Figure 14, where the vertical coordinate ‘success rate’ refers to the ratio of the number of victories of one side to the total number of 1000 confrontation tests. The results of the adversarial tests show that MEOL’s capability is significantly better than the two types of baseline algorithms as well as the three types of tactical intra-options.

Subsequently, the manoeuvre trajectory images of the MEOL against the three tactical intra-options and the two baseline methods were selected at random air combat environment parameters during the confrontation evaluation to illustrate how the MEOL utilized the tactical options during the confrontation.

The trajectory images of the MEOL against the three tactical intra-options are shown in Figure 15, where the self-fighter denotes the MEOL-controlled UCAV. Concurrently, beneath each confrontation track is a visualization of the corresponding MEOL choice of tactics’ intra-option in the time dimension. The utilization of the dark blue line indicates the application of the option, whereas the light yellow line signifies its non-application.

The trajectory of MEOL’s confrontation with angle tactics is illustrated in Figure 15a, while its corresponding option choices in the time dimension are demonstrated in Figure 15d. At the initial time node 0, the two sides present a typical head-on situation, and until time marker 1, the self-fighter’s option choices are focused on angle and snapshot tactics. It is evident that in the aforementioned head-on scenario, the direct utilization of snapshot tactics will yield a superior reward function benefit. However, this approach also renders the self-fighter vulnerable to attack. Consequently, the self-fighter successfully transitions its position through a diving turn manoeuvre, thereby establishing a rear-quarter attack position. The adversary fighter’s strategy is predicated on angle tactic training, the aim of which is to gain a situational advantage in the vertical plane between time mark 1 and 2 through an oblique loop manoeuvre. At this point, the self-fighter is confronted with adversary aircraft seeking to enter the rear hemisphere of the fighter, and gaining an angular advantage is clearly more important. Thus, the angle tactical option for the self-fighter dominates in the second half of the confrontation. Trajectory analysis demonstrates that the horizontal manoeuvres employed by the self-fighter resulted in the adversary fighter moving forward (overshoot), thereby successfully re-establishing the rear-aspect engagement geometry and achieving a successful shot down of the adversary fighter at time marker 3.

In a similar manner, the confrontation trajectory and option-selection process of MEOL and energy tactics are illustrated in Figure 15b and Figure 15e, respectively. The initial posture is a parallel divergence posture, and the adversary fighter trained based on the energy tactic expands its energy advantage by exchanging potential energy for kinetic energy between time markers 0 and 1, so as to reserve energy for the subsequent implementation of the large-overload turning manoeuvre and re-pointing to self-fighter. By analysing the option-selection process, between time markers 0 and 1, the self-fighter primarily selects the energy and angle options, which are visualized in the trajectory diagram by the horizontal turn of our aircraft to re-point at the adversary aircraft. In response to the adversary aircraft’s High Yo-Yo tactical manoeuvres to the rear hemisphere position of the self-fighter at time markers 1 to 2, the self-fighter primarily employs angle and rapid-fire options to augment the AA angle of advantage. This enables the successful execution of the lead turn and the acquisition of a shooting opportunity against the adversary’s fighter at time marker 2.

The trajectory of the confrontation between MEOL and snapshot tactics is illustrated in Figure 15c. It is evident that the adversary fighter tactics, which are trained based on snapshot tactics, exhibit a heightened degree of aggression. Between time markers 0 and 1, the adversary fighter, following an initial head-on attack on the self-fighter, executes a substantial overload turn in the vertical plane to re-establish a head-on position at time marker 1. In the face of the adversary fighter seeking an AA angle advantage, the self fighter prefers to conserve energy and adjust the angle in the early stage of the confrontation, as illustrated in Figure 15f. At time marker 1, when the head-on position is established and the self-fighter is at an altitude disadvantage, the self-fighter successfully performs the barrel roll tactical manoeuvre by utilizing the angle and snapshot options, thereby entering the rear hemisphere area of the adversary fighter. At time marker 2, when the AA angle has an advantage, the self-fighter successfully shoots down the adversary fighter on the side by employing the snapshot option.

5.4. Ablation Study

In order to validate the optimization of MEOL training effectiveness by the automatic curriculum generation mechanism, an ablation experiment was designed in this study. The experimental group adopts the automatic curriculum generation framework based on WGANs, while the control group adopts a baseline training method based on random environmental parameter sampling to conduct a comparative study for three intra-tactical option strategies. The experimental data demonstrate, as illustrated in Figure 16, that the experimental group employing the automatic curriculum generation mechanism exhibits a significant performance advantage in the training task of complex air combat scenarios. The training period required for their strategies to converge to a stable win rate is drastically shortened compared to that of the control group, and the probability of the success of the tactics is relatively elevated for the same number of training steps. These findings confirm that the adversarial training-based course generation mechanism can effectively enhance the tactical execution ability of intelligences in non-stationary combat environments.

6. Conclusions

This study provides empirical evidence that the MEOL framework is an effective solution to the hierarchical decision-making challenges that arise in UCAVs’ WVRAC scenarios. The integration of air combat tactical domain knowledge with hierarchical reinforcement learning enables the proposed method to successfully decouple low-level manoeuvring control from high-level tactical selection, while ensuring dynamic coordination between the two levels. Ablation experiments demonstrate that the WGAN-based automatic course generation mechanism is crucial in overcoming exploration challenges for efficient strategy training under different initial engagement conditions, and that the application of the WGAN technique roughly improves the win rate by about 0.2–0.4 in the training of tactical strategies within the three options. Meanwhile, the results of the confrontation experiments show that the intelligence based on the MEOL framework is significantly better than the intelligence based on a single tactical strategy as well as the option learning frameworks of DAC+PPO and DAC+A2C in terms of in-opportunity aerial combat capability. In the 1000 confrontation experiments conducted separately, the MEOL framework was able to achieve a win rate of 0.583, 0.617, 0.752, 0.605, and 0.68 compared to the DAC+PPO, DAC+A2C, angular tactics, rapid-fire tactics, and energy tactics, respectively, reflecting that it is more capable of exhibiting adaptive tactical switching behaviours in line with expert air combat theory. The framework’s limitations primarily stem from its current reliance on predefined tactical categories and simulated adversarial strategies. Subsequent research will concentrate on extending the methodology to multi-UCAVs’ agent cooperative scenarios and real-time adaptation against evolving adversary tactics. In conclusion, this study provides a concrete pathway for intelligent decision-making on tactics and manoeuvres in UCAVs’ WVRAC.