Imitation-Reinforcement Learning Penetration Strategy for Hypersonic Vehicle in Gliding Phase

Abstract

To enhance the penetration capability of hypersonic vehicles in the gliding phase, an intelligent maneuvering penetration strategy combining imitation learning and reinforcement learning is proposed. Firstly, a reinforcement learning penetration model for hypersonic vehicles is established based on the Markov Decision Process (MDP), with the design of state, action spaces, and composite reward function based on Zero-Effort Miss (ZEM). Furthermore, to overcome the difficulties in training reinforcement learning models, a truncated horizon method is employed to integrate reinforcement learning with imitation learning at the level of the optimization target. This results in the construction of a Truncated Horizon Imitation Learning Soft Actor–Critic (THIL-SAC) intelligent penetration strategy learning model, enabling a smooth transition from imitation to exploration. Finally, reward shaping and expert policies are introduced to enhance the training process. Simulation results demonstrate that the THIL-SAC strategy achieves faster convergence compared to the standard SAC method and outperforms expert strategies. Additionally, the THIL-SAC strategy meets real-time requirements for high-speed penetration scenarios, offering improved adaptability and penetration performance.

1. Introduction

Hypersonic vehicles, with their high speed, wide operational range, and reduced detectability, present formidable capabilities in penetration. These advantages have made them a key area of strategic development and research across major aerospace nations [1,2]. However, as defense weapon systems undergo comprehensive upgrades, including warning detection systems and interception weaponry, the advantages of hypersonic vehicles in terms of difficulty in detection and interception during the gliding phase are gradually diminishing, specifically with the development of detection systems and interception weapons aimed at countering the gliding phase, such as the Hypersonic and Ballistic Tracking Space Sensor (HTBSS) and Glide Phase Interceptor (GPI). The enhanced detection and interception capabilities of these systems have rendered traditional maneuvering penetration strategies during the gliding phase increasingly ineffective [3,4]. Consequently, the probability of hypersonic vehicles encountering terminal-phase interception by defensive missiles has significantly increased. Therefore, addressing the challenge of successfully evading interceptors during the terminal guidance phase under adversarial situations in the gliding phase has become an urgent issue.

In early research, penetration strategies primarily focused on the design of pre-programmed maneuvering patterns, widely utilized in engineering applications, including the step maneuver [5], sinusoidal maneuver [6], weaving maneuver [7,8], spiral maneuver [9,10]. These strategies aimed to alter the vehicle’s flight path, thereby increasing the difficulty for interception systems to predict the trajectory, ultimately depriving them of interception opportunities. However, these penetration strategies require the maneuver timing and patterns to be determined offline in advance, leading to poor operational flexibility and making effective penetration difficult to achieve. To address this issue, penetration guidance laws grounded in modern control theory have been extensively studied. These can be broadly classified into two categories: guidance laws based on optimal control theory and those derived from differential game theory. In term of optimal control theory-based penetration guidance laws, Shinar et al. [11] first proposed the optimal evasion strategy for proportional guidance pursuer under the assumption of two-dimensional linearized kinematics. Furthermore, numerous optimal guidance laws have been developed based on various simplified assumptions [12,13,14,15]. Tabak et al. [13] modeled the interceptor guidance system as an ideal first-order dynamics situation and derived an optimal guidance law for penetration based on the maximum principle. Shaferman [14,15] assumed the interceptor’s guidance law to be known and incorporated it into an attack–defense engagement model, leading to a unilateral penetration model. In term of differential game theory-based penetration guidance laws, the literature [16,17] provided differential game solutions under simplified model assumptions by solving complex optimal problems. Segal et al. [18] proposed a method to solve the high-speed penetration problem for aircraft in a three-dimensional space.

However, the above penetration methods require assumptions that the interceptor’s guidance law is partially or fully known, and simplifications of the interceptor model are often necessary during the modeling process. In practical applications, it is challenging to fully satisfy these conditions, which increases the probability of penetration failure. Furthermore, the optimization problem-solving process is time-consuming and difficult to complete within limited time in practice.

In recent years, rapid advancements in artificial intelligence have facilitated the widespread application of reinforcement learning across various domains, due to its inherent advantages in handling sequential decision-making problems, and it has progressively found application within the penetration domain [19,20,21]. Compared to the aforementioned penetration methods, reinforcement learning-based penetration methods do not require prior knowledge of the interceptor’s guidance strategy or simplified models. These methods transform the penetration strategy problem into an interaction with the environment, aimed at maximizing the reward function, thereby deriving an optimal “end-to-end” penetration strategy. This approach mitigates the computational complexity and time constraints typically associated with traditional methods under complex conditions, providing a novel and efficient solution to the penetration challenges faced by hypersonic vehicles in complex situations.

In particular, Zhuang et al. [22] designed an intelligent penetration strategy utilizing the Proximal Policy Optimization (PPO) algorithm for high-speed fixed-wing unmanned aerial vehicles (UAVs), which resulted in an average improvement of 15.44% in penetration effectiveness. Yan et al. [23] and Guo et al. [24] developed intelligent maneuver strategies for hypersonic vehicles based on the twin delayed deep deterministic policy gradient (TD3) to evade the interception. Hao et al. [25] proposed a method for adjusting evasion guidance law parameters based on reinforcement learning. Zhao et al. [26] proposed a maneuver command decision model utilizing the Soft Actor–Critic (SAC) algorithm. However, since reinforcement learning is an unsupervised learning method, it requires extensive trial and error and exploration of the environment during the learning process. Combined with the complexity of the flight penetration decision-making environment, reinforcement learning-based penetration methods for hypersonic vehicles also face a series of challenges, such as long training time, low efficiency in sample exploration, and difficulties in achieving convergence.

To address these challenges, combining the strengths of traditional penetration methods with reinforcement learning-based methods has emerged as an effective approach to solving penetration problems in complex environments. By imitating the strategies provided by traditional penetration methods, the training process of reinforcement learning can be expedited. In the literature [27], expert knowledge was employed using the Behavior Cloning (BC) imitation learning method for a DDPG-based penetration algorithm, leading to a substantial improvement in the convergence speed of UAV penetration algorithms. To further expedite the learning process, He et al. [28] proposed an innovative learning framework that integrates imitation learning and reinforcement learning. Jiang et al. [29] combined Generative Adversarial Imitation Learning (GAIL) with reinforcement learning methods, enabling the rapid learning of UAV path planning strategies in complex environments. Similarly, Wang et al. [30] combined GAIL with the PPO algorithm, using adversarial training to imitate expert knowledge and accelerate the training of penetration strategies, thus facilitating faster training in complex battlefield environments. However, these approaches heavily depend on the tailored design of reward functions, and the imitation training process based on expert knowledge is intricate. Additionally, challenges such as training instability and mode collapse can arise, which limit the effectiveness of these methods in reducing the difficulty of training reinforcement learning-based penetration strategies.

Motivated by the above observations, this paper proposes an intelligent penetration strategy for hypersonic vehicles in the gliding phase, which combines imitation learning and reinforcement learning. Firstly, a Markov Decision Process (MDP) model for the penetration of hypersonic vehicles in the gliding phase is established. Secondly, an imitation-reinforcement learning model is constructed using the truncated horizon (TH) method. Based on this, a Truncated Horizon Imitation Learning Soft Actor–Critic (THIL-SAC) intelligent penetration strategy is designed, which facilitates the rapid and stable convergence of the penetration strategy learning. The main contributions of this paper are summarized as follows.

• A novel imitation-reinforcement learning method is developed, which unifies the optimization objectives of imitation learning and reinforcement learning through the truncated horizon. By adjusting the truncated horizon length, this method progressively increases the difficulty of learning tasks, providing a new way to enhance the efficiency of reinforcement learning in solving complex problems.
• A result-oriented composite reward function based on the Zero-Effort Miss (ZEM) is designed, which incorporates both terminal rewards and process rewards. The constituent term ZEM of the reward function establishes a direct link between the current penetration situation and the final penetration outcome, enabling a more accurate evaluation of the effectiveness of the penetration actions. This effectively reduces the difficulty of learning the penetration strategy for hypersonic vehicles.
• An intelligent penetration strategy for hypersonic vehicles during the gliding phase, utilizing the THIL-SAC algorithm, is proposed. By employing reward shaping and truncated experience sampling, this approach integrates imitation learning with reinforcement learning, thereby effectively enhancing the convergence rate of penetration strategy learning based on reinforcement learning. Furthermore, this method avoids any need for model simplifications or assumptions, thereby delivering improved penetration performance and greater adaptability.

The remainder of this paper is structured as follows. Section 2 presents the penetration model and MDP model for a hypersonic vehicle in the gliding phase. In Section 3, the penetration strategy based on imitation-reinforcement learning is designed. Section 4 presents simulation experiments and analysis to validate the effectiveness of the proposed methods. Finally, conclusions are offered in Section 5.

2. Materials and Methods

2.1. Penetration Model of Hypersonic Vehicle in Gliding Phase

During the gliding phase of a hypersonic vehicle, its initial penetration attempt fails due to mid-course guidance by the interceptor. Entering the terminal guidance phase, the vehicle maneuvers to evade the interceptor at a distance of 50 km. The relative motion between the hypersonic gliding vehicle and the interceptor is shown in Figure 1. $oxyz$ is the north celestial east coordinate system. H and I represent the hypersonic vehicle and interceptor, respectively. $\mathbf{V},\theta ,$ and $\psi$ are the velocity, the flight path angle, and the heading angle of velocity vector of the hypersonic vehicle. ${\mathbf{V}}_I$ is the velocity of the interceptor. ${\mathbf{R}}_{rel}$ and ${\mathbf{V}}_{rel}$ are the relative position vector and relative velocity vector between them.

To better align with real-world penetration scenarios, this paper considers the path constraints during the flight of the hypersonic vehicle, including the aerodynamic overload limit, the dynamic pressure limit, and the heating rate limit, as well as environmental disturbances and the detection errors of the interceptor. Under these conditions and constraints, the penetration problem studied in this paper can be described as a game-theoretic process, where the hypersonic vehicle exploits its maneuvering capabilities to evade and penetrate against interceptor.

2.1.1. Mathematical Model of Hypersonic Vehicle

Considering the Earth’s rotation, the dynamics model of the hypersonic vehicle during the gliding flight phase is as follows [31].

$$\left\{\begin{array}{cc}\hfill \dot{r}=& Vsin\theta \hfill \\ \hfill \dot{\lambda }=& {\displaystyle \frac{Vcos\varphi sin\psi }{rcos\varphi }}\hfill \\ \hfill \dot{\varphi }=& {\displaystyle \frac{Vcos\theta cos\psi }{r}}\hfill \\ \hfill \dot{V}=& -{\displaystyle \frac{D}{m}}-gsin\theta +r{\Omega }^{2}cos\varphi (cos\varphi sin\theta -cos\theta sin\varphi cos\psi )\hfill \\ \hfill \dot{\theta }=& {\displaystyle \frac{1}{V}}\left[{\displaystyle \frac{L}{m}}cos\sigma +\left(V^2/r-g\right)cos\theta +2\Omega Vcos\varphi sin\psi +\right.\hfill \\ \hfill & \left.{\Omega }^{2}rcos\varphi (cos\theta cos\varphi +sin\theta sin\varphi cos\psi )\right]\hfill \\ \hfill \dot{\psi }=& {\displaystyle \frac{1}{V}}\left[{\displaystyle \frac{Lsin\sigma }{mcos\theta }}+{\displaystyle \frac{V^2}{r}}cos\theta sin\psi tan\varphi -2\Omega V\right.\hfill \\ \hfill & \left.(tan\theta cos\varphi cos\psi -sin\varphi )+{\displaystyle \frac{{\Omega }^{2}r}{cos\theta }}sin\psi sin\varphi cos\varphi \right]\hfill \end{array}\right.$$

(1)

where r is the radial distance. V is the earth-relative velocity. $\lambda$ and $\varphi$ are the longitude and latitude. $\theta$ and $\psi$ are the flight path angle and heading angle. $\sigma$ is the bank angle. m is the mass of the vehicle. $\Omega$ and g are the earth angular velocity and gravitational acceleration. D and L are the aerodynamic drag force and lift force, the magnitude of which is determined by the current attack angle of the vehicle when the altitude and speed are known.

The aerodynamic lift and drag forces are expressed as follows:

$$\left\{\begin{array}{c}L=qS{C}_L\hfill \\ D=qS{C}_D\hfill \end{array}\right.$$

(2)

where $q={\displaystyle \frac{1}{2}}\rho V^2$ is the dynamic pressure, with $\rho$ denoting the atmospheric density. S is the aerodynamic reference area. The lift and drag coefficients, $C_L=f_{C_L}(\alpha ,V)$ and $C_D=f_{C_D}(\alpha ,V)$, are nonlinear functions of the angle of attack $\alpha$ and the velocity V.

To facilitate the calculation of the relative relationship between the hypersonic vehicle and the interceptor, the position of the hypersonic vehicle can be expressed as $X,Y,Z$ in the Cartesian coordinate system as follows.

$$\left\{\begin{array}{c}X=rcos\varphi cos\lambda \hfill \\ Y=rcos\varphi sin\lambda \hfill \\ Z=rsin\varphi \hfill \end{array}\right.$$

(3)

Furthermore, the hypersonic vehicle must satisfy both path constraints and control input constraints. The control input constraints include limitations on the angle of attack and the bank angle. In this paper, the angle of attack follows a predefined nominal profile, while the bank angle constraints will be specified in the subsequent detailed design. The path constraints include the heating rate limit, the aerodynamic overload limit and the dynamic pressure limit. For the sake of convenience in the subsequent design, these constraints are reformulated as bank angle limits in Equation (4), such that satisfying the bank angle limits also ensures compliance with the original path constraints.

$$\left\{\begin{array}{c}{cos}^{-1}\left[\left(g-{\displaystyle \frac{V^2}{r}}\right){\displaystyle \frac{2m{V}^{4.3}}{C_{L}S}}{\left({\displaystyle \frac{k_Q}{{\dot{Q}}_{max}}}\right)}^2\right]=\left|{\sigma }_{{\dot{Q}}_{max}}\left(V\right)\right|\hfill \\ {cos}^{-1}\left[\left(g-{\displaystyle \frac{V^2}{r}}\right){\displaystyle \frac{\sqrt{1+{\left(C_D/C_L\right)}^2}}{g_0{n}_{max}}}\right]=\left|{\sigma }_{n_{max}}\left(V\right)\right|\hfill \\ {cos}^{-1}\left[\left(g-{\displaystyle \frac{V^2}{r}}\right){\displaystyle \frac{m}{C_L{q}_{max}S}}\right]= \left|{\sigma }_{q_{max}}\left(V\right)\right|\hfill \end{array}\right.$$

(4)

where ${\dot{Q}}_{max}$, $n_{max}$, and $q_{max}$ represent the allowable upper bounds of the heating rate limit, the aerodynamic overload limit, and the dynamic pressure limit. $\left|{\sigma }_{{\dot{Q}}_{max}}\left(V\right)\right|$, $\left|{\sigma }_{n_{max}}\left(V\right)\right|$, and $\left|{\sigma }_{q_{max}}\left(V\right)\right|$ represent the upper bounds of the bank angle under these constraints. $k_Q$ is a constant related to the structural characteristics of the vehicle. $g_0$ is the gravitational acceleration at sea level.

2.1.2. Mathematical Model of Interceptor

The dynamics of the interceptor are modelled in the launch coordinate system as follows:

$$\left\{\begin{array}{cc}\hfill {\dot{X}}_I& =V_{I}cos{\theta }_{I}cos{\psi }_I\hfill \\ \hfill {\dot{Y}}_I& =V_{I}sin{\theta }_I\hfill \\ \hfill {\dot{Z}}_I& =-V_{I}cos{\theta }_{I}sin{\psi }_I\hfill \\ \hfill {\dot{V}}_I& =g\left(n_x-sin{\theta }_I\right)\hfill \\ \hfill {\dot{\theta }}_I& ={\displaystyle \frac{g}{V_I}}\left(n_y-cos{\theta }_I\right)\hfill \\ \hfill {\dot{\psi }}_I& =-{\displaystyle \frac{g}{V_{I}cos{\theta }_I}}{n}_z\hfill \end{array}\right.$$

(5)

where $X_I,Y_I,$ and $Z_I$ are the coordinate components of the position of the interceptor in the launch system. $V_I$ is the velocity of interceptor. ${\theta }_I$ is the flight path angle. ${\psi }_I$ is the heading angle. $n_x,n_y,$ and $n_z$ are the overload components in the launch system.

The interceptor uses the most typical and commonly guidance law proportional guidance (PN) to intercept the hypersonic vehicle The specific form is as follows:

$$\left\{\begin{array}{cc}\hfill a_{Iy}& =N{V}_c{\dot{\varsigma }}_e\hfill \\ \hfill a_{Iz}& =N{V}_c{\dot{\varsigma }}_a\hfill \end{array}\right.$$

(6)

where $a_{Iy}$ and $a_{Iz}$ are the acceleration of the interceptor. N is the coefficient of navigation law, which defines the proportionality between the interceptor’s commanded acceleration and the line-of-sight angular rate. In practice, N ranges from 2 to 6, with higher values leading to increased maneuvering. ${\dot{\varsigma }}_e$ and ${\dot{\varsigma }}_a$ are the line-of-sight angular rates. $V_c$ is the approach velocity, and it can be calculated as follows:

$$V_c=-{\displaystyle \frac{{\mathbf{R}}_{rel}}{∥{\mathbf{R}}_{rel}∥}}{\mathbf{V}}_{rel}$$

(7)

where ${\mathbf{R}}_{rel}$ and ${\mathbf{V}}_{rel}$ are the vectors of the relative position and relative velocity.

The efficacy of the interceptor is evaluated by the ZEM, which is determined through the following formula:

$$L_{ZEM}={\displaystyle \frac{∥{\mathbf{R}}_{rel}\times {\mathbf{V}}_{rel}∥}{∥{\mathbf{V}}_{rel}∥}}$$

(8)

where $L_{ZEM}$ is the ZEM, which represents the minimum distance between the hypersonic vehicle and the interceptor, assuming that both stop maneuvering.

Remark 1. 

In this paper, the interceptor model is only utilized to establish the simulation environment. The design of the penetration strategy is independent of the model.

2.2. Design of MDP Model for Hypersonic Vehicle Penetration in Gliding Phase

The solution to reinforcement learning problems is based on the MDP, which can be described by $\left(\mathcal{S},\mathcal{A},\pi ,P,R\right)$. Within the MDP framework, the agent applies an action, $a_t\in \mathcal{A}$, to the environment based on the current state $s_t\in \mathcal{S}$ and policy $\pi$. In the next moment, the environment transitions to a new state, $s_{t+1}\in \mathcal{S}$, according to the state transition probability P influenced by the action, and provides the agent with a reward value, $R_t\in R$, for the current step.

Through the learning process described above, the ultimate objective of the agent is to obtain an optimal policy, ${\pi }^{*}$, that maximizes the cumulative reward $G_0$. To account for foresight in learning, the cumulative reward $G_0$ is calculated using the following discounting method.

$$G_0=\sum _{t=0}^{\infty }{\gamma }^t{R}_t$$

(9)

where $\gamma \in \left(0,1\right]$ represents the discount factor.

In the hypersonic vehicle penetration process during the gliding phase discussed in this paper, the agent is the hypersonic vehicle, while the environment primarily consists of the interceptor. The state transition probability $P=1$ is determined by the model given in Equation (1). The policy represents the learning objective, which can be obtained through reinforcement learning method. Consequently, the designs of the state space, action space, and reward function are critical factors in determining the effectiveness of the hypersonic vehicle’s penetration, which guide the agent’s learning of the optimal policy. The following sections will address the design in detail.

2.2.1. State Space

The state space primarily encompasses information regarding both the hypersonic vehicle and the interceptor. We consider that the ultimate criterion for determining the success of the hypersonic vehicle’s penetration is the relative distance between the two at the time of miss and the relative velocity directly influences the change in this distance. Therefore, the vector of the relative position ${\mathbf{R}}_{rel}$ and relative velocity ${\mathbf{V}}_{rel}$ between the vehicle and the interceptor are selected as the state variable $\mathbf{s}$.

$$\mathbf{s}={[{\mathbf{R}}_{rel},{\mathbf{V}}_{rel}]}^{\mathrm{T}}={\left[r_x,r_y,r_z,v_x,v_y,v_z\right]}^{\mathrm{T}}$$

(10)

where $r_x,r_y,r_z,v_x,v_y,$ and $v_z$ are the components of the relative position vector and the relative velocity vector. Thus, the state space $\mathcal{S}$ for the hypersonic vehicle penetration is defined as the union of all states, $\mathbf{s}$.

2.2.2. Action Space

In accordance with the hypersonic vehicle model in Equation (1), the vehicle’s trajectory is determined once the angle of attack $\alpha$ and angle of bank $\sigma$ are specified. However, $\alpha$ significantly impacts the vehicle’s speed and altitude, making it difficult to meet terminal velocity and remaining range requirements, ultimately leading to an unsuccessful strike. Consequently, the hypersonic vehicle’s penetration during the gliding phase is typically conducted within the horizontal plane [32]. Therefore, in this paper, the vehicle achieves penetration by adjusting the angle of bank $\sigma$.

To ensure that the hypersonic vehicle can still reach the designated reentry point after successfully penetrating defenses and ultimately strike the target, the maximum bank angle for the vehicle’s maneuvering penetration command is constrained. Furthermore, in order to address the practical implementation of the bank angle command, the rate of change in the bank angle is also constrained. Therefore, the action for the hypersonic vehicle penetration is defined as follows:

$$\left\{\begin{array}{c}a={\sigma }_{cmd}\hfill \\ \begin{array}{cc}\hfill \left|{\sigma }_{cmd}\right|<& {\dot{\sigma }}_{max}\hfill \\ \hfill =& min\left\{\left|{\sigma }_{{\dot{Q}}_{max}}\left(V\right)\right|,\left|{\sigma }_{n_{max}}\left(V\right)\right|,\left|{\sigma }_{q_{max}}\left(V\right)\right|,{\sigma }_{m\max }\right\}\hfill \end{array}\hfill \\ \left|{\dot{\sigma }}_{cmd}\right|<{\dot{\sigma }}_{max}\hfill \end{array}\right.$$

(11)

where ${\sigma }_{cmd}$ is the penetration command for the bank angle. ${\sigma }_{max}$ and ${\dot{\sigma }}_{max}$ represent the maximum bank angle and its rate of change. ${\sigma }_{m\max }$ is maximum maneuverable bank angle under the constraint of striking the target. The action space $\mathcal{A}$ for the hypersonic vehicle penetration is thus defined as the union of all actions, a.

2.2.3. Reward Function

The reward function represents a tangible expression of the reinforcement learning objective, determining the direction of the agent’s policy learning and impacting the convergence speed and stability of the learning process, making it a core component of reinforcement learning. For the hypersonic vehicle penetration problem during the gliding phase, the result of the penetration can only be determined at the end, representing a typical sparse reward scenario. Furthermore, the inability to assess the impact of the current action on the final penetration outcome further increases the difficulty of learning the penetration strategy. Inspired by the reward function based on the line-of-sight angular rate in [24], this paper introduces penetration time information into the reward function design, considering the relative azimuthal situation (line-of-sight angle) during the penetration process. To simplify the reward function expression, the ZEM, which can represent the final penetration objective, is employed to unify the azimuthal situation and time information. The following composite reward function is then proposed.

• Terminal reward term
  The objective of the hypersonic vehicle penetration is to maximize the line-of-sight normal displacement (ZEM) between itself and the interceptor at the time of miss. Therefore, the reward function should include a reward term, $r_1$, at the time of miss.

  $$r_1=c_1·{L_{ZEM\_}}_{\mathrm{end}},\begin{array}{ccc}& & \end{array}{c}_1>0$$

  (12)

  where $c_1$ is a positive constant. ${L_{ZEM\_}}_{\mathrm{end}}$ is the ZEM at the time of miss.
• Process reward term
  Since the interceptor uses a near-optimal inverse trajectory for interception, the miss distance is initially at its maximum. As the distance between the hypersonic vehicle and the interceptor decreases, the miss distance will typically decrease as well [23,24]. However, if the rate of decrease in the miss distance slows down or even increases, it indicates that the hypersonic vehicle may eventually succeed in penetrating. Therefore, a process reward term, $r_2$, is introduced to account for the rate of change in the miss distance.

  $$\left\{\begin{array}{cc}\hfill & r_2=c_2·\Delta L_{ZEM},\begin{array}{ccc}& & \end{array}{c}_2>0\hfill \\ \hfill & \Delta L_{ZEM}={L_{ZEM\_}}_t-{L_{ZEM\_}}_{t-1}\hfill \end{array}\right.$$

  (13)

  where $c_2$ is a positive constant. ${L_{ZEM\_}}_t$ is the ZEM at the time of t.
  If only $r_2$ is used as the process reward term, the reward obtained from the agent–environment interactions would almost always be negative, which is detrimental to the diversity of training samples. Therefore, to address this issue, a “dense penalty + sparse reward” reward function format is considered, and an additional reward term, $r_3$, is introduced.

  $$r_3=c_3\begin{array}{cc}& \end{array}\Delta L_{ZEM}>0$$

  (14)

  where $c_3$ is a positive constant.
  In summary, the reward function for the hypersonic vehicle penetration during the gliding phase is as follows:

  $$R_t=r_1+r_2+r_3$$

  (15)

Remark 2. 

The terminal reward $r_1$ directly reflects the penetration effectiveness and determines whether the penetration is successful, thus occupying a dominant position in the overall reward function. On the other hand, the introduction of the process rewards $r_2$ and $r_3$ primarily aims to alleviate the common issue of sparse rewards in reinforcement learning, guiding the agent toward more effective strategy convergence during the learning process. To ensure consistency and stability in the learning objectives, the cumulative value of process rewards should be smaller than or of the same order of magnitude as the terminal reward $r_1$, in order to prevent excessive process incentives from overshadowing the optimization of the terminal goal. Based on the above considerations, we set the weight coefficients $c_1$, $c_2$, and $c_3$ according to the principle of “terminal dominance, process assistance” to achieve a good balance between penetration efficiency and learning stability.

3. Design of Penetration Strategy Based on Imitation-Reinforcement Learning

Reinforcement learning-based intelligent penetration methods require the exploration and learning of the environment from its initial state, which leads to long training times and slow convergence. Additionally, the significant gap between the initial learning objectives and the agent’s capabilities may result in early termination, preventing effective learning.

Therefore, this paper investigates a Truncated Horizon Imitation Learning Soft Actor–Critic (THIL-SAC) method for hypersonic vehicle penetration. This method integrates imitation learning and reinforcement learning through the truncated horizon. In the early stages of penetration strategy training, a “short-sighted” strategy is used, focusing on expert experience imitation and short-term goal achievement. This approach reduces unnecessary exploration and learning difficulty, accelerating the training process. As training progresses, the foresight of strategy learning is gradually enhanced, utilizing the SAC algorithm for comprehensive exploration of the environment to achieve successful penetration.

3.1. Truncated Horizon Imitation-Reinforcement Learning Model

For reinforcement learning, we define the Markov Decision Process ${\mathcal{M}}_0$; then, its state–value function $V^{\pi }$ and state–action value function $Q^{\pi }$ are

$$V^{\pi }\left(s\right)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{R}_t\left(s_t,a_t\right)\mid s_0=s,a\sim \pi \right]$$

(16)

$$Q^{\pi }(s,a)=\mathbb{E}\left[R_t(s_t,a_t)+\gamma {\mathbb{E}}_{s_{t+1}\sim P_{sa}}\left[V^{\pi }\left(s_{t+1}\right)\right]\right]$$

(17)

where $\mathbb{E}$ represents the mathematical expectation.

The objective is to find the strategy that maximizes $V^{\pi }$.

$${\pi }^{*}=arg{max}_{\pi }{V}^{\pi }\left(s\right),\forall s\in \mathcal{S}$$

(18)

For imitation learning, the expert’s experience is represented by fitting a temporal difference method to $V^e$, and combining it with the reward defines the dominance function:

$$A^e\left(s_t,a_t\right)=R_t\left(s_t,a_t\right)+\gamma {\mathbb{E}}_{s_{t+1}\sim P}\left[V^e\left(s_{t+1}\right)\right]-V^e\left(s_t\right)$$

(19)

Then, the objective of the greedy learning of the expert experience directly is as follows [33].

$$\widehat{\pi }\left(s\right)=arg\underset{\pi }{max}{A}^e\left(s,a\right)$$

(20)

According to the theorem of reward shaping [34], the reinforcement learning optimization objective in Equation (18) can be equivalently expressed as follows:

$${\pi }^{*}=arg\underset{\pi }{max}\mathbb{E}\left[\sum _{t=1}^{\infty }{\gamma }^{t-1}{A}^e\left(s_t,a_t\right)\left|s_1=s;a\sim \pi \right.\right]$$

(21)

Since the problem involves the entire MDP, it is an infinite horizon optimization problem, which is difficult to solve. Using the idea of the truncated horizon to transform it into an optimization problem with a finite number of decisions, Equation (21) can be expressed as follows.

$$\begin{array}{cc}\hfill {\tilde{\pi }}^{*}& =arg\underset{\pi }{max}\mathbb{E}\left[\sum _{t=1}^k{\gamma }^{t-1}{A}^e\left(s_t,a_t\right)\left|s_1=s;a\sim \pi \right.\right]\hfill \\ \hfill & \begin{array}{cc}\hfill =arg\underset{\pi }{max}\mathbb{E}[& \sum _{t=1}^k{\gamma }^{t-1}{R}_t\left(s_t,a_t\right)+{\gamma }^k{V}^e\left(s_{k+1}\right)-\hfill \\ \hfill & V^e\left(s_1\right)\left|s_1=s;a\sim \pi \right.]\hfill \end{array}\hfill \end{array}$$

(22)

where $k\ge 1$ is the truncated length. Then, the error due to the transformation of the optimization objective is as follows [35].

$$\left\{\begin{array}{c}J\left({\tilde{\pi }}^{*}\right)\le J\left({\pi }^{*}\right)+{\displaystyle \frac{{\gamma }^k}{1-{\gamma }^k}}\kappa \hfill \\ J\left(\pi \right)=\mathbb{E}\left[V^{\pi }\left(s_0\right)\right]\hfill \end{array}\right.$$

(23)

where $\kappa$ is the error between $V^{{\tilde{\pi }}^{*}}$ and $V^{{\pi }^{*}}$.

In summary, when $k=1$, the optimization objective of Equation (22) is the same as that of Equation (20), and the optimization objective of imitation learning can be expressed by Equation (22). When $k>1$, the strategy obtained by using Equation(22) is better than that of imitation learning. When $k=\infty$, the optimization objective of Equation (22) is equivalent to that of reinforcement learning. Therefore, the optimization objective of reinforcement learning and imitation learning can be expressed uniformly.

3.2. THIL-SAC-Based Penetration Strategy

The THIL-SAC-based penetration strategy for hypersonic vehicles is built upon the SAC algorithm. It reduces the learning difficulty of the penetration strategy by using the truncated horizon method and imitation learning from expert experience. The approach enables the intelligent maneuvering and penetration of the hypersonic vehicle in complex battlefield environments and achieves faster convergence. The overall structure of the algorithm is shown in Figure 2.

Figure 2 illustrates the training process and online application process of the penetration strategy network. In terms of penetration strategy training, the THIL-SAC algorithm can be broadly divided into the foundational SAC algorithm and improvements designed to accelerate algorithm convergence. The improvements are divided into two parts: (1) the efficient utilization of expert experience based on reward shaping and sampling; (2) the optimization of the penetration strategy training process based on the truncated horizon. Together, these elements enable the rapid development of an effective intelligent penetration strategy network. In terms of penetration strategy online application, the states of the hypersonic vehicle and the interceptor missile are fed into the trained strategy network ${\pi }^{\varphi }$, which then outputs the maneuvering penetration command $\sigma$ for the hypersonic vehicle to execute the penetration maneuver.

3.2.1. SAC Algorithm

The penetration mission of hypersonic vehicles is a decision-making problem with continuous action and state spaces, making it suitable for solving using policy gradient methods. However, in classical policy gradient algorithms, the DDPG algorithm is highly sensitive to training hyperparameters; the PPO algorithm faces a serious exploration–exploitation trade-off, which can result in a significant discrepancy between the training policy and the actual interaction policy, leading to difficulties in algorithm convergence. To address the aforementioned issues, this paper adopts the SAC (Soft Actor–Critic) algorithm as the foundational reinforcement learning algorithm for learning penetration strategies for hypersonic vehicles. The SAC algorithm is a reinforcement learning algorithm based on the Actor–Critic framework, which improves the exploration of the environment by incorporating maximum entropy into the learning objective. This allows the algorithm to achieve high rewards while also exhibiting greater robustness [36].

The SAC algorithm employs both the Actor and Critic networks to approximate the policy and value function, respectively. The Actor network generates actions based on the current state, while the Critic network estimates the value function, evaluating the actions suggested by the Actor. Through iterative learning, both networks converge towards the optimal policy and value function. The training process of the algorithm consists of two main steps: policy evaluation and policy improvement.

• Policy evaluation
  The purpose of policy evaluation is to the evaluation accuracy of the Critic network. To fully utilize the available data, increase sampling efficiency, and improve learning stability, the SAC algorithm constructs four identical networks, $Q_{soft}$: two main Q-networks ($Q_i^{{\vartheta }_i}\left(s,a\right);i=1,2$) and two target Q-networks ($Q_i^{{\vartheta }_{tar,i}}\left(s,a\right);i=1,2$), where $\vartheta$ and ${\vartheta }_{tar}$ are the network parameters.
  The main Q-networks update their parameters by minimizing the soft Bellman error, with the loss function given by

  $$\begin{array}{cc}\hfill & J_Q\left({\vartheta }_i\right)={\mathbb{E}}_{\left(s_t,a_t,R_t,s_{t+1}\right)\sim D}\left[{\left(Q_i^{{\vartheta }_i}\left(s_t,a_t\right)-y\left(R_t,s_{t+1}\right)\right)}^2\right]\hfill \end{array}$$

  (24)

  where D is the replay buffer. $y\left(R_t,s_{t+1}\right)$ is the updating target, which can be calculated by the following formula.

  $$\begin{array}{cc}\hfill y\left(R_t,s_{t+1}\right)=& R_t+\gamma (\underset{i=1,2}{min}{Q}_i^{{\vartheta }_{tar,i}}\left(s_{t+1},a_{t+1}\right)-\hfill \\ \hfill & {\alpha }_{T}log{\pi }^{\varphi }\left(a_{t+1}|s_{t+1}\right))\hfill \end{array}$$

  (25)

  where ${\pi }^{\varphi }$ is the policy, and $a_{t+1}$ is the action given in the state $s_{t+1}$ by ${\pi }^{\varphi }$.
  To ensure the accuracy of $Q_i^{{\vartheta }_i}\left(s,a\right)$ and minimize the loss function, it is common to update the parameter ${\vartheta }_i$ by using gradient descent as follows:

  $${\vartheta }_i\leftarrow {\vartheta }_i-{\lambda }_Q\nabla J_Q\left({\vartheta }_i\right)$$

  (26)

  where ${\lambda }_Q>0$ indicates the learning rate of $Q_i^{{\vartheta }_i}\left(s,a\right)$ and satisfies a small value, and $\nabla J_Q\left({\vartheta }_i\right)$ denotes the gradient of $J_Q\left({\vartheta }_i\right)$.
  The $Q_i^{{\vartheta }_{tar,i}}\left(s,a\right)$ parameters are then soft-updated according to the $Q_i^{{\vartheta }_i}\left(s,a\right)$ parameters with the following update equation:

  $${\vartheta }_{tar,i}\leftarrow \tau {\vartheta }_i+\left(1-\tau \right){\vartheta }_i$$

  (27)

  where $\tau$ is the soft update coefficient.
• Policy improvement
  The purpose of policy improvement is to enhance the policy of the agent by updating the policy network ${\pi }^{\varphi }$, where $\varphi$ represents the policy network parameters. ${\pi }^{\varphi }$ is updated by the Kullback–Leibler (KL) divergence between the current policy and the target policy, and the loss function is as follows:

  $$\begin{array}{cc}\hfill J_{\pi }\left(\varphi \right)=& {\mathbb{E}}_{s_t\sim D,a_t\sim {\pi }^{\varphi }}\left[{\alpha }_{T}log\left({\pi }^{\varphi }\left(a_t\right|s_t\right)\right)-\hfill \\ \hfill & \underset{i=1,2}{min}{Q}_i^{{\vartheta }_i}\left(s_t,a_t\right)]\hfill \end{array}$$

  (28)
  In the equation, the temperature coefficient ${\alpha }_T$ is dynamically adjusted based on the state to enhance the randomness of policy exploration. This update rule is as follows:

  $$\left\{\begin{array}{c}{\alpha }_T\leftarrow {\alpha }_T-{\lambda }_{{\alpha }_T}\nabla J\left({\alpha }_T\right)\hfill \\ J\left({\alpha }_T\right)={\mathbb{E}}_{a_t\sim {\pi }^{\varphi }}\left[-{\alpha }_{T}log{\pi }^{\varphi }\left(a_{\iota }|s_{\iota }\right)-{\alpha }_T\overline{H}\right]\hfill \end{array}\right.$$

  (29)
  The update formula for the policy network is

  $$\varphi \leftarrow \varphi -{\lambda }_{\pi }\nabla J_{\pi }\left(\varphi \right)$$

  (30)

  where ${\lambda }_{\pi }>0$ indicates the learning rate.

For the hypersonic vehicle penetration problem, the penetration strategy network for the vehicle is the SAC algorithm’s policy network, which generates the vehicle’s maneuvering and penetration commands, $\sigma$. The network $Q_{soft}$ learns the penetration strategy based on the reward function, which guides the direction of strategy learning.

Considering the complexity of the hypersonic vehicle penetration environment in the gliding phase, where the relationship between the penetration target, current state, and reward is intricate, the learning task is challenging. Without prior experience, the penetration strategy network may engage in excessive exploratory behavior initially, leading to slow convergence or even failure to converge. Therefore, this paper introduces the truncated horizon imitation strategy on top of the SAC algorithm for designing the hypersonic vehicle penetration strategy. This approach employs expert experience and reduces the initial learning difficulty to enhance the speed of strategy learning.

3.2.2. Expert Experience Imitation Based on Reward Shaping and Sampling

To fully utilize and mimic expert experience during the early stages of training, expert experience is incorporated into both the reward function and the experience replay buffer.

Considering the impact of the experience replay buffer on training, during the initial training phase, the buffer is populated with penetration Markov decision data generated by expert experience $\left(s_t,a_{Et},s_{t+1},r_t\right)$, where $a_{Et}$ represents the expert experience data. When sampling data in batches from the experience replay buffer during early training, the expert data will guide the learning of the penetration strategy. As training progresses, the experience replay buffer gradually fills with training data, and the expert experience data are removed, causing their direct influence to diminish until they eventually disappear.

To further leverage expert experience, the expert experience data are used to fit an expert value function ${\widehat{V}}^E\left(s\right)$ via temporal difference (TD) learning. Based on the theorem of reward shaping, the hypersonic vehicle reward function Equation (15) is reconstructed as follows:

$${\overline{R}}_t=R_t+{\widehat{V}}^E\left(s_{t+1}\right)-{\widehat{V}}^E\left(s_t\right)$$

(31)

By utilizing expert experience to guide the reward function calculation, the learning process for the hypersonic vehicle glide phase penetration strategy is accelerated.

3.2.3. Optimization of the Training Process Based on TH

To address the challenges of high learning difficulty and slow training speed in the initial stages of hypersonic vehicle penetration strategy learning, the truncated length k is employed to incrementally increase the complexity of the learning task. This approach transitions the focus from primarily imitation learning based on expert experience during the initial training phase to reinforcement learning aimed at exploring the optimal strategy. The specific implementation steps are as follows:

• Truncated horizon learning framework
  Based on the unified imitation-reinforcement learning framework with the truncated horizon established in Section 3.1, the truncated length k can be introduced into the soft value functions $V_{soft}\left(s_t\right)$ and $Q_{soft}\left(s,a\right)$, which can be rewritten as follows:

  $$Q_{k,soft}\left(s_t,a_t\right)=R_t+{\mathbb{E}}_{\pi }\left[\sum _{l=t}^{t+k}{\gamma }^{l-1}\left({\overline{R}}_l+{\alpha }_{T}H\right)\right]$$

  (32)

  $$V_{k,soft}\left(s_t\right)={\alpha }_{T}log{\int }_{A}exp\left({\displaystyle \frac{1}{{\alpha }_T}}{Q}_{k,soft}\left(s_t,a_t\right)\right)\mathrm{d}a$$

  (33)
  Therefore, using the truncated value functions $Q_{k,soft}$ and $V_{k,soft}$ for penetration strategy learning is equivalent to learning within the truncated horizon length k.
  Considering that in the actual training process, the truncated value functions are calculated from data $\left(s_t,a_t,s_{t+1},{\overline{R}}_t\right)$ sampled from the experience replay buffer, the time step range t of the sampled data determines the learning scope. Therefore, compared to the traditional SAC algorithm, the THIL-SAC algorithm’s experience replay buffer stores data considering the time step t. The truncated experience replay buffer $D^k$ is constructed based on the original experience replay buffer to enable training within the truncated length k. The truncated experience replay buffer $D^k$ is defined as

  $$D^k=\left\{\begin{array}{c}\left(s_t^i,a_t^i,s_{t+1}^i,{\overline{R}}_t^i\right)\\ \left(s_{t+1}^i,a_{t+1}^i,s_{t+2}^i,{\overline{R}}_{t+1}^i\right)\\ ⋮\\ \left(s_{t+k}^i,a_{t+k}^i,s_{t+k}^i,{\overline{R}}_{t+k}^i\right)\end{array}\right\}$$

  (34)

  where i is the number of training episodes.
  At this point, the loss functions for the main Q-network and the target Q-network are as follows:

  $$\begin{array}{c}\hfill J_{Q_k}\left({\vartheta }_i\right)={\mathbb{E}}_{\left(s_t,a_t,{\overline{R}}_t,s_{t+1}\right)\sim D^k}\left[{\left(Q_{k,i}^{{\vartheta }_i}\left(s_t,a_t\right)-y_k\left({\overline{R}}_t,s_{t+1}\right)\right)}^2\right]\end{array}$$

  (35)

  $$\begin{array}{c}\hfill J_{\pi ,k}\left(\varphi \right)={\mathbb{E}}_{s_t\sim D^k,a_t\sim {\pi }^{\varphi }}\left[{\alpha }_{T}log\left({\pi }^{\varphi }\left(a_t\right|s_t\right)-\underset{i=1,2}{min}{Q}_{k,i}^{{\vartheta }_i}\left(s_t,a_t\right)\right]\end{array}$$

  (36)

  where

  $$\begin{array}{cc}\hfill y_k\left({\overline{R}}_t,s_{t+1}\right)=& {\overline{R}}_t+\gamma (\underset{i=1,2}{min}{Q}_{k,i}^{{\vartheta }_{tar,i}}\left(s_{t+1},a_{t+1}\right).-\hfill \\ \hfill & .{\alpha }_{T}log{\pi }^{\varphi }\left(a_{t+1}|s_{t+1}\right))\hfill \end{array}$$

  (37)
  Therefore, the penetration strategy learning network training within the truncated length k can be achieved by the formulas in Equations (32)–(37).
• Adaptive adjustment of truncated length k
  The truncated length k determines the learning range and difficulty during penetration strategy training. To achieve low-difficulty, rapid learning based on expert experience in the initial training phase, and to enable exploration of the optimal penetration strategy using reinforcement learning in the later stages, in this paper, the truncated length k is adjusted as follows:

  $$k=\left\{\begin{array}{c}1,\begin{array}{cccccccccccc}& & & & & & & & & & & \end{array}i<b_0\hfill \\ b_1\left(i-b_0\right),\begin{array}{cc}& \end{array}{b}_0\le i<\left(N+b_0\right)/b_1\hfill \\ N,\begin{array}{ccccccccccc}& & & & & & & & & & \end{array}i\ge \left(N+b_0\right)/b_1\hfill \end{array}\right.$$

  (38)

  where N is the epoch of a single training episode, i is the number of training episodes, and $b_0$ and $b_1$ are positive constants.
  In the THIL-SAC based penetration strategy network training process, the truncated length $k=1$ within episode $b_0$ at the start of training is to satisfy imitation learning of the penetration strategy. As training progresses, the truncated length k is gradually increased in each episode until it reaches the length of a single training episode, N. This gradual increase transitions the learning process from imitation learning to reinforcement learning, enabling the exploration and optimization of the optimal penetration strategy.

3.3. Theoretical Analysis of TH and Expert Imitation in Reducing Learning Difficulty

In Section 3.2, this paper introduced two key components of the THIL-SAC algorithm designed to reduce learning complexity and accelerate training convergence: (i) expert imitation through reward shaping and trajectory sampling (Section 3.2.2), and (ii) training optimization using the TH method (Section 3.2.3). While these strategies intuitively facilitate the learning of breakthrough policies, their theoretical foundations require further elaboration. This section presents a formal analysis of how expert-guided imitation and the truncated horizon length k jointly reduce learning difficulty, highlighting their positive impact on policy convergence speed and sample efficiency.

In the standard SAC algorithm, the policy network ${\pi }^{\varphi }$ is trained by minimizing the KL-regularized expected loss function Equation (32), where the value function $Q_i^{{\vartheta }_i}$ reflects the long-term return over the whole horizon length N. However, the whole optimization horizon introduces high variance in policy gradients due to delayed credit assignment [37,38], resulting in sample complexity scaling as follows [39]:

$$O\left({\displaystyle \frac{A^2{N}^2}{{\epsilon }^2}}\Xi \right)$$

(39)

where A denotes the complexity of the action space. $\epsilon$ is the expected performance error. $\Xi$ is a measure of the policy space complexity. This indicates that the learning complexity is closely related to the horizon length N.

In the proposed THIL-SAC algorithm, instead of optimizing over the entire horizon length N, the update is restricted to a truncated horizon length k, thereby reducing the sampling complexity to

$$O\left({\displaystyle \frac{A^2{k}^2}{{\epsilon }^2}}\Xi \right)$$

(40)

Meanwhile, the truncated horizon leads to the following truncated gradient:

$$\nabla J_{\pi ,k}\approx {\mathbb{E}}_{\tau }\left[\sum _{t=1}^{\left|\tau \right|}\left({\overline{R}}_t\sum _{i=0}^{k-1}\nabla log{\pi }^{\varphi }\left(a_{t-i}|s_{t-i}\right)\right)\right]$$

(41)

where $\left|\tau \right|=N$ represents the length of the trajectory, $\tau$. Similar to Truncated Backpropagation Through Time (TBPTT) [40], a truncating gradient reduces the long-term correlation between value and action, leading to a decrease in gradient variance and consequently lowering the learning difficulty.

In addition, in the THIL-SAC algorithm, the reward function is shaped using expert experience (Equation (35)), with the optimal policy remaining unchanged [34]. However, since the value function ${\widehat{V}}^E$ in Equation (35) is learned through expert experience, it inherently serves as a low-noise, high-quality target signal. As a result, the variance of the shaped reward $\overline{R}$ can be significantly smaller than that of the original reward R. This reduction in variance leads to the improved value function $Q_i^{{\vartheta }_i}$, estimation accuracy, and lower gradient variance, thereby accelerating training convergence [41]. Furthermore, the initial filling and utilization of expert experiences in the replay buffer reduces state–action distribution mismatch and provides high-quality targets early on, which has been shown to improve both convergence rate and value estimation stability [42,43].

In summary, the proposed THIL-SAC algorithm leverages expert experience through reward shaping and experience replay imitation to provide high-quality examples for learning, thereby reducing the error in value function estimation. Building upon this foundation, the algorithm utilizes the TH method to reduce learning sample complexity and the variance in policy gradient calculations, effectively lowering the learning difficulty and enhancing convergence during training.

3.4. Training Process of THIL-SAC-Based Penetration Strategy

The training of the hypersonic vehicle penetration strategy based on THIL-SAC algorithm is divided into two parts: initialization and interactive training. The specific process is described in Algorithm 1.

Algorithm 1 Penetration algorithm based on THIL-SAC.

1: initialize parameters: main Q-networks ${\vartheta }_1,{\vartheta }_2$, policy networks $\varphi$, truncated length $k=1$, target Q-network parameters ${\vartheta }_{tar,1}\leftarrow {\vartheta }_1,{\vartheta }_{tar,2}\leftarrow {\vartheta }_2$, expert value function ${\widehat{V}}^E\left(s\right)$; Clear the experience replay bufferD. 2: for each episode i do 3:     Reset the penetration environment and obtain the initial state $s_t$ Calculate the truncated length k using Equation (38). 4:     for step t in $\left[1,N\right]$ do 5:         Observe state $s_t$ and select action $a_t\sim {\pi }^{\varphi }\left({\mathbf{s}}_t\right)$. 6:         Analyze $a_t$ to get penetration command $\sigma$. 7:         Execute $\sigma$, observe next state $s_{t+1}$, shaped reward ${\overline{R}}_t$. 8:         Store $\left(t,s_t,a_t,s_{t+1},{\overline{R}}_t\right)$ in truncated replay buffer $D^k$. 9:         for each network learning step do 10:            Randomly sample $batch\_size$ samples from $D^k$. 11:            Update main Q-network: 12:            ${\vartheta }_i\leftarrow {\vartheta }_i-{\lambda }_Q\nabla J_Q\left({\vartheta }_i\right),i=1,2$ 13:            Update policy network: 14:            $\varphi \leftarrow \varphi -{\lambda }_{\pi }\nabla J_{\pi }\left(\varphi \right),i=1,2$ 15:            Update temperature coefficient: 16:            ${\alpha }_T\leftarrow {\alpha }_T-{\lambda }_{{\alpha }_T}\nabla J\left({\alpha }_T\right)$ 17:            Update target Q-network: 18:            ${\vartheta }_{tar,i}\leftarrow \tau {\vartheta }_i+\left(1-\tau \right){\vartheta }_i,i=1,2$ 19:         end for 20:     end for 21: end for

4. Simulation Experiments and Analysis

This section will first verify the effectiveness of the proposed reward function, followed by the validation of the proposed penetration strategy. The validation of the penetration strategy encompassed the following aspects:

• The Learning Efficiency of the Penetration Strategy: The SAC algorithm and the proposed algorithm were used to learn penetration strategies under identical conditions, and the learning speeds were compared.
• The Effectiveness of the Penetration Strategy: The trained penetration strategies were applied to a realistic penetration scenario, where the performance of the expert strategy, the SAC-based penetration strategy, and the proposed method were evaluated and compared to demonstrate their effectiveness.

4.1. Simulation Condition

During the gliding phase of the hypersonic vehicle, the penetration game was initiated when the vehicle was 50 km away from the interceptor. The interceptor was modeled as a Standard Missile-6 (SM-6, Raytheon Missiles & Defense, Tucson, AZ, USA), using proportional navigation guidance with a navigation ratio of 5. The hypersonic vehicle was modeled as a CAV-L. To satisfy terminal position and velocity constraints, lateral maneuvers in the horizontal plane (by changing the bank angle) were employed for penetration. At the start of the simulation, the interceptor adopted a head-on intercept against the hypersonic vehicle, and the initial state parameters for both are shown in

Table 1. Initial parameters of hypersonic vehicle and interceptor.

Parameters	Hypersonic Vehicle	Interceptor
Longitude (°)	148.95	149.30
Latitude (°)	23.72	23.99
Altitude (m)	26997.33	29967.40
X-direction speed (m/s)	−2165.21	1404.25
Y-direction speed (m/s)	42.11	−153.45
Z-direction speed (m/s)	−366.06	−48.79

.

The THIL-SAC algorithm includes three types of networks: the Q-network, the policy network, and the expert value function network.

The policy network took $\mathbf{s}$ as the input. The hidden layer consists of 256 neurons, with the Rectified Linear Unit (ReLU) as the activation function. The network output is divided into two layers: an action-mean output layer and a standard deviation output layer. The action (maneuver command for penetration) was obtained through Gaussian sampling of these outputs. To ensure the validity of the output action, the Tanh function was applied to constrain the output within an appropriate range.

The Q-network took the state–action pair $\left(\mathbf{s},a\right)$ as the input. It consists of two hidden layers, each containing 256 neurons and using ReLU as the activation function. The output layer consists of a neuron, responsible for outputting the Q-value.

The expert value function network took the state $\mathbf{s}$ as the input and output the state value. Its structure is identical to that of the Q-network.

The THIL-SAC-based penetration strategy training environment was implemented using the PyTorch 2.0.1 framework. The neural network optimizer used was Adam, with hyperparameters adjusted based on training performance. The final values of the training hyperparameters are listed in

Table 2. Training parameters of THIL-SAC-based algorithm.

Parameters	Value
Policy network learning rate ${\lambda }_{\pi }$	0.0003
Q-network learning rate ${\lambda }_Q$	0.0003
Replay buffer size	1e5
Batch size	256
Discount factor $\gamma$	0.99
Temperature coefficient ${\alpha }_T$	0.12
Soft update coefficient $\tau$	0.01
Parameters of reward function $\{c_1,c_2,c_3\}$	$\{10,1,5\}$
Parameters of truncated length $\{b_0,b_1\}$	$\{50,10\}$

.

Remark 3. 

The above training parameters were obtained through simulation optimization. The simulation optimization process for the reward function parameters $c_1$, $c_2$, and $c_3$ is shown in Appendix A. The results confirm the reward function parameter design principles outlined in Remark 2.

In the subsequent penetration strategy training process, the network structure and parameter settings for the SAC-based penetration strategy training were the same as those for the THIL-SAC-based penetration strategy algorithm.

4.2. Simulation Results and Analysis

4.2.1. Different Reward Functions Comparison

In this section, the influence of different reward functions on the learning of hypersonic vehicle penetration strategies is analyzed to validate the effectiveness of the proposed reward function. Based on the aforementioned simulation scenarios, two sets of simulations were conducted using the SAC algorithm for training. The reward functions employed included the line-of-sight angular velocity-based reward function from Reference [24] and the proposed ZEM-based reward function, with all other conditions kept identical. The variations in the reward function during the training process are presented in Figure 3.

In Figure 3, the average rewards across five runs are shown in solid lines and the range of rewards are shown in the shaded area. Under identical conditions, the reward function proposed in this paper enabled the learning of hypersonic vehicle penetration strategies to converge at approximately 400 episodes. In contrast, when using the reward function from Reference [24], the reward values only began to increase after over 800 episodes and gradually converged to a value lower than that achieved by the proposed method. This is because the proposed reward function is constructed based on the ZEM, which is inherently more aligned with the penetration objective compared to the line-of-sight angular velocity-based reward function. Consequently, the proposed reward function effectively improves the learning efficiency of hypersonic vehicle penetration strategies.

4.2.2. Results of Penetration Strategy Training

In this paper, the optimal penetration guidance law (expert strategy) from the literature [14] was simulated under the initial conditions set in Section 4.1 to obtain expert experience data for the hypersonic vehicle’s penetration. Based on these data, the THIL-SAC-based and SAC-based hypersonic vehicle penetration strategy networks were then trained.

In addition, to further validate the effectiveness of the proposed method, the improved SAC algorithm, ISAC, proposed in [44], which incorporates Prioritized Experience Replay (PER) and Emphasizing Recent Experience (ERE), was also employed for training the penetration strategy networks.

The hypersonic vehicle penetration strategy networks were trained separately using the SAC-based, ISAC-based, and THIL-SAC-based algorithms under identical settings. The reward value variation curves during training are presented in Figure 4.

In Figure 4, the average rewards across five runs are shown in solid lines and the range of rewards are shown in shaded area. When training the hypersonic vehicle penetration strategy network using the hyper-parameters listed in Table 2, all three algorithms (SAC, ISAC, and THIL-SAC) exhibited an overall increasing trend in reward values, eventually stabilizing. This indicates that the agent successfully learned the vehicle penetration strategy.

During the training process using the THIL-SAC-based algorithm, the reward function converged at approximately 200 episodes, compared to around 400 episodes for the SAC-based algorithm and around 300 episodes for the ISAC-based algorithm. This reflects a significant improvement in convergence speed. Moreover, the reward curve of the THIL-SAC algorithm displays a smoother convergence trajectory, suggesting enhanced training stability. Although the final convergence values of the reward functions for all three algorithms were similar, the proposed THIL-SAC-based penetration algorithm demonstrated a clear advantage in terms of convergence speed and stability.

4.2.3. Testing Results of the Penetration Strategy in Typical Scenarios

To validate the effectiveness of the trained penetration strategy network, the trained penetration strategy was used to generate bank angle commands for the hypersonic vehicle’s penetration. Under the initial conditions given in Section 4.1, the expert strategy, the SAC-based penetration strategy network, and the THIL-SAC-based penetration strategy network were used for hypersonic vehicle’s penetration. The results are presented in Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11.

As shown in Figure 5 and Figure 11, under all three penetration strategies, the hypersonic vehicle successfully penetrated the interceptor. At the miss time, the ZEM was 763.76 m for the expert strategy and 972.34 m and 1008.76 m for the SAC-based and THIL-SAC-based strategies, respectively (

Table 3. ZEM of three penetration strategies at miss time.

Strategy	ZEM at the Miss Time
Expert strategy	763.76 m
SAC-based strategy	972.34 m
THIL-SAC-based strategy	1008.76 m

). In Figure 6, the bank angle command curves indicate that the penetration commands generated by the THIL-SAC-based strategy were initially similar to those of the expert strategy. However, in the later stages of penetration, the THIL-SAC-based strategy adjusted the timing and method of maneuvers, resulting in a larger miss distance compared to that in the expert strategy. This is because the THIL-SAC-based strategy optimized the penetration approach without simplifying assumptions about the model, ultimately leading to better performance than the expert strategy based on optimal guidance laws. Under the influence of the penetration command depicted in Figure 6, the normal and lateral overload curves of the hypersonic vehicle are presented in Figure 7 and Figure 8. The normal and lateral overload curves of the interceptor are shown in Figure 9 and Figure 10.

4.2.4. Analysis of the Generalization of the Penetration Strategy Network

To verify the adaptability and effectiveness of the THIL-SAC-based penetration strategy, simulations were conducted under different attack–defense scenarios while considering hypersonic vehicle detection errors. The detection errors follow a normal distribution, with their standard deviation given as follows:

$$\left\{\begin{array}{cc}\hfill & {\sigma }_{R_{rel}}={\sigma }_{R_{rel0}}{\left({\displaystyle \frac{∥{\mathbf{R}}_{rel}∥}{∥{\mathbf{R}}_{rel0}∥}}\right)}^2\hfill \\ \hfill & {\sigma }_{V_{rel}}={\sigma }_{V_{rel0}}{\left({\displaystyle \frac{∥{\mathbf{R}}_{rel}∥}{∥{\mathbf{R}}_{rel0}∥}}\right)}^2\hfill \end{array}\right.$$

(42)

where ${\sigma }_{R_{rel}}$ and ${\sigma }_{V_{rel}}$ are the standard deviation errors for relative distance and relative velocity, with initial values of ${\sigma }_{R_{rel0}}=100\mathrm{m}$ and ${\sigma }_{V_{rel0}}=5\mathrm{m}/\mathrm{s}$.

The variations in these scenarios were achieved by altering the initial position and speed of the interceptor, with the initial deviation settings detailed in

Table 4. Initial deviation of interceptor.

Parameters	Standard Deviation
Initial position deviation along the X-axis	2500 m
Initial position deviation along the Y-axis	2500 m
Initial position deviation along the Z-axis	2500 m
Initial velocity deviation along the X-axis	25 m/s
Initial velocity deviation along the Y-axis	25 m/s
Initial velocity deviation along the Z-axis	25 m/s

.

Under the above conditions, 500 target simulations were conducted. The distribution of the ZEM at the time of miss is shown in Figure 12 and Figure 13.

As shown in Figure 12, among the 500 target simulations, the number of cases where the ZEM was less than 10 m was 102 for the expert strategy and 35 for the THIL-SAC-based strategy. Considering the interceptor’s lethal radius, penetration was deemed successful only if the terminal miss distance exceeded 10 m. Based on this criterion, the penetration success rates for the expert and THIL-SAC-based strategies were 79.6% and 93%, respectively. Figure 13 further shows that the ZEM distribution for the expert strategy was generally lower than that for the THIL-SAC-based strategy, which suggests that the THIL-SAC penetration strategy not only achieved a higher penetration success rate but also demonstrated better adaptability compared to the expert strategy.

Furthermore, on a computer with an AMD Ryzen 7 6800H CPU operating at a base frequency of 3.20 GHz, the THIL-SAC penetration strategy network took 1.479 to 1.534 s to perform 10,000 calculations, with a single calculation taking less than 0.2 milliseconds. This meets the real-time requirements for the highly dynamic penetration of hypersonic vehicles.

5. Conclusions

This paper addresses the problem of penetration in the gliding phase of a hypersonic vehicle by designing a THIL-SAC-based imitation-reinforcement learning approach with progressively increasing learning difficulty. By unifying imitation learning and reinforcement learning objectives within a truncated horizon framework, this approach gradually increases learning difficulty, enabling the rapid training of the hypersonic vehicle’s penetration strategy. The resulting THIL-SAC-based strategy not only ensures successful penetration but also demonstrates strong adaptability. Compared to traditional expert strategies, the THIL-SAC-based method avoids the need for model simplifications or assumptions, leading to enhanced penetration performance and adaptability. Additionally, the THIL-SAC method converges more quickly than SAC-based reinforcement learning methods and requires relatively low computational resources during online application, making it well suited for the real-time demands of highly dynamic penetration scenarios faced by hypersonic vehicles.