A Multi-Constraint Guidance and Maneuvering Penetration Strategy via Meta Deep Reinforcement Learning

Abstract

In response to the issue of UAV escape guidance, this study proposed a unified intelligent control strategy synthesizing optimal guidance and meta deep reinforcement learning (DRL). Optimal control with minor energy consumption was introduced to meet terminal latitude, longitude, and altitude. Maneuvering escape was realized by adding longitudinal and lateral maneuver overloads. The Maneuver command decision model is calculated based on soft-actor–critic (SAC) networks. Meta-learning was introduced to enhance the autonomous escape capability, which improves the performance of applications in time-varying scenarios not encountered in the training process. In order to obtain training samples at a faster speed, this study used the prediction method to solve reward values, avoiding a large number of numerical integrations. The simulation results demonstrated that the proposed intelligent strategy can achieve highly precise guidance and effective escape.

1. Introduction

Hypersonic UAVs mainly glide in near-space [1]. In their early phase, a higher flight velocity is acquired, relying on the thin atmospheric environment, which is an advantage for effectively avoiding the interception of defense systems. At the end of a gliding flight, the velocity is mainly influenced by the aerodynamic force and suffers from the restrictions of heat flow, dynamic pressure, and overload [2]. The velocity advantage leads to penetration becoming more difficult, so orbital maneuvering is applied by UAVs to achieve penetration. The main flight mission is split into avoiding defense system interception and satisfying multiple terminal constraints [3]. The core of this manuscript is designing a penetration guidance strategy via orbital maneuvering capabilities, avoiding interception, and reducing the penetration’s impact on guidance accuracy.

The penetration strategy is summarized as a tactical penetration and a technical penetration strategy [4]. The technical penetration strategy changes the flight path through maneuvering, aiming to increase the missed distance in order to successfully penetrate. Common maneuver manners include the sine maneuver, step maneuver, square wave maneuver, and spiral maneuver [5]. There are some limitations and instability for technical penetration strategies, attributed to the UAV struggling to adopt an optimal penetration strategy according to the actual situation of the offensive and defensive confrontations. Compared with the traditional procedural maneuver strategy, the differential game guidance law has the characteristics of real-time operation and intelligence as a tactical penetration strategy [6]. Penetration problems are essentially regarded as the continuous dynamic conflict problem of multi-party participants, and this strategy is an essential solution for solving multi-party optimal control problems. Applying it to the problem of attack–defense confrontation can not only fully consider the relative information between UAVs and interceptors but also obtain a Nash equation strategy to reduce energy consumption. Many scholars have proposed differential game models of various maneuvering control strategies based on control indexes and motion models. Garcia [7] regarded the scenario of active target defense modeling as a zero-sum differential game, designed a complete differential game solution, and comprehensively considered the optimal strategy of closed-loop state feedback to obtain the value function. In Ref. [8], the optimal guidance problem was studied between an interceptor and an active defense ballistic UAV, and an optimal guidance scheme was proposed based on the linear quadratic differential game method and the numerical solution of Riccati differential equations. Liang [9] mainly analyzed the problem of pursuit and escape attacks of multiple players, inducted the three-body game confrontation into competition and cooperation problems, and solved the optimal solution of multiple players via differential game theory. The above methods are of great significance for analyzing and solving the confrontation process between UAVs and interceptors. Near-space UAVs have the characteristics of high velocity and short time in the phase of attack and defense confrontation terminal guidance [10], and the differential game guidance law struggles to show advantages in this phase. Moreover, the differential game method requires a large amount of calculation, and the bilateral performance indicators are difficult to model [11]; as a result, this theory is unable to be applied in practice.

DRL is a research hotspot in the field of artificial intelligence that has sprung up in recent years, and amazing results have been achieved in robot control, guidance, and control technologies [12]. DRL specifically refers to agents learning in the process of interaction with the environment to find the best strategy to maximize cumulative rewards [13]. With the advantages of dealing with high-dimensional abstract problems and making decisions quickly, DRL provides a new solution for the maneuvering penetration of high-velocity UAVs. In order to solve the problem of intercepting a high maneuvering target, an auxiliary DRL algorithm was proposed in Ref. [14] to optimize the frontal interception guidance control strategy based on a neural network. Simulation results showed that DRL had a higher hit rate and larger terminal interception angle than traditional methods and proximal policy optimization algorithms. Gong [15] proposed an Omni bearing attack guidance law for agile UAVs via DRL, which effectively dealt with aerodynamic uncertainty and strong nonlinearity at a high attack angle. DRL was used to generate a guidance law for the attack angle in an agile turning phase. Furfaro [16] proposed an adaptive guidance algorithm based on classical zero-effort velocity, and the limitations of this algorithm were overcome via RL. A closed-loop guidance algorithm was created that is lightweight and flexible enough to adapt to a given constrained scene.

Compared with differential game theory, DRL is convenient for establishing the performance index function, and it is a feasible method to solve the dynamic programming problem by utilizing the powerful numerical calculation ability of computers to skillfully avoid solving the function analytical solution. However, the traditional DRL has some limitations, such as high sample complexity, low sample utilization, a long training time, and so on. Once the mission changes, the original DRL parameters are hard to adapt to the new mission and need to be learned from scratch. A change in mission or environment will lead to the failure of the trained model and poor generalization ability of the model. In order to solve the existing problems in DRL, researchers introduced meta-learning into DRL and proposed Meta DRL [17]. Lu et al. [18] mainly solved the issues of maximizing the total data collected and avoiding collisions during the guidance flight and improved the adaptability to different missions via meta RL. Hu et al. [19] mainly studied a challenging trajectory design problem, optimized the DBS trajectory, and considered the uncertainty and dynamics of terrestrial users’ service requests. MAML was introduced for the purposes of solving raised POMDPS and enhancing adaptability to balance flight aggressiveness and safety [20]. For suspended payload transportation tasks, the paper [21] proposed a meta-learning approach to improve adaptability. The simulation demonstrated improvements in closed-loop performance compared to non-adaptive methods.

By learning useful meta knowledge from a group of related missions, agents acquire the ability to learn, and the learning efficiency on new missions is improved and the complexity of samples is reduced. When faced with new missions or environments, the network responds quickly based on the previously accumulated knowledge, so only a small number of samples are needed to quickly adapt to the new mission.

Based on the above analysis, this manuscript proposes Meta DRL to solve the UAV guidance penetration strategy, and the DRL is improved, resulting in enhancing the adaptability of UAVs in complex and changeable attack and defense confrontations. In addition, the idea of meta-learning is used to enable UAVs to learn and improve their ability for autonomous flight penetration. The core contributions of this manuscript are as follows:

• By modeling the three-dimensional attack and defense scene between UAVs and interceptors, we analyze the terminal and process constraints of UAVs. A guidance penetration strategy based on DRL is proposed, aiming to provide the optimal solution for maneuvering penetration under a constant environment or mission.
• Meta-learning is used to improve the UAV guidance penetration strategy. The improvement enables the UAV to learn and enhances the autonomous penetration ability.
• Through the simulation analysis, the manuscript analyzes the penetration strategy, explores penetration timing and maneuvering overload, and summarizes the penetration tactics.

2. Modeling of the Penetration Guidance Problem

2.1. Modeling of UAV Motion

The three-degree-of-freedom motion equation is adopted to describe UAVs and the dynamic equation is established in the ballistic coordinating system:

$$\left\{\begin{array}{l}\dot{v}=-\frac{\rho v^2{S}_m{C}_D}{2m}+{g^{\prime }}_r\sin \theta +g_{\omega e}\left(\cos \sigma \cos \theta \cos \varphi +\sin \theta \sin \varphi \right)\\ +{\omega }_e^{2}r\left({\cos }^2\varphi \sin \theta -\cos \varphi \sin \varphi \cos \sigma \cos \theta \right)\\ \dot{\theta }=\frac{\rho v^2{S}_m{C}_L\cos \upsilon }{2mv}+\frac{{g^{\prime }}_r\cos \theta }{v}-2{\omega }_e\sin \sigma \cos \varphi + \frac{v\cos \theta }{r}\\ +\frac{g_{\omega e}}{v}\left(\cos \theta \sin \varphi -\cos \sigma \sin \theta \cos \varphi \right)+\frac{{\omega }_e^{2}r}{v}\left(\cos \varphi \sin \varphi \cos \sigma \sin \theta +{\cos }^2\varphi \cos \theta \right)\\ \dot{\sigma }=-\frac{\rho v^2{S}_m{C}_L\sin \upsilon }{2mv\cos \theta }-\frac{g_{\omega e}\sin \sigma \cos \varphi }{v\cos \theta }+ \frac{{\omega }_e^{2}r\left(\cos \varphi \sin \varphi \sin \sigma \right)}{v\cos \theta }+\frac{v\tan \varphi \cos \theta \sin \sigma }{r}\\ -2{\omega }_e\left(\sin \varphi -\cos \sigma \tan \theta \cos \varphi \right)\\ \dot{\varphi }=\frac{v\cos \theta \cos \sigma }{r}\\ \dot{\lambda }=-\frac{v\cos \theta \sin \sigma }{r\cos \varphi }\\ \dot{r}=v\sin \theta \end{array}\right.$$

(1)

where $v$ is the velocity of UAV relative to the earth, $\theta$ is the velocity slope angle, $\sigma$ is the velocity azimuth, and the positive direction is clockwise from the north. $r$ represents the geocentric distance, and $\left(\lambda ,\varphi \right)$ is the longitude and latitude. The differential argument is flight time $t$. $g_{\omega e}$ is the component of the Earth’s gravitational acceleration in the direction of the Earth’s rotational angular rate ${\omega }_e$, while $g_r^{\prime }$ is the component of the Earth’s gravitational acceleration in the direction of the geo-center. $\rho$ is the density of the atmosphere, while $m$ and $S_m$ are the mass and reference area of UAV. $C_D$ and $C_L$ are the drag and lift coefficient, respectively, relating to the Mach number and attack angle, so the control variable attack angle $\alpha$ is implicit in it, and the other control variable is the bank angle $\upsilon$.

For a hypersonic UAV with a large lift–drag ratio (L/D), the heat flow, overload, and dynamic pressure are considered as flight process constraints, as follows:

$$\left\{\begin{array}{l}{k}_h{\rho }^{1/2}{v}^3\le Q_{s\max }\\ \frac{\rho v^2}{2}\le q_{\max }\\ \frac{\sqrt{D^2+L^2}}{m{g}_0}\le n_{\max }\end{array}\right.$$

(2)

where $Q_{s\max }$, $q_{\max }$, and $n_{\max }$ are the maximum heat flow, dynamic pressure, and overload, respectively; the parameter $k_h$ is a constant coefficient, and g₀ is the gravitational acceleration at sea level. In order to keep the gliding state steady and effectively prevent trajectory jumps, the balance of lift and gravity is required by UAV [22]. Stable gliding is generally considered as quasi-equilibrium gliding (QEGs). At present, the international definition of QEG can be summarized into two types [23]: the velocity slope angle is considered as constant, expressed by $\dot{\theta }=0$, or the flight altitude variance ratio is considered as constant, expressed by $\ddot{h}=0$. Moreover, $\dot{\theta }=0$ was adopted by traditional QEG guidance, and $\ddot{h}=0$ mainly appeared in the analytical prediction guidance of the Mars landing at the end of the last century [24].

For high L/D UAVs flying in the gliding phase, the Coriolis inertial force and centrifugal inertial force caused by the Earth’s rotation are relatively small compared to aerodynamic forces. Therefore, in the design of guidance laws, the Earth can be assumed to be a homogeneous sphere that does not rotate. We further decompose Equation (1) into longitudinal and lateral directions, where the longitudinal motion equation is as follows:

$$\left\{\begin{array}{l}\dot{v}=-\frac{\rho v^2{S}_m{C}_D}{2m}-g\sin \theta \\ \dot{\theta }=\frac{\rho v^2{S}_m{C}_L\cos \upsilon }{2mv}-\frac{g\cos \theta }{v}+\frac{v\cos \theta }{r}\\ \dot{h}=v\sin \theta \\ {\dot{L}}_R=R_{e}v\cos \theta /\left(R_e+h\right)\end{array}\right.$$

(3)

where $h=r-R_e$ represents the flight altitude, and $L_R$ is range. For a gliding UAV flying in close proximity to space, $h\ll R_e$. Therefore, range differentiation can be further described as follows:

$${\dot{L}}_R=v\cos \theta$$

(4)

This manuscript adopts $\dot{\theta }=0$ as the QEG condition. Referring to the second equation in Equation (3), $\dot{\theta }=0$ is transferred into Equation (5).

$$\begin{array}{l}\dot{\theta }=0=\frac{\rho v^2{S}_m{C}_L\cos \upsilon }{2mv}-\frac{g\cos \theta }{v}+\frac{v\cos \theta }{r}\\ \Rightarrow \frac{\rho v^2{S}_m{C}_L\cos \upsilon }{2mv}=\frac{g\cos \theta }{v}-\frac{v\cos \theta }{r}\\ \Rightarrow \frac{\rho v^3{S}_m{C}_L\cos \upsilon }{2}=\frac{mg\cos \theta }{v}-\frac{m{v}^2\cos \theta }{r}\end{array}$$

(5)

where $L=\frac{\rho v^3{S}_m{C}_D}{2}$, hence

$$m(g-\frac{v^2}{r})\cos \theta -L\cos \upsilon =0$$

(6)

2.2. Description of Flight Missions

2.2.1. Guidance Mission

The physical meaning of gliding guidance is eliminating heading errors, satisfying complex process constraints, and minimizing energy loss. The UAV is guided to glide unpowered to the setting terminal target point ($h_f,{\lambda }_f,{\varphi }_f$), satisfying the terminal altitude, longitude and latitude. Hence, the terminal constraints are expressed by Equation (7).

$$\left\{\begin{array}{l}h(L_{Rf})=h_f, \lambda (L_{Rf})={\lambda }_f\\ \varphi (L_{Rf})={\varphi }_f,\Delta {\sigma }_f\le \Delta {\sigma }_{\max }\end{array}\right.$$

(7)

where the terminal range $L_{Rf}$ is given, $\Delta \sigma$ represents the heading error, and $\Delta {\sigma }_{\max }$ is a pre-setting allowable value. The guidance problem is the process of determining $\alpha$ and $\upsilon$.

2.2.2. Penetration Mission

The main indexes are used to judge penetration probability as follows:

(1)

The miss distance $D_{miss}$ with the interceptor at the encounter moment.

(2)

The overload $N_m$ of the interceptor in the last phase.

(3)

The line-of-sight (LOS) angular rates ${\dot{\theta }}_{\mathrm{int}los}$ and ${\dot{\sigma }}_{\mathrm{int}los}$ with the interceptor at the encounter moment.

where $D_{miss}$ directly reflects the result of penetration—the larger the $D_{miss}$, the greater of penetration probability. The $N_m$ and LOS angle indirectly reflect the result of penetration. The larger the ${\dot{\theta }}_{\mathrm{int}los}$ and ${\dot{\sigma }}_{\mathrm{int}los}$, the more difficult it is for the interceptor to successfully intercept, which is attributed to ${\theta }_{\mathrm{int}los}$ and ${\sigma }_{\mathrm{int}los}$ not easily converging to a constant value at the encounter moment. In addition, a larger overload is required to adjust ${\theta }_{\mathrm{int}los}$ and ${\sigma }_{\mathrm{int}los}$ at the end of interception. The larger the $N_m$, the greater the control cost that the interceptor has to pay to complete the interception mission. Once $N_m$ exceeds the overload limit of the interceptor, it indicates that the control is saturated, which demonstrates that the interceptor struggles to intercept based on the current maximum overload constraint. Referring to the size and flight characteristics of UAVs [25], this manuscript assumes that the penetration mission is completed if $D_{miss}$ is greater than 2 m.

The maximum maneuvering overload of UAVs is constrained by the structure and QEG condition [24]. If the lateral maneuver overload is too large, it will cause the UAV to deviate from the course, eventually leading to the failure of the guidance mission. A large longitudinal maneuver amplitude will have a significant impact on the L/D of the UAV, affecting safety. According to the above analysis, the maximum lateral maneuvering overload is set as 2 g, and the maximum longitudinal maneuvering overload is set as 1 g.

2.3. The Guidance Law of Interceptor

The guidance law of the interceptor relies on the inertial navigation system to obtain information such as the position and velocity of the UAV. The relative motion is shown in Figure 1.

P, T, and M, respectively, represent the UAV, target, and interceptor. $r$ is the relative position between the UAV and the interceptor, and $r_t$ is the target.

The LOS angular rate and the approach velocity to the UAV are obtained for the interceptor. Overload control command is derived via generalized proportional navigation guidance (GPNG).

$$\left\{\begin{array}{l}{n}_{y_2}^{*}=K_D\left|\dot{r}\right|{\dot{\theta }}_{\mathrm{int}los}/g_0\\ n_{z_2}^{*}=K_T\left|\dot{r}\right|{\dot{\sigma }}_{\mathrm{int}los}/g_0\end{array}\right.$$

(8)

As shown in Equation (8), $K_D$ and $K_T$ are navigation ratios in the longitudinal and lateral direction, respectively. $\dot{r}$ is the approach velocity. ${\dot{\theta }}_{\mathrm{int}los}$ and ${\dot{\sigma }}_{\mathrm{int}los}$ represent the LOS angular rate in the longitudinal and lateral directions, respectively.

$$\left\{\begin{array}{l}r=\sqrt{{(x-x_m)}^2+{(y-y_m)}^2+{(z-z_m)}^2}\\ {\theta }_{\mathrm{int}los}=\arcsin ((y-y_m)/r)\\ {\sigma }_{\mathrm{int}los}=\arctan ((z-z_m)/(x-x_m))\end{array}\right.$$

(9)

Differentiating Equation (9) with respect to $t$, we obtain the following:

$$\left\{\begin{array}{l}{\dot{\theta }}_{\mathrm{int}los}=\frac{1}{\sqrt{1-{(\frac{y-y_m}{r})}^2}}\cdot \frac{(\dot{y}-{\dot{y}}_m)r-(y-y_m)\dot{r}}{r^2}\\ {\dot{\sigma }}_{\mathrm{int}los}=\frac{1}{1+{(\frac{z-z_m}{x-x_m})}^2}\cdot \frac{(\dot{z}-{\dot{z}}_m)\cdot (x-x_m)-(z-z_m)\cdot (\dot{x}-{\dot{x}}_m)}{{(x-x_m)}^2}\\ \dot{r}=\frac{(x-x_m)\cdot (\dot{x}-{\dot{x}}_m)+(y-y_m)\cdot (\dot{y}-{\dot{y}}_m)+(z-z_m)\cdot (\dot{z}-{\dot{z}}_m)}{r}\end{array}\right.$$

(10)

3. Design of Penetration Strategy Considering Guidance

3.1. Guidance Penetration Strategy Analysis

Generally, the UAV achieves penetration through a velocity advantage or increased maneuvering overload. The former is used in pre-gliding flight, and compared with interceptors, the velocity of UAVs is relatively large, which is conducive to penetrating defenses. More threats to UAVs come from the defense systems of intended targets, resulting in an intercept threat focusing on the end of the glide flight. However, the flight velocity gradually decreases, which is not enough to penetrate escape. Based on the above analysis, the manuscript designs a penetration strategy by increasing the maneuvering overload.

Firstly, in previous research [26], the energy-optimized gliding guidance law was designed. In this study, avoiding the intercept by maneuvering becomes the key point. The real-time flight information of the interceptor cannot easily be accurately obtained by the UAV, while the real-time flight information of the UAV is obtained by the interceptor. Based on the UAV penetration mission, which only knew the initial launch position of the interceptor, this manuscript simulates the attack and defense environment between the UAV and the interceptor, applying the DRL method to solve the overload command of UAV maneuvering penetration. DNN parameters are trained offline, and a maneuvering penetration command is constructed online. The launch position of the interceptor is changeable, and a stable penetration command cannot adapt to the complex penetration environment. To improve the adaptability of DNN parameters, the original DRL is optimized by adopting the idea of meta-learning, and the ontology and environmental information are fully utilized. The optimization of meta-learning enhances flight capability, fast response to complex missions, and flight self-learning. Finally, the UAV guidance penetration strategy based on Meta DRL is proposed in this manuscript, as shown in Figure 2.

As for the gliding guidance mission, DRL and optimal control are used to achieve the guidance penetration mission of the UAV. An optimal guidance command was conducted in previous research [26], which was introduced to satisfy constraints regarding terminal position, altitude, and minimal energy loss. Maneuvering overloads are added between the longitudinal and lateral directions, aiming to achieve the penetration mission at the end of the gliding flight. Maneuvering overloads are solved via DRL. The DNN parameters are optimized by means of meta-learning.

Guidance command $\left(n_y^{\ast }, n_z^{\ast }\right)$ is generated by the optimal guidance strategy, and maneuvering overload command $\left(\Delta n_y,\Delta n_z\right)$ is generated by meta DRL. The flight total overload is shown in Equation (11).

$$\left\{\begin{array}{l}Ny=n_y^{\ast }+\Delta n_y\\ Nz=n_z^{\ast }+\Delta n_z\end{array}\right.$$

(11)

Maneuvering penetration command via meta DRL is the core of this manuscript. The penetration considering guidance is described as a Markov decision process (MDP), which consists of finite-dimensional continuous flight state space, longitudinal and lateral direction overload sets, and a reward function judging the penetration strategy. Flight data generated via numerical integration and SAC networks are introduced to train and learn MDP. Optimizing network parameters via meta-learning aims at adjusting network parameters with very little flight data when the UAV is faced with online mission changes, adapting to the new environment as soon as possible.

3.2. Energy Optimal Gliding Guidance Method

In previous research [26], based on the QEG condition and taking the required overload as the control amount, the performance index with minimum energy loss was established. The optimal longitudinal and lateral overload were designed, respectively, satisfying the constraints on terminal latitude, longitude, altitude, and velocity. The required overload command is shown in Equation (12).

$$\left\{\begin{array}{l}{u}_y^{*}=k\left(C_h{L}_R-C_{\theta }\right)+1\\ u_z^{*}=\frac{{\sigma }_{los}-\sigma }{k\left(L_{Rf}-L_R\right)}\end{array}\right.$$

(12)

where $u_y^{*}=n_y^{*}$ and $u_z^{*}=n_z^{*}$ are the optimal overload between the longitudinal and lateral dimensions. $k=\frac{g_0}{v^2}\approx \frac{g}{v^2}$, $L_R$ is the current range, and $L_{Rf}$ is the total range of the gliding phase. $C_h$ and $C_{\theta }$ are the guidance coefficients based on optimal control, represented as Equation (13).

$$\left\{\begin{array}{l}{C}_h=\frac{6\left(\left(L_R-L_{Rf}\right)\left({\theta }_f+\theta \right)-2h+2h_f\right)}{k^2{\left(L_R-L_{Rf}\right)}^3}\\ C_{\theta }=\frac{2\left(L_R{L}_{Rf}\left(\theta -{\theta }_f\right)-L_{Rf}^2\left(2\theta +{\theta }_f\right)+L_R^2\left(2{\theta }_f+\theta \right)+3\left(L_{Rf}+L_R\right)\left(h_f-h\right)\right)}{k^2{\left(L_R-L_{Rf}\right)}^3}\end{array}\right.$$

(13)

Based on Equation (14), the control variables $\alpha$ and $\upsilon$ are calculated as follows:

$$\left\{\begin{array}{l}\frac{\rho v^2{S}_m{C}_L(Ma,\alpha )}{2g_0}=\sqrt{n_y^{*2}+n_z^{*2}}\\ \upsilon =\arctan \left(\frac{n_z^{*}}{n_y^{*}}\right)\end{array}\right.$$

(14)

where $\alpha$ is obtained by calculating the contrast value in Equation (14).

4. RL Model for Penetration Guidance

The problem of maneuvering penetration is modeled as a series of stationary MDPs [27] with unknown transition probabilities. The continuous flight state space, action set, and reward function for judging the command are determined in this section.

4.1. MDP of the Penetration Guidance Mission

Deterministic MDP with a continuous state and action is defined as $(\mathit{S},\mathit{A},T,R,\gamma )$ via a quintuple. $\mathit{S}$ is described as the continuous state space, $\mathit{A}$ is described as the finite action set, and $T$ is depicted as the state transition function. $\mathit{S}\times \mathit{A}\to T$, reflecting deterministic state transition relationships. $R$ is defined as the immediate reward. $\gamma \in [0,1]$ is the discount factor, to balance immediate and forward reward.

The agent chooses the action $a_t\in \mathit{A}$ at the current state $s_t$, while the state changes from $s_t$ to $s_{t+1}\in \mathit{S}$, and the environment returns an immediate reward $R_t=f(s_t,a_t)$ for the agent. Cumulative rewards are obtained with a controlled action sequence $\tau$, as shown in Equation (15).

$$G\left(s_0,\tau \right)=R_0+\gamma R_1+{\gamma }^2{R}_2+...={\displaystyle \sum _{t=0}^{\infty }{\gamma }^t{R}^t}$$

(15)

The goal of MDP is determining policy, and maximizing the expected accumulated rewards, as follows:

$${\tau }^{\ast }=\arg \underset{\tau }{\max }\left\{\left.G\left(s_0,\tau \right)\right\}\right.$$

(16)

4.1.1. State Space Design

The environment of attack–defense is abstracted as a state space of MDP, which applies guidance for action commands. UAVs struggle to acquire the flight information and guidance laws of interceptors. Hence, the information of the UAV and the target is only considered as state space.

$$\mathit{S}=\left\{\left.x_r,y_r,z_r,{\theta }_{los},{\sigma }_{los}\right\}\right.$$

(17)

where $\left(x_r,y_r,z_r\right)$ represents the relative distance between the UAV and the target under the North East Down (NED) Coordinate System. $\left({\theta }_{los},{\sigma }_{los}\right)$ represents, respectively, the longitudinal and lateral LOS angles. In order to eliminate dimension differences and enhance compatibility among states, $\mathit{S}$ is normalized using Equation (18).

$$\mathit{S}=\left(x_r=\frac{x_r}{x_{r0}}, y_r=\frac{y_r}{y_{r0}}, z_r=\frac{z_r}{z_{r0}}, {\theta }_{los}=\frac{{\theta }_{los}}{2\pi }, {\sigma }_{los}=\frac{{\sigma }_{los}}{2\pi }\right)$$

(18)

4.1.2. Action Space Design

Action is a decision selected by the UAV based on the current state, and action space $\mathit{A}$ is the set of all possible decisions. Overload directly affects the velocity azimuth and slope angle and indirectly affects the gliding flight status; hence, this manuscript determines overload as an intermediate control variable.

On the basis of the optimal guidance law and flight process constraints, longitudinal overload $\Delta n_y$ and lateral overload $\Delta n_{\mathrm{z}}$ are added to the action space of MDP. Considering the safety of the UAV and heading error, this manuscript assumes that the longitudinal and lateral maximum maneuvering overloads are 1 g and 2 g, respectively. $\mathit{A}$ is defined as a continuous set, shown in Equation (19).

$$\mathit{A}=\left\{\begin{array}{l}\Delta n_y\in [-g, g]\\ \Delta n_z\in [-2g, 2g]\end{array}\right.$$

(19)

4.2. Multi-Missions Reward Function Designing

The reward function $f_r$ is an essential section for guiding and training the maneuvering penetration strategy. After execution of the action command, $f_r$ returns a reward value to the UAV, which reflects the fairness and rationality of the action judgment. The rationality of $f_r$ directly affects the training result and determines the efficiency of SAC training. In this manuscript, the aim of $f_r$ is to guide the UAV to achieve the guidance penetration mission while satisfying terminal multi-constraints. Given the requirements of the mission, $f_r$ consists of the miss distance with the interceptor and the terminal deviation from the target.

$$f_r(s,a)=c_1{d}_{miss}-c_2{d}_{error}$$

(20)

where $d_{miss}$ and $d_{error}$ are the miss distance and terminal deviation after execution the action command. The sufficient and necessary condition of satisfying terminal position deviation is eliminating the heading error, which is directly related to the LOS angular rate. Similarly, $d_{miss}$ can be reflected by the LOS angular rate at the encounter time between the UAV and the interceptor. Therefore, the normalized $f_r$ is expressed by Equation (21).

$$f_r=c_1\sqrt{{\left({\overline{\dot{\theta }}}_{\mathrm{int}los}\right)}^2+{\left({\overline{\dot{\sigma }}}_{\mathrm{int}los}\right)}^2}-c_2\sqrt{{\left({\overline{\dot{\theta }}}_{los}\right)}^2+{\left({\overline{\dot{\sigma }}}_{los}\right)}^2}$$

(21)

where ${\overline{\dot{\theta }}}_{\mathrm{int}los}$ represents the normalized LOS angular rate in the longitudinal direction with the interceptor, ${\overline{\dot{\sigma }}}_{\mathrm{int}los}$ represents the normalized LOS angular rate in the lateral direction with the interceptor. ${\overline{\dot{\theta }}}_{los}$ represents the normalized LOS angular rate in the longitudinal direction with the target, and ${\overline{\dot{\sigma }}}_{los}$ represents the normalized LOS angular rate in the lateral direction with the target. In this manuscript, the LOS angular rates at the encounter time and terminal time are solved analytically via numerical calculation.

4.2.1. The Solution of LOS Angular Rate in the Lateral Direction

The attack–defense confrontation model in the lateral direction is shown in Figure 3.

P, T, and M, respectively represent UAV, target, and interceptor. L_Rgo and L_Mgo are represented as the remaining range among UAV, target and interceptor, which are calculated in Equations (22) and (23).

The lateral relative motion model between UAV and the target is shown in Equation (22).

$$\left\{\begin{array}{l}{\dot{L}}_{Rgo}=-v\cos \Delta \delta \\ L_{Rgo}{\dot{\sigma }}_{los}=v\sin \Delta \delta \end{array}\right.$$

(22)

where $\Delta \delta =\sigma -{\sigma }_{los}$, represents the heading error between UAV and the target point. Lateral relative motion model between UAV and interceptor is shown in Equation (23).

$$\left\{\begin{array}{l}{\dot{L}}_{Mgo}=-v\cos \Delta \sigma -v_m\cos \Delta {\sigma }_m\\ L_{Mgo}{\dot{\sigma }}_{\mathrm{int}los}=v\sin \Delta \sigma -v_m\sin \Delta {\sigma }_m\end{array}\right.$$

(23)

where $\Delta \sigma =\sigma -{\sigma }_{\mathrm{int}los}$, represents heading error between UAV and the interceptor. In order to simplify the calculation, the relative motion equation Equation (23) is conducted as Equation (24):

$$\left\{\begin{array}{l}{\dot{L}}_{Mgo}=-v_r\cos \Delta {\sigma }_{mn}\\ L_{Mgo}{\dot{\sigma }}_{\mathrm{int}los}=v_r\sin \Delta {\sigma }_{mn}\end{array}\right.$$

(24)

where $v_r$ and $\Delta {\sigma }_{mn}$ are calculated as Equation (25).

$$\left\{\begin{array}{l}{v}_r=\sqrt{{\left(v\cos {\sigma }_v-v_m\cos {\sigma }_m\right)}^2+{\left(v\sin {\sigma }_v-v_m\sin {\sigma }_m\right)}^2}\\ \Delta {\sigma }_{mn}=\mathrm{atan}\frac{v\sin {\sigma }_v-v_m\sin {\sigma }_m}{v\cos {\sigma }_v-v_m\cos {\sigma }_m}\end{array}\right.$$

(25)

To facilitate the analysis and prediction, ${\dot{\sigma }}_{los}$ is calculated as follows, taking the derivation of the second formula in Equation (22):

$${\dot{L}}_{Rgo}{\dot{\sigma }}_{LOS}+L_{Rgo}{\ddot{\sigma }}_{LOS}=\dot{v}\sin \Delta \sigma +v\Delta \dot{\sigma }\cos \Delta \sigma$$

(26)

Bring the heading error and first formula in Equation (22) into Equation (26), the rate of LOS angular rate is calculated by Equation (27).

$$\begin{array}{l}{\dot{L}}_{Rgo}{\dot{\sigma }}_{los}+L_{Rgo}{\ddot{\sigma }}_{los}=\dot{v}\sin \Delta \sigma +v\Delta \dot{\sigma }\cos \Delta \sigma \Rightarrow \\ {\dot{L}}_{Rgo}{\dot{\sigma }}_{los}+L_{Rgo}{\ddot{\sigma }}_{los}=\frac{\dot{v}}{v}{L}_{Rgo}{\dot{\sigma }}_{los}-{\dot{L}}_{Rgo}{\dot{\sigma }}_{los}+{\dot{L}}_{Rgo}\dot{\sigma }\Rightarrow \\ {\ddot{\sigma }}_{los}=\left(\frac{\dot{v}}{v}-\frac{2{\dot{L}}_{Rgo}}{L_{Rgo}}\right){\dot{\sigma }}_{los}+\frac{{\dot{L}}_{Rgo}}{L_{Rgo}}\dot{\sigma }\end{array}$$

(27)

Defining $T_{goc}$ as the predicted remaining time of flight, $T_{goc}$ is derived via the remaining flight range and variation in range, expressed in Equation (28).

$$T_{goc}=-\frac{L_{Rgo}}{{\dot{L}}_{Rgo}}$$

(28)

Defining the value of state $x={\dot{\sigma }}_{los}$ and the value of control $u=\dot{\sigma }$, the differential of LOS angular rate is obtained, which is shown in Equation (29).

$$\dot{x}=\left(\frac{\dot{v}}{v}+\frac{2}{T_{goc}}\right)x-\frac{1}{T_{goc}}u$$

(29)

In the latter phase of glide flight, $\frac{\dot{v}}{v}$ is an order of magnitude smaller than $\frac{2}{T_{goc}}$. Equation (29) is further simplified to Equation (30).

$$\dot{x}=\frac{2}{T_{goc}}x-\frac{1}{T_{goc}}u$$

(30)

The current remaining flight time T_goc, as shown in Equation (31). a certain time t of future flight starting from the current time t, and the remaining flight time T_go1 at time t satisfy the following relationship:

$$T_{goc}=T_{go1}+t$$

(31)

$d{T}_{go1}=-dt$ represents the derivation of remaining flight time. For a given control input u, the definite integral of Equation (30) is solved, and the calculated result is shown in Equation (32).

$$\begin{array}{cc}\hfill x\left(t\right)& ={\mathrm{e}}^{{\displaystyle \int \frac{2}{T_{go1}}\mathrm{d}t}}\left({\displaystyle \int -\frac{u}{T_{go1}}{\mathrm{e}}^{-{\displaystyle \int \frac{2}{T_{go1}}\mathrm{d}t}}\mathrm{d}t}+C\right)\hfill \\ & ={\mathrm{e}}^{{\displaystyle \int \frac{2}{T_{goc}-t}\mathrm{d}t}}\left({\displaystyle \int -\frac{u}{T_{goc}-t}{\mathrm{e}}^{-{\displaystyle \int \frac{2}{T_{goc}-t}\mathrm{d}t}}\mathrm{d}t}+C\right)\hfill \\ & ={\mathrm{e}}^{-2\ln \left(T_{goc}-t\right)}\left({\displaystyle \int \frac{u}{t-T_{goc}}{\mathrm{e}}^{2\ln \left(t-T_{goc}\right)}\mathrm{d}t}+C\right)\hfill \\ & =\frac{1}{{\left(T_{goc}-t\right)}^2}\left({\displaystyle \int u\left(t-T_{goc}\right)\mathrm{d}t}+C\right)\hfill \\ & =\frac{1}{{\left(T_{goc}-t\right)}^2}\left(u\left(\frac{1}{2}{t}^2-T_{goc}t\right)+C\right)\hfill \end{array}$$

(32)

The LOS angular rate is ${\dot{\sigma }}_{los}$, at the current time t = 0, the constant C is expressed by Equation (33).

$$C={\dot{\sigma }}_{los}{T}_{goc}^2$$

(33)

${\dot{\sigma }}_{los}$ is obtained by Equation (34).

$${\dot{\sigma }}_{los}\left(t\right)=\frac{1}{{\left(T_{goc}-t\right)}^2}\left(u\left(\frac{1}{2}{t}^2-T_{goc}t\right)+{\dot{\sigma }}_{los}{T}_{goc}^2\right)$$

(34)

${\dot{\sigma }}_{los}$ at the terminal time t_f is shown in Equation (35).

$${\dot{\sigma }}_{los}={\dot{\sigma }}_{los}\left(t_f\right)=\frac{1}{{\left(T_{goc}-t_f\right)}^2}\left(u\left(\frac{1}{2}{t}_f^2-T_{goc}{t}_f\right)+{\dot{\sigma }}_{los}{T}_{goc}^2\right)$$

(35)

Similarly, based on above analysis, ${\dot{\sigma }}_{\mathrm{int}los}$ is calculated via the analysis and prediction. The solution is shown in Equation (36).

$${\dot{\sigma }}_{\mathrm{int}los}={\dot{\sigma }}_{\mathrm{int}los}\left(t_{\mathrm{int}f}\right)=\frac{1}{{\left(T_{\mathrm{int}goc}-t_{\mathrm{int}f}\right)}^2}\left(u\left(\frac{1}{2}{t}_{\mathrm{int}f}^2-T_{\mathrm{int}goc}{t}_{\mathrm{int}f}\right)+{\dot{\sigma }}_{\mathrm{int}los}{T}_{\mathrm{int}goc}^2\right)$$

(36)

where $T_{\mathrm{int}goc}$ represents the total encounter time based on the current moment, $t_{\mathrm{int}f}$ represents encounter time with interceptor, and $u$ is the input overload.

4.2.2. The Solution of LOS Angular Rate in the Longitudinal Direction

The attack–defense confrontation model in longitudinal direction is shown in Figure 4.

R_Rgo and R_Mgo represent the remaining range among the UAV, target and interceptor, which are calculated in Equations (37) and (38).

The longitudinal relative motion model between the UAV and the target is shown in Equation (22).

$$\left\{\begin{array}{l}{\dot{R}}_{Rgo}=-v\cos \Delta \theta \\ R_{Rgo}{\dot{\theta }}_{los}=v\sin \Delta \theta \end{array}\right.$$

(37)

The longitudinal relative motion model between the UAV and the interceptor is shown in Equation (38).

$$\left\{\begin{array}{l}{\dot{R}}_{Mgo}=-v\cos \Delta \theta -v_m\cos \Delta {\theta }_m\\ {\dot{R}}_{Mgo}{\dot{\theta }}_{\mathrm{int}los}=v\sin \Delta \theta -v_m\sin \Delta {\theta }_m\end{array}\right.$$

(38)

In order to simplify the calculation, the relative motion equation Equation (38) is expressed as Equation (39).

$$\left\{\begin{array}{l}{\dot{R}}_{Mgo}=-v_r\cos \Delta {\theta }_{mn}\\ R_{Mgo}{\dot{\theta }}_{\mathrm{int}los}=v_r\sin \Delta {\theta }_{mn}\end{array}\right.$$

(39)

where $v_r$ and $\Delta {\theta }_{mn}$ are calculated in Equation (40).

$$\left\{\begin{array}{l}{v}_r=\sqrt{{\left(v\cos {\theta }_v-v_m\cos {\theta }_m\right)}^2+{\left(v\sin {\theta }_v-v_m\sin {\theta }_m\right)}^2}\\ \Delta {\theta }_{mn}=\mathrm{atan}\frac{v\sin {\theta }_v-v_m\sin {\theta }_m}{v\cos {\theta }_v-v_m\cos {\theta }_m}\end{array}\right.$$

(40)

To facilitate the analysis and prediction, ${\dot{\theta }}_{los}$ is calculated as follows. Taking the derivation of the second formula in Equation (37) results in Equation (41):

$${\dot{R}}_{Rgo}{\dot{\theta }}_{LOS}+R_{Rgo}{\ddot{\theta }}_{LOS}=\dot{v}\sin \Delta \theta +v\Delta \dot{\theta }\cos \Delta \theta$$

(41)

Bring the heading error and first formula in Equation (37) results in Equation (41), the rate of the LOS angular rate is calculated using Equation (42).

$$\begin{array}{l}{\dot{R}}_{Rgo}{\dot{\theta }}_{los}+R_{Rgo}{\ddot{\theta }}_{los}=\dot{v}\sin \Delta \theta +v\Delta \dot{\theta }\cos \Delta \theta \Rightarrow \\ {\dot{R}}_{Rgo}{\dot{\theta }}_{los}+R_{Rgo}{\ddot{\theta }}_{los}=\frac{\dot{v}}{v}{R}_{Rgo}{\dot{\theta }}_{los}-{\dot{R}}_{Rgo}{\dot{\theta }}_{los}+{\dot{R}}_{Rgo}\dot{\theta }\Rightarrow \\ {\ddot{\theta }}_{los}=\left(\frac{\dot{v}}{v}-\frac{2{\dot{R}}_{Rgo}}{R_{Rgo}}\right){\dot{\theta }}_{los}+\frac{{\dot{R}}_{Rgo}}{R_{Rgo}}\dot{\theta }\end{array}$$

(42)

Based on the predicted remaining time of flight $T_{goc}$ in lateral prediction, the LOS angular rate is ${\dot{\sigma }}_{los}$, and at the current time t = 0, the constant C is expressed by Equation (43).

$$C={\dot{\theta }}_{los}{T}_{goc}^2$$

(43)

${\dot{\theta }}_{los}$ is obtained using Equation (44).

$${\dot{\theta }}_{los}\left(t\right)=\frac{1}{{\left(T_{goc}-t\right)}^2}\left(u\left(\frac{1}{2}{t}^2-T_{goc}t\right)+{\dot{\theta }}_{los}{T}_{goc}^2\right)$$

(44)

${\dot{\theta }}_{los}$ at the terminal time t_f is shown in Equation (45).

$${\dot{\theta }}_{los}={\dot{\theta }}_{los}\left(t_f\right)=\frac{1}{{\left(T_{goc}-t_f\right)}^2}\left(u\left(\frac{1}{2}{t}_f^2-T_{goc}{t}_f\right)+{\dot{\theta }}_{los}{T}_{goc}^2\right)$$

(45)

Similarly, based on the above analysis, ${\dot{\theta }}_{\mathrm{int}los}$ is calculated via the analysis and prediction. The solution is shown in Equation (46).

$${\dot{\theta }}_{\mathrm{int}los}={\dot{\theta }}_{\mathrm{int}los}\left(t_{\mathrm{int}f}\right)=\frac{1}{{\left(T_{\mathrm{int}goc}-t_{\mathrm{int}f}\right)}^2}\left(u\left(\frac{1}{2}{t}_{\mathrm{int}f}^2-T_{\mathrm{int}goc}{t}_{\mathrm{int}f}\right)+{\dot{\theta }}_{\mathrm{int}los}{T}_{\mathrm{int}goc}^2\right)$$

(46)

where $T_{\mathrm{int}goc}$ represents the total encounter time based on the current moment, $t_{\mathrm{int}f}$ represents encounter time with interceptor, and $u$ is the input overload.

According to the above analysis, $\left({\dot{\sigma }}_{los},{\dot{\theta }}_{los}\right)$ and $\left({\dot{\sigma }}_{\mathrm{int}los},{\dot{\theta }}_{\mathrm{int}los}\right)$ depend on the change in the LOS angle rate. For the guidance mission, $\left({\dot{\sigma }}_{los},{\dot{\theta }}_{los}\right)$ is related to the overload of the UAV, and the smaller the value, the closer the UAV approaches the target at the end of the gliding flight. For the penetration mission, $\left({\dot{\sigma }}_{\mathrm{int}los},{\dot{\theta }}_{\mathrm{int}los}\right)$ is related to the overloads of the UAV and the interceptor, and the greater the value, the higher the costs of the interceptor at the interception terminal phase, and the UAV will break through more easily if the control overload of the interceptor reaches saturation.

5. DRL Penetration Guidance Law

5.1. SAC Training Model

Standard DRL maximizes the sum of expected rewards ${\displaystyle {\sum }_t{\mathbb{E}}_{({\mathrm{s}}_{t,}{\mathrm{a}}_t)\sim {\rho }_{\pi }}\left[f_r\left({\mathrm{s}}_{t,}{\mathrm{a}}_t\right)\right]}$. For the problem of multi-dimensional continuous state inputs and a continuous action output, SAC networks are introduced to solve the MDP model.

Compared with other policy learning algorithms [28], SAC augments the standard RL objective with expected policy entropy using Equation (47).

$$J_{\pi }={\displaystyle {\sum }_t{\gamma }^t{\mathbb{E}}_{({\mathrm{s}}_t,{\mathrm{a}}_t)\sim {\rho }_{\pi }}\left[f_r\left({\mathrm{s}}_t,{\mathrm{a}}_t\right)+\tau \mathcal{H}\left(\pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)\right)\right]}$$

(47)

The entropy term $\tau \mathcal{H}\left(\pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)\right)$ is shown in Equation (48), which represents the stochastic feature of the strategy, balancing the exploration and learning of networks. The entropy parameter $\tau$ determines the relative importance of entropy against the immediate reward.

$$\begin{array}{cc}\hfill \mathcal{H}\left(\pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)\right)& =-{\displaystyle {\int }_{\mathrm{a}\in A}\pi }\left(\left.\mathrm{a}\right|{\mathrm{s}}_t\right)\log \pi \left(\left.\mathrm{a}\right|{\mathrm{s}}_t\right)d\mathrm{a}\hfill \\ & =\underset{\mathrm{a}\sim \pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)}{\mathbb{E}}\left[-\log \pi \left(\left.\mathrm{a}\right|{\mathrm{s}}_t\right)\right]\hfill \end{array}$$

(48)

The optimal strategy of SAC is shown in the Equation (49), aiming at maximizing the cumulative reward and policy entropy.

$${\pi }_{MaxEnt}^{*}=\underset{\pi }{\arg \max }{\displaystyle {\sum }_t{\gamma }^t{\mathbb{E}}_{({\mathrm{s}}_t,{\mathrm{a}}_t)\sim {\rho }_{\pi }}\left[f_r\left({\mathrm{s}}_t,{\mathrm{a}}_t\right)+\tau \mathcal{H}\left(\pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)\right)\right]}$$

(49)

The framework of SAC networks is shown in Figure 5, consisting of an Actor network and a Critic network. The Actor network generates the action, and the environment returns the reward and the next state. All of the ballistics data are stored in the experience pool, including the state, action, reward, and next state.

The Critic network is used to judge the found strategies, which impartially guides the strategy of the Actor network. At the beginning, the Actor network and Critic network are given random parameters. The Actor network struggles to generate the optimal strategy, and Critic network struggles to scientifically judge the strategy of the Actor network. The parameters of the networks need to be updated based on continuously generating data and sampling ballistics data.

For updating the Critic network, it outputs the expected reward $\underset{\mathrm{a}\sim \pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)}{\mathbb{E}}$ based on samples, and the Actor network outputs the action probability, which is depicted by the entropy term $\mathcal{H}\left(\pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)\right)$. Combining $\underset{\mathrm{a}\sim \pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)}{\mathbb{E}}$ with $\mathcal{H}\left(\pi \left(\left.\cdot \right|{\mathrm{s}}_t\right)\right)$, the value function is conducted and shown in Equation (50)

$$Q_{soft}(s_t,a_t)=\underset{(s_t,a_t)\sim {\rho }_{\pi }}{\mathbb{E}}\left[{\displaystyle \sum _{t=0}^{\infty }{\gamma }^{t}r(s_t,a_t)+\tau {\displaystyle \sum _{t=1}^{\infty }{\gamma }^t\mathcal{H}(\pi (\cdot |s_t))}}\right]$$

(50)

We further obtain the Bellman equation, as shown in Equation (51):

$$Q_{soft}(s_t,a_t)=\underset{\begin{array}{l}{s}_{t+1}\sim \rho (s_{t+1}|s_t,a_t)\\ a_{t+1}\sim \pi \end{array}}{\mathbb{E}}\left[r(s_t,a_t)+\gamma \left(Q_{soft}(s_{t+1},a_{t+1})+\tau H(\pi (\cdot |s_{t+1}))\right)\right]$$

(51)

Given by Equation (52), the loss function of the Critic network is acquired as follows:

$$J_Q(\psi )=\underset{\begin{array}{l}(s_t,a_t,s_{t+1})\sim D\\ a_{t+1}\sim \pi \end{array}}{\mathbb{E}}\left[\frac{1}{2}{\left(Q_{soft}(s_t,a_t)-\left(r(s_t,a_t)+\gamma \left(Q_{soft}(s_{t+1},a_{t+1})-\tau \log \left(\pi \left(a_{t+1}|s_{t+1}\right)\right)\right)\right)\right)}^2\right]$$

(52)

For updating the Actor network, the updating strategy is shown in Equation (53)

$${\pi }_{new}=\underset{\pi \in \prod }{\arg \min D_{KL}}\left(\pi (\cdot |s_t)‖\frac{\exp \left(\frac{1}{\tau }{Q}_{soft}^{{\pi }_{old}}(s_t,\cdot )\right)}{Z_{soft}^{{\pi }_{old}}(s_t)}\right)$$

(53)

where $\Pi$ represents the set of strategies, and $Z$ is the partition function, used to normalize the distribution. $D_{KL}$ is the Kullback–Leibler (KL) divergence [29].

Combining the re-parameterization technique with Equation (54), the loss function of the Actor network is obtained as follows:

$$J_{\pi }(\varphi )=\underset{s_t\sim D,{\epsilon }_t\sim N}{\mathbb{E}}[\tau \log \pi (f({\epsilon }_t;s_t)|s_t)-Q_{soft}(s_t,f({\epsilon }_t;s_t))]$$

(54)

in which $a_t=f({\epsilon }_t;s_t)$, and ${\epsilon }_t$ is the input noise, obeying the distribution $N$.

The method of stochastic gradient descent is introduced to minimize the loss function of networks. The optimal parameters of the Actor–Critic networks are obtained by repeating the updating process and passing the parameters to the target networks via soft updating.

5.2. Meta SAC Optimization Algorithm

The learning algorithm in DRL relies on a lot of interaction between the agent and the environment and high training costs. Once the environment changes, the original strategy is no longer applicable and needs to be learned from scratch. The penetration guidance problem under the stable flight environment can be solved using SAC networks. For a changeable flight environment, such as where the initial position of the interceptor changes greatly or the interceptor guidance law deviates greatly from the preset value, the strategy solved via traditional SAC struggles to adapt, which thus requires to restudy and redesign. This manuscript introduces meta-learning to optimize and improve SAC performance. The training goal of Meta SAC is to obtain initial SAC model parameters. When the UAV penetration mission is changed, through a few scenes of learning, the UAV can adapt to the new environment and complete the corresponding guidance penetration mission, without relearning model parameters. Meta SAC can achieve “learn while flying” for UAVs and strengthen the adaptability of UAVs.

The Meta SAC algorithm is shown in Algorithm 1, which is divided into a meta-training and meta-testing phase. The meta-training phase seeks to determine the optimal meta-learning parameters based on multi-experience missions. In the meta-testing phase, the trained meta parameters are interactively learned in the new mission environment to fine-tune the meta parameters.

Algorithm 1 Meta SAC

1: Initialize the experience pool $\Omega$, Storage space N 2: Meta training: 3: Inner loop 4: for iteration k do 5:     sample mission(k) from $\mathcal{T}\sim p\left(\mathcal{T}\right)$ 6:     update actor policy $\Theta$ to ${\Theta }^{\prime }$ using SAC based on mission(k): 7:     ${\Theta }^{\prime }\leftarrow SAC\left(\Theta ,mission\left(k\right)\right)$. 8: Outer loop 9: $\Theta =mmse\left({\displaystyle \sum _{i=1}^k{{\Theta }^{\prime }}_i}\right)$ 10: Generate ${\mathcal{D}}_1$ from $\Theta$ and estimate the reward of $\Theta$. 11: Add a hidden layer feature as a random noise. 12: ${{\Theta }^{\prime }}_i=\Theta +{\alpha }_{\Theta }{\nabla }_{\Theta }{E}_{{\alpha }_t\sim \pi \left({\alpha }_t\left|s_t;\Theta ,z_i\right.\right),z_i\sim \mathcal{N}\left({\mu }_i,{\sigma }_i\right)}\left[{\displaystyle \sum _t{R}_t\left(s_t\right)}\right]$ 13: The meta-learning process of different missions is carried out through SGD. 14: for iteration mission(k) do 15:     $\underset{\Theta }{\min }{\displaystyle \sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{{\Theta }^{\prime }}_i}\right)}={\displaystyle \sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\Theta -\alpha }\nabla \Theta {\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\Theta }\right)\right)}$ 16:     $\Theta =\Theta -\beta \nabla \Theta {\displaystyle \sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{{\Theta }^{\prime }}_i}\right)}$ 17: Meta testing 18: Initialize the experience pool $\Omega$, Storage space N. 19: Load meta training network parameters $\Theta$. 20: Set training parameters. 21: for iteration i do 22:   sample mission from $\mathcal{T}\sim p\left(\mathcal{T}\right)$ 23:   ${\Theta }^{\prime }=SAC\left(\Theta ,mission\left(k\right)\right)$ End for

The basic assumption of Meta SAC is that the experience mission for meta training and the new mission for meta testing obey the same mission distribution $p\left(\mathcal{T}\right)$. Therefore, there are some common characteristics between different missions. In the DRL scenario, our goal is to learn a function $f_{\theta }$ with parameter $\theta$, which can minimize the loss function ${\mathcal{L}}_{\mathcal{T}}$ of a specific mission $\mathcal{T}$. In the meta DRL scenario, our goal is to learn a learning process ${\theta }^{\prime }={\mu }_{\psi }\left({\mathcal{D}}_{\mathcal{T}}^{tr},\theta \right)$, which can quickly adapt to the new mission $\mathcal{T}$ with a very small dataset ${\mathcal{D}}_{\mathcal{T}}^{tr}$. Meta SAC can be summarized as optimizing the parameters $\theta$ and $\psi$ in the learning process, and the optimization equation is shown in Equation (55).

$$\underset{\theta ,\psi }{\min E_{\mathcal{T}\sim P\left(\mathcal{T}\right)}}\left[\mathcal{L}\left({\mathcal{D}}_{\mathcal{T}}^{test},{\theta }^{\prime }\right)\right] \mathrm{s}.\mathrm{t}. {\theta }^{\prime }={\mu }_{\psi }\left({\mathcal{D}}_{\mathcal{T}}^{tr},\theta \right)$$

(55)

where ${\mathcal{D}}_{\mathcal{T}}^{test}$ and ${\mathcal{D}}_{\mathcal{T}}^{tr}$, respectively, represent training and testing missions sampled from $p\left(\mathcal{T}\right)$, and $\mathcal{L}\left({\mathcal{D}}_{\mathcal{T}}^{test},{\theta }^{\prime }\right)$ represents the testing loss function. In the meta-training phase, parameters are optimized via the inner loop and outer loop.

In the inner loop, Meta SAC updates the model parameters with a small amount of randomly selected data for the specific mission $\mathcal{T}$ as the training data, reducing the loss of the model in mission $\mathcal{T}$. In this part, the updating of model parameters is the same as in the original SAC algorithm, and the agent learns several scenes from randomly selected missions.

The minimum mean square error of strategy parameters $\theta$ corresponding to different missions in the inner loop phase is solved to obtain the initial strategy parameters ${\theta }_{ini}$ of the outer loop. In this manuscript, a hidden layer feature is added to the input part of strategy ${\theta }_{ini}$ as a random noise. The random noise is sampled again in each episode, in order to provide a more continuous random exploration in time, which is helpful for agent to adjust their overall strategy exploration according to the current mission MDP. The goal of meta-learning is to enable the agent to learn how to quickly adapt to new missions by simultaneously updating a small amount of gradient of strategy parameters and hidden layer features. Therefore, the $\theta$ of ${\theta }^{\prime }={\mu }_{\psi }\left({\mathcal{D}}_{\mathcal{T}}^{tr},\theta \right)$ includes not only parameters of the neural network, but also the distribution parameters of hidden variables of each mission, namely the mean and variance of the Gaussian distribution, as shown in Equation (56).

$$\begin{array}{l}{\mu }_i^{\prime }={\mu }_i+{\alpha }_{\mu }{\nabla }_{{\mu }_i}{E}_{{\alpha }_t\sim \pi \left({\alpha }_t\left|s_t;\theta ,z_i\right.\right),z_i\sim \mathcal{N}\left({\mu }_i,{\sigma }_i\right)}\left[{\displaystyle \sum _t{R}_t\left(s_t\right)}\right]\\ {\sigma }_i^{\prime }={\sigma }_i+{\alpha }_{\sigma }{\nabla }_{{\sigma }_i}{E}_{{\alpha }_t\sim \pi \left({\alpha }_t\left|s_t;\theta ,z_i\right.\right),z_i\sim \mathcal{N}\left({\mu }_i,{\sigma }_i\right)}\left[{\displaystyle \sum _t{R}_t\left(s_t\right)}\right]\\ {\theta }_i^{\prime }=\theta +{\alpha }_{\theta }{\nabla }_{\theta }{E}_{{\alpha }_t\sim \pi \left({\alpha }_t\left|s_t;\theta ,z_i\right.\right),z_i\sim \mathcal{N}\left({\mu }_i,{\sigma }_i\right)}\left[{\displaystyle \sum _t{R}_t\left(s_t\right)}\right]\end{array}$$

(56)

The model is represented by a parameterized function $f_{\theta }$ with parameter $\theta$, and when it is transferred to a new mission $\mathcal{T}$, model parameter $\theta$ is updated to ${\theta }^{\prime }$ through gradient rise, as shown in Equation (57).

$${\theta }_i^{\prime }=\theta -\alpha {\nabla }_{\theta }{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\theta }\right)$$

(57)

We update step $\alpha$ is a fixed super parameter. Model parameter $\theta$ is updated to maximize the performance $f_{{\theta }_i^{\prime }}$ of different missions, as shown in Equation (58).

$$\underset{\theta }{\min }{\displaystyle \sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{\theta }_i^{\prime }}\right)}={\displaystyle \sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\theta -\alpha }\nabla \theta {\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{\theta }\right)\right)}$$

(58)

The meta-learning process of different missions is carried out through SGD, and the principle of $\theta$ is as follows:

$$\theta =\theta -\beta \nabla \theta {\displaystyle \sum _{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{\theta }_i^{\prime }}\right)}$$

(59)

where $\beta$ is the meta update step.

In the meta-testing phase, a small amount of experience in new missions is used to quickly learn strategies for solving new missions. A new mission may involve completing a new mission goal or achieving the same mission goal in a new environment. The updating process of the model in this phase is the same as the cycle part in the meta-training phase, and by calculating the loss function with the data collected in the new mission and adjusting the model through back propagation, the new mission is adapted by the agent.

6. Simulation Analysis

In this section, the manuscript analyzes and verifies the escape guidance strategy based on Meta SAC. SAC is used to solve a specific escape guidance mission. We conduct comprehensive experiments to verify whether the UAV can complete the guidance escape mission under satisfying terminal and process constraints. Once the UAV guidance escape mission changes, the original strategy based on SAC cannot be easily adapted to the changed mission and thus needs to be relearned and retrained. The manuscript proposes an optimization method via meta-learning that improves the learning ability of UAVs during the training process. This section focuses on verifying the validation of Meta SAC and demonstrating its performance in various new missions. In addition, the maneuvering overload commands under different pursuit evading distances are analyzed in order to explore the influence of different maneuvering timings and distances on the escape results. We use CAV-H to verify the escape guidance performance. The initial conditions, terminal altitude, and Meta SAC training parameters are given in

Table 1. Simulation and Meta SAC training conditions.

Scheme		Meta SAC Training Parameters
UAV initial velocity	4000 m/s	Learning episodes	1000
Initial velocity Inclination	0°	Guidance period	0.1 s
Initial velocity azimuth	0°	Data sampling interval	30 Km
Initial position	(3° E, 1° N)	Discount factor	$\gamma$ = 0.99
Initial altitude	45 Km	Soft update tau	0.001
Terminal altitude	40 Km	Learning rate	0.005
Target position	(0° E, 0° N)	Sampling size for each train	128
Interceptor Initial velocity	1500 m/s	Net layers	2
Initial velocity Inclination	Longitudinal LOS angle	Net nodes	256
Initial velocity azimuth	Lateral LOS angle	Capacity of experience pool	20,000

.

6.1. Validity Verification on SAC

In order to verify the effectiveness of SAC, three different pursuit evading scenarios are constructed, and the terminal reward value, miss distance, and terminal position deviation are analyzed. As shown in Figure 6a, the terminal reward value is poor in the initial phase of training, which demonstrates that the optimal strategy is not found. After 500 episodes, the terminal reward value increases gradually, indicating that a better strategy has been explored and converged. In the last 100 episodes, the optimal strategy is trained and learned, while the network parameters are adjusted to the optimal strategy. As can be seen from Figure 6b, the miss distance is relatively divergent in the first 150 episodes of training, indicating that the Action network in SAC constantly explores new strategies, and the Critic network also learns scientific evaluation criteria. After 500 training episodes, the network gradually learns and trains in the direction of optimal solution. The miss distance at the encounter moment converges to about 20 m. As shown in Figure 6c, the terminal position of the UAV has a large deviation in the early training phase, which is attributed to the exploration of the escape strategy by the network. In the later training phase, the position deviation is less affected by exploration. These pursuit evading scenarios tested in the manuscript can achieve convergence, and the final convergence values are all within 1 m.

In order to verify whether the SAC algorithm can solve the escape guidance strategy that meets the mission requirements in different pursuit and evasion scenarios, the pursuing and evading distance is changed, and the training results are shown in Figure 7. In the medium-range scenario, the miss distance converges to about 2 m, and the terminal deviation converges to about 1 m.

As shown in Figure 8, in long-range attack and defense scenarios, the miss distance converges to about 5 m, and the terminal deviation converges to about 1 m.

Based on the above simulation analysis, SAC is a feasible method to solve the UAV guidance escape strategy. After limited episodes of learning and training, network parameters are converged, which is used to test the flight mission.

6.2. Validity Verification on Meta SAC

When the mission of the UAV changes, the original SAC parameters cannot meet the requirements of the new mission, and thus the parameters need to be retrained and relearned. The SAC proposed in the manuscript is improved via meta-learning. Strong adaptive network parameters are found using learning and training, and when the pursuit evading environment changes, the network parameters are fine-tuned to adapt to the new environment immediately.

Meta SAC is divided into a meta-training phase and a meta-testing phase, and initialization parameters for the SAC network are trained in the meta-training phase, which is fine-tuned by interacting with the new environment in the meta-testing phase. By changing the initial interceptor position, three different pursuit evading scenarios are constructed, which represent a short, medium, and long distance.

The training results of Meta SAC and SAC are compared, and terminal reward values are presented in Figure 9a. Meta SAC is an effective method to speed up the training process, and after 100 episodes, a better strategy is learned by the network and converged gradually. In contrast, the SAC network needs 500 episodes to find the optimal solution. Miss distance is shown in Figure 9b. The better strategy is quickly learned by Meta SAC, which is more effective than the SAC method. Figure 9c shows the terminal deviation between the UAV and the target.

To explore the optimal solution as much as possible, some strategies with large terminal position deviations appear in the training process. As shown in Figure 10b,c, in medium-range attack and defense scenarios, the miss distance converges to about 8 m based on Meta SAC, and the terminal deviation converges to about 1 m.

As shown in Figure 11b,c, in long-range attack and defense scenarios, the miss distance converges to about 10 m based on Meta SAC, and the terminal deviation converges to about 1 m.

According to the theoretical analysis, in the training process, new missions corresponding to the same distribution are used to execute micro-testing using Meta SAC, resulting in more gradient-descending directions of the optimal solution being learned by the network. Combined with the theory analysis and training results, the manuscript demonstrates that meta-learning is a feasible method to accelerate convergence and improve the efficiency of training.

In the previous analysis, when the pursuit evading scenario is changed, network parameters obtained in the meta-training phase are fine-tuned through a few interactions. The manuscript verifies meta testing performance by changing the initial interceptor position, and results compared with the SAC method are shown in

Table 2. Results compared with the SAC method.

Interceptor Initial Position (Km)	Interaction Episodes		Miss Distance (m)		Terminal Deviate (m)
	SAC	Meta SAC	SAC	Meta SAC	SAC	Meta SAC
(0, 30, 0)	74	1	3.78	3.29	0.56	0.61
(2, 30, 6)	75	4	2.80	2.72	0.68	0.72
(4, 30, 12)	59	8	6.93	3.75	0.69	0.58
(6, 30, 18)	59	1	2.71	6.82	0.68	0.72
(8, 30, 24)	26	2	3.16	3.70	0.47	0.50
(10, 30, 30)	58	3	3.50	2.37	0.61	0.64
(12, 30, 36)	67	1	2.86	2.21	0.68	0.45
(14, 30, 42)	56	8	2.18	2.89	0.55	0.61
(16, 30, 48)	69	1	2.73	2.23	0.61	0.72
(18, 30, 54)	106	1	2.45	3.71	0.56	0.63
(20, 30, 60)	94	1	2.7	2.35	0.49	0.54
(22, 30, 66)	59	1	2.23	2.51	0.73	0.71
(24, 30, 72)	62	1	2.11	3.47	0.48	0.67
(26, 30, 78)	63	1	2.04	4.5	0.48	0.57
(28, 30, 84)	63	4	2.64	5.12	0.47	0.40
(30, 30, 90)	63	9	2.95	6.05	0.68	0.47

. Based on the network parameters of the meta-training phase, the strategic solutions for the escape guidance missions are found through training within 10 episodes. On the contrary, network parameters based on SAC need more interaction to find solutions, and the episode of interactions comprises more than 50 episodes. According to the above simulation, the adaptability of Meta SAC is much greater than SAC, and once the escape mission changes, through very few episodes of learning, the new mission is completed by the UAV without re-learning and designing the strategy. The method provides the possibility of realizing UAV learning while flying.

6.3. Strategy Analysis Based on Meta SAC

This section tests the network parameters based on Meta SAC, and analyzes the escape strategy and flight state under different pursuit evading distances. As shown in Figure 12a, for the pursuit evading scene over a short distance, the longitudinal maneuvering overload is larger in the first half of the phase of escape, resulting in the velocity slope angle decreasing gradually. In the second half of the phase of escape, if the strategy is executed under the original maneuvering overload, the terminal altitude constraint cannot be satisfied, and therefore, the overload gradually decreases, and the velocity slope angle is slowly reduced. As shown in Figure 12b, at the beginning of escape, the lateral maneuvering overload is positive, and the velocity azimuth angle constantly increases. With the distance between the UAV and the interceptor reducing, the overload increases gradually in the opposite direction, and the velocity azimuth angle decreases. On the one hand, this can confuse the strategy of the interceptor, while on the other hand, the guidance course is corrected.

As shown in Figure 13a, compared with the pursuit evading scene of a short distance, the medium-distance escape process takes longer, the pursuing time left to the interceptor is longer, and the UAV flies in the direction of increasing the velocity slope angle. The timing of the maximum escape overload corresponding to the medium distance is also different. As shown in Figure 13b, in the first half of the phase of escape, lateral maneuvering overload corresponding to a medium distance is larger than that in a short distance, and in the second half of the phase of escape, the corresponding reverse maneuvering overload is smaller, resulting in the UAV being able to use the longer escape time to slowly correct the course.

As shown in Figure 14, under a long pursuit distance, the overload change of the UAV maneuver is similar to that for a medium range, and the escape timing is basically the same as the escape strategy.

According to the above analysis, the escape guidance strategy via Meta SAC can be used as a tactical escape strategy, and the timing of escape and the maneuvering overload are adjusted in a timely manner under different pursuit evading distances. On the one hand, the overload corresponding to this strategy can confuse the interceptor and cause some interference, and on the other hand, it can take into account the guidance mission, correcting the course deviation caused by escape.

Figure 15a shows the flight trajectory of the interceptor against UAV at the North East Down (NED) coordinate (10 km, 30 km, 30 km), the trajectory point at the encountering moment is shown in Figure 15b, and the miss distance is 19 m in this pursuit evading scene. To verify the scientific and applicability of Meta SAC, the initial position of interceptor is changed. Flight trajectories are shown in Figure 15c,e, and trajectory points at the encountering moment are shown in Figure 15d,f. The miss distances in these two pursuit evading scenarios are 3 m and 6 m, respectively. Based on the CAV-H structure, the miss distance between the UAV and the interceptor is greater than 2 m at the encountering moment, which means that the escape mission is achieved.

Based on the principles of Meta SAC and optimal guidance, flight states are shown in Figure 16. Longitude, latitude, and altitude during the flight of the UAV are shown in Figure 16a,b, under different pursuit evading scenarios, and the terminal position and altitude constraints are met. There is larger amplitude modification in the velocity slope and azimuth angle, which is attributed to the escape strategy via lateral and longitudinal maneuvering, as shown in Figure 16c,d. The total change in velocity slope and azimuth angle is within two degrees, which meets the flight process constraints. Through the analysis of flight states, this escape strategy is an effective measure for guidance and escape with high accuracy.

Flight process deviation mainly includes aerodynamic calculation deviation and output overload deviation. For the aerodynamic deviation, this manuscript uses the interpolation method to calculate it based on the flight Mach number and angle of attack, which may have some deviation. Therefore, when calculating the aerodynamic coefficient, random noise with an amplitude of 0.1 is added to verify whether the UAV can complete the guidance mission. As shown in Figure 17a, aerodynamic deviation noise causes certain disturbances to the angle of attack during the flight. At the 10th second and end of the flight, the maximum deviation of the angle of attack is 2°. However, overall, the impact of aerodynamic deviation on the entire flight is relatively small, and the change in angle of attack is still within the safe range of the UAV. As shown in Figure 17b, due to the constraints of UAV game confrontation and guidance missions, the bank angle during the entire flight process changes significantly, and aerodynamic deviation noise has a small impact on the bank angle. After increasing the aerodynamic deviation noise, the miss distance between the UAV and the interceptor at the time of encounter is 8.908 m, and the terminal position deviation is 0.52 m. Therefore, under the influence of aerodynamic deviation, the UAV can still complete the escape guidance mission.

For the output overload deviation, the total overload is composed of the guiding overload derived from the optimal guidance law and the maneuvering overload output derived from the neural network. Random maneuvering overload with an amplitude of 0.1 is added to verify whether the UAV can complete the maneuver guidance mission. As shown in Figure 18, random overloads are added in the longitudinal and lateral directions, respectively. Through simulation testing, the miss distance between the UAV and the interceptor at the encounter point is 10.51 m, and the terminal deviation of the UAV is 0.6 m. Under this deviation, the UAV can still achieve high-precision guidance and efficient penetration.

7. Conclusions

This manuscript proposes an escape guidance strategy satisfying multiple terminal constraints via SAC. The action space is designed under the UAV process constraint. Considering the fact that real-time interceptor information is hard to obtain in the actual escape process, the state space considering the target and the heading angle is designed. The reward function is an important index function that is used to guide and evaluate the training results. Based on the pursuit evading model, the terminal LOS angle rates in the lateral and longitudinal directions are derived to describe the deviation and miss distance. In order to improve the adaptability of the escape guidance strategy, the manuscript improves SAC via meta-learning and compares meta SAC with SAC. The strong adaptive escape strategy based on Meta SAC is analyzed. In view of the above theoretical numerical analysis, we have obtained the following conclusions:

(1)

The escape guidance strategy based on SAC is a feasible tactical escape strategy that can achieve high-precision guidance while meeting the escape requirements.

(2)

Meta SAC can significantly improve the adaptability of the escape strategy. When the escape mission changes, it can fine-tune network parameters to adapt to the mission through a small number of training epochs. This method provides the possibility for the UAV to learn while flying.

(3)

The strong adaptive escape strategy based on Meta SAC can adjust the escape timing and maneuver overload in real time according to the pursuit evading distance. On the one hand, the overload corresponding to this strategy can confuse the interceptor and cause some interference, while on the other hand, it can take into account the guidance mission, correcting the course deviation caused by escape.