Deep Reinforcement Learning-Based Impact Angle-Constrained Adaptive Guidance Law

Abstract

This study presents an advanced second-order sliding-mode guidance law with a terminal impact angle constraint, which ingeniously combines reinforcement learning algorithms with the nonsingular terminal sliding-mode control (NTSM) theory. This hybrid approach effectively mitigates the inherent chattering issue commonly associated with sliding-mode control while maintaining high levels of control system precision. We introduce a parameter to the super-twisting algorithm and subsequently improve an intelligent parameter-adaptive algorithm grounded in the Twin-Delayed Deep Deterministic Policy Gradient (TD3) framework. During the guidance phase, a pre-trained reinforcement learning model is employed to directly map the missile’s state variables to the optimal adaptive parameters, thereby significantly enhancing the guidance performance. Additionally, a generalized super-twisting extended state observer (GSTESO) is introduced for estimating and compensating the lumped uncertainty within the missile guidance system. This method obviates the necessity for prior information about the target’s maneuvers, enabling the proposed guidance law to intercept maneuvering targets with unknown acceleration. The finite-time stability of the closed-loop guidance system is confirmed using the Lyapunov stability criterion. Simulations demonstrate that our proposed guidance law not only meets a wide range of impact angle constraints but also attains higher interception accuracy and faster convergence rate and better overall performance compared to traditional NTSM and the super-twisting NTSM (ST-NTSM) guidance laws, The interception accuracy is less than 0.1 m, and the impact angle error is less than 0.01°.

1. Introduction

The traditional proportional navigation guidance (PNG) law is simple in form and easy to realize and has been widely used in engineering [1,2]. However, with modern warfare development, targets have become more and more defensive and maneuverable, necessitating missiles that not only achieve precise target strikes but also meet a specified terminal impact angle [3,4]. This is crucial for scenarios such as intercepting missiles with minimal impact angles for direct collision or striking vulnerable sections of targets like ground tanks and aircraft wings with optimal angles.

Current research into guidance laws with impact angle constraints primarily focuses on methods such as proportional navigation guidance’s additional bias term, optimal control theory, and sliding-mode variable structure control theory [5,6,7] among others. The basic idea of the proportional guidance method with an additional bias term [8,9,10] is to introduce an additional bias parameter to design a guidance law that meets the constraints of the miss distance and impact angle. The guidance law designed by this method is simple and easy to implement, but it mainly attacks stationary or slow-moving targets and lacks efficacy against rapid and highly maneuverable targets [11]. The optimal control method is also the most widely studied method; the idea is to find the guidance law with terminal impact angle constraint by minimizing the performance index. In [12], a guidance law with impact angle constraint that can hit maneuvering targets with constant acceleration is proposed based on the linear quadratic optimization theory. Reference [13] proposes a modeling method with the desired line of sight as the control target and designs an optimal guidance law for ballistic forming on the basis of constraining the relative motion direction of the terminal projectile and ballistic overload. The above optimal control methods achieve satisfactory control accuracy, but all of them require an estimation of the time-to-go, which can be challenging to ascertain in real-world scenarios.

Sliding-mode variable structure control has the characteristics of high precision and strong robustness, so it is widely used in the design of guidance laws with constraints [14,15]. In [16,17,18], the authors use a terminal sliding-mode (TSM) control method to design the guidance law, which can guarantee that the impact angle converges to the expected impact angle and the line-of-sight (LOS) angle rate converges to zero in finite time, and they apply the finite-time control theory to solve the specific convergence time. However, the existing TSM and fast terminal sliding-mode (FTSM) controllers have a common shortcoming, that is, the singular problem occurs easily on the terminal sliding-mode surface when the error approaches zero. To solve this problem, NTSM was proposed. NTSM can not only achieve finite time convergence but also avoid singularity problems; references [19,20,21,22] verify the above theory. It should be pointed out that in [20,22], in order to eliminate chattering, a special function is used to replace the sign function, which introduces steady-state errors, and the guidance laws above are all for non-maneuvering targets. For maneuvering targets, the authors of [23] achieve impact angle convergence by selecting the missile’s lateral acceleration to enforce the terminal sliding mode on a switching surface designed using nonlinear engagement dynamics, but they do not take into account the singularity that the proposed design may exhibit in its implementation. A guidance law based on NTSM is proposed in [19], but the maneuvering upper limit of targets needs to be estimated, and there are chattering problems. In order to suppress high-frequency chattering, the boundary-layer method, reaching law method, and observer method are all effective solutions. The authors in [24] used continuous saturation function to approximate the sign function, which reduced the chattering, but this method led to a decrease in the robustness of the system as the boundary layer increased. To solve this problem, reference [25] designed a second-order sliding-mode guidance law with continuous characteristics based on the high-order sliding-mode algorithm. The super-twisting algorithm was adopted as the reaching law of the sliding-mode control to eliminate the discontinuous term in the guidance law and thus weaken the chattering. However, the traditional super-twisting algorithm has the shortcomings of a slow convergence speed when the system state is far from the equilibrium point and cannot make full use of the missile overload capacity [26]. Nowadays, with powerful nonlinear approximation and data representation methods, data-driven reinforcement learning(RL) methods have attracted considerable attention in the design of guidance law, and model-free RL has been implemented in several algorithms for finding optimal solutions, making complex decisions, and in self-learning systems which improve behavior based on previous experiences [27]. Traditional reinforcement learning methods are suited for discrete environments with small action and sample spaces. However, real-world tasks, such as guidance and control challenges, involve large state spaces and continuous action domains with complex, nonlinear dynamics, which conventional methods find difficult to manage. Deep reinforcement learning (DRL) addresses these limitations by integrating deep neural networks with reinforcement learning, thereby enabling a more robust approach to handling the complexities of continuous and nonlinear systems. In recent years, many authors [28,29,30,31,32] have used Q-learning, deep Q-leaning network (DQN), deep deterministic policy gradient (DDPG), twin-delayed deep deterministic policy gradient (TD3), and other RL algorithms to train neural network models. The parameters of the missile’s LOS rate, position vector, and so on are used as state variables in a Markov decision as inputs, then the trained neural network can directly map the guidance command, which provides a new idea for the design of guidance law under multi-constraint conditions. Gaudet et al. [33] improved the Proximal Policy Optimization (PPO) algorithm and proposed a three-dimensional guidance law design framework based on DRL for interceptors trained with the PPO algorithm. A comparative analysis showed that the deduced guidance law also had better performance and efficiency than the extended Zero-Effort-Miss (ZEM) policy. In a similar way, Gaudet et al. [34] developed a new adaptive guidance system using reinforcement meta-learning with a recursive strategy and a value function approximator to complete the safe landing of an agent on an asteroid with unknown environmental dynamics. This is also a good method to solve the continuous control problem, but the PPO algorithm uses a random policy to explore and exploit rewards, and this method to estimate the accurate gradient requires a large number of random actions to find the accurate gradient. Therefore, such random operation reduces the algorithm’s convergence speed. In [35], an adaptive neuro-fuzzy inference sliding-mode control (ANFSMC) guidance law with impact angle constraint is proposed by combining the sliding-mode control method with an adaptive neuro-fuzzy inference system (ANFIS), and the ANFIS is introduced to adaptively update the additional control command and reduce the high-frequency chatter of the sliding-mode control (SMC), which enhances the robustness and reduces the chattering of the system; numerical simulations also verify the effectiveness of the proposed guidance law. In [36], the author proposes a three-dimensional (3D) intelligent impact-time control guidance (ITCG) law based on nonlinear relative motion relationship with the field of view (FOV) strictly constrained. By feeding back the FOV error, the modified time bias term including a guidance gain which can accelerate the convergence rate is incorporated into 3D-PNG; then, the guidance gain is obtained by DDPG in the RL framework, which makes the proposed guidance law achieve a smaller time error and have less energy loss. But in [37], the authors pointed out that the DDPG had the problem of an overestimation of the Q value, which may not give the optimal solution. Moreover, the DDPG uses a replay buffer to remove the correlations existing in the input experience (i.e., the experience replay mechanism) [38,39] and uses the target network method to stabilize the training process. As an essential part of the DDPG, experience replay significantly affects the performance and speed of the learning process by choosing the experience used to train the neural network.

Based on the above problems, this paper designs a novel adaptive impact angle-constrained guidance law based on DRL and the super-twisting algorithm. It introduces a parameter $\gamma$ to replace the fractional power equal to one half in the conventional super-twisting algorithm, which improves the guidance performance and the convergence rate of the system state by adaptively adjusting this parameter. Drawing on the insights from [37,40], we apply the preferential sampling or prioritized experience replay mechanism [41] to enhance the TD3 algorithm and then obtain the adaptively optimal parameters $\gamma$ using our improved DRL algorithm. Compared to the sliding-mode guidance law based on the traditional super-twisting algorithm, our algorithm is more effective in reducing chattering, has better guidance accuracy, and achieves better constraint effects on the terminal impact angle. Moreover, the trained neural network exhibits strong generalization capabilities in unfamiliar scenarios. Finally, we design a GSTESO, different from the modified ESO in [42] that only uses nonlinear terms; the proposed GSTESO can provide faster convergence rate when the estimation error is far away from the equilibrium, and it does not depend on an upper bound of uncertainty. Furthermore, in comparison to the modified ESOs incorporating both linear and nonlinear terms discussed in [43], the GSTESO we propose does not introduce any additional parameters that require adjustment.

2. Preliminaries and Problem Setup

2.1. Relative Kinematics and Guidance Problem

The missile–target relative motion relationship is established in the inertial coordinate system, as shown in Figure 1, where $Oxy$ is the ground inertial coordinate system, r is the missile–target relative distance, q is the missile–target LOS angle, M and T are the interceptor and the target, $\sigma$ and ${\sigma }_T$ are the flight path angle of the missile and the target, respectively, V and $V_T$ represent the velocity of the missile and the target, respectively, $a_M$ and $a_T$ represent the normal acceleration of the missile and the target, respectively, $\eta$ and ${\eta }_T$ are the parameters of leading angles of the missile and the target. It is stipulated that all angles in Figure 1 are positive counterclockwise.

According to the missile–target relative motion relationship in Figure 1, the dynamic model of the missile terminal guidance phase is obtained as

$$\dot{r}=-V_{T}cos({\sigma }_T-q)-Vcos(\sigma -q)$$

(1)

$$\dot{q}=[V_{T}sin({\sigma }_T-q)-Vsin(\sigma -q)]/r$$

(2)

$$\dot{\sigma }={\displaystyle \frac{a_M}{V}}$$

(3)

$${\dot{\sigma }}_T={\displaystyle \frac{a_T}{V_T}}$$

(4)

where $\dot{r}$ is the relative speed between the missile and the target; $\dot{q}$ is the LOS rate. According to the constant-bearing course, a guidance law is designed to make the LOS angle rate converge in a finite time to ensure that the missile can accurately attack the target.

The impact angle is defined as the angle between the missile velocity vector and the target velocity vector at the interception of the guidance phase during the missile’s attack on the target. The moment when the missile intercepts the target is denoted as $t_f$, and the desired impact angle of the missile in the guidance process is denoted as ${\sigma }_f$.

Lemma 1 

([44]). For a pre-designated target with an expected impact angle, there always exists a unique desired terminal LOS angle. Hence, the control of the intercept angle can be transformed into the control of the final LOS angle.

Assuming that the desired LOS angle is $q_f$, the objective of this paper translates into designing a continuous guidance law that achieves $q\to q_f$, $\dot{q}\to 0$ in finite time.

2.2. Kinematic Equations for LOS Angle Tracking Error

We take the first derivative of Equation (1) and Equation (2), respectively, to obtain

$$\ddot{r}=r{\dot{q}}^2+a_{Tr}-a_r$$

(5)

$$\ddot{q}={\displaystyle \frac{-2\dot{r}\dot{q}}{r}}-{\displaystyle \frac{a_q}{r}}+{\displaystyle \frac{a_{Tq}}{r}}$$

(6)

where $a_r$, $a_{Tr}$ is the acceleration of the missile and the target along the LOS angle, respectively, $a_r=asin(q-\sigma )$, $a_{Tr}=a_{T}sin(q-{\sigma }_T)$; $a_q$, $a_{Tq}$ is the acceleration of the missile and the target normal to the LOS angle direction, respectively. For most tactical missiles controlled by aerodynamic forces, the axial acceleration along the velocity direction is often uncontrollable, so this paper only designs the guidance law based on Equation (6).

Let e be the LOS angle tracking error, and the LOS angle rate $\dot{e}=\dot{q}$. Then, the LOS angle second-order dynamic error equation can be obtained as:

$$\ddot{e}={\displaystyle \frac{-2\dot{r}\dot{q}}{r}}-{\displaystyle \frac{a_q}{r}}+{\displaystyle \frac{a_{Tq}}{r}}$$

(7)

As shown in Equation (7), when $q-\sigma =\pm {\displaystyle \frac{\pi }{2}} \mathrm{rad}$, $a_q=0 \mathrm{m}/{\mathrm{s}}^2$, the missile cannot be controlled by the controller at this point, so it is regarded as a singular point. In order to avoid this situation, $a_q$ is used as a virtual control input, and Equation (7) can be rewritten as follows:

$$\ddot{e}={\displaystyle \frac{-2\dot{r}\dot{q}}{r}}-{\displaystyle \frac{a}{r}}+{\displaystyle \frac{a-a_q+a_{Tq}}{r}}$$

(8)

2.3. Deep Reinforcement Learning

The deep reinforcement learning algorithm is an algorithm based on the actor–critic network structure, which can update policy and value function at the same time and is suitable for solving decision problems in the continuous task space. Using the DDPG [45] algorithm as an example, the actor network uses policy functions to generate action $a=\pi \left(s\right)$, which is executed by agents to interact with the environment. The critic network uses the state-action value function $Q(s,a)$ to evaluate the performance of the action and instruct the actor network to choose a better action for the next stage.

The actor network uses a policy gradient algorithm to update parameter $\varphi$:

$${\nabla }_{\varphi \pi }J=\mathrm{E}\left[{\nabla }_{a}Q(s,a){|}_{a=\pi \left(s\right)}{\nabla }_{\theta \pi }\pi \left(s\right)\right]$$

(9)

The actor network uses the estimated value $Q(s,a)$ obtained by the critic network to select actions, so the estimate should be as close to the target value as possible, which is defined by:

$$y_t=R_t+\tau Q^{\prime }(s_{t+1},{\pi }^{\prime }\left(s_{t+1}\right))$$

(10)

where $R_t$ represents the current reward, $s_{t+1}$ represents the next state, $Q^{\prime }(·)$ is the state-action estimate obtained by the critic target network, and ${\pi }^{\prime }\left(s_{t+1}\right)$ is the estimated action obtained by the actor target network. All target networks use the “soft update” method to update network parameters.

The training of the critic network is based on minimizing the following loss function $L\left(w\right)$ to update the new parameter w:

$$L\left(w\right)=\mathrm{E}\left[{(y_t-Q_w(s_t,a_t))}^2\right]$$

(11)

When the two networks obtain the optimal parameters, the whole algorithm can obtain the optimal strategy, which can be regarded as a mapping from state to action, so that the missile can select the most valuable action according to the current environment state.

3. Guidance Law Design with Impact Angle Constraint

3.1. Disturbance Differential Observer Design and Stability Analysis

The observer proposed in this paper is the GSTESO, whose role is to observe the disturbance in real time. Its input is the measurable system state parameters, and its output is the observed value of the disturbance, which is used for system compensation.

Consider the following system:

$$\dot{z}=u+d\left(t\right)$$

(12)

where z is the controlled variable, u is the control variable, and d is the disturbance.

Assuming that the upper limit of target acceleration $a_T$ is known [46], and due to the limitation of physical conditions, there is always an upper limit of ${\dot{a}}_T$. Therefore, assuming that ${\dot{a}}_T$ is known, it is easy to know that the disturbance term $a_{Tq}$ also has an upper limit, that is:

$$a_M-a_q+a_{Tq}<L$$

(13)

Let system disturbance $d\left(t\right)=a_M-a_q+a_{Tq}$; Equation (6) becomes

$$\ddot{q}=\ddot{e}={\displaystyle \frac{-2\dot{r}\dot{q}}{r}}-{\displaystyle \frac{a_M}{r}}+{\displaystyle \frac{d\left(t\right)}{r}}$$

(14)

Let $z=r\dot{q}$; Equation (6) can be written as

$$\dot{z}=-\dot{r}\dot{q}-a_M+d\left(t\right)$$

(15)

According to Equations (12) and (15), the proposed GSTESO can be constructed as follows:

$$\left\{\begin{array}{c}\dot{\widehat{z}}=\widehat{d}\left(t\right)-\dot{r}\dot{q}-a_M+k_3{\zeta }_1\left(e_1\right)\hfill \\ \dot{\widehat{d}}\left(t\right)=k_4{\zeta }_2\left(e_1\right)\hfill \end{array}\right.$$

(16)

where $e_1=\widehat{z}-z$, and the nonlinear function is designed as

$$\left\{\begin{array}{c}{\zeta }_1\left(e_1\right)={⌊e_1⌉}^{{\displaystyle \frac{1}{2}}}+e_1\hfill \\ {\zeta }_2\left(e_1\right)={\displaystyle \frac{3}{2}}{⌊e_1⌉}^{{\displaystyle \frac{1}{2}}}+{\displaystyle \frac{1}{2}}·sign\left(e_1\right)+e_1\hfill \end{array}\right.$$

(17)

where the symbol ${⌊·⌉}^x$ is defined as ${\left|·\right|}^x·\mathrm{sgn}(·)$, and the parameter of GSTESO can be selected as $k_3=2{\epsilon }_0$, $k_4={\epsilon }_0^2$.

Similarly, let $e_2=\widehat{d}\left(t\right)-d\left(t\right)$; the estimation error system can be written as:

$$\left\{\begin{array}{c}{\dot{e}}_1=k_3{\zeta }_1\left(e_1\right)+e_2\hfill \\ {\dot{e}}_2=k_4{\zeta }_2\left(e_1\right)-\dot{d}\left(t\right)\hfill \end{array}\right.$$

(18)

Next, we verify the stability of the GSTESO. Firstly, let us introduce a new state vector as follows

$${\eta }^{\mathrm{T}}=[\zeta \left(e_1\right),e_2]$$

(19)

The Lyapunov function is constructed as

$$V_1={\eta }^{\mathrm{T}}P\eta$$

(20)

where P is a symmetric and positive definite matrix, and the derivative of $\eta$ is

$$\begin{array}{c}\dot{\eta }=\omega \left(e_1\right)\left[\begin{array}{c}{k}_3\zeta \left(e_1\right)+e_2\\ k_4\zeta \left(e_1\right)-\dot{d}\left(t\right)/\omega \left(e_1\right)\end{array}\right]\hfill \\ =\omega \left(e_1\right)(A\eta -B\Lambda )\hfill \end{array}$$

(21)

where $\omega \left(e_1\right)=0.5|e_1{|}^{-0.5}+1$, $\Lambda =\dot{d}\left(t\right)/\zeta \left(e_1\right)$

$$A=\left[\begin{array}{cc}{k}_1& 1\\ k_2& 0\end{array}\right],B=\left[\begin{array}{c}0\\ 1\end{array}\right]$$

(22)

Inspired by [43], we set $\Delta (\zeta ,\Lambda )={\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]}^T\left[\begin{array}{cc}C& D\\ D& -\theta \end{array}\right]\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]$, where $\theta \ge 0$; then, it can be guaranteed that $\Delta (\zeta ,\Lambda )\ge 0$ by choosing proper matrices C and D.

Theorem 1 

([43]). Suppose there exists a positive definite symmetric matrix P and positive constant κ satisfying the following matrix inequality:

$$\left[\begin{array}{cc}{A}^{T}P+PA+\kappa P+C& PB+D^T\\ B^{T}P+D& -\theta \end{array}\right]\le 0$$

(23)

System (18) is finite-time stable.

$$\begin{array}{cc}\hfill {\dot{V}}_1& =\omega \left(e_1\right)\left[{\eta }^T(A^{T}P+PA)\eta +\Lambda B^{T}P\eta +{\eta }^{T}PB\Lambda \right]\hfill \\ \hfill & =\omega \left(e_1\right){\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]}^T\left[\begin{array}{cc}{A}^{T}Q+QA& QB\\ B^{T}Q& 0\end{array}\right]\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]\hfill \\ \hfill & \le \omega \left(e_1\right)\left\{{\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]}^T\left[\begin{array}{cc}{A}^{T}Q+QA& QB\\ B^{T}Q& 0\end{array}\right]\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]+\Delta (\eta ,\Lambda )\right\}\hfill \\ \hfill & \le \omega \left(e_1\right){\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]}^T\left[\begin{array}{cc}-\kappa P& 0\\ 0& 0\end{array}\right]\left[\begin{array}{c}\eta \\ \Lambda \end{array}\right]\hfill \\ \hfill & =-{\displaystyle \frac{\kappa }{2{\left|e_1\right|}^{1/2}}}V-\kappa V\hfill \end{array}$$

(24)

According to Equation (20), we can obtain the final result as follows:

$${\psi }_{min}\left\{P\right\}{∥\eta ∥}^2\le {\eta }^{T}P\eta \le {\psi }_{max}\left\{P\right\}{∥P∥}^2$$

(25)

where $\psi \left\{P\right\}$ is the eigenvalue of matrix P, and ${∥\eta ∥}^2=\left|e_1\right|+2{\left|e_1\right|}^{(3/2)}+e_1^2+e_2^2$.

Therefore, ${\left|e_1\right|}^{(1/2)}\le (V_1^{(1/2)})/({\psi }_{min}^{(1/2)}\left\{P\right\})$ holds, and Equation (24) can be rewritten as

$${\dot{V}}_1\le -{\displaystyle \frac{\kappa {\psi }_{min}^{1/2}\left\{P\right\}}{2}}{V}^{1/2}-\kappa V$$

(26)

According to the finite-time stability criterion mentioned in [47], the error system (18) achieves finite-time stability.

3.2. Guidance Law Design

The sliding-mode control can be divided into two phases: the first phase is the reaching period, during which the system state converges from the initial state to the sliding-mode surface under the action of the auxiliary control term. The second phase is the sliding phase, in which the system state slides along the sliding-mode surface to the equilibrium point under the action of the equivalent control term to achieve state convergence.

Terminal sliding-mode control uses a nonlinear function as a sliding surface, which can make the system state converge in finite time, but this method has a singularity problem. In order to avoid the singularity problem and obtain higher control accuracy and faster convergence speed [48], this paper uses the non-singular terminal sliding-mode surface to design the guidance law. The sliding surface is designed by:

$$c=e+{\displaystyle \frac{1}{\beta }}{\left|\dot{e}\right|}^{\alpha }\mathrm{sgn}\left(\dot{e}\right)$$

(27)

where $\alpha ,\beta$ are design parameters, $\beta >0,1<\alpha <2$.

The super-twisting algorithm is a second-order sliding-mode algorithm, which has been widely used in control problems because of its superior characteristics such as effectively reducing the chattering, strong robustness, and high-precision control.

The adaptive super-twisting algorithm is as follows:

$$\left\{\begin{array}{c}{\dot{y}}_1=-k_1{\left|y_1\right|}^{1-1/\gamma }\mathrm{sgn}\left(y_1\right)+y_2\hfill \\ {\dot{y}}_2=-k_2\mathrm{sgn}\left(y_1\right)\hfill \end{array}\right.$$

(28)

The algorithm converges in finite time. The proof can be found in reference [49,50].

According to Equations (27) and (28) and reference [51], the proposed RLST-NTSM is given by:

$$a_M=-2\dot{r}\dot{q}+{\displaystyle \frac{\beta r}{\alpha }}{\left|\dot{e}\right|}^{2-\alpha }\mathrm{sgn}\left(\dot{e}\right)+z_1+k_1{\left|c\right|}^{(1-1/\gamma )}\mathrm{sgn}\left(c\right)+k_2\int {\displaystyle \frac{\mathrm{sgn}\left(c\right)}{r}}dt$$

(29)

where $k_1,k_2$ are design parameters, $\gamma$ is the introduced adaptive parameter, and $z_1$ is the output of the GSTESO Equation (16).

Remark 1. 

While the traditional super-twisting method directly sets the parameter to 2, we introduced a free parameter γ in Equation (28). Various papers [26,52,53,54,55,56,57] have proposed different adaptive methods to improve the super-twisting sliding-mode guidance law, each with its own specific form. However, most of these methods involve an adaptive gain adjustment of the parameter $k_1$. The study [50] already showed that for the traditional super-twisting algorithm, as long as $k_1$ was sufficiently large, the algorithm could converge. As for the impact of parameter variation involved in the equation $1-1/\gamma$ on control effectiveness, there is currently no literature studying this aspect, with only a few references (e.g., [58]) mentioning choosing a value slightly greater than 0.5 for $1-1/\gamma$. This suggests that some researchers may have already found that directly selecting the value as $1-1/\gamma =0.5$ may not be appropriate. Therefore, this paper employed DRL algorithms to adaptively adjust the selection of this parameter through model-free learning to achieve adaptive tuning.

3.3. Stability Analysis

Before delving into the analysis, let us introduce Lemma 2 [59]:

Lemma 2 

([59]). Suppose $v\left(x\right)$, defined on $U\in {\mathrm{R}}^n$, $C^1$, is a smooth positive definite function, and $v\left(x\right)+{\beta }_1{v}^{\beta 2}\left(x\right)$ is a negative semi-definite function on $U\in {\mathrm{R}}^n$ for ${\beta }_2\in (0,1)$ and ${\beta }_1\in {\mathrm{R}}^{+}$; then, there exists an area $U_0\in {\mathrm{R}}_n$ such that any $v\left(x\right)$ which starts from $U_0$ can reach $v\left(x\right)=0$ in finite time $T_r$, and the settling time $T_r$ can be estimated by

$$T_r\le {\displaystyle \frac{v^{1-{\beta }_2}\left(x_0\right)}{{\beta }_1(1-{\beta }_2)}}$$

(30)

where $v\left(x_0\right)$ is the initial value of $v\left(x\right)$.

Taking the derivative of Equation (27) with respect to time, we obtain

$$\dot{c}=\dot{e}+{\displaystyle \frac{\alpha }{\beta }}{\left|\dot{e}\right|}^{\alpha -1}\ddot{e}=\dot{e}+{\displaystyle \frac{\alpha }{\beta }}{\left|\dot{e}\right|}^{\alpha -1}({\displaystyle \frac{-2\dot{r}\dot{q}}{r}}-{\displaystyle \frac{a}{r}}+{\displaystyle \frac{f}{r}})$$

(31)

Substituting Equation (27) into Equation (29), we obtain

$$\begin{array}{c}\dot{c}=\dot{e}+{\displaystyle \frac{\alpha }{\beta }}{\left|\dot{e}\right|}^{\alpha -1}({\displaystyle \frac{f}{r}}-{\displaystyle \frac{z_1}{r}}-{\displaystyle \frac{\beta }{\alpha }}{\left|\dot{e}\right|}^{2-\alpha }\mathrm{sgn}\left(\dot{e}\right)-{\displaystyle \frac{k_1}{r}}{\left|c\right|}^{1-1/\gamma }\mathrm{sgn}\left(c\right)-{\displaystyle \frac{k_2}{r}}\int {\displaystyle \frac{\mathrm{sgn}\left(c\right)}{r}}dt)\hfill \\ ={\displaystyle \frac{\alpha }{\beta r}}{\left|\dot{e}\right|}^{\alpha -1}(f-z_1-k_1{\left|c\right|}^{1-1/\gamma }\mathrm{sgn}\left(c\right)-k_2\int {\displaystyle \frac{\mathrm{sgn}\left(c\right)}{r}}dt)\hfill \end{array}$$

(32)

According to Section 3.1, the estimation error of the GSTESO in (16) is dynamically finite-time stable, when the observer observation error tends to 0, and Equation (32) can be simplified as follows

$$\dot{c}={\displaystyle \frac{\alpha }{\beta r}}{\left|\dot{e}\right|}^{\alpha -1}(-k_1{\left|c\right|}^{1-1/\gamma }\mathrm{sgn}\left(c\right)-k_2\int {\displaystyle \frac{\mathrm{sgn}\left(c\right)}{r}}dt)$$

(33)

Referring to reference [49], we introduce a new state vector as follows

$$\xi ={[{\left|x_1\right|}^{1-1/\gamma }\mathrm{sgn}\left(x_1\right),x_2]}^{\mathrm{T}}$$

(34)

where $x_1=c$, $x_2=-{\displaystyle \frac{\alpha k_2}{\beta }}\int {\displaystyle \frac{\mathrm{sgn}\left(c\right)}{r}}dt+{\displaystyle \frac{\alpha }{\beta }}(f-z_1)$.

Taking the derivative of Equation (34) with respect to time, we obtain

$$\dot{\xi }={\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r}}{\left|x_1\right|}^{-{\displaystyle \frac{1}{\gamma }}}A\xi$$

(35)

$$A=\left[\begin{array}{cc}-{\displaystyle \frac{\alpha }{\beta }}(1-{\displaystyle \frac{1}{\gamma }})& 1-{\displaystyle \frac{1}{\gamma }}\\ -{\displaystyle \frac{\alpha k_2}{\beta }}& 0\end{array}\right]$$

(36)

Since $k_1>0,k_2>0,\beta >0,1<\alpha <2,\gamma >1$, it is easy to prove that A is a Hurwitz matrix. Thus, there exists a unique matrix $P=P^{\mathrm{T}}$ and P is the solution of the following Lyapunov equation.

$$A^{\mathrm{T}}P+PA=-Q$$

(37)

where every $Q=Q^{\mathrm{T}}>0$.

For the stability analysis of Equation (35), we choose the following Lyapunov function:

$$V={\xi }^{\mathrm{T}}P\xi$$

(38)

Taking the derivative of Equation (38) and substituting Equations (34) and (35), we obtain

$$\begin{array}{cc}\hfill \dot{V}& ={\dot{\xi }}^{\mathrm{T}}P\xi +{\xi }^{\mathrm{T}}P\dot{\xi }\hfill \\ \hfill & ={\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r}}{\left|x_1\right|}^{-1/\gamma }{\xi }^{\mathrm{T}}(A^{\mathrm{T}}P+PA)\xi \hfill \\ \hfill & =-{\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r}}{\left|x_1\right|}^{-1/\gamma }{\xi }^{\mathrm{T}}Q\xi \hfill \end{array}$$

(39)

From the definition of $\xi$, we know that ${\left|x_1\right|}^{1-1/\gamma }\le {∥\xi ∥}_2$, where ${∥·∥}_2$ denotes the L2-norm of the vector; then, Equation (39) becomes

$$\begin{array}{cc}\hfill \dot{V}& \le {\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r}}{{∥\xi ∥}_2}^{-{\displaystyle \frac{1}{\gamma -1}}}{\xi }^{\mathrm{T}}Q\xi \hfill \\ \hfill & \le -{\lambda }_{min}\left(Q\right){\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r}}{{∥\xi ∥}_2}^{2-{\displaystyle \frac{1}{\gamma -1}}}\hfill \\ \hfill & \le -{\lambda }_{min}\left(Q\right){\lambda }_{max}^{-[1-{\displaystyle \frac{1}{2(\gamma -1)}}]}\left(P\right){\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r}}{V}^{1-{\displaystyle \frac{1}{2(\gamma -1)}}}\hfill \\ \hfill & \le -{\lambda }_{min}\left(Q\right){\lambda }_{max}^{-[1-{\displaystyle \frac{1}{2(\gamma -1)}}]}\left(P\right){\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r_{max}}}{V}^{1-{\displaystyle \frac{1}{2(\gamma -1)}}}\hfill \end{array}$$

(40)

where ${\gamma }_{min}(·)$ and ${\gamma }_{max}(·)$ are the minimum eigenvalue and maximum eigenvalue of matrix $(·)$, ${\lambda }_{min}\left(Q\right){\lambda }_{max}^{-[1-{\displaystyle \frac{1}{2(\gamma -1)}}]}\left(P\right){\displaystyle \frac{{\left|\dot{e}\right|}^{\alpha -1}}{r_{max}}}\ge 0$, $1-1/\left[2\right(\gamma -1\left)\right]\in (0,1)$. According to Lemma 3, when $\left|\dot{e}\right|\ne 0$, the equilibrium point of the system in Equation (35) is finite-time stable, and the convergence time $T_c$ is satisfied

$$T_c\le T_r+{\displaystyle \frac{V^{1-\frac{1}{2(\gamma -1)}}\left(T_r\right)}{{\lambda }_{min}\left(Q\right){\lambda }_{max}^{-[1-\frac{1}{2(\gamma -1)}]}\left(P\right)\frac{{\left|\dot{e}\right|}^{\alpha -1}}{r_{max}}\frac{1}{2(\gamma -1)}}}$$

(41)

where $V\left(T_r\right)$ denotes the value of V at time $t_r$. It follows from the definition of $\dot{\xi }$ that Equation (27) for the nonsingular terminal sliding-mode surface can also converge in finite time. Once the sliding-mode surface is reached, Equation (27) is obtained

$$e+{\displaystyle \frac{1}{\beta }}{\left|\dot{e}\right|}^{\alpha }\mathrm{sgn}\left(\dot{e}\right)=0$$

(42)

In summary, the system in Equation (34) is finite-time stable.

3.4. Algorithm Design for Reinforcement Learning

Reinforcement learning considers the paradigm of interaction between an agent and its environment, and its goal is to make the agent take the behavior that maximizes the benefit, so as to obtain the optimal policy. Most reinforcement learning algorithms are based on a Markov decision process (MPD). The MPD model is often used to establish the corresponding mathematical model for a decision problem with uncertain state transition probability and state space and action space and to solve the reinforcement learning problem by solving the mathematical model. If the simulation covers all the state spaces encountered by the guidance law in practice and there is sufficient sampling density, the resulting guidance law is the optimal one with respect to the modeled missile and environment dynamics.

To better solve the problem in this paper, we chose the LOS angle rate, LOS angle tracking error, as well as the sliding-mode value as the state space $S:\left[\dot{q},e,c\right]$, which can also cover the whole guidance process. Then, the neural network was used to map the state of each step out of the parameter $\gamma$ in Equation (28). Through Equation (40), we know $1-1/\left[2\right(\gamma -1\left)\right]\in (0,1)$, so $\gamma >1.5$ can be obtained; thus, we designed the action space as $A:\left[\gamma \right]\in (1.6,3.6)$. Considering the overload problem in real flight, we restricted the action space to 200 m/s².

The reward function acts as a feedback system to indicate from the environment whether the missile performance is good or bad. In order to further reduce the chattering problem and improve the guidance accuracy, the reward function designed in this work was divided into two parts.

We shaped the shaping reward function as follows:

$$R_{\mathrm{shaping}}=\left\{\begin{array}{cc}{\displaystyle \frac{0.0001}{\left|c\right|}},\hfill & \mathrm{if}\mathrm{dr}<0\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right.$$

(43)

$$R_{\mathrm{terminal}}=\left\{\begin{array}{cc}{\displaystyle \frac{100}{\left|r\right|}},\hfill & \mathrm{if}r<10\hfill \\ 0,\hfill & \mathrm{if}\mathrm{otherwise}\hfill \end{array}\right.$$

(44)

$$R=\omega R_{\mathrm{shaping}}+R_{\mathrm{terminal}}$$

(45)

where $\omega >1$ is the weight parameter, and the maximum values of $R_{\mathrm{shaping}}$ and $R_{\mathrm{terminal}}$ were set to 100 and 10,000, respectively.

In standard reinforcement learning, the aim is to maximize the long-term reward for the agent’s interaction with the environment. The goal of the agent is to learn a policy $\pi :S\to A$ that maximizes the long-term reward: $G_t={\displaystyle \sum _{i=t}^T}{\tau }^{i-t}R\left(s_i,a_i\right)$, where T is the termination time step, $R\left(s_i,a_i\right)$ is the reward the agent would receive for performing action $a_i$ in state $s_i$, and $\tau$ is the discount factor $\tau \in (0,1)$. The action value function is used to represent the expected long-term reward for performing action $a_i$ in state $s_i$:

$$Q(s_t,a_t)=\mathrm{E}[G_t|s=s_t,a=a_t]=E[\sum _{i=t}^T{\tau }^{i-t}R\left(s_i,a_i\right)]$$

(46)

Using the Bellman equation to compute the optimal behavioral value function, the above equation becomes

$$Q\ast (s_t,a_t)=\mathrm{E}[R(s_t,a_t)+\tau \underset{\pi }{max}Q\ast (s_{t+1},a_{t+1})]$$

(47)

The reinforcement learning algorithm used in this paper was based on the TD3 algorithm. Compared with the DDPG algorithm mentioned in Section II, this algorithm has three differences: (1) it uses two critic network modules to estimate the Q value, (2) it has delayed policy updates, and (3) it uses target policy smoothing.

The first is also known as Clipped Double-Q learning. TD3 has two critic networks and two critic target networks. Both target networks produce a value and use the smaller one as the new target.

$$y=R+\tau \underset{i=1,2}{min}{Q}_{ti}(s^{\prime },a\left(s^{\prime }\right))$$

(48)

where $Q_{t1}$, $Q_{t2}$ are the Q values generated by the two critic target networks, respectively, and $s^{\prime }$ represents the next state. When updating the policy, because the actor network module selects the maximum Q value in the current state, Equation (48) can effectively reduce the overestimation error. Then, we take the above equation and calculate the mean squared error (MSE) of the values generated by the two critic networks, and we can obtain the loss function of the critic network.

$$L\left({\theta }_{e=1,2}\right)={\mathrm{E}}_{(s,a,r,s^{\prime })\sim D}\left[{(y-Q_{e=1,2}(s,a))}^2\right]$$

(49)

where ${\theta }_{e=1,2}$ stands for the parameters of the two critic networks, D stands for the replay buffer, $Q_1(·)$ and $Q_2(·)$ are the Q values produced by the two critic models, respectively.

The next point is the target policy smoothing; if the D estimate produces an incorrect peak, the policy network is immediately affected, and then the performance of the whole algorithm is affected, so clipping noise was added to the target policy generation operation. Then, the action was clipped within the valid range, and Equation (48) was formulated as follows:

$$y=R+\tau \underset{i=1,2}{min}{Q}_{ti}(s^{\prime },\mathrm{clip}({\pi }_{\varphi }\left(s^{\prime }\right)+\mathrm{clip}(\epsilon ,-c,c),a_{\mathrm{low}},a_{\mathrm{high}}))$$

(50)

where ${\pi }_{\varphi }(·)$ is the policy generated by the actor network, $\mathrm{clip}(\epsilon ,-c,c)$ is the clipping noise, and $a_{\mathrm{low}}$ and $a_{\mathrm{high}}$ refer to the upper and lower bounds of the action space, respectively. Adding noise at the target value is equivalent to adding a regularization term to the critic. In the process of sampling multiple times, $a^{\prime }$ obtains slightly different values according to different sampling noise, and y evaluates the average of the value functions of these sampling points. Therefore, y reflects the overall situation of the value functions around $a^{\prime }$. This way, it is not affected by some extreme value functions, and the stability of critic learning is enhanced.

The last point is to delay policy update. DDPG and TD3 both focus on how to learn a good critic, because the more accurate the critic estimates, the better the policy obtained by actor training. Therefore, if the critic is poorly learned and its output has large errors, the actor is also affected by preorder errors in the learning process. TD3 proposes that the critic should be updated more frequently than the actor, that is, the policy should be updated after getting an accurate estimate of the value function, and in this paper, the value function was updated five times before the policy function was updated.

3.5. Prioritized Experience Replay

The core concept of prioritized experience replay is to replay more frequently experiences associated with very successful attempts or extremely poor performance. Therefore, the key issue lies in defining the criteria for determining experience value. In most reinforcement learning algorithms, the TD error is commonly used to update the estimation of the action value function $Q(s,a)$. The magnitude of the TD error acts as a correction for the estimation and implicitly reflects the agent’s ability to learn from experiences. A larger absolute TD error implies a more significant correction to the expected action value. Furthermore, experiences with high TD errors are more likely to possess a high value and be associated with highly successful attempts. Similarly, experiences with large negative TD errors represent situations where the agent performs poorly, and these conditions are learned by the agent. Prioritized experience replay helps gradually realize the actual consequences of incorrect behavior in corresponding states and prevents repeating those mistakes under similar circumstances, thereby enhancing overall performance, so these unfavorable experiences are also considered valuable. In this paper, we adopted the absolute TD error $\left|\delta \right|$ as an index for evaluating experience value. The TD error ${\delta }_j$ of experience j can be calculated as:

$${\delta }_j=y-Q_{e=1,2}(s,a)$$

(51)

where y is obtained from Equation (49), and $Q_{e=1,2}(·)$ is the Q value generated by the two critic models, respectively.

We defined the probability of sampling experience j as

$$P\left(j\right)={\displaystyle \frac{D_j^{\rho }}{{\sum }_k{D}_k^{\rho }}}$$

(52)

where $D_j={\displaystyle \frac{1}{\mathrm{rank}\left(j\right)}}>0$, $k>0$, $\mathrm{rank}\left(j\right)$ represents the priority of experience j in replay buffer, and the parameter $\rho$ is used to adjust the priority level. The definition of sampling probability can be regarded as a method to add a random factor when selecting experiences, because even experiences with a low TD error can still have the probability of replay, which ensures the diversity of sampling experiences. This diversity helps to prevent any overfitting of the neural network.

Because our priority experience replay changed the sampling method, the importance sampling weight was used to correct the error introduced by the priority replay, and the loss function of the network for gradient training was calculated to reduce the error rate of the model. The importance sampling weights were calculated as follows:

$${\omega }_j={(N·P\left(j\right))}^{-\zeta }$$

(53)

where $\zeta$ is the adjustment factor of ${\omega }_j$.

Based on the several algorithms introduced above, an adaptive integrated guidance algorithm with impact angle constraint based on deep reinforcement learning is presented in the following Figure 2.

The TD3 algorithm with prioritized experience replay (PERTD3) is shown in Algorithm 1.

Algorithm 1 Guidance law’s learning algorithm based on the PERTD3 algorithm

1: Initialize critic networks $Q_{\theta 1},Q_{\theta 2}$, and actor network ${\pi }_{\varphi }$ with random parameters ${\theta }_1,{\theta }_2,\varphi$ 2: Initialize target networks parameters ${\theta }_1^{\prime }\leftarrow {\theta }_1,{\theta }_2^{\prime }\leftarrow {\theta }_2,{\varphi }^{\prime }\leftarrow \varphi$ 3: Initialize replay buffer $\mathcal{D}$ 4: for t = 1 to T do 5:    $ENV.reset\left(\right)$, reset the environment initial value 6:    while True do 7:      Sample experience j with probability $P\left(j\right)$ 8:      $\tilde{a}\leftarrow {\pi }_{{\varphi }^{\prime }}\left(s\right)+\epsilon$, $\epsilon \sim \mathrm{clip}(\mathcal{N}(0,\tilde{\sigma }),-c,c)$ 9:      if $t>M$ then 10:         for $j=1,K$ do 11:           $\tilde{a}\leftarrow {\pi }_{{\varphi }^{\prime }}\left(s\right)+\epsilon$,$\epsilon \sim \mathrm{clip}(\mathcal{N}(0,\tilde{\sigma }),-c,c)$ 12:           Obtain Q values by the two critic target networks as Equation (48) 13:           $y=R+\tau \underset{i=1,2}{min}{Q}_i(s^{\prime },a\left(s^{\prime }\right))$ 14:           Calculate corresponding importance-sampling weight ${\omega }_j$ and the TD error 15:           ${\delta }_j$ as Equations (51) and (53) 16:           ${\omega }_j={(N·P\left(j\right))}^{-\zeta }$ 17:           ${\delta }_j=y_j-Q_{\theta }\left(s,a\right)$ 18:         end for 19:         Calculate the loss function $L\left({\theta }_{e=1,2}\right)$ as Equation (49) 20:         $L\left({\theta }_{e=1,2}\right)={\mathrm{E}}_{(s,a,r,s^{\prime })\sim D}\left[{(y-Q_{e=1,2}(s,a))}^2\right]$ 21:         Update the critic network using gradient descent as in reference [32] 22:         $\begin{array}{c}{\nabla }_{{\theta }_e}L\left({\theta }_e\right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{\delta }_i{\nabla }_{{\theta }_e}{Q}_{{\theta }_e}\left(s_i,a_i\right),{{\theta }_e}^{\prime }={\theta }_e+{\alpha }_{{\theta }_e}{\nabla }_{{\theta }_e}L\left({\theta }_e\right)\end{array}$ 23:         if t mod d then 24:           Update $\varphi$ by the deterministic policy gradient as Equation (9): 25:           ${\nabla }_{\varphi }J\left({\pi }_{\varphi }\right)={\left.N^{-1}\sum {\nabla }_a{Q}_{{\theta }_1}(s,a)\right|}_{a={\pi }_{\varphi }\left(s\right)}{\nabla }_{\varphi }{\pi }_{\varphi }\left(s\right)$ 26:           ${\varphi }^{\prime }=\varphi +{\alpha }_{\varphi }{\nabla }_{\varphi }J\left({\pi }_{\varphi }\right)$ 27:           Update target networks as in reference [32]: 28:           ${\theta }_e^{\prime }\leftarrow \tau {\theta }_e+(1-\tau ){\theta }_e^{\prime }$ 29:           ${\varphi }^{\prime }\leftarrow \tau \varphi +(1-\tau ){\varphi }^{\prime }$ 30:         end if 31:      end if 32:    end while 33: end for

4. Simulation and Analysis

In this section, we considered two maneuver modes, a sinusoidal maneuver and a constant maneuver, and then tested them from three aspects: flight trajectory, normal acceleration, and LOS angle rate. Finally, the performance of the three guidance laws was analyzed in terms of miss distance.

First, when the target was on a sinusoidal maneuver, the PERTD3 and TD3 algorithms were both trained 10,000 times using the parameters in

Table 1. Dynamic model’s initial simulation parameters.

Parameters	Value
Missile position (m)	(0, 0)
Missile speed (m/s)	500
Target speed (m/s)	300
Relative range (m)	15,000
LOS angle (°)	45
Missile flight path angle (°)	60
Target flight path angle (°)	180
Impact angle (°)	Random (0, 90)

and

Table 2. Hyperparameters for TD3 algorithm [32].

Parameters	Value
Number of hidden layers	2
BATCH_SIZE	32
Size of experience buffer	50,000
Learning rate for actor	${10}^{-5}$
Learning rate for critic	$2\times {10}^{-5}$
Policy noise	0.2
Noise bound	0.5
Soft replacement $\tau$	0.99
Discounting factor $\gamma$	0.95
Delay steps	5
Gradient optimizer	Adam

. Both actor network and critic network were implemented by a three-layer fully connected neural network, Critic1 and Critic2 used the same network structure; the network structure is shown in Figure 3. Except for the output layer of critic, the neurons in all other layers were activated by a ReLu function, i.e.,

$$\mathrm{ReLu}(x)=\left\{\begin{array}{c}x,\mathrm{if}x\ge 0\\ 0,\mathrm{if}x<0\end{array}\right.$$

(54)

The output layer of the critic network was activated by the tanh function, which is defined as

$$tanh\left(x\right)={\displaystyle \frac{e^x-e^{-x}}{e^x+e^{-x}}}$$

(55)

The policy and value functions were periodically updated during optimization after accumulating trajectory rollouts of the replay buffer size.

As shown in Figure 4, our improved deep reinforcement learning algorithm was better than two other existing deep reinforcement learning algorithms in our written dynamic environment. The guidance algorithm based on PERTD3 and TD3 basically converged when approaching 800 steps, but the TD3 algorithm with prioritized experience replay could achieve higher scores and had higher stability, while the DDPG algorithm did not converge at the end.

To verify the robustness of our algorithm, we took the following two maneuvers and the case of the required impact angle constraint, and case 1 was not used during training but was used to test the generalization of our reinforcement learning algorithms.

Case 1: The target performed a constant maneuver $a_T=20 \mathrm{m}/{\mathrm{s}}^2$, i.e., the impact angle constraint was $0^{\circ }<q_f<{90}^{\circ }$ for every ${30}^{\circ }$.

Case 2: The target performed a constant maneuver $a_T=50sin\left(0.5t\right)\mathrm{m}/{\mathrm{s}}^2$, i.e., the impact angle constraint was $0^{\circ }<q_f<{90}^{\circ }$ for every ${30}^{\circ }$.

As shown in Figure 5 and Figure 6, the simulation results in case 1 are shown in Figure 5, and the simulation results in case 2 are shown in Figure 6. From Figure 5a,c,d and Figure 6a,c,d,

Table 3. Statistics of guidance simulation results in case 1.

LOS Angle Expectation (°)	0	30	60	90
Miss distance (m)	0.0173	0.0173	0.0047	0.0174
LOS angle error (°)	−0.00001	−0.00006	−0.00006	0.00011

and

Table 4. Statistics of guidance simulation results in case 2.

LOS Angle Expectation (°)	0	30	60	90
Miss distance (m)	0.0071	0.0103	0.0152	0.0080
LOS angle error (°)	0.00014	−0.00004	0.00014	0.00043

, it can be seen that the missile could accurately hit the target with the desired terminal LOS angle in a limited time, the LOS angle rate converged to 0°/s, and the LOS angle converged to the expected value under different desired LOS angles, which verified the superstrong control ability of the guidance law for the miss distance and the impact angle. From Figure 5b and Figure 6b, it can be seen that the missile acceleration command was smooth. Since the control law was continuous with respect to the sliding variable, it can be seen that the sliding variable converged to zero continuously and smoothly in finite time in Figure 5e and Figure 6e, and the chattering phenomenon did not appear in the control channel. It can be seen from Figure 5f and Figure 6f that the disturbance differential observer could achieve a great estimation of the system uncertainty. As can be seen from the overall Figure 6, when facing untrained scenes, our improved reinforcement learning algorithm could still hit the target, and the LOS angle converged, which proves that our algorithm has generalization and practical significance.

In Figure 7, Figure 7a is the parameter change for case 1, and Figure 7b is the parameter change for case 2. It can be seen from the figure that the parameter change finally converged to between two and three, and the change process was smooth, which is of practical engineering significance.

In order to show the advantages of our guidance law, we compared with ST-NSTM and NSTM guidance laws, and we set the miss distance to less than 0.1 m. Our algorithm could call the RLST-NTSM guidance law. The parameters of Equation (20) were borrowed from the literature [58], where $\beta =1$, $\alpha =1.4$, and then we chose $k_1=1200$, $k_2=800$, after the comparison. The $\alpha ,\beta ,k_1,k_2$ in the ST-NTSM guidance law was the same as before, and the traditional super-twisting non-singular terminal sliding-mode guidance law (ST-NTSM [51]) was formulated as:

$$a=-2\dot{r}\dot{q}+{\displaystyle \frac{\beta r}{\alpha }}{\left|\dot{e}\right|}^{2-\alpha }\mathrm{sgn}\left(\dot{e}\right)+z_1+k_1{\left|c\right|}^{1/2}\mathrm{sgn}\left(c\right)+k_2\int {\displaystyle \frac{\mathrm{sgn}\left(c\right)}{r}}dt$$

(56)

where $z_1$ is the output of the finite-time convergence disturbance observer (FTDO). According to the reference [58], the specific form of the FTDO [60] introduced by the author in his article is as follows:

$$\begin{array}{c}{\dot{z}}_0=v_0-\dot{r}\dot{q}-a\hfill \\ v_0=-{\lambda }_2{L}^{1/3}{\left|z_0-x\right|}^{2/3}\mathrm{sgn}(z_0-\theta )+z_1\hfill \\ {\dot{z}}_1=v_1,v_1=-{\lambda }_1{L}^{1/2}{\left|z_1-v_0\right|}^{1/2}\mathrm{sgn}(z_1-v_0)+z_2\hfill \\ {\dot{z}}_2=-{\lambda }_{0}L\mathrm{sgn}(z_2-v_1)\hfill \end{array}$$

(57)

where the choice of $\lambda$ in the equation can be obtained from [58]. The parameter L is used to adjust the transient performance of the estimation process and can be set to 200, which makes the observer output $z_1$ converge to $d\left(t\right)$ in finite time.

In addition, another extended state observer based on the traditional super-twisting extended state observer (STESO) [42] was also added for comparison; it was formulated as:

$$\left\{\begin{array}{c}\dot{\widehat{z}}=\widehat{d}\left(t\right)-rq-a+k_3{\left|e_1\right|}^{1/2}\mathrm{sgn}\left(e_1\right)\hfill \\ \dot{\widehat{d\left(t\right)}}=k_4\mathrm{sgn}\left(e_1\right)\hfill \end{array}\right.$$

(58)

where $k_3$, $k_4$ were the same as in the GSTESO.

And the NSTM guidance law was of the form:

$$a=-2\dot{r}\dot{q}+{\displaystyle \frac{\beta r}{\alpha }}{\left|\dot{e}\right|}^{2-\alpha }\mathrm{sgn}\left(\dot{e}\right)+K\mathrm{sgn}\left(c\right)$$

(59)

where the values of $\alpha ,\beta$ were also the same as before, and the value of the switching gain K was 400 according to reference [58].

As shown in

Table 5. Statistics of simulation results for case 2 with different parameter $\gamma$ values.

Parameter $\mathit{\gamma }$	1.7	2.3	2.6	2.9	3.2
LOS angle expectation (°)	90	90	90	90	90
Miss distance (m)	0.1223	0.0834	0.2577	0.0513	0.1666
LOS angle error (°)	0.0040	0.00001	0.000005	0.0006	0.0028
Sliding variable	0.3430	0.0015	0.0002	−0.0187	−0.0004

and Figure 8, we simulated different $\gamma$ values corresponding to ST-NTSM for case 2. It can be seen from Table 5 that no matter how the parameter $\gamma$ value was selected, the accuracy of ST-NTSM was not as high as our algorithm, only the values of 2.3 and 2.6 qualified, given that the miss distance had to be less than 0.1 m. Figure 8 also shows that ST-NTSM still had a small number of chattering problems. And as the parameter $\gamma$ increased, the convergence rate of the guidance law was slower, while our guidance law converged faster, and the chattering phenomenon did not occur.

The simulation results for case 2 with $q_f={90}^{\circ }$ are shown in Figure 9 and

Table 6. Statistics of simulation results for case 2 with different parameter $\gamma$ values.

	RLST-NTSM	ST-NTSM	NTSM
LOS angle expectation (°)	90	90	90
Miss distance (m)	0.0080	0.042	92.64
LOS angle error (°)	0.00004	0.0020	−104.2
Sliding variable	−0.0001	−20.55	−18.01

. The RLST-NTSM proposed in this paper was better than the other two guidance laws in terms of accuracy and convergence of the final LOS angle error, and Figure 9b shows that the missile guided by NTSM did not hit the target, while the performance of ST-NTSM was relatively good. However, according to Figure 9a,d, there was still a certain chattering phenomenon, and the proposed algorithm further reduced the system chattering. It can also be seen from Figure 9c that RLST-NTSM reached the equilibrium state of the system earlier than ST-NTSM. From Figure 9d,e, it can be seen that the sliding-mode variable and LOS angle rate of ST-NTSM and NTSM did not converge well, which would greatly affect their practical engineering application. Finally, it should be noted that although the three observers could be directly added to any guidance law, it can be seen from Figure 9f–h that the GSTESO performed better than the other two observers, as its initial uncertainty observation error was minimal.

As overload saturation emerged in the simulation, we eventually decided to consider the constraints of the autopilot in a simplified manner; combined with Equation (14), we studied the following augmented system.

$$\left\{\begin{array}{c}\dot{a}=x_4\hfill \\ {\dot{x}}_4=-{\omega }_n^{2}a+{\omega }_n^{2}u+h\hfill \\ \dot{h}=H\hfill \end{array}\right.$$

(60)

where $h=d_a-2{\xi }_n{\omega }_n\dot{a}$, $d_a$ is the modeling error, ${\xi }_n$ and ${\omega }_n$ are the damping ratio and natural frequency of the autopilot.

This paper employed an autopilot approach similar to the one described in reference [61], and its formulation is given by

$$\begin{array}{c}u={\omega }_n^{-2}[{\omega }_n^{2}a-z_{23}+L_{22}\mathrm{fal}(e_{21},{\vartheta }_{21},\eta )-L_{21}^{2}e21\hfill \\ -H_2+{\dot{\phi }}_2-m_1{s}_2-m_{2}si{g}^{\chi }{s}_2]\hfill \\ s_2={\epsilon }_2+{\alpha }_2{\epsilon }_1+{\alpha }_2{\beta }_2\left({\epsilon }_1\right)\hfill \\ {\beta }_2\left({\epsilon }_1\right)=\left\{\begin{array}{c}si{g}^{\chi }{\epsilon }_1\mathrm{if}\left|{\epsilon }_1\right|\ge {\eta }_2\hfill \\ b_1{\epsilon }_1+b_{2}si{g}^2{\epsilon }_1\mathrm{if}\left|{\epsilon }_1\right|<{\eta }_2\hfill \end{array}\right.\hfill \end{array}$$

(61)

where $H_2={\alpha }_2{\epsilon }_2+{\alpha }_2{\dot{\beta }}_2\left({\epsilon }_1\right)$, ${\epsilon }_1=z_{21}-{\phi }_1,{\epsilon }_2={\dot{z}}_{21}-{\phi }_2$.

We selected the virtual control signal $a_c$ as in Equation (29). The first and second derivatives of $a_c$ were estimated using the following filter instructions:

$$\begin{array}{c}{\dot{\phi }}_1={\phi }_2\hfill \\ {\dot{\phi }}_2=-2\xi \omega {\phi }_2-{\omega }^2({\phi }_1-a)\hfill \end{array}$$

(62)

where $b_1=(2-\chi ){\eta }^{\chi -1},b_2=(1-\chi ){\eta }^{\chi -2}$, $\xi =0.707$ and $\omega =35$ are the damping ratio and natural frequency of the command filter, respectively, ${\phi }_1$ and ${\phi }_2$ are the estimate and the corresponding derivative, respectively, ${\alpha }_2$ and $\eta$ are positive constants. The system was observed using the following ESO

$$\left\{\begin{array}{c}{z}_{21}=z_{22}-L_{21}{e}_{21}\hfill \\ z_{22}=-{\omega }_n^{2}a+{\omega }_n^{2}u+z_{23}-L_{22}\mathrm{fal}(e_{21},{\vartheta }_{21},\eta )\hfill \\ z_{23}=-L_{23}\mathrm{fal}(e_{21},{\vartheta }_{22},\eta )\hfill \\ e_{21}=z_{21}-a\hfill \end{array}\right.$$

(63)

where $e_{21}=z_{21}-a,e_{22}=z_{22}-x_4$ are the estimate errors, $L_{21},L_{22}$, and $L_{23}$ are the positive observer gains to be designed, $0<{\vartheta }_{21},{\vartheta }_{22}<1$, and $\eta$ is a small positive constant.

The stability of the autopilot component can be established by consulting reference [61], and a detailed analysis was not conducted. The autopilot parameters were ${\omega }_n=35\mathrm{rad}/\mathrm{s},{\xi }_n=0.8,d_a=0.1{\omega }_n^{2}u$. The design parameters for implementing guidance law (61) were set to ${\alpha }_2=0.5,\chi =0.6,{\eta }_2=0.8,m_1=200,$ and $m_2=50$. The parameters of the ESOs were selected as $L_{21}=80,L_{22}=800,L_{23}=20,000,{\vartheta }_{21}=0.7,\eta =0.02$.

As illustrated in Figure 10 and

Table 7. Simulation results considering the autopilot lag for case 2.

	RLST-NTSM Considering Autopilot Lag
LOS angle expectation (°)	0
Miss distance (m)	0.045
LOS angle error (°)	−0.0305
Sliding variable	−0.0013

, this paper focused on simulating the scenario that produced the maximum normal overload, specifically, $q_f={90}^{\circ }$ in case 2; the results showed that considering the autopilot lag characteristics did not affect the performance of the proposed guidance law.

5. Conclusions

In this study, we proposed a novel guidance law, named RLST-NTSM, which combined deep reinforcement learning with sliding-mode guidance. That guidance law was specifically designed for the interception of maneuvering targets with terminal LOS angle constraints. The RLST-NTSM guidance law enhanced the traditional ST-NTSM by introducing an adaptive gamma $\gamma$ parameter, which was optimized through a DRL algorithm to achieve optimal adaptive control. Through the design of the reward function, it effectively reduced the chattering phenomenon and improved the convergence rate of the system state; this is beneficial to improve the stability of the missile body during the actual flight. Furthermore, a GSTESO was employed to estimate and compensate for system disturbances. Distinguished from conventional STESO and FTDO, the GSTESO provided stronger robustness and disturbance rejection ability. Utilizing Lyapunov’s stability theory, we proved the finite-time convergence of the closed-loop guidance system. Simulation results showed that our algorithm greatly improved the overall performance compared with the traditional ST-NTSM and NTSM, and the interception accuracy was improved by at least five times. In response to the emergence of overload saturation within our simulations, we considered the delay characteristics of the autopilot in the concluding segment of our study. Our analysis revealed that these constraints did not impair the guidance performance. In the future, the primary objective of the subsequent phase of research will be to expand the guidance environment into a three-dimensional space. Following this, we intend to deploy the guidance algorithm in a real-world setting to conduct experimental trials.