A Nonlinear Finite-Time Robust Differential Game Guidance Law

Abstract

In this paper, a robust differential game guidance law is proposed for the nonlinear zero-sum system with unknown dynamics and external disturbances. First, the continuous-time nonlinear zero-sum differential game problem is transformed into solving the nonlinear Hamilton–Jacobi–Isaacs equation, a time-varying cost function is developed to reflect the fixed terminal time, and the robust guidance law is developed to compensate for the external disturbance. Then, a novel neural network identifier is designed to approximate the unknown nonlinear dynamics with online weight tuning. Subsequently, an online critic neural network approximator is presented to estimate the cost function, and time-varying activation functions are considered to deal with the fixed final time problem. An adaptive weight tuning law is given, where two additional terms are added to ensure the stability of the closed-loop nonlinear system and so as to meet the terminal cost at a fixed final time. Furthermore, the uniform ultimate boundedness of the closed-loop system and the critic neural network weights estimation error are proven based upon the Lyapunov approach. Finally, some simulation results are presented to demonstrate the effectiveness of the proposed robust differential game guidance law for nonlinear interception.

1. Introduction

In the modern battlefield, when the missile executes the interception mission, the fuel consumption, rapid attitude maneuver, and other uncertain factors may lead to the nonlinear dynamic characteristics of the system. Moreover, external disturbances, which may cause the system to lose control, are important factors in practice applications. Furthermore, the missile intercepts maneuvering targets with a fixed terminal time, which is a more effective attack. Therefore, it is of great significance to study the finite-time robust differential game law in nonlinear circumstances with external disturbance.

Traditional interception guidance strategies include proportional navigation guidance (PNG) [1] and augmented proportional navigation guidance (APNG) [2,3]. In these works, the interception problem was simplified into a two-dimensional plane and the target aircrafts were assumed to be non-maneuvering targets. With the development of the modern control theory, optimal guidance laws (OGLs), sliding mode control (SMC) guidance laws, and linear quadratic differential game (LQDG) guidance laws are investigated to solve the missile interception problem. In [4], first-order interceptor dynamics were taken into account, and an optimal guidance law (OGL) was proposed to control the impact time and impact angle. In [5], a combination of a line-of-sight (LOS) rate shaping technique and a second-order SMC was proposed. However, the target position was assumed to be known in advance. In [6], three-party pursuit and evasion guidance strategies were derived from a linear dynamic system, analytical solutions were derived via the Game Algebraic Riccati equation (GARE) and LQDG guidance laws were proposed. However, it is almost impossible, or even impossible, to acquire the analytic solution, due to the inherent nonlinearity. To circumvent the inherent nonlinearity of the system, in [7], an SDRE guidance law was proposed by solving the nonlinear Hamilton–Jacobi–Isaacs (HJI) equation. However, the external disturbance was neglected and it was difficult for the analytical solution to find its Nash equilibrium. Recently, an artificial neural network [8] has been applied to solve the nonlinear HJI equation for an unknown system. However, the finite-horizon optimal guidance law still remains an unsolved problem.

To solve the GARE, adaptive dynamic programming (ADP) techniques serve as powerful tools to solve the optimal control problems. In [9], a novel online mode-free reinforcement learning algorithm was proposed to solve the multiplayer non-zero-sum games, and the algebraic Riccati equation (ARE) was solved by an iterative algorithm. However, external disturbances were not considered. Unfortunately, in practice applications, external disturbances always exist. In [10], the author proposed a data-driven value iteration (VI) algorithm to handle the adaptive CT linear optimal output regulation. The optimal feedback control gain was learned by an online value iteration algorithm. However, the iterative algorithm requires a significant number of iterations within a fixed sampling interval to guarantee the stability of the system. In the above references, no matter what the mode-free integral RL algorithm or the data-driven value iteration algorithm, all the proposed methods were based on a solvable ARE. However, in most practice applications, external disturbances and parameter uncertainties always exist, which the ARE cannot obtain, in addition to the optimal controller.

For other contemporary approaches related to ADP, one can refer to refs. [11,12]. However, these are not applicable to practice applications with unknown or inaccurate system dynamics. The neuro-dynamic programming (NDP) technique is a typical method used to solve the nonlinear Hamilton–Jacobi–Isaacs (HJI) equation. In [13], the policy or value iteration-based NDP scheme was proposed to obtain the finite-horizon $\epsilon$-optimal control for the discrete-time nonlinear system by using offline neural network (NN) training. In [14], an online algorithm based on policy iteration was proposed to attain the synchronous optimal policy with infinite horizon time for nonlinear systems with known dynamics. However, inadequate iterations within a sampling interval may lead to instability. To avoid this problem, in [15], a time-based NDP method was studied, and iteration-based optimal solutions were replaced by using the previous history of system states and cost function approximations. However, this NDP scheme was not suitable for finite-time nonlinear control; furthermore, external disturbances were not considered and only infinite-time optimal control was studied. In [16], an online concurrent learning NDP algorithm was presented to solve the two-player zero-sum game of nonlinear CT systems with unknown system dynamics and three NN approximators were tuned to learn about the value function corresponding to the optimal control strategies. However, two or more NNs led to an increase in computational complexity.

Although the above research has solved the HJB problem to a certain extent, unfortunately, there are few kinds of literature involved in the missile–target interception problem. In [17], a novel sliding mode adaptive neural network guidance law was proposed to intercept highly maneuvering targets. Aimed at the external disturbance caused by the target maneuvering, the RBF neural network was adopted to eliminate estimation errors without prior information about the target. Similarly, in [18], an adaptive NN-based scheme with an estimation cost function was proposed for solving the interception problem of the spacecraft with limited communication and external disturbances, but it did consider the target maneuverability. However, in these studies, only the missile interception strategy is considered, and the target control strategy is ignored.

In this paper, the nonlinear system of missile–target engagement with external disturbances is considered, and a time-varying cost function is designed for satisfying the terminal interception time. The uncertain nonlinear two-player zero-sum game is developed via the HJI equation based on differential game theory. The main contributions of this paper include two aspects. First, for solving the external disturbances problem, unlike the work [14], an extended robust interception guidance strategy for the missile to intercept the maneuvering target within a fixed final time is proposed. Second, two novel NNs are designed, one online NN identifier is designed to approximate the unknown nonlinear system, and the other critic NN is developed to approximate the cost function without policy or value iterations, while online learning is adopted. Finally, the nonlinear finite-time robust differential game guidance law is proposed.

The advantages of the proposed method of this paper are listed as follows:

(1)

The nonlinear guidance law for missile–target engagement within a fixed interception time is studied by using the differential game theory based on neuro-dynamic programming (NDP). More importantly, a time-varying cost function is reconstructed by a critic NDP with two additional terms added to ensure the stability of the nonlinear system and to meet the fixed interception time, which implies that the missile can intercept the target at different terminal times.

(2)

In practical applications, there are always external disturbances, and a robust interception guidance law is proposed to deal with this problem. Furthermore, inspired by the work [19], our proposed method extends the controller by considering the target.

(3)

Unlike the discrete system, our proposed method is the CT. Moreover, compared with the existing work [15,16], a clear advantage of our method is that a simpler critic NN structure is designed; thus, the computational burden is alleviated.

The remainder of this paper is organized as follows. The statement of the normal guidance problem is presented in part II. The robust control strategy of the nonlinear system with external disturbances is developed in part III. In part IV, an online NN identifier and a novel NDP-based approximator are presented. The stability of the nonlinear system is proved in part V. The nonlinear model of the missile–target engagement is established and numerical experiments are carried out to evaluate the performance of the proposed robust differential game guidance strategy in part VI. Part VII presents some conclusions.

2. Problem Formulation

In this paper, the two players are the missile and the target, which are described in detail in part VI. For the finite-time nonlinear two-player zero-sum differential game, the object of the missile input $\overline{u}\left(t\right)$ is to minimize the cost function, while the target input $\overline{w}\left(t\right)$ is to maximize the cost function in a specified time. The continuous-time (CT) uncertain nonlinear two-player zero-sum differential game is now presented as

$$\dot{x}\left(t\right)=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)\left(\overline{u}\left(t\right)+\overline{d}\left(x\right)\right)+k\left(x\left(t\right)\right)\left(\overline{w}\left(t\right)+\overline{p}\left(x\right)\right)$$

(1)

where $x\left(t\right)\in R^n$, $\overline{u}\left(t\right)\in R^m$, and $\overline{w}\left(t\right)\in R^q$ represent the system state vector, the control input of the missile and the target, respectively. $f\left(x\right)\in R^n$ denotes the internal system dynamics, $g\left(x\right)\in R^{n\times m} , \mathrm{and} k\left(x\right)\in R^{n\times q}$ are the control coefficient matrices, with $f\left(x\right),g\left(x\right) \mathrm{and} k\left(x\right)$ are locally Lipschitz, respectively. $\overline{d}\left(x\right) \mathrm{and} \overline{p}\left(x\right)$ represent external disturbances, which are both bounded by known functions $d_M\left(x\right)$ and $p_M\left(x\right)$, i.e., $\Vert \overline{d}\left(x\right)\Vert \le d_M\left(x\right)$ and $\Vert \overline{p}\left(x\right)\Vert \le p_M\left(x\right)$. Furthermore, we assume that $d\left(x\right)=R_1^{\frac{1}{2}}\overline{d}\left(x\right)$ and $p\left(x\right)=\gamma \overline{d}\left(x\right)$, with $R_1$ and $\gamma$ are symmetric positive definite matrices, respectively.

The nominal system (without external disturbances) of the system (1) can be described as

$$\dot{x}\left(t\right)=f\left(x\left(t\right)\right)+g\left(x\left(t\right)\right)u\left(t\right)+k\left(x\left(t\right)\right)w\left(t\right)$$

(2)

We assume that $f+gu+kw$ is Lipschitz continuous on a set $\mathsf{\Omega }$ and the system (2) is controllable.

Considering external disturbances in the nonlinear system (1), for the nominal system (2), the finite-time two-player zero-sum differential game cost function is defined as

$$\begin{array}{c}V\left(x,t_0\right)=\phi \left(x\left(t_f\right),t_f\right)+{{\displaystyle \int }}_{t_0}^{t_f}r\left(x\left(t\right),u\left(t\right),w\left(t\right)\right)dt\\ V\left(x,t_f\right)=\phi \left(x\left(t_f\right),t_f\right)\end{array}$$

(3)

where $r\left(x\left(t\right),u\left(t\right),w\left(t\right)\right)=Q\left(x\right)+u^T{R}_{1}u-\frac{1}{2{\gamma }^2}{w}^{T}w$, with $Q\left(x\right)=d_M^2\left(x\right)+R_2{p}_M^2\left(x\right)+x^T{Q}_{1}x$; $Q_1$ is a semi-positive function. The terminal cost, external disturbances, control efforts of the missile and the target, the system state, and fixed terminal time are chosen as the performance evaluation indicators. $\phi \left(x\left(t_f\right),t_f\right)$ reflects the terminal cost between the missile with the target. The term $Q\left(x\right)$ reflects external disturbances and the system state simultaneously, which is positively defined. Moreover, $R_2$ represents the influence of the target disturbance. $R_1$ reflects the missile control effort. $\gamma$ reflects the target control effort. All parameters $Q\left(x\right)$, $R_1,$ $R_2,$ and $\gamma$ are positively defined. The goal of this paper is to find the saddle point of the cost function (3).

Remark 1.

First, unlike the infinite-time scenario, the terminal cost$V\left(x,t_0\right)$is a time-varying function.$\phi \left(x\left(t_f\right),t_f\right)$is needed to guarantee the finite-time scenario. Second, external disturbances are considered in the cost function via adopting a positive constant$R_2$and the robust control problem can be addressed by designing the finite-time guidance strategy of the nominal system (2).

Assuming that $V\left(x,t\right)\in C^1$, an infinitesimal equivalent to (3) can be derived as

$$-\frac{\partial V\left(x,t\right)}{\partial t}=r\left(x,u,d\right)+\frac{\partial V^T\left(x,t\right)}{\partial x}\left(f\left(x\right)+g\left(x\right)u+k\left(x\right)w\right)$$

(4)

when $t_0=t_f$, the terminal cost function can be expressed as

$$V\left(x,t_f\right)=\Psi \left(x\left(t_f\right),t_f\right)$$

(5)

The Hamiltonian function of the nonlinear system (2) can be defined as

$$H\left(x,u,v\right)=V_t+r\left(x,u,v\right)+V_x^T\left(f\left(x\right)+g\left(x\right)u\left(t\right)+k\left(x\right)w\left(t\right)\right)$$

(6)

where $V_t=\frac{\partial V\left(x,t\right)}{\partial t}$ and $V_x=\frac{\partial V\left(x,t\right)}{\partial x}$. It can be clearly observed that the Hamiltonian function includes a time-dependent term $V_t$.

In the Nash equilibrium theory, the saddle point with respect to the optimal control pair $\left(u\left(t\right),w\left(t\right)\right)$ can be obtained by

$$H\left(x,u^{*},w\right)\le H\left(x,u^{*},w^{*}\right)\le H\left(x,u,w^{*}\right)$$

(7)

If one recalls the classical optimal game theory, both optimal controllers can be solved by using stationary conditions $\frac{\partial H\left(·\right)}{\partial u}=0$ and $\frac{\partial H\left(·\right)}{\partial w}=0$, which yields

$$\{\begin{array}{c}{u}^{*}\left(x\right)=-\frac{1}{2}{R}_1^{-1}{g}^T\left(x\right)V_x^{*}\hfill \\ w^{*}\left(x\right)={\gamma }^2{k}^T\left(x\right) V_x^{*}\hfill \end{array}$$

(8)

where $V^{*}\left(x,t\right)$ is the optimal two-player zero-sum game cost function, which is the saddle point of the cost function, such that

$$\begin{array}{c}{V}^{*}\left(x,t\right)=\underset{w\left(t\right)}{\max }\underset{u\left(t\right)}{\min }V\left(x,t\right)\\ V^{*}\left(x,t_f\right)=\phi \left(x\left(t_f\right),t_f\right)\end{array}$$

(9)

By substituting optimal strategy (8) into Equation (4), the Hamilton–Jacobi–Isaacs (HJI) equation reduces to

$$\begin{array}{c}{V}_t^{*}+Q\left(x\right)+V_x^{*}{}^{T}f\left(x\right)-\frac{1}{4}{V}_x^{*}{}^{T}g\left(x\right)R_1^{-1}{g}^T\left(x\right)V_x^{*}\\ +\frac{{\gamma }^2}{2}{V}_x^{*}{}^{T}k\left(x\right)k^T\left(x\right)V_x^{*}=0\end{array}$$

(10)

Remark 2.

For the linear system case, the HJB equation can be easily solved by the Riccati equation [19]. However, it is difficult or even impossible to attain the mathematical solution of the HJI Equation (10), when system dynamics exist in nonlinear terms. Moreover, the fixed final time $t_f$ is provided in this nonlinear system for solving the inadequate iterations problem, a novel time-based online optimal guidance law design is proposed and the system dynamics are demonstrated.

3. The Nonlinear Finite-Time Robust Differential Game Guidance Law

In this part, the nonlinear finite-time robust differential game guidance law is presented. First, for coping with the robust stabilization problem of the system (1) with external disturbances, a robust controller is designed. Then, the finite-time NDP-based optimal guidance strategy is designed.

3.1. Robust Controller Design of Uncertain Nonlinear Differentia Games

By extending the work [20], two feedback gains ${\pi }_1 \mathrm{and}\frac{1}{2{\pi }_2}$ are added to the optimal feedback control (8) of the system (2) for the missile and the target, respectively. The robust optimal feedback control yields as follows:

$$\{\begin{array}{c}{\overline{u}}^{*}\left(x\right)={\pi }_1{u}^{*}\left(x\right)=-\frac{1}{2}{\pi }_1{R}_1^{-1}{g}^T\left(x\right)V_x^{*}\hfill \\ {\overline{w}}^{*}\left(x\right)=\frac{1}{2{\pi }_2}{w}^{*}\left(x\right)=\frac{1}{2{\pi }_2}{\gamma }^2{k}^T\left(x\right) V_x^{*}\hfill \end{array}$$

(11)

Here, some lemmas are presented for indicating that the robust optimal control has an infinite gain margin.

Lemma 1.

For the nominal system (2), the optimal control strategy given by (11) can ensure that the closed-loop system is asymptotically stable for${\pi }_1\ge \frac{1}{2}$and${\pi }_2\ge 1$.

Proof of Lemma 1.

The optimal cost function $V^{*}\left(x,t\right)=J^{*}\left(x,t\right)$ is selected as the Lyapunov function. In light of (3), it is easily found that $V^{*}\left(x,t\right)$ is positive definite. By combining (10) and (11), the derivative of the $V^{*}\left(x,t\right)$ along the trajectory of the closed-loop system yields

$$\begin{gathered} {\dot{J}}^{*}\left(x,t\right)=V_t^{*}+V_x^{*}{}^T\left(f\left(x\right)+g\left(x\right)\overline{u}\left(t\right)+k\left(x\right)\overline{w}\left(t\right)\right) \\ =V_t^{*}-Q\left(x\right)-V_t^{*}+\frac{1}{4}{V}_x^{*}{}^{T}g\left(x\right)R_1^{-1}{g}^T\left(x\right)V_x^{*} \\ -\frac{{\gamma }^2}{2}{V}_x^{*}{}^{T}k\left(x\right)k^T\left(x\right)V_x^{*}-\frac{1}{2}{\pi }_{1}g\left(x\right)R_1^{-1}{g}^T\left(x\right)V_x^{*}+\frac{1}{2{\pi }_2}{\gamma }^{2}k\left(x\right)k^T\left(x\right) V_x^{*} \\ \begin{array}{c}=-Q\left(x\right)-\frac{1}{2}\left({\pi }_1-\frac{1}{2}\right){\Vert R_1^{-\frac{1}{2}}{g}^T\left(x\right)V_x^{*}\Vert }^2-\frac{1}{2}\left(1-\frac{1}{{\pi }_2}\right){\Vert \gamma k^T\left(x\right)V_x^{*}\Vert }^2\end{array} \end{gathered}$$

(12)

Hence, whenever ${\pi }_1\ge \frac{1}{2}$, ${\pi }_2\ge 1$ and $x\ne 0,$ ${\dot{J}}^{*}\left(x,t\right)<0$. □

Theorem 1.

For the system (1), there exists two positive gains,${\pi }_1^{*}\ge 1$and${\pi }_2^{*}\ge \frac{2R_2}{2R_2+1}$, with$R_2>1$, such that for any${\pi }_1>{\pi }_1^{*}$and${\pi }_2>{\pi }_2^{*}$, the robust control (11) ensures that the closed-loop system (1) is asymptotically stable.

Proof of Theorem 1.

The optimal cost function $L\left(x,t\right)=V^{*}\left(x,t\right)$ is selected as the Lyapunov function, and the derivative of the $V^{*}\left(x,t\right)$ along the trajectory of the closed-loop system can be obtained as

$$L\left(x,t\right)=V_t^{*}+V_x^{*}{}^T\left(f\left(x\right)+g\left(x\right)\left(\overline{u}\left(t\right)+\overline{d}\left(x\right)\right)+k\left(x\right)\left(\overline{w}\left(t\right)+\overline{p}\left(x\right)\right)\right)$$

(13)

Based on (12), (13) can be rewritten as

$$\begin{gathered} \dot{L}\left(x,t\right)=-Q\left(x\right)-\frac{1}{2}\left({\pi }_1-\frac{1}{2}\right){\Vert V_x^{*}{}^{T}g\left(x\right)R_1^{-\frac{1}{2}}\Vert }^2-\frac{1}{2}\left(1-\frac{1}{{\pi }_2}\right){\Vert V_x^{*}{}^{T}g\left(x\right)\gamma \Vert }^2+V_x^{*}{}^{T}g\left(x\right)\overline{d}\left(x\right)+V_x^{*}{}^{T}k\left(x\right)\overline{p}\left(x\right) \\ \le -x^T{Q}_{1}x-\left(\frac{1}{2}\left({\pi }_1-\frac{1}{2}\right){\Vert V_x^{*}{}^{T}g\left(x\right)R_1^{-\frac{1}{2}}\Vert }^2+\frac{1}{2}\left(1-\frac{1}{{\pi }_2}\right){\Vert V_x^{*}{}^{T}g\left(x\right)\gamma \Vert }^2-\Vert V_x^{*}{}^{T}g\left(x\right)R_1^{-\frac{1}{2}}\Vert d_M\left(x\right) \\ -\Vert V_x^{*}{}^{T}k\left(x\right)\gamma \Vert d_M\left(x\right)+d_M^2\left(x\right)+R_2{d}_M^2\left(x\right)\right) \end{gathered}$$

(14)

If $z={\left[d_M\left(x\right), p_M\left(x\right),\Vert V_x^{*}{}^{T}g\left(x\right)R_1^{-\frac{1}{2}}\Vert , \Vert V_x^{*}{}^{T}k\left(x\right)\gamma \Vert \right]}^T$; thus, (14) can be rewritten as

$$\dot{L}\left(x,t\right)\le -x^T{Q}_{1}x-z^T\mathsf{\Theta }z$$

(15)

where $\mathsf{\Theta }=\left[\begin{array}{cccc}1& 0& -\frac{1}{2}& 0\\ 0& R_2& 0& -\frac{1}{2}\\ -\frac{1}{2}& 0& \frac{1}{2}\left({\pi }_1-\frac{1}{2}\right)& 0\\ 0& -\frac{1}{2}& 0& \frac{1}{2}\left(1-\frac{1}{{\pi }_2}\right)\end{array}\right]$.

According to the Lyapunov function, the determinant $\mathsf{\Theta }\ge 0$ represents the $\dot{L}\left(x,t\right)\le 0$ and implies that the closed-loop system is asymptotically stable. Thus, it can be concluded that ${\pi }_1^{*}\ge 1$ and ${\pi }_2^{*}\ge \frac{2R_2}{2R_2+1}$ can ensure the positive definiteness of $\mathsf{\Theta }$. When ${\pi }_1>{\pi }_1^{*}$ and ${\pi }_2>{\pi }_2^{*}$, the closed-loop system is asymptotically stable. □

3.2. Finite-Time NDP-Based Optimal Guidance Strategy

First, a novel online NN identifier is proposed to approximate the unknown system dynamics. Next, a critical NDP-based approximator is utilized to estimate the cost function within a fixed final time and an online adaptive weight tuning law is proposed with additional terms to guarantee the stability of the nonlinear system. Finally, combining the identified system and estimated cost function, the finite-horizon optimal differential guidance strategy is derived.

3.2.1. NN Identifier

System dynamics are necessary for developing guidance laws of the nonlinear two-player zero-sum differential games. However, system dynamics may be unknown in practice applications. To overcome this problem, a novel online NN identifier is designed. Based on the NN universal function approximation property, the nonlinear system can be represented as

$$\begin{gathered} f\left(x\right)=W_f^T{\sigma }_{f\left(x\right)}+{\epsilon }_f \\ g\left(x\right)=W_g^T{\sigma }_{g\left(x\right)}+{\epsilon }_g \\ k\left(x\right)=W_k^T{\sigma }_{k\left(x\right)}+{\epsilon }_k \end{gathered}$$

(16)

where $W_f\in R^{f\times N},W_g\in R^{g\times N}$, and $W_k\in R^{k\times N}$ are ideal weight matrices. ${\sigma }_{f\left(x\right)}\in R^N, {\sigma }_{g\left(x\right)}\in R^N$, and ${\sigma }_{k\left(x\right)}\in R^N$ denote NN activation function vectors, N is the number of hidden layer neurons, and ${\epsilon }_f, {\epsilon }_g,$ and ${\epsilon }_k$ represent NN approximation errors.

Then, the nominal system (2) can be represented by using (16) as

$$\dot{x}=f\left(x\right)+g\left(x\right)u+k\left(x\right)w={\left[\begin{array}{c}{W}_f\\ W_g\\ W_k\end{array}\right]}^T\left[\begin{array}{ccc}{\sigma }_{f\left(x\right)}& 0& 0\\ 0& {\sigma }_{g\left(x\right)}& 0\\ 0& 0& {\sigma }_{k\left(x\right)}\end{array}\right]\left[\begin{array}{c}1\\ u\\ w\end{array}\right]+{\epsilon }_f+{\epsilon }_{g}u+{\epsilon }_{k}w=W_I^T{\sigma }_{I\left(x\right)}\overline{\xi }+{\epsilon }_I$$

(17)

Because the ideal NN weights are typically unknown, we define the state estimator as follows:

$$\dot{\widehat{x}}={\widehat{W}}_I^T{\sigma }_{I\left(x\right)}\overline{\xi }+K\tilde{x}$$

(18)

where ${\widehat{W}}_I$ represents the estimate of the $W_I$. $\tilde{x}=x-\widehat{x}\in R^N$ denotes the state estimation error. $K$ is a design parameter, which can guarantee the stability of the NN identifier.

From (17) and (18), the derivative of the state estimation error yields

$$\dot{\tilde{x}}=\dot{x}-\dot{\widehat{x}}={{\tilde{W}}_I}^T{\sigma }_{I\left(x\right)}\overline{\xi }+{\epsilon }_I-K\tilde{x}$$

(19)

Inspired by [9], in order to make the approximated NN identifier weight matrix close to its ideal value, the online tuning law is given by

$${\dot{\widehat{W}}}_I\left(t\right)=-{\alpha }_1{\widehat{W}}_I\left(t\right)+{\sigma }_{I\left(x\right)}\overline{\xi }\tilde{x}$$

(20)

where ${\alpha }_1$ is the learning rate of the critical NN.

Next, by defining ${\tilde{W}}_I=W_I-{\widehat{W}}_I$, the identifier weight estimation error yields the following equation by using (20):

$$\dot{{\tilde{W}}_I}\left(t\right)={\alpha }_1{\widehat{W}}_I\left(t\right)-{\sigma }_{I\left(x\right)}\overline{\xi }\tilde{x}$$

(21)

Theorem 2.

For the NN identifier (18), let the initial ideal critic NN weight$W_I$reside in a compact set by selecting the proposed NN weight tuning law provided (20). There exists a positive tuning parameter (${\alpha }_1>0$). Then, the identification (19) and the weight estimation errors${\tilde{W}}_I\left(t\right)$are uniformly ultimately bounded (UUB) with a fixed terminal time.

Proof of Theorem 2.

Let the Lyapunov function candidate be as follows:

$$J_I=\frac{1}{2}{\tilde{x}}^T\tilde{x}+\frac{1}{2}tr\left({{\tilde{W}}_I}^T{\tilde{W}}_I\right)$$

(22)

Then,

$$\begin{gathered} \dot{J_I}={\tilde{x}}^T\dot{\tilde{x}}+tr\left({{\tilde{W}}_I}^T\dot{{\tilde{W}}_I}\right)\le \frac{1}{2}{\Vert \tilde{x}\Vert }^2+\frac{1}{2}{\Vert {\tilde{\epsilon }}_I\Vert }^2-{\lambda }_{min}\left(K\right){\Vert \tilde{x}\Vert }^2+{\alpha }_{1}tr\left({{\tilde{W}}_I}^T{\tilde{W}}_I\right)-{\alpha }_{1}tr\left({{\tilde{W}}_I}^T{\tilde{W}}_I\right) \\ \le -\left({\lambda }_{min}\left(K\right)-\frac{1}{2}\right){\Vert \tilde{x}\Vert }^2-\frac{{\alpha }_1}{2}{\Vert {\tilde{W}}_I\Vert }^2+{\epsilon }_{IM} \end{gathered}$$

(23)

where ${\epsilon }_{IM}=\frac{{\alpha }_1}{2}{W}_I^2+\frac{1}{2}{\Vert {\epsilon }_I\Vert }^2$, the eigenvalue K and ${\alpha }_1$ are the design parameters that guarantee the stability of the system. Therefore, when ${\lambda }_{min}\left(K\right)\ge \frac{1}{2},$ and ${\alpha }_1\ge 0$, the following inequalities hold

$$\Vert \tilde{x}\Vert >\sqrt{\frac{{\epsilon }_{IM}}{\left({\lambda }_{min}\left(K\right)-\frac{1}{2}\right)}} \mathrm{or} \Vert {\tilde{W}}_I\Vert >\sqrt{\frac{2{\epsilon }_{IM}}{{\alpha }_1}}$$

(24)

It can be observed from (24) that $\Vert \tilde{x}\Vert$ bound can be decreased by increasing the eigenvalue K. Therefore, $\Vert \tilde{x}\Vert$ is quantified by selecting the minimum eigenvalue ${\lambda }_{min}\left(K\right)$. According to (19) and the relationship between $\Vert \tilde{x}\Vert$ and $\Vert {\tilde{W}}_I\Vert$, the smaller $\Vert \tilde{x}\Vert$ will enforce the $\Vert {\tilde{W}}_I\Vert$ to converge into a small bound. Therefore, we have $\dot{J_I}\le 0$ and it can be concluded that $\Vert \tilde{x}\Vert$ and $\Vert {\tilde{W}}_I\Vert$ are UUB.

This completes the proof. □

3.2.2. NDP-Based Guidance Strategy

In order to confront the nonlinear HJI function, according to the universal approximation property of the neural network, the optimal cost function can be reconstructed by a critic NDP on a compact set as

$$V\left(x,t\right)=W_V^{T}h\left(x,t_f-t\right)+{\epsilon }_V\left(x,t\right)$$

(25)

where $W_V\in R^{V\times N} is$ the ideal weight matrix. $h\left(x,t_f-t\right)\in R^N$ denotes NN activation function vectors, N is the number of hidden layer neurons, and ${\epsilon }_v$ represents the NN approximation error.

Thus, the terminal cost function can be expressed as

$$V\left(x,t_f\right)=W_V^{T}h\left(x\left(t_f\right),0\right)+{\epsilon }_V\left(x,t_f\right)$$

(26)

Remark 3.

The critic NN activation function$h\left(x,t\right)$and its gradient$\nabla h\left(x,t\right)$are upper bounded, i.e.,$\Vert h\left(x,t\right)\Vert \le h_M$and$\Vert \nabla h\left(x,t\right)\Vert \le h_{dM}$, with$h_M$and$h_{dM}$positive constants. The critic NN weight W is upper bounded, i.e.,$\Vert W_V\Vert \le W_{VM}$, with$W_{VM}$being a positive constant. The critic NN approximation error${\epsilon }_V\left(x\right)$and its gradient$\nabla {\epsilon }_V\left(x\right)$are upper bounded, i.e.,$\Vert {\epsilon }_V\left(x\right)\Vert \le {\epsilon }_{VM}$and$\Vert \nabla {\epsilon }_V\left(x\right)\Vert \le {\epsilon }_{dVM}$, with${\epsilon }_{VM}$and${\epsilon }_{dVM}$positive constants. Obviously, the activation function is a time-varying function, which can maintain the fixed final time.

Next, the partial derivation of $V\left(x,t\right)$ with respect to x and t can be obtained, respectively.

$$\begin{array}{c}{V}_x=\nabla h_x^T\left(x,t_f-t\right)W_V+{\nabla }_x{\epsilon }_V\left(x,t\right)\\ V_t=\nabla h_t^T\left(x,t_f-t\right)W_V+{\nabla }_t{\epsilon }_V\left(x,t\right)\end{array}$$

(27)

where $\nabla h_x^T\left(x,t_f-t\right)=\partial h\left(x,t_f-t\right)/\partial x$, $\nabla h_t^T\left(x,t_f-t\right)=\partial h\left(x,t_f-t\right)/\partial t$, ${\nabla }_x{\epsilon }_V\left(x,t\right)=\partial {\epsilon }_V\left(x,t_f-t\right)/\partial x$, and ${\nabla }_t{\epsilon }_V\left(x,t\right)=\partial {\epsilon }_V\left(x,t_f-t\right)/\partial t$.

Therefore, by substituting (27) into (8), we then obtain the differential game guidance strategy as

$$\{\begin{array}{c}{u}^{*}\left(x\right)=-\frac{1}{2}{R}_1^{-1}{g}^T\left(x\right)V_x^{*}=-\frac{1}{2}{R}_1^{-1}{g}^T\left(x\right)\left(\nabla h_x^T\left(x,t_f-t\right)W_V+{\nabla }_x{\epsilon }_V\left(x,t\right)\right)\\ w^{*}\left(x\right)={\gamma }^2{k}^T\left(x\right) V_x^{*}={\gamma }^2{k}^T\left(x\right)\left(\nabla h_x^T\left(x,t_f-t\right)W_V+{\nabla }_x{\epsilon }_V\left(x,t\right)\right)\end{array}$$

(28)

By substituting (28) into (10), the HJI function can be rewritten as

$$\begin{gathered} H\left(x,u,w\right)=W_V^T\nabla h_t^T\left(x,t_f-t\right)+Q\left(x\right)+W_V^T\nabla h_x\left(x,t_f-t\right)f\left(x\right) \\ \begin{array}{c}-\frac{1}{4}{W}_V^T\nabla h_x\left(x,t_f-t\right)D_1\left(x\right)\nabla h_x^T\left(x,t_f-t\right)W_V+\frac{{\gamma }^2}{2}{W}_V^T\nabla h_x\left(x,t_f-t\right)k\left(x\right)k^T\left(x\right)\nabla h_x^T\left(x,t_f-t\right)W_V+{\epsilon }_{HJB}\left(x,t\right)\end{array} \end{gathered}$$

(29)

where $D_1\left(x\right)=g\left(x\right)R_1^{-1}{g}^T\left(x\right)$ and

$$\begin{gathered} {\epsilon }_{HJB}\left(x,t\right)={\nabla }_t{\epsilon }_V\left(x,t\right)+\frac{1}{2}{W}_V^{T}h\left(x,t_f-t\right){\nabla }_x{\epsilon }_V\left(x,t\right)D_1\left(x\right){\nabla }_x{\epsilon }_V\left(x,t\right)-{\gamma }^2{W}_V^{T}h\left(x,t_f-t\right){\nabla }_x{\epsilon }_V\left(x,t\right)D_1\left(x\right){\nabla }_x{\epsilon }_V\left(x,t\right) \\ -\frac{1}{2}{\nabla }_x{\epsilon }_V^T\left(x,t\right)D_1\left(x\right)\nabla h_x^T\left(x,t_f-t\right)W_V-\frac{1}{4}{\nabla }_x{\epsilon }_V^T\left(x,t\right)D_1\left(x\right){\nabla }_x{\epsilon }_V\left(x,t\right) \\ +{\gamma }^2{\nabla }_x{\epsilon }_V^T\left(x,t\right)k\left(x\right)k^T\left(x\right)\nabla h_x^T\left(x,t_f-t\right)W_V+\frac{{\gamma }^2}{2}{\nabla }_x{\epsilon }_V^T\left(x,t\right)k\left(x\right)k^T\left(x\right){\nabla }_x{\epsilon }_V\left(x,t\right) \\ +\frac{1}{4}{W}_V^T\nabla h_x\left(x,t_f-t\right)D_1\left(x\right){\nabla }_x{\epsilon }_V\left(x,t\right) \\ +{\nabla }_x{\epsilon }_V^T\left(x,t\right)f\left(x\right)-\frac{{\gamma }^2}{2}{W}_V^T\nabla h_x\left(x,t_f-t\right)k\left(x\right)k^T\left(x\right){\nabla }_x{\epsilon }_V\left(x,t\right) \end{gathered}$$

(30)

Because ideal NN weights are typically unknown, we define the estimated cost function $\widehat{V}\left(x,t\right)$ as follows

$$\widehat{V}\left(x,t\right)={\widehat{W}}_V^{T}h\left(x,t_f-t\right)$$

(31)

The estimated terminal cost function is

$$\widehat{V}\left(x,t_f\right)={\widehat{W}}_V^{T}h\left(\widehat{x}\left(t_f\right),0\right)$$

(32)

where ${\widehat{W}}_V$ denotes the estimate of the $W_V$ and $h\left(\widehat{x}\left(t_f\right),0\right)$ is the activation function with the estimated terminal state $\widehat{x}\left(t_f\right)$.

Next, the partial derivation of the estimated cost function $\widehat{V}\left(x,t\right)$ with respect to x and t can be obtained, respectively.

$$\begin{array}{c}{\widehat{V}}_x=\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V\\ {\widehat{V}}_t=\nabla h_t^T\left(x,t_f-t\right){\widehat{W}}_V\end{array}$$

(33)

where ${\widehat{V}}_t=\frac{\partial V\left(x,t\right)}{\partial t}$ and ${\widehat{V}}_x=\frac{\partial V\left(x,t\right)}{\partial x}$.

Then, by applying (33) to (8), the estimated differential game guidance strategy can be rewritten as

$$\{\begin{array}{c}\widehat{u}\left(x\right)=-\frac{1}{2}{R}_1^{-1}{g}^T\left(x\right){\widehat{V}}_x=-\frac{1}{2}{R}_1^{-1}{g}^T\left(x\right)(\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V\\ \widehat{w}\left(x\right)={\gamma }^2{k}^T\left(x\right){\widehat{V}}_x={\gamma }^2{k}^T\left(x\right)\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V\end{array}$$

(34)

By applying (34) to (10), the estimated HJB function yields

$$\begin{gathered} \widehat{H}\left(x,u,w\right)={\widehat{W}}_V^T\nabla h_t^T\left(x,t_f-t\right)+{\widehat{W}}_V^T\nabla h_x\left(x,t_f-t\right)\widehat{f}\left(x\right) \\ \begin{array}{c}-\frac{1}{4}{\widehat{W}}_V^T\nabla h_x\left(x,t_f-t\right){\widehat{D}}_1\left(x\right)\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V+\frac{{\gamma }^2}{2}{\widehat{W}}_V^T\nabla h_x\left(x,t_f-t\right)\widehat{k}\left(x\right){\widehat{k}}^T\left(x\right)\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V=e_c\end{array} \end{gathered}$$

(35)

In order to obtain the optimal differential game guidance strategy, we define the estimated terminal cost error as

$$e_{tf}=\Psi \left(x\left(t_f\right),0\right)-{\widehat{W}}_V^{T}h\left(\widehat{x}\left(t_f\right),0\right)=W_V^T\tilde{h}\left(x\left(t_f\right),0\right)+{\tilde{W}}_V^{T}h\left(\widehat{x}\left(t_f\right),0\right)+{\epsilon }_{t_f}$$

(36)

where $\tilde{h}\left(x\left(t_f\right),0\right)=h\left(x\left(t_f\right),0\right)-h\left(\widehat{x}\left(t_f\right),0\right)$.

Moreover, in order to guarantee the estimated NN weight ${\widehat{W}}_V$, which is approximated to the ideal NN weight $W_V$, and after combining the time-varying nature of the cost function and the estimated terminal cost error, the total NN approximation error is defined as

$$E_{total}=\frac{1}{2}{e}_c^T{e}_c+\frac{1}{2}{e}_{t_f}^4$$

(37)

By using the gradient descent algorithm, a novel simplified weight tuning law is proposed with additional terms to ensure the stability of the nonlinear system (1), as follows:

$$\begin{gathered} {\dot{\widehat{W}}}_V=-{\alpha }_2\frac{\left(\widehat{\omega }+\frac{1}{2}sng\left(\widehat{\omega }\right)\right)}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}{e}_c+{\alpha }_3\frac{\left(\widehat{\xi }+\frac{1}{2}sng\left(\widehat{\xi }\right)\right)}{{\left(1+{\widehat{\xi }}^T\widehat{\xi }\right)}^2}{e}_{t_f}^3 \\ \begin{array}{c}+\frac{{\alpha }_4}{2}\nabla h_x^T\left(x,t_f-t\right)\left({\widehat{D}}_1\left(x\right)-2{\gamma }^2{\widehat{k}}^T\left(x\right)k^T\left(x\right)\right)Q\left(x,t\right)\end{array} \end{gathered}$$

(38)

where $\widehat{\omega }=\nabla h_t^T\left(x,t_f-t\right)+\nabla h_x\left(x,t_f-t\right)\widehat{f}\left(x\right)+\frac{1}{2}\nabla h_x\left(x,t_f-t\right)\left({\widehat{D}}_1\left(x\right)+{\widehat{k}}^T\left(x\right)k^T\left(x\right)\right)\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V$, $\widehat{\xi }=h\left(\widehat{x}\left(t_f\right),t_f\right)$

Remark 4.

It is important to mention that the first term in (38) is used to minimize the squared residual error. The second term is used to minimize the terminal cost estimation error. The last term is used to guarantee that the system states remain bounded.

4. Stability Analysis

In order to prove the stability of the weight tuning law and the nonlinear system by choosing the optimal strategy, and without loss of generality, the weight estimation error of the critic NN is defined as ${\tilde{W}}_V=W_V-{\widehat{W}}_V$; then, we have ${\dot{\tilde{W}}}_V=-{\dot{\widehat{W}}}_V$. Therefore, the approximated HJI is

$$\begin{gathered} \widehat{H}\left(x,u,w\right)=-{\tilde{W}}_V^T\nabla h_t^T\left(x,t_f-t\right)-{\tilde{W}}_V^T\nabla h_x\left(x,t_f-t\right)\widehat{f}\left(x\right)-W_V^T\nabla h_x\left(x,t_f-t\right)\tilde{f}\left(x\right) \\ +\frac{1}{2}{\tilde{W}}_V^T\nabla h_x\left(x,t_f-t\right)\left({\widehat{D}}_1\left(x\right)-2{\gamma }^2{\widehat{k}}^T\left(x\right)k^T\left(x\right)\right)\nabla h_x^T\left(x,t_f-t\right){\widehat{W}}_V \\ \begin{array}{c}+\frac{1}{4}{W}_V^T\nabla h_x\left(x,t_f-t\right)\left(\tilde{D}\left(x\right)-2{\gamma }^2{\tilde{k}}^T\left(x\right)\tilde{k}\left(x\right)-4{\gamma }^2{\tilde{k}}^T\left(x\right)\widehat{k}\left(x\right)\right)\nabla h_x^T\left(x,t_f-t\right)W_V+{\epsilon }_{HJB}\end{array} \end{gathered}$$

(39)

Assumption 1.

For the nonlinear system (1) with the cost function (2) and the optimal guidance law (34), let the value function$V\left(x,t\right)$be the Lyapunov function and continuously differentiable, when

$$\dot{V}\left(x,t\right)=V_x\left(x,t\right)\dot{x}+V_t\left(x,t\right)+V_x\left(x,t\right)\left(f\left(x\right)+g\left(x\right)u^{*}+k\left(x\right)w^{*}\right)<0$$

(40)

Exists and it is clear that the inequation holds

$$V_t\left(x,t\right)+V_x\left(x,t\right)\left(f\left(x\right)+g\left(x\right)u^{*}+k\left(x\right)w^{*}\right)<-Q\left(x,t\right)$$

(41)

Theorem 3.

For the nonlinear system (1) with the ideal HJI Equation (9), let the updated law for the NN-based identifier and NDP-based cost function approximator be provided by (20) and (38), respectively and the estimated optimal guidance laws are given in (34). The existence of the positive constants$B_{Q_x}$,$B_{W_V}$,$B_{\tilde{x}}$and$B_{W_I}$mean that the identification error, weight estimation error and the controller are UUB.

Proof of Theorem 3.

Select the Lyapunov candidate function as

$$J_V=J_{aV}+J_{bV}+J_{cV}+J_{dV}$$

(42)

where $J_{aV}=\frac{{\alpha }_2}{4}{\left({\tilde{x}}^T\Lambda \tilde{x}\right)}^2+\frac{{\alpha }_3}{4}{\left(tr\left({{\tilde{W}}_I}^T\Lambda {\tilde{W}}_I\right)\right)}^2$, $J_{bV}=\frac{1}{2}{\tilde{W}}_V^T\Pi {\tilde{W}}_V$, $J_{cV}={\alpha }_{4}V\left(x,t\right)$, $J_{dV}=\frac{1}{2}{\tilde{x}}^T\Xi \tilde{x}+\frac{1}{2}tr\left({{\tilde{W}}_I}^T\Lambda {\tilde{W}}_I\right)$.

It can be observed that $J_V>0$.

First, the derivation of $J_{aV}$ with respect to time is given by

$$\begin{gathered} {\dot{J}}_{aV}\le {\Vert \Lambda \Vert }^2{\alpha }_2\left({\tilde{x}}^T\tilde{x}\right){\tilde{x}}^T\dot{\tilde{x}}+{\Vert \Lambda \Vert }^2{\alpha }_{3}tr\left({{\tilde{W}}_I}^T{\tilde{W}}_I\right)tr\left({{\tilde{W}}_I}^T\dot{{\tilde{W}}_I}\right) \\ \le -{\alpha }_2\left({\lambda }_{min}\left(K_I\right)-\frac{3}{2}-\frac{{\alpha }_3}{4{\alpha }_2}\right){\Vert \Lambda \Vert }^2{\Vert \tilde{x}\Vert }^4+\frac{1}{4}{\Vert \Lambda {\epsilon }_I\Vert }^2{\Vert \tilde{\mathrm{u}}\Vert }^2 \\ -\frac{{\alpha }_3}{2}\left(\frac{{\alpha }_I}{2}-1-\frac{{\alpha }_2+{\alpha }_3}{4{\alpha }_3}{\Vert {\zeta }_I\Vert }^4\right){\Vert \Lambda \Vert }^2{\Vert {\tilde{W}}_I\Vert }^4+{\epsilon }_{cM} \end{gathered}$$

(43)

where ${\alpha }_2>0$, ${\alpha }_3>0$ and ${\alpha }_I>4+\frac{{\alpha }_2+{\alpha }_3}{4{\alpha }_3}\Vert {\zeta }_I\Vert {}^4$, $K_I$ satisfies ${\lambda }_{min}\left(K_I\right)>\frac{3}{2}+\frac{{\alpha }_3}{4{\alpha }_2},$ ${\epsilon }_{cM}=\frac{{\alpha }_3{\alpha }_I}{4}{\Vert \Lambda \Vert }^2{W}_{I,M}^4+\frac{{\Vert \Lambda \Vert }^2{\alpha }_2\Vert {\epsilon }_I\Vert {}^2}{4}$.

The derivation of $J_{bV}$ is governed by

$$\begin{gathered} {\dot{J}}_{bV}={\tilde{W}}_V^T\Pi {\dot{\tilde{W}}}_V \\ \le \Vert \Pi \Vert \left({\tilde{W}}_V^T{\alpha }_2\frac{\widehat{\omega }}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}{e}_c-{\alpha }_3{\tilde{W}}_V^T\frac{\widehat{\xi }}{{\left(1+{\widehat{\xi }}^T\widehat{\xi }\right)}^2}{e}_{t_f}-\frac{{\alpha }_4}{2}{\tilde{W}}_V^T\nabla h_x^T\left(x,t_f-t\right)\left({\widehat{D}}_1\left(x\right)+\widehat{k}\left(x\right)k^T\left(x\right)\right)Q\left(x,t\right)\right) \\ \le -\frac{{\alpha }_2\Vert \Pi \Vert }{8}\frac{{\tilde{W}}_V^T\widehat{\omega }{\widehat{\omega }}^T{\tilde{W}}_V}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}+\frac{{\alpha }_2{W}_{VM}^4{\lambda }_{max}^2\left(R^{-1}\right)\nabla h_{xM}^4{\sigma }_{I,M}^4}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}{\Vert {\tilde{W}}_I\Vert }^4 \\ -\frac{{\alpha }_4\Vert \Pi \Vert }{2}{\tilde{W}}_V^T\nabla h_x^T\left(x,t_f-t\right)\left({\widehat{D}}_1\left(x\right)-2{\gamma }^2\widehat{k}\left(x\right)k^T\left(x\right)\right)Q\left(x,t\right)+{\epsilon }_{VH} \end{gathered}$$

(44)

where

$$0<{\alpha }_2\le \frac{3{\alpha }_2{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2\left({\lambda }_{min}\left(\widehat{\xi }{\widehat{\xi }}^T\right)+\frac{1}{2}\right)}{{\lambda }_{max}^2\left(\nabla h_x\left(x,t_f-t\right)\right)\left({\widehat{D}}_1\left(x\right)-2{\gamma }^2\widehat{k}\left(x\right)k^T\left(x\right)\right)\nabla h_x^T\left(x,t_f-t\right){\left(1+{\widehat{\xi }}^T\widehat{\xi }\right)}^2}$$

and ${\epsilon }_{VH}=\frac{{\alpha }_2\Vert \Pi \Vert }{2}\frac{{\epsilon }_{HJB}^2}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}+3{\left(W_V^T\tilde{h}\left(x\left(t_f\right),0\right){\tilde{h}}^T\left(x\left(t_f\right),0\right)W_V+{\epsilon }_V^2\left(x,t_f\right)\right)}^2+\frac{15}{2}{\left({\alpha }_3\frac{{\epsilon }_V^2\left(x,t_f\right)}{{\left(1+{\widehat{\xi }}^T\widehat{\xi }\right)}^2}+\frac{W_V^T\tilde{h}\left(x\left(t_f\right),0\right){\tilde{h}}^T\left(x\left(t_f\right),0\right)W_V}{{\left(1+{\widehat{\xi }}^T\widehat{\xi }\right)}^2}\right)}^2+\frac{{\alpha }_2{W}_{VM}^4{\lambda }_{max}^4\left(R^{-1}\right)h_{xM}^4{\epsilon }_{IH}^4}{2\left({\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2\right)}$. ${\lambda }_{min}\left(R^{-1}\right)$ and ${\lambda }_{max}\left(R^{-1}\right)$ is the minimum and the maximum eigenvalue of the matrix $R^{-1}$.

Next, we have

$$\begin{gathered} {\dot{J}}_{cV}={\alpha }_4\left(V_t\left(x,t\right)+V_x\left(x,t\right)\left(f\left(x\right)+g\left(x\right)\widehat{u}+k\left(x\right)\widehat{w}\right)\right) \\ ={\alpha }_4(V_t\left(x,t\right)+V_x\left(x,t\right)\left(f\left(x\right)-\frac{1}{2}g\left(x\right)R^{-1}{\widehat{g}}^T\nabla h_x\left(x,t_f-t\right){\widehat{W}}_V+{\gamma }^{2}k\left(x\right){\widehat{k}}^T\left(x\right)\nabla h_x\left(x,t_f-t\right){\widehat{W}}_V\right) \end{gathered}$$

(45)

Thus,

$$\begin{gathered} \dot{J_V}={\dot{J}}_{aV}+{\dot{J}}_{bV}+{\dot{J}}_{cV}+{\dot{J}}_{dV} \\ \le {\alpha }_4\left(V_t\left(x,t\right)+V_x\left(x,t\right)\left(f\left(x\right)-\frac{1}{2}g\left(x\right)R^{-1}{\widehat{g}}^T\left(x\right)\right)\nabla h_x\left(x,t_f-t\right){\widehat{W}}_V+{\gamma }^{2}k\left(x\right){\widehat{k}}^T\left(x\right)\nabla h_x\left(x,t_f-t\right){\widehat{W}}_V\right) \\ +\frac{{\alpha }_2{W}_{VM}^4{\lambda }_{max}^4\left(R^{-1}\right)\nabla h_{xM}^4{\sigma }_{I,M}^4}{16{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}\Vert \Lambda \Vert {\Vert {\tilde{W}}_I\Vert }^4-\frac{{\alpha }_I}{2}\Vert \Xi \Vert {\Vert {\tilde{W}}_I\Vert }^2+\Vert \Xi \Vert {\epsilon }_{IM}-\frac{1}{2}({\alpha }_2\frac{{\widehat{\omega }}^T\widehat{\omega }}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2} \\ +{\alpha }_3\frac{{\widehat{\xi }}^T\widehat{\xi }}{{\left(1+{\widehat{\xi }}^T\widehat{\xi }\right)}^2})\Vert \Pi \Vert \Vert {\tilde{W}}_V\Vert {}^2-\frac{{\alpha }_4}{2}{\tilde{W}}_V^T\nabla h_x^T\left(x,t_f-t\right)\left({\widehat{D}}_1\left(x\right)-2{\gamma }^2\widehat{k}\left(x\right)k^T\left(x\right)\right)Q\left(x,t\right)+{\epsilon }_{VH} \\ -{\alpha }_2\left({\lambda }_{min}\left(K_I-\frac{3}{2}-\frac{{\alpha }_3\Vert {\zeta }_I\Vert {}^4}{8{\alpha }_2}\right){\Vert \Lambda \Vert }^2{\Vert \tilde{x}\Vert }^4+{\epsilon }_{IM}\right)-\frac{{\alpha }_3}{4}\left({\alpha }_I-\frac{3}{2}-\frac{{\alpha }_2}{{\alpha }_3}\Vert {\zeta }_I\Vert {}^4\right){\Vert \Lambda \Vert }^2\Vert {\tilde{W}}_I\Vert {}^4-\left({\lambda }_{min}\left(K_I\right)-\frac{1}{2}\right)\Vert \Xi \Vert {\Vert \tilde{x}\Vert }^2 \\ \le -\frac{{\alpha }_4}{5}{\Vert Q\left(x,t\right)\Vert }^2-\frac{5{\alpha }_4\left(W_{VM}^2\nabla h_{xM}^2+1\right)W_{IM}^2{\lambda }_{max}^2\left(R^{-1}\right){\sigma }_{I,M}^4}{16}\Vert {\tilde{W}}_I\Vert {}^2 \\ -\frac{{\alpha }_2}{4}\Vert \Lambda \Vert \frac{{\widehat{\omega }}^T\widehat{\omega }}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}\Vert {\tilde{W}}_V\Vert {}^2-\frac{{\alpha }_2}{8}\Vert \Lambda \Vert \frac{1}{{\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2}\Vert {\tilde{W}}_V\Vert {}^2+{\epsilon }_{TC} \end{gathered}$$

(46)

where

$$0<{\alpha }_4\le \min (\frac{5{\lambda }_{min}\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)+\frac{5}{2}}{32\nabla h_{xM}^6{Q}_{min}{W}_{VM}^4{\lambda }_{max}^2\left(R^{-1}\right){\left(1+{\widehat{\omega }}^T\widehat{\omega }\right)}^2},1)$$

$$\Xi =\frac{5{\alpha }_4\left(W_{VM}^2\nabla h_{xM}^2+1\right)W_{IM}^2{\sigma }_{I,M}^4{\lambda }_{max}^2\left(R^{-1}\right)}{4{\alpha }_I}I$$

$$\begin{gathered} \Pi =\frac{5}{8{\alpha }_2{W}_{VM}^4{\lambda }_{max}^2\left(R^{-1}\right)\nabla h_{xM}^4}I \\ {\epsilon }_{TC}={\epsilon }_{WM}+\Vert \Xi \Vert {\epsilon }_{IM}+{\epsilon }_{cM}+\frac{5{\alpha }_4}{4}\Vert {\epsilon }_{xM}\Vert {}^2 \end{gathered}$$

${\epsilon }_{IM}={\epsilon }_{fM}+{\epsilon }_{gM}{u}_M^{*}+{\epsilon }_{hM}{d}_M^{*}$. $u_M^{*}$ and $d_M^{*}$ are the upper bounds of the optimal guidance strategy $u^{*}$ and $d^{*}$.

Therefore, when the following conditions hold, the first derivation of $J_V$ is less than zero.

$$\begin{gathered} \Vert Q\left(x,t\right)\Vert >\sqrt{\frac{5{\epsilon }_{TC}}{{\alpha }_4}}=B_{Q_x} \mathrm{or} \Vert {\tilde{W}}_V\Vert >\sqrt{\frac{2{\epsilon }_{TC}}{{\alpha }_4\nabla h_x^2}}=B_{W_V} \mathrm{or} \Vert \tilde{x}\Vert >\sqrt{\frac{{\epsilon }_{TC}}{({\lambda }_{min}\left(K_I-\frac{1}{2}\right))\Vert \Xi \Vert }}=B_{\tilde{x}} \mathrm{or} \Vert {\tilde{W}}_I\Vert > \\ \sqrt{\frac{16{\epsilon }_{TC}}{5{\alpha }_4\left(W_{VM}^2\nabla h_x^2+1\right)W_{IM}^2{\lambda }_{max}^2\left(R^{-1}\right){\sigma }_{I,M}^2}}=B_{W_I} \end{gathered}$$

This completes the proof. □

Remark 5.

The eigenvalue K,${\alpha }_1$,${\alpha }_2$, ${\alpha }_3$, and${\alpha }_4$are the tuning parameters for guaranteeing the lower bound of $B_{Q_x}$,$B_{W_V}$, $B_{\tilde{x}}$and$B_{W_I}$, which can quantify the bound of the system. In additions, from the proof, we can observe that the estimated optimal guidance laws given in (34) can ensure the system is UUB. Thus, by combining the robust optimal feedback control (11), the complete nonlinear finite-time robust differential game guidance law is provided.

5. Application

A missile–target engagement scenario is considered in this section. The engagement geometry of the missile–target is shown in Figure 1, where the X-Y plane represents the Cartesian reference frame. The variables $V \mathrm{and} A$ denote the speed and the normal acceleration of the missile and the target, respectively. α and β denote the flight angles of the missile and the target, respectively. The variables r and θ represent the missile–target distance and the line of sight (LOS) angle, and LOS angular rate is $\dot{\theta }$ donated by $\sigma$. $u$ and $w$ are the control vectors perpendicular to the velocity of the missile, and the target.

The engagement occurs in the terminal guidance phase and all the participators are assumed to neglect the effect of gravity and have constant velocity. The nonlinear kinematics of the missile and the target, in a polar coordinate system, is given by

$$V_r=\dot{r}=V_{T}cos\left(\beta -\theta \right)-V_{M}cos\left(\alpha -\theta \right)$$

(47)

$$\sigma =\dot{\theta }=(V_{T}sin\left(\beta -\theta \right)-V_{M}sin\left(\alpha -\theta \right))/r$$

(48)

where $V_r$ represents the closing velocity.

The first-order dynamics of the missile are considered, and the motions of the missile are as follows:

$${\dot{x}}_M=V_{M}cos\alpha$$

(49)

$${\dot{y}}_M=V_{M}sin\alpha$$

(50)

$$\dot{\alpha }=\frac{a_M}{V_M}$$

(51)

$${\dot{a}}_M=\frac{u_M-a_M}{{\tau }_M}$$

(52)

where $\left(x_M,y_M\right)$ is the position of the missile along with the Cartesian reference frame. $a_M$ is the lateral acceleration of the missile and ${\tau }_M$ is a time constant.

Similarly, the motion equations of the target are as follows:

$${\dot{x}}_T=V_{T}cos\beta$$

(53)

$${\dot{y}}_T=V_{T}sin\beta$$

(54)

$$\dot{\beta }=\frac{a_T}{V_T}$$

(55)

$${\dot{a}}_T=\frac{u_T-a_T}{{\tau }_T}$$

(56)

where $\left(x_T,y_T\right)$ is the position of the target along with the Cartesian reference frame. $a_M$ is the lateral acceleration of the target and ${\tau }_T$ is a time constant.

To obtain the capture zone, the guidance principle is adopted as follows.

Definition 2.

Zero effort miss distance (ZEMD) is the closest distance between the missile and the target at an instant$t$, while the missile and the target do not impose any control and the agents continue to perform their scheduled maneuver strategy from the current time to the endgame. The ZEMD is computed as

$$r_{miss}\left(t\right)=\frac{r^2\sigma }{\sqrt{V_r^2+r^2{\sigma }^2}}$$

(57)

In this scenario, the missile applies its optimal strategy (34) to minimize the ZEMD, while the target applies its optimal strategy (34) to maximize the ZEMD. In addition, from (57), we can observe that if $\sigma$ tends to be zero, the ZEMD also tends to be zero. Meanwhile, the closing $V_r$ is also less than zero. Therefore, to ensure that the missile successfully intercepts the target, the following two conditions hold, which will be verified in simulations:

$$\sigma \to 0, V_r<0$$

(58)

Based on the two conditions mentioned above, by choosing the $x={\left[\theta \sigma \right]}^T$ as the system state, differentiating the Equation (48) with respect to time, the nonlinear system can be given by

$$\{\begin{array}{c}\dot{x_1}=x_2\\ \dot{x_2}=-\frac{2V_r}{r}{x}_2-\frac{cos\left(\beta -\theta \right)}{r}{u}_M+\frac{cos\left(\alpha -\theta \right)}{r}{w}_T\end{array}$$

(59)

Remark 6.

From (59), it can be observed that when$r$is close to 0, the nonlinear terms$f\left(x\right), g\left(x\right), and k\left(x\right)$tend to be infinite. It will mean that the system is broken and the proposed differential game guidance laws do not work. Thus, a minimized ZEMD ($r_{min}$) should be designed, which means that when the displacement between the missile and the target is less or equal to the$r_{min}=0.5 m$, the missile completes the interception mission. Moreover, in the practice application, the missile has an actual killing radius; thus, this design is reasonable.

Remark 7.

From (59), it also can be found that when$\left|\left(\beta -\theta \right)\right|=\pi /2$and$\left|\left(\alpha -\theta \right)\right|=0$, no matter what the proposed guidance laws change, the nonlinear system is an unstable equilibrium. Therefore, the domain where the differential game-based guidance laws are applicable is given by

$$\phi =\left\{x: \left|\left(\beta -\theta \right)\right|\ne \frac{\pi }{2},\left|\left(\alpha -\theta \right)\right|\ne 0,r\ne 0,V_r<0\right\}$$

(60)

6. Simulation Results

In this section, some experiments are designed to verify the proposed finite-time robust differential game guidance strategy. To further explain the performance of the proposed differential game guidance strategy based on NDP, some simulation experiments are designed and analyzed. The initial engagement occurs in the terminal guidance phase. The speeds of the missile and the target are V_M = 700 m/s, and V_T = 400 m/s, respectively. The initial position of the target is at (0 m, 0 m), and the position of the missile is (2500 m, 0 m). The initial flight path angles of the missile and the target are $\alpha =70°$ and $\beta =150°$, respectively. The time constants are ${\tau }_M=0.1 \mathrm{and} {\tau }_T=0.1$, respectively. The weight parameters $R_1=5$, $R_2=50,\gamma =10 \mathrm{and} t_f=7 \mathrm{s}$ are set. $Q_x=20{\left(x_1^2+x_2^2+{\tau }^2\right)}^2$, where $\tau =t_f-t$.

Furthermore, the initial parameters of the critic NN are selected inside $w_V\in \left[0 1\right]$ randomly and the critic NN identifier activation function is designed as ${\sigma }_{I\left(x\right)}={\left[1 x_1^2 x_1{x}_2 x_2^2 x_1^3 x_1^2{x}_2 x_1{x}_2^2 x_2^3 x_1^4 x_1^3{x}_2 x_1^2{x}_2^2 x_2 x_2^3 x_2^4\right]}^T$, which refers to [18,21]. In addition, polynomial basis functions can better approximate the nonlinear system. The initial parameters of the critic NN are $w_V={\left[10 10 10 10 10 10 10 10\right]}^T$ and the critic NN vector activation function for estimating the cost function is designed as

$$h\left(x,t_f-t\right)={\left[x_1^2\exp \left(-\tau \right) x_2^2\exp \left(-\tau \right) x_1{x}_2\tau x_1^4\exp \left(-\tau \right) x_2^2\exp \left(-\tau \right) x_1^3{x}_2 x_1^2{x}_2 x_1{x}_2^2\right]}^T.$$

The learning rates are ${\alpha }_1=0.01, {\alpha }_2=0.05, {\alpha }_3=0.55 \mathrm{and} , {\alpha }_4=0.15$, respectively. The experiment environments are performed on a PC platform with i7-9750H CPU, using Matlab 2020b.

6.1. Effect of the Proposed Finite-Time Differential Game Guidance Strategy without Unknown Uncertainties

In this case, the performance of the proposed finite-time differential game guidance strategy is verified without unknown uncertainties and the missile and the target choose the optimal guidance strategy (34). More importantly, the ZEMD and the lateral acceleration of the agents are the performance evaluation indicators and we have rewritten the physical meaning. The performance of the ZEMD tends to be zero, indicating that the missile intercepts the target successfully. The performance of the lateral accelerations of the agents is limited in the range ±100 g and stable change indicates that the agents can be reasonably controlled. The simulation results for this engagement scenario are shown in Figure 2, Figure 3 and Figure 4.

Figure 2a shows the trajectories of the missile and the target by selecting the proposed finite-time differential game guidance strategy (34). Figure 3b presents the change in the relative distance between the missile and the target and it can be observed that the ZEMD is less than 0.5 m. By bombining (a) and (b), the results reveal that the missile intercepts the target successfully.

Both the angular rate and the relative velocity are presented in Figure 3. From Figure 3a, it can be observed that the state variable (the angular rate) bound is zero at about 4.5 s, which leads the ZEMD to be zero according to the Equation (57) and Figure 2b. $\sigma$ tends to be zero, which can ensure that the missile intercepts the target. Furthermore, the negative $V_r$ guarantees the relative distance between the missile and the target tends to be zero in Figure 3b. Moreover, sharp changes between $\sigma$ and $V_r$ are reasonable due to the system characteristics. Furthermore, it also verifies Definition 2, which mentioned that $\sigma \to 0, \mathrm{and} V_r<0$.

Figure 4 presents lateral acceleration curves of the target and the missile. It can be observed that both lateral accelerations are maintained within a reasonable range, which can ensure that the missile intercepts the target. However, due to the system characteristics, the lateral accelerations decrease sharply at the end of the engagement.

The convergence curves of critic NN weights are presented in Figure 5 and it can be observed that the critic NN weights finally converge to stable coefficients. The result reveal that critic NN weights can guarantee the stability of the closed-loop nonlinear system.

6.2. Engagement without Unknown Uncertainties for a Maneuvering Target

To further verify the effectiveness of the proposed guidance strategy dealing with the target executing other forms of target maneuvers, the following experiment is presented. In this experiment, the target is expected to perform a sin-wave maneuver with a magnitude of 10 g and the missile still selects the guidance law (34). The simulation results for this engagement scenario are shown in Figure 6, Figure 7 and Figure 8.

Figure 6a shows the engagement trajectories of the missile confronting the target, which is expected to perform a sin-wave maneuver by selecting the proposed guidance law (34). Figure 6b presents the change in the relative distance between the missile and the target and it can be observed that the final ZEMD is less than 0.5 m. The results reveal that the missile intercepts the maneuvering target successfully.

Curves of the angular rate and the range rate are presented in Figure 7. The results reveal the same conclusion as case 1.

Figure 8 presents the lateral acceleration curves of the target and the missile. It can be observed that both lateral accelerations are maintained within a reasonable range, which can ensure that the missile intercepts the target. Furthermore, compared with Figure 4a, due to the maneuvering target, higher acceleration demands of the missile are needed.

6.3. Engagement with Unknown Uncertainties for a Maneuvering Target

To further verify how the proposed finite-time robust guidance strategy deals with unknown uncertainties, the following experiment is presented. In this experiment, the missile selects the robust differential game guidance law (11), and the target is expected to perform a sin-wave maneuver with a magnitude of 10 g. Furthermore, external disturbances, with a uniform distribution between −0.2 and 0.2, are considered in both input vectors. The simulation results for this engagement scenario are shown in Figure 9, Figure 10 and Figure 11.

Figure 9a shows the trajectories of the missile confronting the target, which is expected to perform a sin-ware maneuver by selecting the proposed guidance law. Figure 9b presents the change in the relative distance between the missile and the target and it can be observed that the miss distance is less than 0.5 m. By combining (a) and (b), the results reveal that the missile intercepts the maneuvering target successfully.

Curves of the angular rate and the range rate are presented in Figure 10. The results reveal the same conclusion as case 1.

Figure 11 presents the acceleration demands and the control requirements of the target and the missile. It can be observed that the lateral accelerations are maintained within a reasonable range, which can that ensure the missile intercepts the target. Furthermore, when the missile experiences external disturbances, the proposed robust differential game guidance strategy can successfully intercept maneuvering targets with external disturbances.

6.4. Effect of the Proposed Robust Optimal Differential Game Guidance Strategy with Other Methods

To confirm the advantage of the proposed robust optimal differential game guidance law, we offer a comparative experiment. In this experiment, the target selects the proposed robust optimal differential game guidance strategy (11), while the missile chooses the proposed differential game guidance strategy (11), i.e., the OGL in [4], and the conventional differential game guidance law (CDGGL) in [18], respectively. To further illustrate the advantages of the proposed differential game guidance strategy, the control effort J is defined as $J={{\displaystyle \int }}_0^{\infty }{u}^T\left(\tau \right)u\left(\tau \right)d\tau .$ The comparison results for this engagement scenario are shown in Figure 12.

The engagement trajectories and the control effort of the missile are presented in Figure 12a. It can be observed that the OGL, the CDGGL, and the proposed robust optimal differential game guidance strategy both can successfully intercept the target, with a miss distance equal to 0.985, 1.218, and 0.0024 m, respectively. Furthermore, the curves of the missile control effort are presented in Figure 12b. It can be shown that the minimal control effort is our proposed robust optimal differential game guidance strategy, the second control effort is the CDGGL in [18], and the maximum control effort is the OGL in [4]. All control efforts are limited to the setting range (100 g). More importantly, it also can be observed that before the missile intercepts the target, the control effect we proposed is always at its minimum at different times. For example, when t = 4.8 s, the control effort of our method is minimal. Thus, the missile uses our method to intercept the target, which saves more energy. Furthermore, the OGL in [4] is larger than the proposed robust optimal differential game guidance strategy, which means that the missile may not successfully intercept the target when using the OGL in some acceleration-limited scenes. In general, our proposed guidance law is superior to the OGL in [4] and the CDGGL in [18]. To further illustrate the superiority of the proposed robust optimal differential game guidance strategy, the energy consumption and the simulation time are compared in 100 average experiments. The comparison results are shown in

Table 1. Comparison results of OGL, CDGGL, and our method.

Method	Control Effort	Computation Time (t)	Number of Tests
OGL in [4]	980.56	1.647	100
CDGGL in [18]	570.64	1.083	100
Our method	218.845	0.451	100

.

From Table 1, it is easily found that less control effort is needed in our proposed method compared with the OGL in [4] and the CDGGL in [18], which implies that our proposed guidance law can reduce unnecessary energy consumption. The missile can intercept the target with minimal control. Moreover, the computation time of our proposed method is shorter compared with the OGL in [4] and the CDGGL in [18], which means that our proposed method can make the confrontation strategy in the shortest time. In general, our proposed guidance law can intercept targets with minimal control effort and a fast response.

7. Conclusions

In this paper, a finite-time robust differential game law for the nonlinear two-player zero-sum game is proposed for unknown system dynamics with external disturbance. The robustness and optimality of the guidance strategy are proven under a time-varying cost function. In order to solve the fixed terminal time constraint, the HJI equation is solved by a critic NN with time-varying activation functions, and extra weight tuning terms are introduced to guarantee the stability of the closed-loop system and the minimization of the HJI approximation error. The NN identifier estimates the nonlinear system with an online tuning law, which can be subsequently utilized in the finite-time guidance strategy design. The proposed scheme yields an online guidance strategy design that enjoys great practical benefits. Finally, in the missile–target engagement experiments, the missile with the proposed differential game guidance law successfully intercepts the target in different forms of maneuvers and all the results verify the theoretical claims.