Attitude-Tracking Control for Over-Actuated Tailless UAVs at Cruise Using Adaptive Dynamic Programming

Abstract

Using adaptive dynamic programming (ADP), this paper presents a novel attitude-tracking scheme for over-actuated tailless unmanned aerial vehicles (UAVs) that integrates control and control allocation while accounting for nonlinearity and nonaffine control inputs. The proposed method uses the idea of nonlinear dynamic inversion to create an augmented system and converts the optimal tracking problem into an optimal regulation problem using a discounted performance function. Drawing inspiration from incremental control, this method achieves optimal tracking control for the nonaffine system by simply using a critic-only structure. Moreover, the unique design of the performance function ensures robustness against model uncertainties and external disturbances. The ADP method was found to outperform traditional control architectures that separate control and control allocation, achieving the same level of attitude-tracking performance through a more optimized approach. Furthermore, unlike many recent optimal controllers for nonaffine systems, our method does not require any model identifiers and demonstrates robustness. The superiority of the ADP-based approach is verified through two simulated scenarios, and its internal mechanism is further discussed. The theoretical analysis of robustness and stability is also provided.

1. Introduction

The tailless unmanned aerial vehicle (UAV) has garnered immense attention due to its promising potential in both civil and military aviation. Its superior aerodynamic efficiency in comparison to traditional designs offers benefits such as improved voyage, carrying capacity, and stealth performance. This has led to the emergence of several tailless UAVs such as Boeing’s X-45 A, X-45B/C, Lockheed Martin’s RQ-170 Sentinel, BAE’s Taranis, and Dassault’s NEURON.

The Innovative Control Effector (ICE) aircraft [1,2], developed through research by Lockheed Martin, stands out among the latest tailless aerial vehicles. ICE is equipped with as many as 11 effectors relocated compactly on its main wing, which makes the system over-actuated. Such a unique layout was designed to investigate and measure the aerodynamics and performance of various low-observable tailless configurations using innovative control effectors. After decades of research, ICE has been found to have excellent maneuverability and stealth performance, making it a good choice for future UAV design.

The unique configuration of ICE allows its effectors to achieve goals beyond providing aerodynamic moments, such as minimizing drag or maximizing lift [3]. However, this configuration also poses challenges for control. Like other tailless vehicles, ICE suffers from problems such as poor static stability and coupling between longitudinal and lateral dynamics. Additionally, the redundant effectors of ICE require dealing with the control allocation problem, which involves selecting appropriate effectors and providing deflection commands to generate the required moments. However, the compact layout of ICE’s effectors results in strong coupling effects between them, causing the control inputs to appear nonlinear in the system, which means that the system is nonaffine. As a result, the control allocation of ICE is an extremely challenging task.

Since the proposal of the concept of ICE, researchers in the flight control field have been paying constant attention to it [4,5,6]. In 2017, Niestroy et al. [7] published detailed aerodynamic data for ICE, which enabled the construction of a highly precise control-oriented nonlinear model and the development of advanced control algorithms. Many researchers have developed different control algorithms for the nonlinear ICE model. In a recent study, He et al. [8] proposed an altitude tracker for ICE using the well-known decoupling conditions for nonaffine systems [9,10,11,12,13,14], while other researchers prefer incremental control methods.

The principle behind incremental control methods is timescale separation, which makes use of Taylor expansion. Incremental control methods can transform nonaffine systems into incremental affine forms. Therefore, the complexity of nonlinear optimization in CA can be avoided. This approach is highly effective for dealing with nonaffine systems [15]. Recently, there have been several advancements in incremental control for ICE. Stolk et al. [3] proposed a minimum drag CA method based on incremental nonlinear dynamic inversion. Matamoros [16] implemented an incremental nonlinear CA in ICE, resulting in improved tracking and CA performance. Sun et al. [17] improved the CA of ICE using hierarchical multi-objective optimization and adaptive incremental backstepping. Additionally, He et al. [14] extended the incremental control to the outer-loop control of ICE trajectory tracking using the pseudo-control hedging technique and relaxing the need for a timescale separation principle.

The reason incremental control is effective for nonaffine systems is it makes good use of the partial derivative of control inputs, $\frac{\partial f(x,u)}{\partial u}$. This is obtained through the digital differentiation of aerodynamic data. It is worth noting that obtaining the partial derivative of control inputs may be challenging in some systems. However, with the advancements in wind-tunnel tests, we can obtain more accurate and economic aerodynamic data for flight control. When combined with model identification techniques, incremental control has great potential for the future.

Most of the aforementioned control methods separate the command tracking and control allocation (CA) tasks. The command tracker provides the aerodynamic coefficients command ${\mathit{\tau }}_{\mathit{c}}$ to ensure that the reference signal ${\mathit{x}}_{\mathit{c}}$ is accurately tracked, while the CA determines the specific effector deflection $\mathit{u}$ based on the aerodynamic coefficients command and its objective function. This framework is highly convenient for incorporating established flight control algorithms, and the CA can be viewed as an optimization problem that can leverage the well-established optimization theory. As a result, this framework is preferred by most researchers.

However, there are some aspects of the above framework that could be improved. The obvious drawback is that the CA is designed to minimize the objective function consisting of the moments tracking error ${\mathit{\tau }}_{\mathit{e}}=\mathit{\tau }-{\mathit{\tau }}_{\mathit{c}}$ and the effector deflection $\mathit{u}$. From an input and output perspective, the moments tracking error is only an intermediate value, and what truly matters is the reference signal tracking error ${\mathit{x}}_{\mathit{e}}=\mathit{x}-{\mathit{x}}_{\mathit{c}}$. Therefore, the ideal objective function should take both ${\mathit{x}}_{\mathit{e}}$ and the effector deflection into consideration, instead of just ${\mathit{\tau }}_{\mathit{e}}$. Additionally, most existing flight control algorithms are based on Lyapunov theory and can only take into account the convergence of ${\mathit{x}}_{\mathit{e}}$. To make these algorithms compatible with the over-actuated UAV, the above framework must be adopted, and the second goal is left to the CA.

Meanwhile, the above framework takes two steps to give effector deflection, increasing the computational time. Hou et al. [18] introduced the recurrent neural network in CA and claimed that the recurrent neural network model could be solved in parallel to meet the real-time requirement. Still, this approach has only been validated in a linearized model, where the computational load is inherently small.

Hence, it is imperative to develop more reasonable and effective frameworks that abandon the meaningless intermediate values, therefore ensuring convergence of ${\mathit{x}}_{\mathit{e}}$ and achieving the second goal in one step. Optimal control is a promising option in this regard. Unlike the objective function of CA that only considers ${\mathit{\tau }}_{\mathit{e}}$ and the second goal, the performance function of optimal control can incorporate the tracking error and any other desired second goals. This means that the command tracking and CA can be described using a single equation. However, solving the nonlinear Hamilton–Jacobi–Bellman (HJB) equation remains a formidable challenge.

Adaptive dynamic programming(ADP) provides new ideas for solving the nonlinear HJB equation. ADP is a heuristic algorithm for solving optimal control. Compared with other heuristic algorithms, such as reinforcement learning, ADP is supported by optimal control theory, so it shows better convergence and stability and is more suitable for flight control. The first application of ADP in optimal control could be seen in a study by Werbos [19]. ADP’s basic idea is to use sampling data to drive a neural network to approximate the optimal value function. In this way, APD turns the backward-in-time dynamic programming process into a forward-in-time manner and greatly expands the application of optimal control. For theoretical studies of ADP, Wei and Liu [20] give the stability analysis of policy iterative APD, and the stability proof of value iterative ADP is given by Al-Tamimi and Lewis [21]. Moreover, the researchers also proposed different frameworks of ADP, such as heuristic dynamic programming [22], dual heuristic programming [23], and globalized dual heuristic programming [24]. These studies lay the foundation of ADP, and a more detailed review of recent studies on ADP can be found in the paper by Liu et al. [25].

Model identification is a commonly adopted technique in recent applications of ADP in practical systems [26,27]. Model identification is an effective method for enhancing the robustness of ADP, but it requires introducing an identifier network. Compared to basic ADP, which uses only a critical network to approximate the value function, the incorporation of additional networks significantly increases the computational complexity. Therefore, approaches to alleviate the computational burden, such as the event-trigger technique [28], are necessary for these methods.

However, it is often overlooked that the optimal control itself could be robust with an appropriate design of performance function. This idea is illustrated in a book by Lin [29], and the author systematically discusses how to handle disturbance and model uncertainty in an optimal control way. This way, the ADP could enjoy robustness while avoiding heavy computational burdens. However, the author also points out that it is still an open question how to apply similar approaches to a nonaffine system. Most ADPs are developed to address the optimal regulation problem, but for flight control, the optimal tracking control has a more practical use. With the development of the aviation industry, modern flight control is no longer satisfied with just ensuring flight stability. Many researchers [30,31,32,33,34] began to consider how to track the command signal optimally.

In the control field, optimal tracking control has attracted increased attention. One of the most common optimal tracking methods is the combination of feedforward control and feedbackward control [35,36,37,38,39,40]. The feedforward control is a traditional steady-state tracking controller to ensure the command reference signal is tracked. ADP is used in the feedbackward control to stabilize the transient error optimally. With the help of the traditional steady-state tracking controller, this optimal tracker shows good stability. Nevertheless, this optimal tracker is not suitable for ICE. Designing a traditional steady-state tracking controller for ICE has already been an arduous task, and the control allocation still needs to be considered in this process.

Some ADP-based optimal trackers do not rely on the feedforward control [41,42,43]. These studies applied a discounted performance function to ensure the boundness of the optimal value function in the infinite-time process and constructed an augmented system using the error dynamic and reference signal dynamic to transform the tracking problem into the regulation problem. However, the dynamic of the reference signal is unavailable in these methods, limiting the use of these ADPs.

From the other view, nonlinear dynamic inversion [44], as a tried-and-tested control method in the flight control area, constructs the dynamic of the desired signal using state value and command signal, which provides a new idea to overcome this drawback. Moreover, these ADPs also cannot address the nonaffine system, so they cannot be directly used for ICE.

To apply ADP in ICE, the nonaffine control input must be considered. Recent optimal trackers for nonaffine systems can be grouped into two types. One type is mainly for single-input systems, which decouple the nonaffine system into an affine system with model uncertainties [45,46,47], but it is not easy to extend such a method to the multi-input system. Using decoupling conditions, these methods show robustness, but neglecting model details also makes their optimization performance poor. The other type uses the other neural network, known as the actor network, to handle nonlinearity in control input and update the policy through gradient-base algorithm [48,49,50]. This method performs well but also needs more data and training, which undoubtedly increases the computational burden. Of course, some tricks commonly used in reinforcement learning [34] could also help improve the convergence rate of the method, but this also causes the lack of stability proofs.

Motivated by the aforementioned studies, this article proposes a critic-only ADP technique for the attitude tracking of ICE featured by nonaffine control inputs and redundant effectors. Through ADP, our approach integrates control and control allocation so that the same performance can be achieved at a cost less than conventional methods. By the idea of nonlinear dynamic inversion, an augmented system is constructed. The optimal tracking problem is transformed into an optimal regulation problem with discounted performance function, and the command dynamic is avoided. Inspired by the successful use of $\frac{\partial f(x,u)}{\partial u}$ in incremental control, we introduce $\frac{\partial f(x,u)}{\partial u}$ into APD, letting our method handle the nonaffine system in a simple way. Moreover, this article proves that for the control of the nonaffine system, the robust tracking problem could be equivalent to the optimal tracking problem with an augmented cost. This provides another way to improve the robustness of ADP, and complex model identification methods can be avoided.

The rest of the paper is arranged as follows: Section 2 introduces the aerodynamic model of the UAV. Section 3 gives the problem formulation and shows at the theoretical level that the optimal control with a specially designed performance function is equivalent to robust control. Section 4 presents the control scheme and stability analysis. Section 5 presents two simulations that validate the superiority of our method over the conventional approach and demonstrate its robustness, respectively. Finally, Section 6 gives the conclusion and the outlook for the next steps of research.

2. Model Description

This section introduces the ICE model. The basic parameters of ICE can be found in

Table 1. The basic parameters of ICE.

Parameter	Nomenclature	Value	Unit
b	Lateral–directional reference length, span	37.50	ft
$\overline{c}$	Mean aerodynamic chord	28.75	ft
m	Weight	32,750	LBF
$I_{yy}$	Pitch Moment of Inertia	78,451	$\mathrm{slug}·{\mathrm{ft}}^2$
$I_{xz}$	Cross Product of Inertia	−525	$\mathrm{slug}·{\mathrm{ft}}^2$
$I_{xx}$	Roll Moment of Inertia	35,479	$\mathrm{slug}·{\mathrm{ft}}^2$
$I_{zz}$	Yaw Moment of Inertia	110,627	$\mathrm{slug}·{\mathrm{ft}}^2$
S	Reference area	808.60	${\mathrm{ft}}^2$
$X_{th}$	Moment arm for thrust vectoring	18.75	ft
$X_{cg}$	Gravity center	38.84%$\overline{c}$
$X_{ac}$	Aerodynamic center	38.00%$\overline{c}$

[7], while more detailed information on the modeling of effectors can be found in Chapter 3 of [3]. Due to space constraints, this information will not be repeated here.

Figure 1 displays the layout of ICE, which features a high-sweep, tailless flying wing with a leading-edge sweep of 65 deg and 25 deg chevron shaping on the trailing edge. ICE is equipped with 13 independent effectors, including two pairs of leading-edge flaps (LEF), a pair of spoiler slot deflectors (SSD), a pair of all-moving tips (AMT), a pair of elevon (ELE), a pair of ganged pitch flaps (PF), and multi-axis thrust vectoring (MTV). Since this paper is focused on the cruising stage, MTV will not be taken into account.

The deflection ranges of the effectors are, inboard LEF: 0–40 deg, outboard LEF: ±40 deg, ELE: ±30 deg, PF: ±30 deg, AMT: ±60 deg, SSD: 0–60 deg. The rate limits on the leading-edge devices are 40 deg/s and on all the other surfaces 150 deg/s.

The modeling of the UAV is based on the following two assumptions: 1st, the UAV flies in the atmosphere, and the atmosphere is incompressible; 2nd, the UAV’s body is rigid. Please note that only the body of the UAV is considered a rigid body, but the effectors are deformable.

Remark 1.

In previous studies [8,44,51], the MTV was used solely during the UAV’s vigorous maneuvers or when other effectors were saturated. However, this paper proposes a cruise-oriented approach where ADP enables a superior trade-off between effector deflection and tracking error. This effectively eliminates the need for MTV, ensuring that effector saturation is avoided.

The motion equation of 6-DOF UAV model is given below [14,18,52], the nomenclature of the variables in the following equation can be found in

Table 2. Nomenclature of variables.

Parameter	Nomenclature	Unit
F	= sum of aerodynamic force and thrust	LBS
$l,m,n$	= aerodynamic rolling, pitching, and yaw moment	LBS·ft
$p,q,r$	= body-axis roll, pitch, and yaw rate	rad/s
V	= airspeed	ft/s
$\gamma$	= flight path angle	rad
$\chi$	= flight path sideslip angle	rad
$\alpha$	= angle of attack	rad
$\beta$	= sideslip angle	rad
$\mu$	= bank angle about the velocity vector	rad

.

$$\left[\begin{array}{c}\dot{V}\\ \dot{\chi }\\ \dot{\gamma }\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{M}& 0& 0\\ 0& \frac{1}{MV{c}_{\gamma }}& 0\\ 0& 0& \frac{-1}{MV}\end{array}\right]·\left[{\mathit{T}}_{\mathit{ve}}{\mathit{G}}_{\mathit{f}}+\mathit{F}\right]$$

(1)

where V, $\gamma$, and $\chi$ are airspeed, flight path angle, and ground tracking angle, respectively. $s_{★}$ and $c_{★}$ represent $sin★$ and $cos★$. $\mathit{F}$ donate the sum of aerodynamic force and thrust, which could be approximated through accelerometers, and ${\mathit{G}}_{\mathit{f}}={\left[00Mg\right]}^T$ represents gravitational forces. Define $\mu$, $\alpha$, and $\beta$ as the bank angle, angle of attack, and sideslip angle, then the dynamic of $\left[\mu \alpha \beta \right]$ is:

$$\left[\begin{array}{c}\dot{\mu }\\ \dot{\alpha }\\ \dot{\beta }\end{array}\right]={\left[\begin{array}{ccc}{c}_{\alpha }{c}_{\beta }& 0& s_{\alpha }\\ s_{\beta }& 1& 0\\ s_{\alpha }{c}_{\beta }& 0& -c_{\alpha }\end{array}\right]}^{-1}\left(-{\mathit{T}}_{bv}^T\left[\begin{array}{c}-\dot{\chi }{s}_{\gamma }\\ \dot{\gamma }\\ \dot{\chi }{c}_{\gamma }\end{array}\right]+\left[\begin{array}{c}p\\ q\\ r\end{array}\right]\right)$$

(2)

where $p,q,$ and r are the body-axis roll, pitch, and yaw rates. The expression ${\mathit{T}}_{\mathit{vb}}$ is the transformation matrix from the body frame to the velocity frame, and ${\mathit{T}}_{\mathit{ve}}$ is the transformation matrix from the earth frame to the velocity frame. These matrices are given in [53]:

$${\mathit{T}}_{vb}=\left[\begin{array}{ccc}{c}_{\alpha }{c}_{\beta }& s_{\beta }& s_{\alpha }{c}_{\beta }\\ -c_{\alpha }{s}_{\beta }{c}_{\mu }+s_{\alpha }{s}_{\mu }& c_{\beta }{c}_{\mu }& -s_{\alpha }{s}_{\beta }{c}_{\mu }-c_{\alpha }{s}_{\mu }\\ -c_{\alpha }{s}_{\beta }{s}_{\mu }-s_{\alpha }{c}_{\mu }& c_{\beta }{s}_{\mu }& -s_{\alpha }{s}_{\beta }{s}_{\mu }+c_{\alpha }{c}_{\mu }\end{array}\right]$$

(3)

$${\mathit{T}}_{ve}=\left[\begin{array}{ccc}{c}_{\chi }{c}_{\gamma }& s_{\chi }{c}_{\gamma }& s_{\gamma }\\ -s_{\chi }& c_{\chi }& 0\\ c_{\chi }{s}_{\gamma }& s_{\chi }{s}_{\gamma }& c_{\gamma }\end{array}\right]$$

(4)

The dynamic of $\left[pqr\right]$ is:

$$\left[\begin{array}{c}\dot{p}\\ \dot{q}\\ \dot{r}\end{array}\right]={\mathit{J}}^{-1}({\mathit{M}}_{\mathit{a}}-\left[\begin{array}{c}p\\ q\\ r\end{array}\right]\times \mathit{J}\left[\begin{array}{c}p\\ q\\ r\end{array}\right])$$

(5)

where $\mathit{J}$ is rotary inertia, defined as:

$$\mathit{J}=\left[\begin{array}{ccc}{I}_{xx}& 0& I_{xz}\\ 0& I_{yy}& 0\\ I_{zx}& 0& I_{zz}\end{array}\right]$$

(6)

and ${\mathit{M}}_{\mathit{a}}=\left[lmn\right]$ is the aerodynamic moment, defined by:

$${\mathit{M}}_{\mathit{a}}=\left[\begin{array}{c}l\\ m\\ n\end{array}\right]=\overline{q}S\left[\begin{array}{c}b·C_l\\ \overline{c}·C_m\\ b·C_n\end{array}\right]=\overline{q}S\left[\begin{array}{c}b·\left[(C_{l,base}(\alpha ,\beta ,V)+\sum _{i=1}^j{C}_{l,i}(\alpha ,\beta ,\mathit{\delta }))\right]\\ \overline{c}·\left[(C_{m,base}(\alpha ,\beta ,V)+\sum _{i=1}^j{C}_{m,i}(\alpha ,\beta ,\mathit{\delta }))\right]\\ b·\left[(C_{n,base}(\alpha ,\beta ,V)+\sum _{i=1}^j{C}_{n,i}(\alpha ,\beta ,\mathit{\delta }))\right]\end{array}\right]$$

(7)

where the $\overline{q}$ is the dynamic pressure, b is span, $\overline{c}$ is the mean aerodynamic chord, $\mathit{\delta }\in {\mathit{\delta }}_{\mathit{max}}$ represents the deflection of effectors, ${\mathit{\delta }}_{\mathit{max}}$ is the deflection range of effectors [3], $C_{·,base}(\alpha ,\beta ,V)$ and ${\sum }_{i=1}^j{C}_{·,i}(\alpha ,\beta ,\mathit{\delta })$ are the aerodynamic coefficients generated by the body and control surfaces.

The coordinate system involved in the above kinetic equations is shown in Figure 2. Equation (1) is defined in the tangent-plane coordinate system, which is aligned as a geographic system but has its origin fixed at a point of interest on the spheroid; Equation (2) is defined in the wind-axes system, and Equation (5) is defined in the body-fixed coordinate system. The relationship between the wind-axes system and the body-fixed coordinate system is shown in Figure 1. The origin of both is at the UAV’s center of gravity, but the X-axis of the body-fixed coordinate system points in the direction of the nose, and the X-axis of the wind-axes system points in the direction of the relative wind [53].

To facilitate the design of the attitude tracker, we first construct a control-oriented model that considers disturbances and model uncertainty caused by inaccurate aerodynamic parameters. Define ${\mathit{x}}_{\mathbf{1}}=\left[\mu \alpha \beta \right]$, ${\mathit{x}}_{\mathbf{2}}=\left[pqr\right]$, then

$$\left\{\begin{array}{cc}\hfill {\dot{\mathit{x}}}_{\mathbf{1}}& ={\mathit{f}}_{\mathbf{1}}\left({\mathit{x}}_{\mathbf{1}}\right)+{\mathit{g}}_{\mathbf{1}}\left({\mathit{x}}_{\mathbf{1}}\right){\mathit{x}}_{\mathbf{2}}\hfill \\ \hfill {\dot{\mathit{x}}}_{\mathbf{2}}& ={\mathit{f}}_{\mathbf{2}}\left({\mathit{x}}_{\mathbf{2}}\right)+{\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })+{\mathit{d}}_{\mathit{t}}\hfill \end{array}\right.$$

(8)

where

$${\mathit{f}}_{\mathbf{1}}\left({\mathit{x}}_{\mathbf{1}}\right)={\left[\begin{array}{ccc}{c}_{\alpha }{c}_{\beta }& 0& s_{\alpha }\\ s_{\beta }& 1& 0\\ s_{\alpha }{c}_{\beta }& 0& -c_{\alpha }\end{array}\right]}^{-1}·-{\mathit{T}}_{bv}^T\left[\begin{array}{c}-\dot{\chi }{s}_{\gamma }\\ \dot{\gamma }\\ \dot{\chi }{c}_{\gamma }\end{array}\right]$$

(9)

$$\begin{array}{c}{\mathit{g}}_{\mathbf{1}}\left({\mathit{x}}_{\mathbf{1}}\right)={\left[\begin{array}{ccc}{c}_{\alpha }{c}_{\beta }& 0& s_{\alpha }\\ s_{\beta }& 1& 0\\ s_{\alpha }{c}_{\beta }& 0& -c_{\alpha }\end{array}\right]}^{-1}\end{array}$$

(10)

$$\begin{array}{c}{\mathit{f}}_{\mathbf{2}}\left({\mathit{x}}_{\mathbf{2}}\right)={\mathit{J}}^{-1}{\mathit{x}}_{\mathbf{2}}\times \mathit{J}{\mathit{x}}_{\mathbf{2}}\end{array}$$

(11)

$$\begin{array}{c}{\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })={\mathit{J}}^{-1}{\mathit{M}}_{\mathit{a}}-{\mathit{g}}_{\mathbf{2},\mathit{e}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })\end{array}$$

(12)

$$\begin{array}{c}{\mathit{d}}_{\mathit{t}}={\mathit{g}}_{\mathbf{2},\mathit{e}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })+\mathit{d}\end{array}$$

(13)

where ${\mathit{d}}_{\mathit{t}}$ stand for total uncertainty, ${\mathit{g}}_{\mathbf{2},\mathit{e}}$ represents the control effectiveness that aerodynamic data fail to curve, and d represents the external disturbances.

3. Problem Formulation

The control structure is shown in Figure 3, donating ${·}^c$ as the command signal. The control system consists of two parts, attitude control using NDI and angular rate control using ADP. The attitude control is to give the proper angular command so that the UAV can track the attitude command, assuming that the derivative of the attitude command is known, and gives the angular rate command as [44]:

$${\mathit{x}}_{\mathbf{2}}^{\mathit{c}}={{\mathit{g}}_{\mathbf{1}}}^{-1}\left({\mathit{x}}_{\mathbf{1}}\right)·({\dot{\mathit{x}}}_{\mathbf{1}}^{\mathit{d}}-{\mathit{f}}_{\mathbf{1}}\left({\mathit{x}}_{\mathbf{1}}\right))$$

(14)

where ${\mathit{x}}_{\mathbf{2}}^{\mathit{c}}={[p_c,q_c,r_c]}^T$.

The goal of angular rate control is to give effectors’ deflection so that the angular rate command can be tracked optimally according to the performance function. First, we constructed an augmented system:

$$\begin{array}{c}{\dot{\mathit{x}}}_{\mathbf{2}}={\mathit{f}}_{\mathbf{2}}\left({\mathit{x}}_{\mathbf{2}}\right)+{\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })\end{array}$$

(15)

$$\begin{array}{c}{\dot{\mathit{x}}}_{\mathbf{2}}^{\mathit{d}}=\mathit{\lambda }({\mathit{x}}_{\mathbf{2}}^{\mathit{c}}-{\mathit{x}}_{\mathbf{2}})\end{array}$$

(16)

where $\mathit{\lambda }$ is positively defined, and ${\mathit{x}}_{\mathbf{2}}^{\mathit{d}}$ represents the desired angular rate.

Define $\mathit{X}\left(t\right)={[{{\mathit{x}}_{\mathbf{2}}}^T,{{\mathit{x}}_{\mathbf{2}}^{\mathit{d}}}^T]}^T$, and rewrite the augmented system in the compact form:

$$\begin{array}{c}\dot{\mathit{X}}=\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C}+\mathit{D}\end{array}$$

(17)

where $\mathit{F}\left(\mathit{X}\right)=\left[\begin{array}{c}{\mathit{f}}_{\mathbf{2}}\left({\mathit{x}}_{\mathbf{2}}\right)\\ -\mathit{\lambda }{\mathit{x}}_{\mathbf{2}}\end{array}\right]$, $\mathit{G}\left(\mathit{X}\right)=\left[\begin{array}{c}{\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })\\ {\mathbf{0}}_{\mathbf{3}\times \mathbf{1}}\end{array}\right]$, $\mathit{C}=\left[\begin{array}{c}{\mathbf{0}}_{\mathbf{3}\times \mathbf{1}}\\ \mathit{\lambda }{\mathit{x}}_{\mathbf{2}}^{\mathit{c}}\end{array}\right]$, $\mathit{D}=\left[\begin{array}{c}{\mathit{d}}_{\mathit{t}}\\ {\mathbf{0}}_{\mathbf{3}\times \mathbf{1}}\end{array}\right]$.

Remark 2.

From Equation (14), it could be known that it is difficult to obtain the dynamic of ${\mathit{x}}_{\mathbf{2}}^{\mathit{c}}$ because of its dependence on the second order derivative of ${\mathit{x}}_{\mathbf{1}}^{\mathit{d}}$. In many control methods, such as incremental control methods [14], ${\dot{\mathit{x}}}_{\mathbf{1}}^{\mathit{c}}$ is readily available, but ${\dot{\mathit{x}}}_{\mathbf{2}}^{\mathit{c}}$ is usually obtained through digital differentiation, which is not only sensitive to noise but also increases the computational complexity. Therefore, in ADP, the dynamic of ${\mathit{x}}_{\mathbf{2}}^{\mathit{d}}$ is constructed using NDI, and the use of ${\ddot{\mathit{x}}}_{\mathbf{1}}^{\mathit{d}}$ could be avoided in this way.

To facilitate the following analysis, the following assumptions are given:

Assumption 1.

The total model uncertainty is bounded, there exist ${\mathit{D}}_{\mathit{max}}>0$, let $\parallel \mathit{D}\parallel <{\mathit{D}}_{\mathit{max}}$.

Assumption 2.

$\frac{\partial {\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })}{\partial \mathit{\delta }}\in {\mathbb{R}}_{3\times 11}$ is line nonsingular, i.e., for $\parallel \mathit{D}\parallel <{\mathit{D}}_{\mathit{max}}$, there exist $|\Delta \mathit{\delta }|<\overline{\mathit{\delta }}$, $\overline{\mathit{\delta }}>0$ such that $\frac{\partial {\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })}{\partial \mathit{\delta }}\Delta \mathit{\delta }=\mathit{D}$.

Remark 3.

From Equation (8), it is obvious that the total model uncertainties can be equated to time-varying aerodynamic moment disturbances. To put it bluntly, Assumption 2 means that such disturbances are contained in the attainable moments set of ICE, and considering the redundant effectors that ICE is equipped with, its attainable moments set [54] has already been greatly expanded. From Equation (13), it also could be found that ${\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })$ could only reflect the aerodynamic characteristic of ICE to a certain degree. Except for the accuracy loss in the wind-tunnel test, the raw aerodynamic data also must be well-tailored to make it more suitable for flight control design, and the aerodynamic that ${\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })$ failed to reflect are seen as model uncertainties. Therefore, it is reasonable to say that ${\mathit{g}}_{\mathbf{2}}({\mathit{x}}_{\mathbf{1}},\mathit{\delta })$ has the properties stated in Assumption 2.

Specifically, the angular rate control is to ensure the ${\mathit{x}}_{\mathbf{2}}$ tracks the ${\mathit{x}}_{\mathbf{2}}^{\mathit{c}}$ by minimizing the following performance function. Consider that when the command signal does not converge to 0, the control input also tends not to be 0. To ensure the boundness of the performance function, the following discounted performance function is introduced:

$$J(\mathit{X},\mathit{\delta })={\int }_t^{\infty }{e}^{-\upsilon (\tau -t)}({\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{{\mathit{e}}_{\mathbf{2}}}^T\mathit{Q}{\mathit{e}}_{\mathbf{2}}+{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta })d\tau$$

(18)

where ${\mathit{e}}_{\mathbf{2}}={\mathit{x}}_{\mathbf{2}}-{\mathit{x}}_{\mathbf{2}}^{\mathit{d}}$, $\mathit{Q}$ and $\mathit{R}$ are positively defined matrices, $\upsilon >0$ is the discounted factor. Compared to the conventional performance function, the upper bound of model uncertainties $\overline{\mathit{\delta }}$ is introduced into the performance function.

Remark 4.

As a tried-and-tested control algorithm, NDI has been used in flight control for decades. In NDI, by feeding back the error between ${\mathit{x}}_{\mathbf{2}}$ and ${\mathit{x}}_{\mathbf{2}}^{\mathit{d}}$, the tracking of ${\mathit{x}}_{\mathbf{2}}^{\mathit{c}}$ could be achieved. Therefore, in the above performance function design, we also use the feedback of ${\mathit{x}}_{\mathbf{2}}^{\mathit{d}}$ instead of ${\mathit{x}}_{\mathbf{2}}^{\mathit{c}}$.

Since the exact value of model uncertainties is unavailable, only the optimal tracker for the following nominal system that excludes the model uncertainties can be obtained:

$$\begin{array}{c}\dot{\mathit{X}}=\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C}\end{array}$$

(19)

However, the optimal tracker designed for the nominal system using the performance function in Equation (18) is capable of handling model uncertainties. This point will be elaborated on later, and the optimal tracker for the nominal system is derived as follows:

Define ${\mathit{Q}}_{\mathit{T}}={[{\mathit{I}}_{\mathbf{3}\times \mathbf{3}},-{\mathit{I}}_{\mathbf{3}\times \mathbf{3}}]}^T\mathit{Q}[{\mathit{I}}_{\mathbf{3}\times \mathbf{3}},-{\mathit{I}}_{\mathbf{3}\times \mathbf{3}}]$, then the discounted performance function could be modified as:

$$J(\mathit{X},\mathit{\delta })={\int }_t^{\infty }{e}^{-\upsilon (\tau -t)}({\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}+{\mathit{\delta }}^{\mathit{T}}\mathit{R}\delta )d\tau$$

(20)

Then define the optimal value function:

$$V\left(\mathit{X}\right)=\min J(\mathit{X},\mathit{\delta })$$

(21)

Remark 5.

It is worth noting that the constraints imposed by the effector are not taken into account when solving the optimal control problem. However, it is possible to effectively prevent effector saturation by adjusting the weights in the performance function. This approach is widely adopted in the solution of optimization problems.

Differentiating Equation (20) and noting Equation (21) give the following Hamilton–Jacobi–Bellman (HJB) equation:

$$H(V,\mathit{\delta })\triangleq {\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}+{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }-\upsilon V+V_X^T(\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C})=0$$

(22)

where $V_X=\frac{\partial V}{\partial \mathit{X}}$. Applying stationarity condition $\partial H(V,\mathit{\delta })/\partial \mathit{\delta }=0$, then we have the optimal tracker:

$${\mathit{\delta }}^{*}=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{G}}_{\mathit{\delta }}^{\mathit{T}}{V}_X$$

(23)

where ${\mathit{G}}_{\mathit{\delta }}=\frac{\partial \mathit{G}(\mathit{X},\mathit{\delta })}{\partial \mathit{\delta }}$. If the HJB equation is solved, the optimal control can be obtained. Therefore, the main issue of this paper is to solve the HJB equation using ADP.

Remark 6.

From Equation (23), it can be found that the only difference between the proposed optimal control for multi-input nonaffine systems and traditional optimal control lies in the use of ${\mathit{G}}_{\mathit{\delta }}^{\mathit{T}}$, which is obtained through digital differentiation of aerodynamic data. In the theoretical study of the optimal control of nonaffine systems [49,50,55,56,57], the nonaffine part of the system is usually treated as completely unknown. Therefore, complex model identification methods are needed in these studies. However, for realistic control systems, even if the accurate analytical model is not available, the data-based model can be built by observing the input and output of the system. With the improvement of wind-tunnel tests for UAV modeling, the aerodynamic data we obtain is more accurate than ever. The aforementioned theoretical studies, however, have not made sufficient use of these data. This is undoubtedly a huge waste. In this paper, the wind-tunnel data are used to assist ADP. By introducing ${\mathit{D}}_{\mathit{max}}$ into the performance function, the ADP shows robustness against uncertainties in wind-tunnel data. Compared with online model identification, many more computational resources could be saved in this way.

The following lemma shows that with the performance function Equation (20), the optimal tracker in Equation (23) shows robustness against model uncertainties.

Lemma 1.

Assume that the optimal control ${\mathit{\delta }}^{*}$ of the nominal system (19) with performance function (20) exist, ${\mathit{\delta }}^{*}$ could also make the system with model uncertainties (17) asymptotically stable.

Proof. 

To facilitate the proof, an auxiliary system is proposed:

$$\left\{\begin{array}{cc}\hfill \dot{\mathit{X}}& =\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C}+\mathit{D}\hfill \\ \hfill \dot{a}& =-\frac{1}{2}\upsilon a\hfill \end{array}\right.$$

(24)

It is obvious that the subsystem $\dot{a}=-\frac{1}{2}\upsilon a$ is asymptotically stable. Therefore, as long as the auxiliary system is proven to be asymptotically stable, Lemma 1 holds. Consider the optimal value function in Equation (21), for all $\mathit{X}\ne 0$, $V>0$ and $V=0$ only when $\mathit{X}=0$. Therefore, choose the Lyapunov function as $V_R=a^{2}V$, and the time derivation of $V_R$ is:

$$\begin{array}{cc}\hfill {\dot{V}}_R& =a^2\dot{V}-a^2\upsilon V\hfill \\ & =a^2[V_X^T(\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C})+V_X^T\mathit{D}-\upsilon V]\hfill \end{array}$$

according to the HJB equation and stationarity condition, we have:

$$\begin{array}{cc}\hfill {\dot{V}}_R& =a^2[-{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}-{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}-{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }+\upsilon V-2{\mathit{\delta }}^T{\mathit{R}}^{\frac{1}{2}}·{\mathit{R}}^{\frac{1}{2}}{\mathit{G}}_{\mathit{\delta }}^{+}\mathit{D}-\upsilon V]\hfill \end{array}$$

where the ${\mathit{G}}_{\mathit{\delta }}^{+}$ is the Moore-Penrose inverse of ${\mathit{G}}_{\mathit{\delta }}$. According to Assumption 2 ${\mathit{G}}_{\mathit{\delta }}^{+}\mathit{D}=\Delta \mathit{\delta }$, such that:

$$\begin{array}{cc}\hfill {\dot{V}}_R& =a^2[-{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\Delta \mathit{\delta }}^T\mathit{R}\Delta \mathit{\delta }-{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}-{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }-2{\mathit{\delta }}^T{\mathit{R}}^{\frac{1}{2}}·{\mathit{R}}^{\frac{1}{2}}\Delta \mathit{\delta }-\Delta {\mathit{\delta }}^{\mathit{T}}\mathit{R}\Delta \mathit{\delta }]\hfill \\ & =a^2[-{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\Delta \mathit{\delta }}^T\mathit{R}\Delta \mathit{\delta }-{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}-{({\mathit{R}}^{\frac{1}{2}}\mathit{\delta }+{\mathit{R}}^{\frac{1}{2}}\Delta \mathit{\delta })}^T({\mathit{R}}^{\frac{1}{2}}\mathit{\delta }+{\mathit{R}}^{\frac{1}{2}}\Delta \mathit{\delta })]\hfill \end{array}$$

(25)

according to Assumption 2, $|\Delta \mathit{\delta }|<\overline{\mathit{\delta }}$ and $\mathit{R}$ is positively definite, $-{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\Delta \mathit{\delta }}^T\mathit{R}\Delta \mathit{\delta }<0$. Therefore, it is clear that ${\dot{V}}_R$ is negative definite, the auxiliary system is asymptotically stable, and Lemma 1 is proven.

4. Main Result

As mentioned above, the main issue of this paper is solving the HJB equation and obtaining the optimal value function. The following single-layer neural network(NN) is applied to approximate the optimal value function:

$$V\left(\mathit{X}\right)={\mathit{W}}^{\mathit{T}}\mathsf{\Psi }\left(\mathit{X}\right)+\varsigma$$

(26)

where $\mathsf{\Psi }\left(\mathit{X}\right)\in {\mathbb{R}}^l$ is the activation function vector, l is the number of neurons, $\varsigma$ is the approximation error, and the derivative to state of $V\left(\mathit{X}\right)$ is:

$$V_X=\nabla \mathsf{\Psi }\mathit{W}+\nabla \varsigma$$

(27)

where $\mathit{W}\in {\mathbb{R}}^l$ is unknown ideal weights, $\nabla \mathsf{\Psi }=\frac{\partial {\mathsf{\Psi }}^{\mathit{T}}}{\partial \mathit{X}}$, $\nabla \varsigma =\frac{\partial \varsigma }{\partial \mathit{X}}$.

Suppose that the optimal value function is continuous and is defined on a bounded closed interval, according to the Weierstrass approximation theorem [58]. As l increases, the optimal value function can be uniformly approximated by the NN with arbitrarily high preciseness, which means that $\varsigma$ and $\nabla \varsigma$ can be arbitrarily small. In practice, we use a critical NN to approximate the optimal value function:

$$\widehat{V}\left(\mathit{X}\right)={\widehat{\mathit{W}}}^{\mathit{T}}\mathsf{\Psi }\left(\mathit{X}\right)$$

(28)

where $\widehat{\mathit{W}}$ is the estimation of unknown weights $\mathit{W}$. Then the near-optimal control $\widehat{\mathit{\delta }}$ can be obtained:

$$\widehat{\mathit{\delta }}=-\frac{1}{2}{\mathit{R}}^{-1}{\mathit{G}}_{\mathit{\delta }}^{\mathit{T}}\nabla \mathsf{\Psi }\widehat{\mathit{W}}$$

(29)

In this paper, ADP updates the critical NN online using sampling data. As more sampling data ADP obtains, $\widehat{\mathit{W}}$ would approach $\mathit{W}$ gradually. The rest of this section contains two parts. The first part introduces the update law of critical NN. The second part is stability analysis, in which we will discuss why the $\widehat{\mathit{W}}$ could approach $\widehat{\mathit{W}}$ and why the system stays stable.

4.1. Update Law for Critical NN

The goal of the update scheme is to minimize $\tilde{\mathit{W}}=\mathit{W}-\widehat{\mathit{W}}$ the estimation error of unknown NN weights. To derive the update law, substitute Equations (26) and (27) into HJB Equation (22), we have:

$$\begin{array}{ccc}\hfill 0=H(V,\mathit{\delta })& =\hfill & {\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}+{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }-\upsilon ({\mathit{W}}^T\mathsf{\Psi }\left(\mathit{X}\right)+\varsigma )\hfill \\ & & +({\mathit{W}}^T\nabla {\mathsf{\Psi }}^{\mathit{T}}+\nabla {\varsigma }^T)(\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C})\hfill \\ & =\hfill & \Lambda +{\widehat{\mathit{W}}}^T\mathsf{{\rm Y}}+{\tilde{\mathit{W}}}^T\mathsf{{\rm Y}}+{\varsigma }_H\hfill \end{array}$$

(30)

where

$$\begin{array}{cc}\hfill \Lambda & ={\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}+{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}+{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }\hfill \end{array}$$

(31)

$$\begin{array}{cc}\hfill \mathsf{{\rm Y}}& =-\upsilon \mathsf{\Psi }\left(\mathit{X}\right)+\nabla {\mathsf{\Psi }}^{\mathit{T}}(\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C})\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill {\varsigma }_H& =-\upsilon \varsigma +\nabla {\varsigma }^T(\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C})\hfill \end{array}$$

(33)

From Equation (30), both the estimation error $\tilde{\mathit{W}}$ and the weights of critical NN $\widehat{\mathit{W}}$ appear linearly. As discussed in Section 2, ${\varsigma }_H$ is bounded and tends to be zero as l increases. Moreover, the rest of variables $\mathsf{{\rm Y}}$, $\Lambda$ are accessible. This allows us to design the update law that minimizes the estimation error. Therefore, we define the following filter:

$$\left\{\begin{array}{cc}\hfill \dot{\mathbf{P}}& =-k\mathbf{P}+\mathsf{{\rm Y}}{\mathsf{{\rm Y}}}^T,\mathbf{P}\left(0\right)=0\hfill \\ \hfill \dot{\mathbf{Q}}& =-k\mathbf{Q}+\mathsf{{\rm Y}}\Lambda ,\mathbf{Q}\left(0\right)=0\hfill \end{array}\right.$$

(34)

where $k\in {\mathbb{R}}^{+}$. The solution of the filter is:

$$\left\{\begin{array}{cc}\hfill \mathbf{P}& ={\int }_0^t{e}^{-k(t-\tau )}\mathsf{{\rm Y}}{\mathsf{{\rm Y}}}^{T}d\tau \hfill \\ \hfill \mathbf{Q}& ={\int }_0^t{e}^{-k(t-\tau )}\mathsf{{\rm Y}}\Lambda d\tau \hfill \end{array}\right.$$

(35)

According to Equations (30) and (35), we have:

$$0=\mathbf{Q}+\mathbf{P}\widehat{\mathit{W}}+\mathbf{P}\tilde{\mathit{W}}+\mathit{\mu }$$

(36)

where $\mathit{\mu }={\int }_0^t{e}^{-k(t-\tau )}\mathsf{{\rm Y}}{\varsigma }_{H}d\tau$. According to the above analysis, with sufficient neurons, $\varsigma \to 0$ and $\nabla \varsigma \to 0$. Therefore, it is reasonable to find that $\mathit{\mu }$ is bounded, i.e., $\left|\mathit{\mu }\right|<\overline{\mu }$, where $\overline{\mathit{\mu }}\in {\mathbb{R}}^{+}$.

Define an auxiliary vector $\mathbf{M}\in {\mathbb{R}}^l$:

$$\mathbf{M}=\mathbf{Q}+\mathbf{P}\widehat{\mathit{W}}$$

(37)

According to Equation (36), $\mathbf{M}$ could be seen as the estimation error of NN weights:

$$\mathbf{M}=-\mathbf{P}\tilde{\mathit{W}}-\mathit{\mu }$$

(38)

Hence, we could obtain the online update law of $\widehat{\mathit{W}}$ as:

$$\dot{\widehat{\mathit{W}}}=-\mathit{K}\mathbf{M}$$

(39)

where $\mathit{K}$ is constant positively defined matrix.

The update law in this paper is fundamentally different from some of the current methods [41,42] that employ the gradient-based algorithm to minimize the bellman error and only could ensure ultimately uniform boundlessness (UUB). The update law used here intends to use measurable state values and the weights of critical NN to represent the unknown estimation error of NN weights. In this way, the estimation error of NN weights can be ensured to be asymptotically convergent, and good convergence makes this update rate more suitable for online control systems. In what follows, we will show that $\widehat{\mathit{W}}$ could converge to the domain of $\mathit{W}$.

Remark 7.

The idea of this paper is to use the optimal tracker of nominal systems with modified performance functions to handle the systems with disturbances and model uncertainties. This idea can be well applied in linear systems [29] since the optimal control for linear nominal systems is easy to obtain by solving the Recatii equation. For the nonlinear system, however, it needs to solve the HJB equation to obtain the optimal control. For algorithms such as ADP that solve the HJB equation online, it gives the approximate solution of the HJB equation based on the sampled data. Still, for the systems that suffer from disturbance, it is impossible to measure the state value of the nominal system. According to Equations (31), (32) and (34), the information that update law uses include the quadratic function of effector deflection ${\mathit{\delta }}^T\mathit{R}\mathit{\delta }$, the nominal system dynamic $\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C}$, and the quadratic function of state value ${\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}$. The ${\mathit{\delta }}^T\mathit{R}\mathit{\delta }$ and $\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},\mathit{\delta })+\mathit{C}$ are directly accessible. Only the ${\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}$ is affected by disturbances and may influence the update law.

To address this problem, the filter system in Equation (34) is used here. Instead of using these values directly, the values used by the update law are processed by the filter system. In this way, the influence of disturbance on ${\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}$ could be mitigated to a certain extent and make the update law more applicable. To further illustrate this opinion, the following example is introduced:

For the system with disturbance:

$$\left\{\begin{array}{cc}\hfill {\dot{x}}_1& =-x_1+x_2+d\hfill \\ \hfill {\dot{x}}_2& =x_3\hfill \\ \hfill {\dot{x}}_3& =-x_2\hfill \end{array}\right.$$

(40)

where d is the disturbance.

Figure 4 shows the unfiltered value of $x_1^2$, and the value of $x_1^2$ filtered by $\frac{1}{0.8s+1}$. The real system is affected by disturbance, while the nominal system is not. From Figure 4a, there is a distinct difference between the unfiltered $x_1^2$ of the real system and the nominal system. But by filtering the two signals, the difference between the two becomes significantly smaller, as shown in Figure 4b. By choosing the parameters of the filter wisely, the filtered $x_1^2$ of the real system could approximate the $x_1^2$ of the nominal system pretty well.

It should be noted that since the filter used in this paper is only the simplest first-order low-pass filter, this method could achieve quite a good result when the frequency of disturbances is higher than the frequency of state. Considering that high-frequency disturbance is a common kind of disturbance in control systems, the applicability of this method is acceptable. To cope with more complex situations, more targeted filters can be used according to the knowledge of the disturbance.

4.2. Stability Analysis

To prove that our approach can solve the optimal control online while making the tracking error converge, the following stability analysis is performed. Accordingly, modify the nominal system (24) in the following closed-loop form:

$$\left\{\begin{array}{cc}\hfill \dot{\mathit{X}}& =\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},{\mathit{\delta }}^{*})+\mathit{C}+\mathit{G}(\mathit{X},\widehat{\mathit{\delta }})-\mathit{G}(\mathit{X},{\mathit{\delta }}^{*})\hfill \\ \hfill \dot{a}& =-\frac{1}{2}\upsilon a\hfill \end{array}\right.$$

(41)

Assumption 3.

There exist constants $\overline{f},\overline{g}\in {\mathbb{R}}^{+}$ such that $\parallel \mathit{F}\left(\mathit{X}\right)\parallel ⩽\overline{f}\parallel \mathit{X}\parallel$, $\parallel \mathit{G}(\mathit{X},\mathit{\delta })\parallel ⩽\overline{g}\parallel \mathit{X}\parallel$.

Assumption 4.

The $\mathit{G}(\mathit{X},\mathit{\delta })$ is Lipschitz continuous with respect to δ, i.e., $\parallel \mathit{G}(\mathit{X},{\mathit{\delta }}_{\mathbf{1}})-\mathit{G}$$(\mathit{X},{\mathit{\delta }}_{\mathbf{2}})\parallel ⩽L\parallel {\mathit{\delta }}_{\mathbf{1}}-{\mathit{\delta }}_{\mathbf{2}}\parallel$, where $L\in {\mathbb{R}}^{+}$.

Assumption 5.

The command signal is bounded, i.e., there exists a constant $\overline{c}\in {\mathbb{R}}^{+}$ such that $\parallel \mathit{C}\parallel ⩽\overline{c}$.

Assumption 6.

The activation function vector $\mathsf{\Psi }\left(\mathit{X}\right)$ and its derivative $\nabla \mathsf{\Psi }$ is bounded.

Lemma 2.

For the PE condition is satisfied for the regressor Υ of NN, the optimal control and the update law could stabilize the tracking error of the nominal system (19), and the near-optimal control converges to a bounded neighborhood around optimal control, i.e., ${\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}\mathit{X}⩽{\xi }_X$ and $\parallel \widehat{\mathit{\delta }}-{\mathit{\delta }}^{*}\parallel ⩽{\xi }_{\delta }$, where ${\xi }_X,{\xi }_{\delta }\in {\mathbb{R}}^{+}$.

Proof. 

The Lyapunov function is constructed as:

$$J=J_1+J_2$$

(42)

where $J_1=\frac{1}{2}{\tilde{\mathit{W}}}^{\mathit{T}}{\mathit{K}}^{-\mathbf{1}}\tilde{\mathit{W}}$, $J_2={\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}\mathit{X}+{\zeta }_2{a}^{2}V$.

According to the Lemma 1 in [37], if the PE condition is satisfied for the regressor $\mathsf{{\rm Y}}$ of NN, then $\mathbf{P}$ is positively defined, and since the inequality $2ab⩽\eta a^2+\frac{b^2}{\eta }$, the derivative of $J_1$ is:

$$\begin{array}{cc}\hfill {\dot{J}}_1& =-{\tilde{\mathit{W}}}^{\mathit{T}}\mathit{K}\tilde{\mathit{W}}+{\tilde{\mathit{W}}}^{\mathit{T}}\mathit{\mu }\hfill \\ & ⩽-({\lambda }_{min}\left(\mathbf{P}\right)-{\eta }_1)\parallel \tilde{W}{\parallel }^2+\frac{1}{{\eta }_1}\parallel \mathit{\mu }\parallel \hfill \\ & ⩽-({\lambda }_{min}\left(\mathbf{P}\right)-{\eta }_1){\parallel \tilde{W}\parallel }^2+\frac{1}{{\eta }_1}{\overline{\mu }}^2\hfill \end{array}$$

(43)

where ${\lambda }_{min}\left(\mathbf{P}\right)$ represents minimum eigenvalue of $\mathbf{P}$, and ${\eta }_1\in {\mathbb{R}}^{+}$. According to Equations (22) and (24), the derivative of $J_2$ is:

$$\begin{array}{ccc}\hfill {\dot{J}}_2& =\hfill & 2{\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}\dot{\mathit{X}}-\upsilon a^{2}V+a^2\dot{V}\hfill \\ & =\hfill & 2{\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}[\mathit{F}\left(\mathit{X}\right)+\mathit{G}(\mathit{X},{\mathit{\delta }}^{*})+\mathit{C}+\mathit{G}(\mathit{X},\widehat{\mathit{\delta }})-\mathit{G}(\mathit{X},{\mathit{\delta }}^{*})]-\upsilon a^{2}V\hfill \\ & & +a^2(-{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}-{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}-{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }+\upsilon V)\hfill \\ & =\hfill & 2{\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}\mathit{F}\left(\mathit{X}\right)+2{\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}\mathit{G}(\mathit{X},{\mathit{\delta }}^{*})+2{\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}\mathit{C}+2{\zeta }_1{\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}(\mathit{G}(\mathit{X},\widehat{\mathit{\delta }})-\mathit{G}(\mathit{X},{\mathit{\delta }}^{*}))\hfill \\ & & -{\zeta }_2{a}^2{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}-{\zeta }_2{a}^2{\mathit{X}}^T{\mathit{Q}}_{\mathit{T}}\mathit{X}-{\zeta }_2{a}^2{\mathit{\delta }}^{\mathit{T}}\mathit{R}\mathit{\delta }\hfill \\ & ⩽\hfill & {\zeta }_1({\eta }_2\parallel {\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}{\parallel }^2+\frac{1}{{\eta }_2}{\parallel \mathit{F}\left(\mathit{X}\right)\parallel }^2)+{\zeta }_1({\eta }_3\parallel {\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}{\parallel }^2+\frac{1}{{\eta }_3}\parallel \mathit{G}(\mathit{X},{\mathit{\delta }}^{*}){\parallel }^2)\hfill \\ & & +{\zeta }_1({\eta }_4\parallel {\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}{\parallel }^2+\frac{1}{{\eta }_4}{\parallel \mathit{C}\parallel }^2)\hfill \\ & & +{\zeta }_1({\eta }_5\parallel {\mathit{X}}^{\mathit{T}}{\mathit{Q}}_{\mathit{T}}{\parallel }^2+\frac{1}{{\eta }_5}\parallel (\mathit{G}(\mathit{X},{\mathit{\delta }}^{*})-\mathit{G}(\mathit{X},\widehat{\mathit{\delta }})){\parallel }^2)\hfill \\ & & -{\zeta }_2{a}^2{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}-{\zeta }_2{a}^2{\lambda }_{min}\left({\mathit{Q}}_{\mathit{T}}\right){\parallel \mathit{X}\parallel }^2-{\zeta }_2{a}^2{\lambda }_{min}\left(\mathit{R}\right){\parallel {\mathit{\delta }}^{*}\parallel }^2\hfill \end{array}$$

(44)

According to Assumptions 3–5, we have:

$$\begin{array}{ccc}\hfill {\dot{J}}_2& ⩽\hfill & [({\zeta }_1{\eta }_2+{\zeta }_1{\eta }_3+{\zeta }_1{\eta }_4+{\zeta }_1{\eta }_5){\lambda }_{max}^2\left({\mathit{Q}}_{\mathit{T}}\right)+\frac{{\zeta }_1}{{\eta }_2}{\overline{f}}^2+\frac{{\zeta }_1}{{\eta }_3}{\overline{g}}^2-{\zeta }_2{a}^2{\lambda }_{min}\left({\mathit{Q}}_{\mathit{T}}\right)]{\parallel \mathit{X}\parallel }^2\hfill \\ & & +\frac{{\zeta }_1}{{\eta }_5}{L}^2{b}_w\parallel \tilde{\mathit{W}}{\parallel }^2+\frac{{\zeta }_1}{{\eta }_5}{L}^2{b}_{\varsigma }{\parallel \nabla \varsigma \parallel }^2-{\zeta }_2{a}^2{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}-{\zeta }_2{a}^2{\lambda }_{min}\left(\mathit{R}\right){\parallel {\mathit{\delta }}^{*}\parallel }^2\hfill \end{array}$$

(45)

where $b_w=\parallel \frac{1}{2}{\mathit{R}}^{-1}{\mathit{G}}_{\mathit{\delta }}^{\mathit{T}}\nabla \mathsf{\Psi }\parallel$, $b_{\varsigma }=\parallel \frac{1}{2}{\mathit{R}}^{-1}{\mathit{G}}_{\mathit{\delta }}^{\mathit{T}}\parallel$ according to Assumptions 4 and 6, $b_w$ and $b_{\varsigma }$ are bounded.

Therefore, the derivative of J is:

$$\begin{array}{cc}\hfill \dot{J}& ={\dot{J}}_1+{\dot{J}}_2⩽-{\aleph }_1\parallel \mathit{X}\parallel -{\aleph }_2\parallel \tilde{\mathit{W}}\parallel -{\aleph }_3\parallel {\mathit{\delta }}^{*}\parallel +{\aleph }_4\hfill \end{array}$$

(46)

where

$$\begin{array}{cc}\hfill {\aleph }_1& =-({\zeta }_1{\eta }_2+{\zeta }_1{\eta }_3+{\zeta }_1{\eta }_4+{\zeta }_1{\eta }_5){\lambda }_{max}^2\left({\mathit{Q}}_{\mathit{T}}\right)-\frac{{\zeta }_1}{{\eta }_2}{\overline{f}}^2-\frac{{\zeta }_1}{{\eta }_3}{\overline{g}}^2+{\zeta }_2{a}^2{\lambda }_{min}\left({\mathit{Q}}_{\mathit{T}}\right)\hfill \\ \hfill {\aleph }_2& ={\lambda }_{min}\left(\mathbf{P}\right)-{\eta }_1-\frac{{\zeta }_1}{{\eta }_5}{L}^2{b}_w\hfill \\ \hfill {\aleph }_3& ={\zeta }_2{a}^2{\lambda }_{min}\left(\mathit{R}\right)\hfill \\ \hfill {\aleph }_4& =+\frac{1}{{\eta }_1}{\overline{\mu }}^2+\frac{{\zeta }_1}{{\eta }_5}{L}^2{b}_{\varsigma }{\parallel \nabla \varsigma \parallel }^2-{\zeta }_2{a}^2{\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}\hfill \end{array}$$

(47)

By designing the parameters wisely, it could be ensured that $B_1,B_2,B_3>0$. $B_4$ is mainly influenced by critical NN’s estimation error, which would converge to zero as the number of neurons increases.

According to Equation (47), if the following inequalities hold, ${\dot{J}}_2$ would be negative defined:

$$\begin{array}{c}\hfill -{\aleph }_1\parallel \mathit{X}\parallel +{\aleph }_4<0\\ \hfill -{\aleph }_2\parallel \tilde{\mathit{W}}\parallel +{\aleph }_4<0\\ \hfill -{\aleph }_3\parallel {\mathit{\delta }}^{*}\parallel +{\aleph }_4<0\end{array}$$

(48)

Therefore, according to Lyapunov theory, the closed-loop system is stable, and the weights error of NN $\tilde{\mathit{W}}$ is bounded, consider the estimation error of critical NN is also bounded, the near-optimal control will converge to a bounded neighborhood of optimal control. □

5. Simulation Verification

This section presents two representative simulations to illustrate the effectiveness of the ADP-based integrated-control-and-control-allocation scheme. The simulations are conducted using fixed-step ode4(Runge–Kutta) solver. The fixed-step size is 0.01 s. The block diagram of the 6-DOF UAV simulation model can be found in Figure 10 of the report given by Niestroy [7]. Simulation 1 compares our control scheme with conventional incremental dynamic inversion and pseudo-inverse control allocation(INDIPI) to verify the optimality of our method. Simulation 2 aims to test the robustness of the proposed control scheme. It is assumed that the leading-edge actuators are represented by the transfer function $\frac{\left(18\right)\left(100\right)}{\left(\right(s+18\left)\right(s+100\left)\right)}$ while all the other actuators, including thrust vectoring, as $\frac{\left(40\right)\left(100\right)}{\left(\right(s+40\left)\right(s+100\left)\right)}$.

The initial condition of the UAV is $V\left(0\right)=1240$ ft/s, $\chi \left(0\right)=0$, $\gamma \left(0\right)=0$, $\beta \left(0\right)=-0.0196$ deg, $\alpha \left(0\right)=3.759$ deg, $\mu \left(0\right)=0.304$ deg, $p\left(0\right)=0$, $q\left(0\right)=0$, $r\left(0\right)=0$, and the initial height is $H\left(0\right)=$ 10,000 ft. The angle of attack command ${\alpha }_c$ is generated by a band-limited white noise pass the second order filter $\frac{0.5}{5s^2+2s+0.5}$, the bank angle signal command ${\mu }_c=6\sin \left(0.5t\right)$, and the sideslip angle command ${\beta }_c=0$.

The control parameters are set as $\mathit{\lambda }=3·{\mathit{I}}_{\mathbf{3}\times \mathbf{3}}$, $\mathit{R}={\mathit{I}}_{\mathbf{11}\times \mathbf{11}}$, $\mathit{Q}=5·{\mathit{I}}_{\mathbf{3}\times \mathbf{3}}$, $\mathit{K}=500·{\mathit{I}}_{\mathbf{7}\times \mathbf{7}}$, $\upsilon =1$. Define $e_1=p-p_c$, $e_2=q-q_c$, $e_3=r-r_c$, then the activation function vector are designed as $\mathsf{\Psi }\left(\mathit{X}\right)={[e_1^2,e_2^2,e_3^2,e_1^2{e}_2,e_1{e}_2{e}_3,e_3^2{e}_2,e_2^2{e}_3]}^T$, with initial weights $\widehat{\mathit{W}}\left(0\right)={[1000,1000,1500,0,0,0,0]}^T$.

Remark 8.

We designed the activation function vector as above because it would be easier to find the initial admissible control policy [59]. Clearly, there exists $\frac{\partial e_1^2}{\partial p}=2e_1$. In this sense, the initial control is equivalent to proportional control. Compared with dealing with complex nonlinear feedback, finding an admissible proportional control law is much easier.

Model uncertainty exists in both simulations. As mentioned above, model uncertainty mainly comes from inaccurate aerodynamic data, which are used to obtain ${\mathit{G}}_{\mathit{\delta }}$. Since the aerodynamic data provided by Niestroy is a series of discrete points, it takes interpolation so that these data are of practical use. In both simulations, different interpolation methods are used for controller design and model construction to simulate that controller cannot access accurate aerodynamic data. Specifically, the cubic spline is applied for model construction, and linear interpolation is used in controller design.

Remark 9.

The cubic spline is applied for model construction because the actual aerodynamic data should be continuous and smooth. Meanwhile, using linear interpolation in control could save online computational load. As mentioned in [14], different interpolation methods can cause errors of up to 30%. Take the aerodynamic data of a set of all-moving wingtips as an example, as shown in Figure 5, from which it can be founded that the slopes of the tangents of linear interpolation and cubic spline, i.e., $tan{\tau }_2$ and $tan{\tau }_1$, are different.

5.1. Simulation 1

In this simulation, the ADP-based control scheme is compared with INDIPI [60]. Specifically, our approach and INDIPI are to track the same attitude command, and the performance of both is judged according to the tracking performance, flight quality, and control input. Considering that inaccurate control effectiveness could cause INDIPI to lose stability and this simulation is not to compare the robustness of INDIPI and ADP-based method, INDIPI could obtain accurate model information in this simulation, and model uncertainties only influence our method. Set $\overline{\mathit{\delta }}$ as 11-dimensional vector ${[5.6,5.6,\cdots ,5.6]}^T$. Please note that the same NDI scheme is applied in attitude control of the ADP-based method and INDIPI.

The result is shown in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16 and Figure 17. First, from Figure 17, ADP-based method shows good convergence. Moreover, the proposed method outperforms INDIPI in three ways: superior flight quality, intelligence, and better effector deflection pattern.

From Figure 9, Figure 10, Figure 11 and Figure 12, the flight quality under ADP control is better than that of INDIPI. From Figure 9, the p signal under ADP control is steadier. After the adjustment period before 5s, the p signal under ADP control keeps steady, while the p signal fluctuates under INDIPI control, such as around 38 s and 44 s. From Figure 10, q chattered all the time under the control of INDIPI, such chattering also can be found in effector deflection as shown in Figure 16, and this can cause fatigue of the effector, which is very dangerous in reality, while the effector deflection under ADP control is more fluent. From Figure 11, it can be found that there is a sudden change in r signal under INDIPI control at 20 s, 29 s, 36 s, and 44 s. It also could be seen that the r signal under ADP control also appears to fluctuate, but, differently from the sudden change under INDIPI control exhibited all the time, it can be found that the fluctuation under ADP control is becoming lighter.

According to the above description, the control performance of our method is better compared to INDIPI. However, our method goes far beyond that. With the help of ADP, our method shows intelligence, i.e., it could improve its policy according to its experience.

The specific manifestations of the ADP-based method’s intelligence are p signal under ADP control only fluctuates once around 5 s. After that, it is always very smooth, and, compared with the sudden change in r under INDIPI control exhibited all the time, the fluctuation of r signal under ADP control becomes lighter as the control system runs.

A more extreme example is introduced to further illustrate the intelligence of ADP, as shown in Figure 14, which shows the p signal under the proportional control that adopts the initial weights of critical NN. Comparing Figure 9 and Figure 14 it can be found that no matter whether under ADP control or proportional control, fluctuation occurred around 5 s. However, such fluctuation only occurs once under ADP control; for proportional control, such fluctuation occurs repeatedly and eventually leads to losing control. From Figure 17, it can be seen that the critical NN weights undergo a large adjustment at 5 s, after which there is no more fluctuation similar to at 5 s. ADP could learn from such fluctuation; therefore, the subsequent policy is more suitable for flight control with a broader flight envelope.

From Figure 15 and Figure 16, the effector deflection generated by INDIPI and ADP shows different patterns. In Figure 16, it can be seen that only three effectors participate in the process under INDIPI control, even though some effectors appear to be saturated. However, for the ADP, under the modulation of the performance function, more effectors participate in the control process, and the effector deflection amplitude is significantly smaller than that of the INDIPI. From Figure 13, it also can be seen that the weighted quadratic sum of effector deflection given by ADP is much smaller than INDIPI.

Overall, integrated-design ADP’s performance is better than conventional INDIPI’s. Compared with INDIPI, ADP allows for a trade-off between tracking performance and effector deflection. The performance function dominates such a trade-off, so ADP would not waste too many resources to pursue tiny improvements in tracking performance. Coupling with its learning mechanism, ADP can achieve the same tracking performance as INDIPI in an optimal manner.

5.2. Simulation 2

Simulation 2 discusses the robustness of the proposed method and the effect of $\overline{\mathit{\delta }}$. The UAV suffers from aforementioned model uncertainties and external disturbances $d={[0.06sin\left(20t\right),0.04cos\left(20t\right),0.03sin\left(20t\right)]}^T$. The UAV is to follow the same attitude command as Simulation 1 and set $\overline{\mathit{\delta }}$ as ${[17.8,17.8,\cdots ,17.8]}^T$, the result is shown in Figure 18, Figure 19, Figure 20, Figure 21, Figure 22, Figure 23, Figure 24 and Figure 25.

With the help of ADP, the tracking performance of our method in the presence of external disturbances is unaffected, as shown in Figure 18, Figure 19 and Figure 20. However, some small chattering can be observed in the angular rate signal, as depicted in Figure 21, Figure 22 and Figure 23, which is typical for a UAV subject to external disturbance. Nevertheless, this does not compromise the stability of the closed-loop system. Figure 24 demonstrates that more effectors are involved in controlling external disturbances. Most importantly, the convergence of critical NN weights remains satisfactory, as demonstrated by Figure 25.

From the stability analysis, it can be found that our method’s robustness comes from $\overline{\mathit{\delta }}$. In the following, the performance of different $\overline{\mathit{\delta }}$ is tested.

We begin by testing $\overline{\mathit{\delta }}={[5.6,5.6,\cdots ,5.6]}^T$. Due to space limitations, we only present the convergence of the critical NN weights in Figure 26. As shown in Figure 26, the convergence of the critical NN in this result is initially similar to Simulation 1. However, the critical NN weights do not ultimately converge due to external disturbances.

From a theoretical perspective, Lemma 1 can explain the non-convergence of the algorithm, as the closed-loop system’s stability can only be guaranteed when $\overline{\mathit{\delta }}$ is of sufficient magnitude.

From the other point of view, if the UAV experiences intense external disturbances, the initial sampled data may not provide enough information for ADP to update critical NN weights. This is particularly true when the UAV is required to track random commands, as the old critical neural network weights may not be equipped to handle new, unforeseen scenarios. As a result, the NN weights may take longer to converge, making it difficult to maintain control of the UAV. In this sense, ${\overline{\mathit{\delta }}}^T\mathit{R}\overline{\mathit{\delta }}$ not only acts as a means of compensation for external disturbances that may initially affect performance but can also be seen as an estimation of the potential impact of such disturbances on performance function. This helps the ADP better understand the current situation, allowing the weights of the critical NN to converge more quickly to a stable value.

Unlike the affine systems, where the upper bound on the effect of external disturbances on the performance function is easily ascertained [29], for the nonaffine system, the design of $\overline{\mathit{\delta }}$ is more rely on the experience. Still, considering that $\overline{\mathit{\delta }}$ has the actual physical definition, it would not be too hard to find a proper $\overline{\mathit{\delta }}$.

The above result shows that $\overline{\mathit{\delta }}$ must exceed the upper limit of external disturbance effects so that ADP can show robustness.

However, $\overline{\mathit{\delta }}$ should not exceed a reasonable value either. To illustrate this point, we conducted a convergence test of the critical neural network with $\overline{\mathit{\delta }}={[56.4,56.4,\cdots ,56.4]}^T$, which is very large. The result is depicted in Figure 27. It can be observed that the system experiences a significant shift in critical NN weights, leading to a collapse within 3 s. From the analysis of Equation (34), having a too large $\overline{\mathit{\delta }}$ can result in $\mathbf{P}$ not being positively definite. More bluntly, too large $\overline{\mathit{\delta }}$ could dominate the dynamics of critical neural network weights, causing the ignored of sampled data that aid in policy improvement.

In conclusion, Simulation 2 demonstrates that our method can effectively withstand model uncertainties and external disturbances, given the appropriate selection of $\overline{\mathit{\delta }}$.

6. Conclusions and Outlook

The proposed method uses ADP to integrate control and control allocation, resulting in superior performance compared to conventional methods. Without using any model identification techniques, the ADP-based method exhibits strong convergence and robustness in the face of external disturbance and model uncertainty. Additionally, it presents a novel approach to flight control for over-actuated UAVs with nonaffine control inputs. From a control-theory perspective, the paper presents a straightforward yet efficient optimal tracking method for nonaffine systems, with theoretical evidence verifying its robustness. Specifically, this study has two key advantages in comparison to existing research. First, our method achieves better performance than traditional control architectures that separate control and control allocation by using a more optimized approach. Second, unlike many current optimal controllers for nonaffine systems, our method remains robust and does not depend on any model identifiers.

The proposed method has certain limitations that require attention. First, this method is only aimed at the cruise stage, as the nonlinear characteristics of the aircraft during this phase are not as prominent, and the optimal value function is relatively simple and can be well-fitted by a polynomial network. However, if large maneuvering flight is required, a more complex network structure needs to be introduced. This inevitably requires an improvement in the weight update rate to ensure system stability. Second, selecting the initial value for the critical network can be challenging when a complex network is used since the convergence of this method relies on the proper choice of the network’s initial value. Thirdly, the design of the $\overline{\mathit{\delta }}$ is still heavily reliant on empirical knowledge. As demonstrated in the simulation section, a $\overline{\mathit{\delta }}$ that is too small may weaken robustness, while too large $\overline{\mathit{\delta }}$ may harm the closed-loop stability. Lastly, there is a dearth of real-world validation of this method. The external perturbations applied in the simulation offer only a limited exhibition of robustness and stability since the external interferences experienced by a UAV, in reality, are much more complex.

The next-step studies should focus on the following aspects: First, more complex neural networks can be introduced to further approximate the value function and handle more complex situations. Second, it would be very worthy work to introduce some intelligent algorithms to help design the performance function. Thirdly, only linear filters, as shown in Equation (34), are used in this paper, making our method better when facing high-frequency disturbances. In future studies, more advanced filters could be introduced to improve the performance of the ADP-based method when facing various disturbances. More importantly, it would be very expected that the performance of our method can be validated in real flight experiments.