Model-Free Gradient-Based Adaptive Learning Controller for an Unmanned Flexible Wing Aircraft

Abstract

Classical gradient-based approximate dynamic programming approaches provide reliable and fast solution platforms for various optimal control problems. However, their dependence on accurate modeling approaches poses a major concern, where the efficiency of the proposed solutions are severely degraded in the case of uncertain dynamical environments. Herein, a novel online adaptive learning framework is introduced to solve action-dependent dual heuristic dynamic programming problems. The approach does not depend on the dynamical models of the considered systems. Instead, it employs optimization principles to produce model-free control strategies. A policy iteration process is employed to solve the underlying Hamilton–Jacobi–Bellman equation using means of adaptive critics, where a layer of separate actor-critic neural networks is employed along with gradient descent adaptation rules. A Riccati development is introduced and shown to be equivalent to solving the underlying Hamilton–Jacobi–Bellman equation. The proposed approach is applied on the challenging weight shift control problem of a flexible wing aircraft. The continuous nonlinear deformation in the aircraft’s flexible wing leads to various aerodynamic variations at different trim speeds, which makes its auto-pilot control a complicated task. Series of numerical simulations were carried out to demonstrate the effectiveness of the suggested strategy.

1. Introduction

Various Approximate Dynamic Programming (ADP) methods have been employed to solve the optimal control problems for single and multi-agent systems [1,2,3,4,5,6]. They are divided into different classes according to the way the temporal difference equations and the associated optimal strategies are evaluated. The ADP approaches that consider gradient-based forms provide fast converging approaches, but they require the complete knowledge of the dynamical model of the system under consideration [7]. The solution of the flexible wing control problem requires model-free approaches, since the aerodynamics of the flexible wing aircraft are highly nonlinear and they variate continuously [8,9,10,11,12,13,14,15,16]. This type of aircraft has large uncertainties embedded in their aerodynamic models. Herein, an online adaptive learning approach, based on a gradient structure, is employed to solve the challenging control problem of flexible wing aircrafts. This approach does not need any of the aerodynamic information of the aircraft. It is based on a model-free control strategy approximation.

Several ADP approaches have been adopted to solve the difficulties associated with the dynamic programming solutions which involve the curse of dimensionality in the state and action spaces [2,3,4,5,17,18]. They are employed in different applications such as machine learning, autonomous systems, multi-agent systems, consensus and synchronization, and decision making problems [19,20,21]. Typical optimal control methods tend to solve the underlying Hamilton–Jacobi–Bellman (HJB) equation of the dynamical system by applying the optimality principles [22,23]. An optimal control problem is usually formulated as an optimization problem with a cost function that identifies the optimization objectives and a mathematical process to find the respective optimal strategies [6,7,18,22,23,24,25,26,27,28]. To implement the optimal control solutions stemming from the ADP approaches, numerous solving frameworks are considered based on combinations of Reinforcement Learning (RL) and adaptive critics [1,5,18,25,27]. Reinforcement Learning approaches use various forms of temporal difference equations to solve the optimization problems associated with the dynamical systems [1,18]. This implies finding ways to penalize or reward the attempted control strategies to optimize a certain objective function. This is accomplished in a dynamic learning environment where the agent applies its acquired knowledge to update its experience about the merit of using the attempted policies. RL methods implement the temporal difference solutions using two main coupled steps. The first approximates the value of a given strategy, while the second approximates the optimal strategy itself. The sequence of these coupled steps can be implemented with either value or policy iteration method [18]. RL has also been proposed to solve problems with multi-agent structures and objectives [29] as well as cooperative control problems using dynamic graphical games [21,26,30]. Action Dependent Dual Heuristic Dynamic Programming (ADDHP) depends on the system’s dynamic model [7,26,28]. Herein, the relation between the Hamiltonian and Bellman equation is used to solve for the governing costate expressions and hence a policy iteration process is proposed to find an optimal solution. Dual Heuristic Dynamic Programming (DHP) approaches for graphical games are developed in [21,26,30]. However, these approaches require in-advance knowledge of the system’s dynamics and, in some cases of the multi-agent systems, they rely on complicated costate structures to include the neighbors influences.

Adaptive critics are typically implemented within reinforcement learning solutions using neural network approximations [18,27]. The actor approximates the optimal strategy, while the value of the assessed strategy is approximated by the critic [18]. Real-time optimal control solutions using adaptive critics are introduced in [3]. Adaptive critics provide prominent solution frameworks for the adaptive dynamic programming problems [31]. They are employed to produce expert paradigms that can undergo learning processes while solving the underlying optimization challenges. Moreover, they have been invoked to solve a wide spectrum of optimal control problems in continuous and discrete-time domains, where actor-critic schemes are evoked within an Integral Reinforcement Learning context [32,33]. An action-dependent solving value function is proposed to play some zero-sum games in [34], where one critic and two actors are adapted forward in time to solve the game. An online distributed actor-critic scheme is suggested to implement a Dual Heuristic Dynamic Programming solution for the dynamic graphical games in [7,24] without overlooking the neighbors’ effects, which is a major concern in the classical DHP approaches. The solution provided by each agent is implemented by single actor-critic approximators. Another actor-critic development is applied to implement a partially-model-free adaptive control solution for a deterministic nonlinear system in [35]. A reduced solving value function approach employed an actor-critic scheme to solve the graphical games, where only partial knowledge about the system dynamics is necessary [26]. An actor-critic solution framework is adopted for an online policy iteration process with a weighted-derivative performance index form in [33]. A model-free optimal solution for graphical games is implemented using only one critic structure for each agent in [25]. The recent state-of-the-art adaptive critics implementations for numerous reinforcement learning solutions for the feedback control problems are surveyed in [36]. These involve the regulation and tracking problems for single- as well as multi-agent systems [36].

Flexible wing aircraft are usually modeled as two-mass systems (fuselage and wing). Both masses are coupled via different kinematic and dynamic constraints [8,13,14,15,37]. They involve the kinematic constraint at the connection point of the hang strap [38,39]. The keel tube works as a symmetric axis for this type of aircraft. The basic theoretical and experimental developments for the aerodynamic modeling aspects of the flexible wing systems are introduced in [8,13,14,15,40,41]. Several wind tunnel experiments have been introduced for the hang glider in [14]. An approximate modeling approach of the flexible wing’s aerodynamics led to equations of motion for the lateral and longitudinal directions with small perturbation models in [42]. The modeling process for the hang glider assumed a rigid wing modeling process, where the derivatives, due to the aerodynamics, were added at the last stage [11,12]. A comprehensive decoupled aerodynamic model for the hang glider is presented in [43]. A nine-degree-of-freedom aerodynamic model that employs a set of nonlinear state equations is developed in [38,39]. The control of the flexible wing aircraft follows a weight shift mechanism, where the lateral and longitudinal maneuvers or the roll/pitch control mechanism is achieved by changing the relative centers of gravity of the wing and the fuselage systems [9,10,13,14,37,44]. The geometry of the flexible wing’s control arm influences the maximum allowed control moments [9]. The reduced center of gravity magnifies the static pitch stability [9]. Frequency response-based approaches are adopted to study the stability of flexible wing systems in [11,12]. The longitudinal stability of a fixed wing system can be used to understand that of the flexible wing vehicle provided some conditions are satisfied [37]. The lateral stability margins are shown to be larger compared to conventional fixed wing aircraft.

The contribution of this work is four-fold:

• An online adaptive learning control approach is proposed to solve the challenging weight-shift control problem of flexible wing aircraft. The approach uses model-free control structures and gradient-based solving value functions. This serves as a model-free solution framework for the classical Action Dependent Dual Heuristic Dynamic Programming problems.
• The work handles many concerns associated with implementing value and policy iteration solutions for ADDHP problems, which either necessitate partial knowledge about the system dynamics or involve difficulties in the evaluations of the associated solving value functions.
• The relation between a modified form of Bellman equation and the Hamiltonian expression is developed to transfer the gradient-based solution framework from the Bellman optimality domain to an alternative domain that uses Hamilton–Jacobi–Bellman expressions. This duality allows for a straightforward solution setup for the considered ADDHP problem. This is supported by a Riccati development that is equivalent to solving the underlying Bellman optimality equation.
• The proposed solution that is based on the combined-costate structure is implemented using a novel policy iteration approach. This is followed by an actor-critic implementation that is free of the computational expensive matrix inverse calculations.

The paper is organized as follows: Section 2 briefly explains the weight shift control mechanism of a flexible wing aircraft. Section 3 highlights the model-based solutions within the framework of optimal control theory along with the existing challenges. Section 4 discusses the duality between the Hamiltonian function and Bellman equation leading to the Hamilton–Jacobi–Bellman formulation, which is used to generalize the Action Dependent Dual Heuristic Dynamic Programming solution with a policy iteration process. Section 5 introduces the model-free gradient-based solution and the underlying Riccati development. Section 6 demonstrates the adaptive critics implementations for the proposed model-free gradient-based solution. Section 7 tests the validity of the introduced online adaptive learning control approach by applying it on two case studies. Finally, the paper is concluded with some concluding remarks in Section 8.

2. Control Mechanism of a Flexible Wing Aircraft

This section briefly introduces the idea of weight shift control along with a basic aerodynamic model of a flexible wing system. Herein, a flexible wing aircraft is modeled as a two-mass system (fuselage/pilot and wing) coupled through nonlinear kinematic constraints at the hang strap, as shown in Figure 1. The flexible wing is connected to the fuselage through a control bar. The aerodynamic forces are controlled via a weight shift mechanism, where the fuselage’s center of gravity “floats” with respect to that of the wing [8,9,10,11,12,37,44]. Such a system is governed by complex aerodynamic forces which makes it difficult to model to a satisfactory accuracy. Consequently, model-based control approaches may not be appropriate for the auto-pilot control of such systems.

In this framework, the longitudinal and lateral motions are controlled through the force components applied on the control bar of the hang glider [38,39]. This development takes into account a nine-degree-of-freedom model that considers the kinematic interactions and the constraints between the fuselage and the wing at the hang point, as shown in Figure 1. The longitudinal and lateral dynamics are referred to the wing’s frame, where the forces (nonlinear state equations) at the hang point are substituted for by some transformations in the wing’s frame [39].

The decoupled longitudinal and lateral aerodynamic models satisfy the following assumptions [39]:

• The hang strap works as a kinematic constraint between the decoupled wing/fuselage systems.
• The fuselage system is assumed to be a rigid body connected to the wing system via a control triangle and a hang strap.
• The force components applied on the control bar are the input control signals.
• External forces, such as the aerodynamics and gravity, the associated moments, and the internal forces, are evaluated for both fuselage and wing systems.
• The fuselage’s pitch–roll–yaw attitudes and pitch–roll–yaw attitude rates are referred to the wing’s frame of motion through kinematic transformations.
• The complete aerodynamic model of the aircraft is reduced by substituting for the internal forces at the hang strap using the action/reaction laws.
• The pilot’s frames of motion (i.e., longitudinal and lateral states) are referred to the respective wing’s frames of motion.

The dynamics of the flexible wing aircraft are decoupled into longitudinal and lateral systems, such that [9,37,39,44].

$$\begin{array}{c}\hfill {\delta }_{(k+1)}^{Lo/La}=A^{Lo/La}{\delta }_k^{Lo/La}+B^{Lo/La}{u}_k^{Lo/La},\end{array}$$

(1)

where ${\delta }^{Lo}={\left[\begin{array}{cccccc}\hfill {\nu }_{aw}& {\nu }_{nw}& {\dot{\theta }}_w\hfill & {\dot{\theta }}_{fw}& {\theta }_{fw}& {\theta }_w\end{array}\right]}^T$ is the longitudinal state vector, ${\delta }^{La}={\left[\begin{array}{cccccccc}\hfill {\nu }_{lw}& {\dot{\varphi }}_w& {\dot{\psi }}_w\hfill & {\dot{\varphi }}_{fw}& {\dot{\psi }}_{fw}& {\varphi }_{fw}& {\psi }_{fw}& {\varphi }_w\end{array}\right]}^T$ is the lateral state vector, the force ${\displaystyle T_{cq}=\frac{1}{2}\left(T_{Rq}+T_{Lq}\right)}$ is the collective force in direction q, the force ${\displaystyle T_{dq}=\frac{1}{2}\left(T_{Rq}-T_{Lq}\right)}$ is the differential force in direction q, $u^{Lo}={\left[\begin{array}{cc}\hfill T_{cx}& T_{cz}\end{array}\right]}^T$ represents the longitudinal control signals, and $u^{La}={\left[\begin{array}{ccc}\hfill T_{cy}& T_{dx}& T_{dz}\hfill \end{array}\right]}^T$ denotes the lateral control signals.

The modeling results of the flexible wing aircraft are based on the experimental and theoretical studies of [9], where the control mechanism employs force components on the control bar [39].

3. Optimal Control Problem

This section explains the main challenges associated with the optimal solution of the control problem using Action Dependent Dual Heuristic Dynamic Programming approaches. It should justify the need for a model-free gradient-based optimal control solution.

3.1. Bellman Equation Formulation

Consider a flexible wing hang glider characterized by the following discrete-time state space equation:

$${\delta }_{k+1}=A{\delta }_k+B{u}_k,$$

(2)

where ${\delta }_k\in R^n$ is a vector of the (longitudinal/lateral) states and $u_k\in R^m$ is a vector of the (longitudinal/lateral) force components applied on the control bar, A and B are the (longitudinal/lateral) state space matrices, and k is the time index.

A quadratic convex performance index is introduced to assess the quality of the taken control actions, such that

$${\displaystyle J=\sum _{k=0}^{\infty }F({\delta }_k,u_k),}$$

(3)

where F is a convex utility function given by

$${\displaystyle F({\delta }_k,u_k)=\frac{1}{2}\left({\delta }_k^{T}Q{\delta }_k+u_k^{T}R{u}_k\right),}$$

(4)

where $Q\ge 0\in R^{n\times n}$ and $R>0\in R^{m\times m}$ are symmetric time-invariant positive semi-definite and positive definite weighting matrices, respectively.

The structure in Equation (3) is used to suggest a solution form. First, the solving value function $V({\delta }_k,u_k)$ is assumed to depend on the the state ${\delta }_k$ and the control strategy $u_k$ so that

$${\displaystyle V({\delta }_k,u_k)=\sum _{i=k}^{\infty }F({\delta }_i,u_i).}$$

(5)

This yields a temporal difference (Bellman) equation defined by

$${\displaystyle V({\delta }_k,u_k)=\frac{1}{2}\left({\delta }_k^{T}Q{\delta }_k+u_k^{T}R{u}_k\right)+V({\delta }_{k+1},u_{k+1}).}$$

(6)

The value function in Equation (5) is assumed to have the following form

$$\begin{array}{c}\hfill {\displaystyle V({\delta }_k,u_k)=\frac{1}{2}\left[{\delta }_k^{\mathrm{T}}{u}_k^{\mathrm{T}}\right]S\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right],}\end{array}$$

(7)

where $S=\left[\begin{array}{cc}{S}_{\delta \delta }\hfill & S_{\delta u}\hfill \\ S_{u\delta }\hfill & S_{uu}\hfill \end{array}\right]$.

3.2. Model-Based Policy Formulation

Herein, a model-based optimal control strategy and the associated costate equation are derived by applying the Bellman’s optimality principles to Bellman equation (Equation (6)). Below, a model-free policy solution is introduced. To evaluate the optimal control strategy, the optimality principles are applied to $V(\dots )$.

$$\begin{array}{c}{\displaystyle {\mathrm{argmin}}_{u_k}V({\delta }_k,u_k)=\frac{\partial V({\delta }_k,u_k)}{\partial u_k}=0}\\ \Rightarrow u^o=-R^{-1}{B}^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}V({\delta }_{k+1},u_{k+1}),\end{array}$$

(8)

where ${\nabla }_{\delta u_{k+1}}V({\delta }_{k+1},u_{k+1})=S·{\left[{\delta }_{k+1}^T{u}_{k+1}^T\right]}^T$. Applying this model-based optimal policy in Equation (6) yields the following Bellman’s optimality equation:

$$V^o\left({\delta }_k,u_k^o\right)=\frac{1}{2}\left({\delta }_k^{\mathrm{T}}Q{\delta }_k+u_k^{o\mathrm{T}}R{u}_k^o\right)+V^o\left({\delta }_{(k+1)},u_{(k+1)}^o\right).$$

(9)

The gradient-based solution requires the knowledge of the costate equation associated with the system in Equation (2). The costate equation is evaluated as follows

$${\nabla }_{{\delta }_k}V({\delta }_k,u_k)=Q{\delta }_k+A^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}V({\delta }_{k+1},u_{k+1}),$$

(10)

The main concern about this gradient-based development is that both the optimal strategy in Equation (8) and the associated costate Equation (10) depend on the dynamical model of the system (i.e., A and B). The following development shows how it is possible to avoid this shortcoming using dynamical information in deciding on the optimal control strategies.

3.3. Model-Free Policy Formulation

In the sequel, a model-free policy structure is introduced along with the optimal control solution algorithms. Applying the Bellman’s optimality principles [22] yields the optimal control strategy $u_k^o$ so that

$${\mathrm{argmin}}_{u_k}V({\delta }_k,u_k)={\mathrm{argmin}}_{u_k}\frac{1}{2}\left[{\delta }_k^{\mathrm{T}}{u}_k^{\mathrm{T}}\right]S\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right].$$

Note that the optimality principle is applied to the left-hand-side of Equation (6). This yields the following model-free control policy

$$u_k^o=K·{\delta }_k,$$

(11)

where the control gain K is given by $K=-S_{u_k{u}_k}^{-1}·S_{u_k{\delta }_k}$. Substituting Equation (11) into Equation (6) yields a dual (equivalent) Bellman’s optimality equation (Equation (9)). The Bellman optimality equation (Equation (9)) will be used to propose different Action Dependent Dual Heuristic Dynamic solutions for the optimal control problem in hand, as shown below.

To propose gradient-based solutions, the gradient of the Bellman equation (Equation (6)) with respect to the state ${\delta }_k$ is calculated.

$${\nabla }_{{\delta }_k}V({\delta }_k,u_k)=Q{\delta }_k+A^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ K\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}V({\delta }_{k+1},u_{k+1}),$$

(12)

where ${\nabla }_{{\delta }_k}V({\delta }_k,u_k)=\partial V({\delta }_k,u_k)/\partial {\delta }_k$ and ${\nabla }_{\delta u_k}V({\delta }_k,u_k)=S·{\left[{\delta }_k^T{u}_k^T\right]}^T$, $\forall k$.

The optimal strategy (Equation (11)) and the costate (Equation (12)) are used to propose different gradient-based solution forms. These are generalizations of the ADDHP solution, where a slight modification on the approximation of the control policy is introduced. In the sequel, solutions based on value iteration and policy iteration processes are presented.

Remark 1.

Although Algorithm 1 and 2 use model-free policy structures (Equations (14) and (16)), the gradient expressions (Equations (13) and (15)) depend on the system’s drift dynamics (matrix A), which is a real challenge for systems with uncertain or unknown dynamics. Moreover, it is difficult to evaluate the matrix S, and so V, in Equation (15) using the policy iteration process. As such, a new approach is required to benefit the gradient-based solution form without the need for a system’s dynamic model. To do that, a dual development using the Hamiltonian framework is needed.

Algorithm 1 Value Iteration Gradient-based Solution

Initialize ${\nabla }_{{\delta }_k}{V}^0({\delta }_k,u_k)$ and $u_k^o$. Evaluate ${\nabla }_{{\delta }_k}{V}^{\ell +1}(..)$ using $$\begin{array}{c}{\nabla }_{{\delta }_k}{V}^{\ell +1}({\delta }_k,u_k)=Q{\delta }_k^{\ell }+A^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}{K}^{\ell }\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_k}{V}^{\ell }({\delta }_{k+1},u_{k+1}),\end{array}$$ (13) where ℓ is the iteration index. Update the approximation of the optimal strategy using $$\begin{array}{c}\hfill u_k^{\ell +1}=-{\left[S_{u_k{u}_k}^{-1}·S_{u_k{\delta }_k}\right]}^{\ell +1}·{\delta }_k.\end{array}$$ (14) Halt on convergence of $\parallel S^{\ell +1}(..)-S^{\ell }(..)\parallel$.

Algorithm 2 Policy Iteration Gradient-based Solution

Initialize ${\nabla }_{{\delta }_k}{V}^0({\delta }_k,u_k)$ and use admissible $u_k^o$. Evaluate ${\nabla }_{{\delta }_k}{V}^{\ell }(..)$ using $$\begin{array}{c}{\nabla }_{{\delta }_k}{V}^{\ell }({\delta }_k,u_k)=Q{\delta }_k^{\ell }+A^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ K^{\ell }\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_k}{V}^{\ell }({\delta }_{k+1},u_{k+1}).\end{array}$$ (15) Update the approximation of the optimal strategy using $$\begin{array}{c}\hfill u_k^{\ell +1}=-{\left[S_{u_k{u}_k}^{-1}·S_{u_k{\delta }_k}\right]}^{\ell }·{\delta }_k.\end{array}$$ (16) Halt on convergence of $\parallel S^{\ell +1}(..)-S^{\ell }(..)\parallel$.

4. Hamiltonian-Jacobi–Bellman Formulation

The following Hamilton–Jacobi and Hamilton–Jacobi–Bellman developments are necessary to propose the model-free ADDHP control solutions. They find the relation between the costate variable of the Hamiltonian function and the solving value function through Bellman equation via a Hamilton–Jacobi framework. Then, the Hamilton–Jacob-Bellman development is used to propose the model-free ADDHP solution.

4.1. The Hamiltonian Mechanics

Optimal control problems, in general, are solved using the Hamiltonian mechanics, where the necessary conditions of optimality are found by means of Lagrange dynamics [22]. The objective of the optimization problem is to chose a policy ${\mu }_k$ to minimize a cost function F such that ${\mathrm{argmin}}_{{\mu }_k}F({\delta }_k,{\mu }_k)$, subject to the following constraints:

$$\begin{array}{c}{\mu }_k=\chi \left({\delta }_k\right)=C{\delta }_k,\\ {\delta }_{(k+1)}\equiv \varrho ({\delta }_k,{\mu }_k),\end{array}$$

(17)

where $\chi \in R^{m\times 1}$ and $\varrho \in R^{n\times 1}$ are some mapping functions, and $C\in R^{m\times n}$ is a row gain matrix.

The Hamiltonian expression for the problem is given by

$$\begin{array}{c}\hfill H({\delta }_k,{\lambda }_{(k+1)},{\mu }_k)={\lambda }_{(k+1)}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ {\mu }_{k+1}\hfill \end{array}\right]+F({\delta }_k,{\mu }_k),\end{array}$$

(18)

where ${\lambda }_k\in R^{(n+m)\times 1}$ is the Lagrange multiplier or the costate variable. Merging Equation (17) into Equation (18) leads to

$$\begin{array}{c}\hfill H({\delta }_k,{\lambda }_{(k+1)},{\mu }_k)={\lambda }_{(k+1)}^{\mathrm{T}}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]{\delta }_{k+1}+F({\delta }_k,{\mu }_k).\end{array}$$

(19)

Remark 2.

Similar to the optimal policy in Equation (8) derived using Bellman equation (Equation (6)), an optimal model-based control strategy based on the Hamiltonian can be obtained so that

$$\begin{array}{c}{\displaystyle {\mathrm{argmin}}_{{\mu }_k}H({\delta }_k,{\nabla }_{\delta {\mu }_{k+1}}V({\delta }_{k+1},{\mu }_{k+1}),{\mu }_k)=\frac{\partial H(\dots )}{\partial {\mu }_k}=0}\\ {\displaystyle \Rightarrow {\mu }^{*}=-R^{-1}{B}^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial {\mu }_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta {\mu }_{k+1}}V({\delta }_{k+1},{\mu }_{k+1}).}\end{array}$$

(20)

The following Hamilton–Jacobi theorem finds the relation between the costate variable ${\lambda }_k$ and the value function $V({\delta }_k,u_k)$, $\forall k$.

Theorem 1.

Let the Hamiltonian function be given by Equation (18) and the value function $V\left({\delta }_k,{\mu }_k\right)$ be defined by Equation (6). Then, this value function satisfies the following Hamilton–Jacobi equation:

$${\displaystyle V\left({\delta }_{k+1},{\mu }_{k+1}\right)-V\left({\delta }_k,{\mu }_k\right)+H\left({\delta }_k,{\nabla }_{\delta {\mu }_{k+1}}V({\delta }_{k+1},{\mu }_{k+1}),{\mu }_k\right)-{\nabla }_{\delta {\mu }_{k+1}}V{({\delta }_{k+1},{\mu }_{k+1})}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ {\mu }_{k+1}\hfill \end{array}\right]=0,}$$

(21)

where ${\lambda }_{k+1}={\nabla }_{\delta {\mu }_{k+1}}V({\delta }_{k+1},{\mu }_{k+1})$.

Proof. 

The augmented value function $V({\delta }_k,{\mu }_k)$ is

$$V\left({\delta }_k,{\mu }_k\right)=\sum _{l=k}^{\infty }\left\{F\left({\delta }_k,{\mu }_k\right)+{\lambda }_{(l+1)}^{\mathrm{T}}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]\left[\varrho ({\delta }_{\ell },{\mu }_{\ell })-{\delta }_{(l+1)}\right]\right\}.$$

(22)

The Hamiltonian in Equation (18) is rearranged such that

$$H({\delta }_{\ell },{\lambda }_{(\ell +1)},{\mu }_{\ell })={\lambda }_{(\ell +1)}^{\mathrm{T}}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]{\delta }_{\ell +1}+F({\delta }_{\ell },{\mu }_{\ell }).$$

Using this expression into the augmented function in Equation (22) yields

$$\begin{array}{c}\hfill V\left({\delta }_{k+1},{\mu }_{k+1}\right)-V\left({\delta }_k,{\mu }_k\right)+H\left({\delta }_k,{\lambda }_{(k+1)},{\mu }_k\right)-{\lambda }_{(k+1)}^{\mathrm{T}}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]{\delta }_{(k+1)}=0.\end{array}$$

(23)

Finding the gradient of Equation (23) with respect to ${\delta }_{(k+1)}$ yields

$${\displaystyle {\nabla }_{{\delta }_{k+1}}V\left({\delta }_{k+1},{\mu }_{k+1}\right)+{\left(\frac{\partial {\lambda }_{k+1}}{\partial {\delta }_{k+1}}\right)}^{\mathrm{T}}\left(\frac{\partial H\left({\delta }_k,{\lambda }_{k+1},{\mu }_k\right)}{\partial {\lambda }_{k+1}}\right)-\left({\left(\frac{\partial {\lambda }_{k+1}}{\partial {\delta }_{k+1}}\right)}^{\mathrm{T}}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]{\delta }_{k+1}+{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]}^{\mathrm{T}}{\lambda }_{k+1}\right)=0.}$$

This equation can be rearranged such that

$${\nabla }_{{\delta }_{k+1}}V\left({\delta }_{k+1},{\mu }_{k+1}\right)-{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]}^{\mathrm{T}}{\lambda }_{k+1}+{\left(\frac{\partial {\lambda }_{k+1}}{\partial {\delta }_{k+1}}\right)}^{\mathrm{T}}\left(\frac{\partial H\left({\delta }_k,{\lambda }_{k+1},{\mu }_k\right)}{\partial {\lambda }_{k+1}}-\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]{\delta }_{k+1}\right)=0.$$

$$\frac{\partial H\left({\delta }_k,{\lambda }_{k+1},{\mu }_k\right)}{\partial {\lambda }_{k+1}}=\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]{\delta }_{k+1}\Rightarrow {\nabla }_{{\delta }_{k+1}}V\left({\delta }_{k+1},{\mu }_{k+1}\right)={\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]}^{\mathrm{T}}{\lambda }_{k+1}.$$

This expression is equivalent to

$$\left[I_{n\times n}{C}^{\mathrm{T}}\right]S\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ {\mu }_{k+1}\hfill \end{array}\right]={\left[\begin{array}{c}{I}_{n\times n}\hfill \\ C\hfill \end{array}\right]}^{\mathrm{T}}{\lambda }_{k+1},$$

which yields

$${\lambda }_{k+1}=S\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ {\mu }_{k+1}\hfill \end{array}\right].$$

(24)

Then, the costate variable ${\lambda }_{k+1}$ can be written in terms of the gradient of the value function ${\nabla }_{\delta u_{k+1}}V(\dots )$, such that

$${\lambda }_{k+1}={\nabla }_{\delta {\mu }_{k+1}}V({\delta }_{k+1},{\mu }_{k+1})=\left[\begin{array}{c}{\displaystyle \frac{\partial V({\delta }_{k+1},{\mu }_{k+1})}{{\partial \delta }_{k+1}}}\hfill \\ {\displaystyle \frac{\partial V({\delta }_{k+1},{\mu }_{k+1})}{{\partial \mu }_{k+1}}}\hfill \end{array}\right].$$

Therefore, the value function $V(\dots )$ satisfies the HJ equation (Equation (21)). □

4.2. Hamiltonian–Bellman Solutions Duality

The following results show the conditions at which the Hamiltonian and Bellman-based solutions are dual.

Theorem 2.

(a) Let $\widehat{V}\left(\dots \right)$ satisfy the following Hamilton–Jacobi–Bellman equation

$$\begin{array}{c}H\left({\delta }_k,{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1}^o),u_k^o\right)=0\hfill \end{array}$$

(25)

$$\begin{array}{c}\Rightarrow {\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1}^o)}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}^o\hfill \end{array}\right]+F({\delta }_k,u_k^o)=0,\hfill \end{array}$$

(26)

with $\widehat{V}\left(0\right)=0$, where

$$u^o=-R^{-1}{B}^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1}).$$

(27)

Then, $\widehat{V}(\dots )$ satisfies the Bellman optimality (Equation (9)).

(b) Let $(A,B)$ be reachable. If $V^{*}(\dots )$ satisfies Equation (9), then it satisfies the Hamilton–Jacobi–Bellman Equation (25).

Proof. 

(a) The value function $\widehat{V}\left({\delta }_k\right)$ with the optimal policy $u^o$ (Equation (27)) satisfies the HJB equation (Equation (25)). Then, Theorem 1 yields

$$\widehat{V}\left({\delta }_{k+1},u_{k+1}^o\right)-\widehat{V}\left({\delta }_k,u_k^o\right)=-{\nabla }_{\delta u_{k+1}^o}\widehat{V}{({\delta }_{k+1},u_{k+1}^o)}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}^o\hfill \end{array}\right].$$

Therefore, $\widehat{V}(\dots )$ satisfies Equation (9).

(b) The Hamiltonian with the value function $\widehat{V}(\dots )$, arbitrary policy $u_k$, and optimal policy $u_k^o$, evaluated using the optimal value function $\widehat{V}(\dots )$ yields

$$\begin{array}{c}H\left({\delta }_k,{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1}),u_k\right)=H\left({\delta }_k,{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1}^o),u_k^o\right)+{\displaystyle \frac{1}{2}}{(u_k-u_k^o)}^{\mathrm{T}}R(u_k-u_k^o)\\ +{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1})}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}\hfill \end{array}\right]-{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1}^o)}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}^o\hfill \end{array}\right].\end{array}$$

$H\left({\delta }_k,{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_k^o),u_{k+1}^o\right)=0$. Then,

$$\begin{array}{c}H\left({\delta }_k,{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1}),u_k\right)={\displaystyle \frac{1}{2}}{(u_k-u_k^o)}^{\mathrm{T}}R(u_k-u_k^o)+{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1})}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}\hfill \end{array}\right]\\ -{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1}^o)}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}^o\hfill \end{array}\right].\end{array}$$

(28)

Bellman equation (Equation (6)) can be rearranged such that

$$\begin{array}{c}V({\delta }_k,u_k)=\frac{1}{2}\left({\delta }_k^{T}Q{\delta }_k+u_k^{T}R{u}_k\right)+V({\delta }_{k+1},u_{k+1})+{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1})}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}\hfill \end{array}\right]\\ -{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1})}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}\hfill \end{array}\right].\end{array}$$

(29)

Equations (28) and (29), and the results from Theorem 1, yield

$$V({\delta }_k,u_k)=V({\delta }_{k+1},u_{k+1})+{\displaystyle \frac{1}{2}}{(u_k-u_k^o)}^{\mathrm{T}}R(u_k-u_k^o)-{\nabla }_{\delta u_{k+1}}\widehat{V}{({\delta }_{k+1},u_{k+1}^o)}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ u_{k+1}^o\hfill \end{array}\right].$$

Applying the optimality principles (i.e., taking the derivative of $V(\dots )$ with respect to $u_k$) leads to the optimal value function $V^{*}(\dots )$ and the respective optimal policy $u_k^{*}$.

$$\begin{array}{c}{\displaystyle \frac{\partial V({\delta }_k,u_k)}{\partial u_k}=0\Rightarrow u_k^{*}-u_k^o=-R^{-1}{B}^{\mathrm{T}}\times }\\ \left({\left|{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}{V}^{*}({\delta }_{k+1},u_{k+1})\right|}_{u=u^{*}}-{\left|{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1})\right|}_{u=u^o}\right).\end{array}$$

(30)

The Hessian of the Hamiltonian (as a function of ${\delta }_k$, $\widehat{V}(\dots )$ and $u_k$) and the Hessian of the Bellman equation (as a function of ${\delta }_k$, $V^{*}(\dots )$ and $u_k$) are given by

$$\begin{array}{cc}\hfill \frac{{\partial }^{2}H(\dots )}{\partial u_k^2}& =\frac{{\partial }^2\widehat{V}(\dots )}{\partial u_k^2}=R+B^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}\widehat{S}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]B,\hfill \\ \hfill \frac{{\partial }^2{V}^{*}(\dots )}{\partial u_k^2}& =R+B^{\mathrm{T}}{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{S}^{*}\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]B,\hfill \end{array}$$

where $\widehat{S}>0$ and $S^{*}>0$ are the positive-definite solution matrices associated with $\widehat{V}(\dots )$ and $V^{*}(\dots )$, respectively.

Thus, ${\partial }^{2}H(\dots )/\partial u_k^2>0$ and ${\partial }^2\widehat{V}(\dots )/\partial u_k^2>0$. Therefore, the optimal policies $u_k^{*}$ and $u_k^o$ are unique and $u_k^{*}=u_k^o$. Consequently, according to Equation (30), $V^{*}(\dots )$ satisfies the HJB equation (Equation (25)) (i.e., $V^{*}(\dots )=\widehat{V}(\dots )$) if the system is reachable. This can be explained by incorporating the difference between the costate equations evaluated by the Hamiltonian function and Bellman Equation in Equation (30) so that

$$\begin{array}{c}{\displaystyle {\nabla }_{{\delta }_k}{V}^{*}({\delta }_k,u_k)-{\nabla }_{{\delta }_k}\widehat{V}({\delta }_k,u_k)=}\hfill \\ A^{\mathrm{T}}\left({\left|{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}{V}^{*}({\delta }_{k+1},u_{k+1})\right|}_{u=u^{*}}-{\left|{\left[\begin{array}{c}{I}_{n\times n}\hfill \\ {\displaystyle \frac{\partial u_{k+1}}{\partial {\delta }_{k+1}}}\hfill \end{array}\right]}^{\mathrm{T}}{\nabla }_{\delta u_{k+1}}\widehat{V}({\delta }_{k+1},u_{k+1})\right|}_{u=u^o}\right).\hfill \end{array}$$

These results conclude the duality between the Hamiltonian function and Bellman equation for the Action Dependent Dual Heuristic Dynamic Programming solutions. □

5. The Adaptive Learning Solution and Riccati Development

This section introduces the online gradient-based model-free adaptive learning solution which uses the previous HJB development. Then, a Riccati development for the underlying optimal control problem is introduced (it is equivalent to solving the underlying Bellman’s optimality (Equation (9)) or the HJB equation (Equation (25)).

5.1. Model-Free Gradient-Based Solution

The results of Theorem 2 are used to develop a gradient-based algorithm which generalizes the ADDHP solution for the optimal control problem using a model-free policy structure. This adaptive learning solution is based on an online policy iteration process. The duality between the Hamilton–Jacobi–Bellman (HJB) equation (Equation (25)) and Bellman optimality (Equation (9)) is leveraged to propose a gradient-based approach that leads to a model-free control strategy. Algorithm 3 is as follows:

Algorithm 3 Online Policy Iteration Process

Initialize the costate $\nabla V^0\left({\delta }_k\right)$ and the policy $u_k^o$. Evaluate ${\nabla }_{\delta u}{V}^{\ell }(..)$ $${\nabla }_{\delta u_{k+1}}{V}^{\ell }{({\delta }_{k+1},u_{k+1}^{\ell })}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\\ u_{k+1}^{\ell }\end{array}\right]=-F({\delta }_k,u_k^{\ell }).$$ (31) Update the approximation of the optimal strategy, $$\begin{array}{c}\hfill u_k^{\ell +1}=-{\left[S_{u_k{u}_k}^{-1}·S_{u_k{\delta }_k}\right]}^{\ell }·{\delta }_k.\end{array}$$ (32) Terminate on convergence of $\parallel \nabla V^{\ell +1}(..)-\nabla V^{\ell }(..)\parallel$.

5.2. Riccati Development

The following result shows the equivalent Riccati development of the underlying optimal control solution.

Theorem 3.

Let the solution of Equation (9), or equivalently Equation (25), be given by $V({\delta }_k,u_k)=\frac{1}{2}\left[{\delta }_k^T{u}_k^T\right]\mathsf{\Psi }\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right]$ and the optimal strategy follows Equation (8). Then, there is a Riccati solution that is given by

$${\mathsf{\Psi }}^{r+1}=\left[\begin{array}{cc}Q+A^{\mathrm{cT}}{\tilde{\mathsf{\Psi }}}^r{A}^c& A^{\mathrm{T}}{\tilde{\mathsf{\Psi }}}^r{B}^c\\ B^{\mathrm{cT}}{\tilde{\mathsf{\Psi }}}^r{A}^c& R+B^{\mathrm{cT}}{\tilde{\mathsf{\Psi }}}^r{B}^c\end{array}\right].$$

(33)

Note that the parameters of Equation (33) are defined in the proof below.

Proof. 

The optimal policy in Equation (8) can be written as $u_k=-\widehat{\mathsf{\Psi }}{\delta }_k,$ where $\mathsf{\Psi }=\left[\begin{array}{cc}{\mathsf{\Psi }}_{\delta \delta }\hfill & {\mathsf{\Psi }}_{\delta u}\hfill \\ {\mathsf{\Psi }}_{u\delta }\hfill & {\mathsf{\Psi }}_{uu}\hfill \end{array}\right]$ and $\widehat{\mathsf{\Psi }}={\mathsf{\Psi }}_{uu}^{-1}{\mathsf{\Psi }}_{u\delta }$.

Therefore, the value function $V_{(k+1)}$ can be expressed as

$$V\left({\delta }_{(k+1)},u_{(k+1)}\right)=\frac{1}{2}{\delta }_{(k+1)}^{\mathrm{T}}\tilde{\mathsf{\Psi }}{\delta }_{(k+1)},$$

(34)

where $\tilde{\mathsf{\Psi }}={\mathsf{\Psi }}_{\delta \delta }-{\mathsf{\Psi }}_{\delta u}{\mathsf{\Psi }}_{uu}^{-1}{\mathsf{\Psi }}_{u\delta }.$

Substituting the policy in Equation (8) and the value function (Equation (34)) into Equation  (2), yields

$${\delta }_{(k+1)}=A{\delta }_k-B{R}^{-1}{B}^{\mathrm{T}}\tilde{\mathsf{\Psi }}(A{\delta }_k+B{u}_k).$$

Then,

$${\delta }_{(k+1)}=A^c{\delta }_k+B^c{u}_k,$$

(35)

where $A^c=A-B{R}^{-1}{B}^{\mathrm{T}}\tilde{\mathsf{\Psi }}A$ and $B^c=-B{R}^{-1}{B}^{\mathrm{T}}\tilde{\mathsf{\Psi }}B$.

Substituting Equations (35) and (34) into Bellman equation (Equation (9)) leads to

$${\displaystyle V\left({\delta }_k,u_k\right)=\frac{1}{2}\left({\delta }_k^{\mathrm{T}}Q{\delta }_k+u_k^{\mathrm{T}}R{u}_k\right)+\frac{1}{2}{\delta }_k^{\mathrm{T}}{A}^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{A}^c{\delta }_k+\frac{1}{2}{u}_k^{\mathrm{T}}{B}^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{B}^c{u}_k+\frac{1}{2}{u}_k^{\mathrm{T}}{B}^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{A}^c{\delta }_k+\frac{1}{2}{\delta }_k^{\mathrm{T}}{A}^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{B}^c{u}_k.}$$

Therefore,

$$\left[{\delta }_k^T{u}_k^T\right]\mathsf{\Psi }\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right]=\left[{\delta }_k^T{u}_k^T\right]\left[\begin{array}{cc}Q+A^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{A}^c& A^{\mathrm{T}}\tilde{\mathsf{\Psi }}{B}^c\\ B^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{A}^c& R+B^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{B}^c\end{array}\right]\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right].$$

Then,

$$\mathsf{\Psi }=\left[\begin{array}{cc}Q+A^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{A}^c& A^{\mathrm{T}}\tilde{\mathsf{\Psi }}{B}^c\\ B^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{A}^c& R+B^{\mathrm{cT}}\tilde{\mathsf{\Psi }}{B}^c\end{array}\right].$$

This equation yields the Riccati form of Equation (33). □

6. Adaptive Critics Implementations

This section shows the neural network approximation for the online policy iteration solution proposed by Algorithm 3. This implementation represents the optimal value approximation separately form the policy approximation. However, they are both coupled through the Bellman equation or the Hamiltonian function.

6.1. Actor-Critic Neural Networks Implementation

Herein, a simple layer of actor and critic neural network structures is considered. The actor is used to approximate the optimal strategy of Equation (32) while the critic approximates the optimal value in Equation (31). The learning environment involves selecting the values that minimize a cost function along with the associated approximation of the optimal strategies resulting from the feedback and the assessment of the taken strategies. This is done online in real-time where the system dynamics are not required. The weights are adapted through a gradient descent approach.

The value function $V({\delta }_k,u_k)$ is approximated by the following quadratic form:

$$\widehat{V}(.|W_c)=\frac{1}{2}\left[{\delta }_k^T{u}_k^T\right]W_c^{\mathrm{T}}\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right],$$

where $W_c^T=\left[\begin{array}{cc}{W}_{c\delta \delta }^T& W_{c\delta u}^T\\ W_{cu\delta }^T& W_{cuu}^T\end{array}\right]\in R^{(n+m)\times (n+m)}$ is a critic weight matrix.

Consequently, the approximation of ${\nabla }_{\delta u_k}\widehat{V}(\dots )$ follows

$${\nabla }_{\delta u_k}\widehat{V}(.|W_c)=W_c^{\mathrm{T}}\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right].$$

Similarly, the optimal strategy of Equation (32) is approximated by $\widehat{u}=W_a^{\mathrm{T}}{\delta }_k$, where $W_a^T\in R^{m\times n}$ is the actor’s weight matrix.

To proceed with the policy iteration solution of Algorithm 3, the matrix $W_c$ needs to be transformed to a vector form, such that

$$W_c^{\mathrm{T}}\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right]={\overline{W}}_c^T{\overline{\gamma }}_k,$$

where ${\overline{W}}_c^T\in R^{1\times (n+m)(n+m+1)/2}$ and ${\overline{\gamma }}_k\in R^{(n+m)(n+m+1)/2\times 1}$ are the vector transformations of the matrix $W_c$ (upper triangle entries) and its respective combination vector evaluated using the entries from $\left[\begin{array}{c}{\delta }_k\hfill \\ u_k\hfill \end{array}\right]$.

This can be used to formulate Equation (31), such as

$${\overline{W}}_c^T{\overline{\gamma }}_{(k+1)}+\frac{1}{2}({\delta }_k^{\mathrm{T}}Q{\delta }_k+{\widehat{u}}_k^{\mathrm{T}}R{\widehat{u}}_k)=0.$$

Therefore, the target value of the critic approximation of $-{\nabla }_{\delta {\widehat{u}}_{k+1}}\widehat{V}{({\delta }_{k+1},{\widehat{u}}_{k+1})}^{\mathrm{T}}\left[\begin{array}{c}{\delta }_{k+1}\hfill \\ {\widehat{u}}_{k+1}\hfill \end{array}\right]$ is expressed as

$$T^{critic}=\frac{1}{2}({\delta }_k^{\mathrm{T}}Q{\delta }_k+{\widehat{u}}_k^{\mathrm{T}}R{\widehat{u}}_k).$$

(36)

Similarly, the target value of the actor approximation, or the optimal strategy, is defined by

$$T^{actor}=-\left[{W_{c{u}_k{u}_k}^T}^{-1}·{W^T}_{c{u}_k{\delta }_k}\right]{\delta }_k.$$

(37)

The error in the critic approximation is

$${\epsilon }^{critic}=\zeta \left(-{\overline{W}}_c^T{\overline{\gamma }}_{k+1}-T^{critic}\right),$$

(38)

where $\zeta (\dots )$ is a stacking factor that stores $(n+m)\times (n+m+1)/2$ consecutive values of its argument.

In a similar fashion, the error in the actor’s approximation may be written as

$${\epsilon }^{actor}=W_a^{\mathrm{T}}{\delta }_k-T^{actor}.$$

(39)

A gradient decent tuning approach is used to tune the actor and the critic weights as follows:

$$\begin{array}{cc}\hfill W_a^{(\ell +1)\mathrm{T}}& =W_a^{\ell \mathrm{T}}-{\eta }_a{\epsilon }^{actor}{\delta }_k^{\ell \mathrm{T}},\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill {\overline{W}}_c^{(\ell +1)\mathrm{T}}& ={\overline{W}}_c^{\ell \mathrm{T}}-{\eta }_c{\epsilon }^{critic}\zeta {\overline{\gamma }}_{(k+1)}^{\ell },\hfill \end{array}$$

(41)

where ${\eta }_a$ and ${\eta }_c$ are the learning rates for the actor and critic weights, respectively.

The following Algorithm 4 shows the online implementation of Algorithm 3 using the actor-critic neural network approximations.

Algorithm 4 Online Model-Free Actor-Critic Neural Network Solution

Initialize the neural network weights $W_a^0$ and $W_c^0$. Start outside loop (ℓ iterations) (a) Initialize the states ${\delta }_0^0$.  Start inner loop (q iterations) Transfer the outer critic weights $W_c^q=W_c^{\ell }$. Evaluate ${\delta }_{(k+1)}^q$ and ${\widehat{u}}^q$ using Equations (2) and (37). End inner loop when $q=(n+m)\times (n+m+1)/2$. (b) Evaluate the critic weights using Equation (41). (c) Update the actor weights using Equation (40). Terminate on convergence of $∥W_c^{\ell +1}(\dots )-W_c^{\ell }(\dots )∥$.

7. Simulation Results

A flexible wing hang glider is used to validate the developed model-free online adaptive learning approach [9]. The continuously varying dynamics of the flexible wing system poses a challenging control problem. This means that the controller operates in a highly uncertain dynamical environment.

The simulation results highlight the stability properties achieved by the controller in addition to monitoring its robustness against the disturbances and the dynamics’ uncertainties. Two simulation scenarios are considered: Case I shows the controller’s performance in nominal conditions (i.e., at a certain trim speed), while Case II tests the robustness of the developed controller by comparing its performance to the classical Riccati control approach under various disturbances.

7.1. Simulation Parameters

The longitudinal and lateral state space matrices of the flexible wing system, $A^{Lo},B^{Lo},A^{La},$ and $B^{La}$, at a given trim speed, are used to generate the online measurements [9].

The actor and critic learning parameters are set to ${\eta }_a={\eta }_c=0.001$. The weight matrices for the longitudinal $(R^{Lo},Q^{Lo})$ and lateral $(R^{La},Q^{La})$ directions are taken as

$$\begin{array}{ll}{R}^{Lo}& ={10}^{-4}\times [\begin{array}{cc}0.1000& 0.4000\end{array}],\\ Q^{Lo}& [\begin{array}{cccccc}0.0100& 0.0400& 0.1013& 0.1013& 0.4053& 0.4053\end{array}],\\ R^{La}& ={10}^{-4}\times [\begin{array}{ccc}0.0250& 0.1000& 0.4000\end{array}],\\ Q^{La}& [\begin{array}{cccccccc}0.0400& 0.1013& 0.1013& 0.1013& 0.1013& 0.4053& 0.4053& 0.4053\end{array}].\end{array}$$

The eigenvalue structures of the simulated case studies are given the following graphical notations. The open-loop eigenvalues are denoted by □. The * refer to the closed-loop eigenvalues during the learning process. The eigenvalues resulting from the model-free approach are symbolized by ×. The eigenvalues evaluated by the Riccati solution are shown as ∘.

7.2. Simulation Case I

This case shows the simulation outcome when the adaptive learning algorithm is applied to control the decoupled longitudinal and lateral dynamical systems in real-time. The open- and closed-loop poles are tabulated in

Table 1. Open and closed-loop eigenvalues (Case I).

Longitudinal Dynamics
Open-Loop System	$0.9394,0.9950e^{\pm 0.0431}$
	$0.9954,0.9994e^{\pm 0.0084}$
Closed-Loop system	$0.9053,0.9904e^{\pm 0.0082}$
	$0.9932e^{\pm 0.0454},0.9937$
Lateral Dynamics
Open-Loop System	$0.7687,0.9945e^{\pm 0.0120}$
	$0.9971e^{\pm 0.0539}$
	$1.0000e^{\pm 0.0036},1.0016$
Closed-Loop system	$0.7684,0.9937e^{\pm 0.0116}$
	$0.9957e^{\pm 0.0535}$
	$0.9961e^{\pm 0.0035},0.9995$

. The online controller was able to asymptotically stabilize the longitudinal and lateral open-loop systems. The dominant modes are damped much faster than the open-loop system, as shown by the eigenvalue structures in Figure 2. This is further emphasized by Figure 3,

The state space matrices for the longitudinal decoupled dynamics $A^{Lo}$ and $B^{Lo}$ are $$A^{Lo}=\left[\begin{array}{cccccc}0.9906& 0.0272& -0.0982\hfill & 0.0006& 0.1400& -0.0927\\ \hfill -0.0065& 0.9828& 0.0737\hfill & 0.0002& 0.0504& -0.0312\\ \hfill 0.0018& 0.0084& 0.9501\hfill & 0.0000& 0.0018& -0.0002\\ \hfill 0.0057& -0.0024& 0.0990\hfill & 0.9990& -0.1735& -0.0002\\ \hfill 0.0000& -0.0000& 0.0005\hfill & 0.0100& 0.9991& -0.0000\\ \hfill 0.0000& 0.0000& 0.0097\hfill & 0.0000& 0.0000& 1.0000\end{array}\right],$$ $$B^{Lo}=\left[\begin{array}{rr}-0.0004& 0.0002\\ \hfill -0.0002& 0.0001\\ \hfill -0.0005& 0.0002\\ \hfill 0.0011& -0.0005\\ \hfill 0.0000& -0.0000\\ \hfill -0.0000& 0.0000\end{array}\right].$$ The state space matrices for the lateral decoupled dynamics $A^{La}$ and $B^{La}$ are $$A^{La}=\left[\begin{array}{rrrrrrrr}0.9923& 0.0437& -0.0939\hfill & -0.0012& 0.0000& -0.2393& -0.0806& 0.0924\\ \hfill -0.0150& 0.7661& 0.0816\hfill & 0.0000& -0.0000& 0.0019& 0.0003& -0.0007\\ \hfill 0.0009& -0.0009& 0.9949\hfill & 0.0000& 0.0000& 0.0011& 0.0004& 0.0000\\ \hfill 0.0078& 0.2202& -0.0748\hfill & 0.9985& 0.0000& -0.2791& -0.0937& 0.0004\\ \hfill -0.0060& -0.0796& 0.0328\hfill & 0.0000& 1.0000& 0.0027& -0.0004& -0.0003\\ \hfill 0.0001& 0.0013& -0.0005\hfill & 0.0100& -0.0037& 0.9986& -0.0005& 0.0000\\ \hfill -0.0000& -0.0004& 0.0002\hfill & 0.0000& 0.0106& 0.0000& 1.0000& -0.0000\\ \hfill -0.0001& 0.0088& 0.0038\hfill & 0.0000& -0.0000& 0.0000& 0.0000& 1.0000\end{array}\right],$$ $$B^{La}=\left[\begin{array}{rrr}\hfill -0.0008& -0.0000& 0.0002\hfill \\ \hfill 0.0002& -0.0000& -0.0000\hfill \\ \hfill -0.0000& 0.0000& -0.0000\hfill \\ \hfill -0.0010& -0.0000& 0.0002\hfill \\ \hfill 0.0006& -0.0003& -0.0000\hfill \\ \hfill -0.0000& 0.0000& 0.0000\hfill \\ \hfill 0.0000& -0.0000& -0.0000\hfill \\ \hfill 0.0000& -0.0000& -0.0000\hfill \end{array}\right].$$

where the dynamics and the input force control signals are shown to be stable. The adaption process of the actor and the critic neural network weights for both motion frames are shown in Figure 4. The plots demonstrate the converging behavior of the controller. Note that the weights appear in groups as they are too close to show individually. The actor weights are updated after $(n+m)\times (n+m+1)/2$ steps.

7.3. Simulation Case II

This case tests the robustness of the online reinforcement learning algorithm against the uncertainties in the dynamic environment of the flexible wing system (i.e., the matrices $A^{Lo/La}$ and $B^{Lo/La}$) on top of the disturbances in the longitudinal and lateral states ${\delta }^{Lo/La}$. The dynamic uncertainties and disturbances in the states are sampled from a normal Gaussian distribution with amplitudes of up to $\pm 50\%$ and $\pm 20\%$ of the nominal values, respectively. This scenario combines the Riccati classical control technique and the developed online adaptive learning approach such that

$${\delta }_{k+1}=(A+{\tilde{A}}_k)({\delta }_k+{\tilde{\delta }}_k)+B{u}_k^{Ric}+(\tilde{B}-B)u_k^{RL}.$$

where ${\tilde{A}}_k$ and ${\tilde{B}}_k$ are the real-time uncertainties in the drift dynamics A and the control input matrices B. The terms $u_k^{Ric}$ and $u_k^{RL}$ are the control input signals calculated by the Riccati and the online reinforcement learning approaches, respectively.

The eigenvalue structures in

Table 2. Open and closed-loop eigenvalues (Case II).

Longitudinal Dynamics
Open-Loop System	$0.9387,0.9938,0.9940e^{\pm 0.0404}$
(Disturbed System)	$1.0002e^{\pm 0.0080}$
Closed-Loop System	$0.8289,0.9551,0.9818e^{\pm 0.0308}$
(Riccati Solution)	$0.9953e^{\pm 0.0092}$
Closed-Loop System	$0.8271,0.9533,0.9824e^{\pm 0.0298}$
(Model-Free Solution)	$0.9932e^{\pm 0.0104}$
Lateral Dynamics
Open-Loop System (Disturbed System)	$$\begin{gathered} 0.7346,0.9972e^{\pm 0.0578} \\ 0.9948e^{\pm 0.0121}, \\ 0.9996e^{\pm 0.0038},1.0021 \end{gathered}$$
Open-Loop System (Riccati Solution)	$$\begin{gathered} 0.6685,0.7963,0.9778e^{\pm 0.0254} \\ 0.9944e^{\pm 0.0126} \\ 0.9750,0.9945 \end{gathered}$$
Closed-Loop system (Model-Free Solution)	$$\begin{gathered} 0.6622,0.7956,0.9492 \\ 0.9799e^{\pm 0.0156},0.9942e^{\pm 0.0102} \\ 0.9942 \end{gathered}$$

reveal that the disturbed open-loop systems are unstable. However, the combined approach was able to asymptotically stabilize them. Furthermore, it is able to provide faster longitudinal and lateral dominant modes compared to those obtained by the Riccati solution. This is further emphasized by the dynamics and the force control signals shown in Figure 5. The comparison between the eigenvalue structures obtained in Table 1 and Table 2 reveals that the combined approach resulted in faster response compared to the Riccati solution and the original response of the longitudinal and the lateral systems. The convergence behavior of the actor-critic weights is shown in Figure 6, where the adaptation of the weights takes longer this time due to the higher complexity of the problem in hand. The eigenvalues evolution during the learning process is shown in Figure 7. The eigenvalues eventually converge to a stable region.

8. Conclusions

A novel online policy iteration process is developed to generalize model-free gradient based solutions for optimal control problems. The approach is considered a sub-class of the classical action dependent dual heuristic dynamic programming. The mathematical layout showed the duality between the Hamilton–Jacobi–Bellman formulation and the underlying model-free Bellman’s optimality setup. Unlike traditional costate-based solutions, the suggested method does not depend on the system’s dynamics. A Riccati solution is developed and is shown to be equivalent to solving the Bellman’s optimality equation. Artificial neural network-based approximations are employed to provide a real-time implementation of the policy iteration solution. This is accomplished using separate neural network structures to approximate the optimal strategy and the associated gradient of the solving value function. The performance of the proposed control scheme is demonstrated on a flexible wing aircraft. The simulation scenarios proved the effectiveness of the proposed controller under a wide range of uncertainties and disturbances in the system dynamics.