Inducing Optimality in Prescribed Performance Control for Uncertain Euler–Lagrange Systems

Abstract

The goal of this paper is to find a stabilizing and optimal control policy for a class of systems dictated by Euler–Lagrange dynamics, that also satisfies predetermined response criteria. The proposed methodology builds upon two stages. Initially, a neural network is trained online via an iterative process to capture the system dynamics, which are assumed to be unknown. Subsequently, a successive approximation algorithm is applied, employing the acquired dynamics from the previous step, to find a near-optimal control law that takes into consideration prescribed performance specifications, such as convergence speed and steady-state error. In addition, we concurrently guarantee that the system evolves exclusively within the compact set for which sufficient approximation capabilities have been acquired. Finally, we validate our claims through various simulated studies that confirm the success of both the identification process and the minimization of the cost function.

1. Introduction

Finding an optimal control solution for Euler–Lagrange systems can be challenging, owing to the nonlinear nature of the Hamilton–Jacobi–Bellman (HJB) equation. Such complication is also aggravated by the absence of an accurate knowledge of the system dynamics. Towards this direction, multiple methods have been proposed in the recent related literature. Adaptive dynamic programming (ADP) [1] is a prominent class of such algorithms, which evolved from the field of dynamic programming [2] and shares many common tools with the area of reinforcement learning [3]. Owing to their data-driven nature, ADP algorithms that are based on actor–critic schemes and the broad use of neural networks are free from both the “curse of dimensionality” and the “curse of modeling”, which plague traditional dynamic programming techniques. The ADP algorithm that was first introduced by Werbos in [4] for the adaptive approximation of the Bellman equation has evolved beyond its original application on discrete-time systems [5,6], to include algorithms for continuous time. The ever-increasing prominence of neural network programming has further contributed to the success of this particular field of optimal control, exploiting neural networks’ ability to function as universal approximators [7]. The existing literature on the subject is rich, tackling problems such as stabilization [8,9], regulation [10,11] and tracking [12,13,14,15,16].

The regulation problem is examined in [10], aiming to find a near-optimal online tracker using a policy iteration technique. The solutions to the regulation and HJB equations are successively approximated via neural networks. In a more recent work on the regulation problem [11], a value iteration algorithm is proposed, with ensures the convergence rate of the tracking error through the selection of an appropriate gain, free from the hurdle of finding an appropriate initial admissible policy. Regarding ADP methods for tracking problems, in [13], an optimal tracking controller with guaranteed ultimately bounded tracking error is proposed; however, the employed method requires the exact knowledge of the system dynamics, a condition that is often hard to satisfy in practice. Optimal tracking control with completely unknown dynamics is proposed in [14], where the ADP-based controller is used along with a steady-state controller to shape tracking error both in the transient as well as the steady state. Similarly, in [15], an online adaptive tracking controller is designed for completely unknown dynamics, by merging a steady-state and an optimal controller. Another approach for the nonlinear optimal control problem, which differs from the conventional actor–critic networks, introduces an identifier module that is tasked with learning the system dynamics. In [17], this structure is utilized for a partially unknown system, whereas in [15] an algorithm with an identifier–critic structure is applied on a system of completely unknown dynamics. In this technique, the use of the actor is rendered obsolete, reducing the computational burden.

2. State of the Art and Contributions

A common factor in the aforementioned works is that none of them take into consideration performance specifications, such as convergence speed and steady-state error. One way to achieve this is through the incorporation of the prescribed performance control (PPC) technique [18]. The PPC scheme has been utilized for Euler–Lagrange systems with unknown dynamics in several works [19,20,21]. In [19], a sliding-mode control technique is employed along with a novel prescribed performance function to guarantee finite-time stability. In [20], the saturated PPC problem is addressed by using a controller that utilizes an adaptive multilayer neural network of adjustable weights to compensate for state-dependent uncertainties of the system. Furthermore, ref. [21] presents a reinforcement learning-based control scheme that has the optimizing ability for a trade-off between performance and control cost. Nevertheless, note that optimality is only addressed in the latter.

In this paper, we examine the optimal regulation problem and propose an algorithm that approximates the solution to the HJB equation without assuming any a priori knowledge of the system dynamics. In contrast to the vast majority of previous related works, the corresponding control policy is not only optimal with respect to the chosen cost function, but also satisfies prescribed performance criteria that have been set by the designer beforehand through the appropriate selection of certain control parameters. This is achieved by incorporating the PPC formulation within the definition of the optimal control problem. In addition, a significant part of our contribution lies in the provable guarantee of the system’s trajectories evolution, strictly within the set for which the approximation capabilities of the adopted neural network structures are sufficient. This is necessary to ensure the stability of the derived optimal control policy, which is the primary requirement in control system design [22].

The proposed method follows a two-phase strategy. First, via a system identification phase, a neural network is utilized to learn the unknown open-loop system dynamics, even though it might be unstable (i.e., probing signals that are common in the related identification literature are not sufficient to excite the system dynamics as they may lead to instability). The novelty of our approach lies in the fact that it can be employed to learn the unknown dynamics even in the case of an open-loop unstable system. Moreover, the neural network is fitted over a series of trajectories that cover the desired subset of the state space in its entirety to ensure that the extracted knowledge is valid over the entire set and not only around the neighborhood of a single trajectory, as commonly met in related works [10]. It should be noted that knowledge of the system dynamics is not required to achieve prescribed performance [23]. However, to achieve optimality, the system dynamics are necessary since they relate the input–output mapping which is involved in the adopted cost function, as it will be made clear in the sequel. Despite the approximation error between the real and the acquired dynamics, the induction of optimality with prescribed performance in the derived control strategy provides robustness when dealing with such uncertainty. In the second phase, the optimal cost function and policy are approximated through an iterative technique that converges uniformly to their actual values. A constrained least-squares problem is solved at each iteration, and upon convergence, we obtain the optimal controller’s parameters which also ensure that the system’s trajectories evolve strictly within the set for which the approximation of the cost function is valid (an issue that is ignored in the related literature).

Our key contributions in this work can be summarized as follows:

• An identification framework that is able to retrieve the unknown system dynamics even in the case of open-loop instability.
• A successive approximation algorithm that aims to obtain a near-optimal control law, while incorporating prescribed performance specifications.
• A method that guarantees the evolution of the system’s trajectories strictly within the set for which the approximation capabilities of the identification structure are sufficient.

The overall methodology and its effectiveness are demonstrated through extensive simulation studies on a pendulum and a two-degree-of-freedom robotic manipulator.

3. Problem Formulation and Preliminaries

Consider an n-degree-of-freedom Euler–Lagrange system that obeys the following dynamic model:

$$M\left(q\right)\ddot{q}+C(q,\dot{q})\dot{q}+G\left(q\right)=\tau$$

(1)

where $q,\dot{q}\in {\mathbb{R}}^n$ denote the generalized state vector (e.g., position and velocity), $M\left(q\right)$ is a positive definite inertia matrix, $C(q,\dot{q})$ is the matrix that describes the Coriolis centrifugal phenomena, $G\left(q\right)$ is the vector describing the influence of gravity, and $\tau$ is the torque that acts as the system’s input.

Assumption 1. 

The states q (position) and $\dot{q}$ (velocity) are available for measurement, but $\ddot{q}$ (acceleration) is not.

We also define the Lipschitz continuous functions $f(q,\dot{q})\in {\mathbb{R}}^n$ and $g\left(q\right)\in {\mathbb{R}}^{n\times n}$ to describe the system drift dynamics $f(q,\dot{q})=-M^{-1}\left(q\right)(C(q,\dot{q})\dot{q}+G\left(q\right))$ and the input vector field $g\left(q\right)=M^{-1}\left(q\right)$, respectively. Thus, system (1) may be reformulated as:

$$\ddot{q}=f(q,\dot{q})+g\left(q\right)\tau$$

(2)

Assumption 2. 

The system (2) is robustly stabilizable [24], that is, there exists a continuous control law $\tau (q,\dot{q})$ and a compact set $\mathsf{\Omega }\subset {\mathbb{R}}^n$ such that for any initial condition $\tilde{q}\left(0\right)={[q\left(0\right),\dot{q}\left(0\right)]}^T$, the solutions of the closed-loop system starting from $\tilde{q}\left(0\right)$ exist for all $t\ge 0$ and satisfy $\parallel \tilde{q}{\left(t\right)\parallel }_{\mathsf{\Omega }}\le \beta (\parallel \tilde{q}\left(0\right){\parallel }_{\mathsf{\Omega }},t)$, where β is a class $\mathcal{K}\mathcal{L}$ function and ${\parallel •\parallel }_{\mathsf{\Omega }}$ denotes ${\parallel •\parallel }_{\mathsf{\Omega }}=inf\{\parallel •-\zeta \parallel :\zeta \in \mathsf{\Omega }\}$.

Assumption 3. 

No a priori knowledge regarding $f(q,\dot{q})$ and $g\left(q\right)$ is available besides the Lipschitz continuity.

Our objective is to design an optimal (with respect to a state and input integral cost) control strategy within a compact workspace ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}\subset {\mathbb{R}}^{2n}$ that drives the system towards a fixed configuration ${\overline{q}}_d\in {\mathsf{\Omega }}_q$ with predefined transient and steady-state performance (i.e., minimum convergence rate and steady-state error). Before we proceed with the formulation of the optimal control problem, we first give a brief presentation of the PPC technique.

3.1. Prescribed Performance Control (PPC)

The PPC strategy [18] enables the tracking of a reference trajectory with the system’s response fulfilling predefined performance criteria during the transient and the steady state, without requiring any knowledge of the system dynamics. For the case of a generic scalar error $\sigma \left(t\right)$, the prescribed performance is achieved if the error remains bounded within a predefined region that is formed by decaying functions of time, as illustrated in Figure 1:

$$-\rho \left(t\right)<\sigma \left(t\right)<\rho \left(t\right),\forall t⩾0$$

(3)

The function $\rho \left(t\right)$ is a smooth, bounded, strictly positive and decreasing function of time, called performance function, and is chosen as $\rho \left(t\right)=({\rho }_0-{\rho }_{\infty })e^{-lt}+{\rho }_{\infty }$, with ${\rho }_0,{\rho }_{\infty }$ and l being positive gains that are chosen to satisfy the designer’s specifications. Specifically, ${\rho }_{\infty }={lim}_{t\to \infty }\rho \left(t\right)$ is selected according to the maximum allowable tracking error at the steady state; l determines a lower bound on the speed of convergence; and ${\rho }_0$ affects the maximum overshoot and is selected such that ${\rho }_0>\left|\sigma \left(0\right)\right|$.

In our case, we define the scalar tracking error ${\sigma }_i\left(e_i\left(t\right)\right)={\tilde{\lambda }}^T{e}_i\left(t\right)$ where ${\tilde{\lambda }}^T=[\lambda ,1]$ with $\lambda$ being a positive constant and $e_i\left(t\right)={[q_i-q_{d_i},{\dot{q}}_i-{\dot{q}}_{d_i}]}^T$ with $q_{d_i}\left(t\right)$, $i=1,\dots ,n$ being the reference trajectories. Notice that the performance specifications imposed on the errors ${\sigma }_i$ are easily translated into equivalent performance specifications of the errors $e_i$ as mentioned in Lemma 1 in [23]. The intrinsic property behind PPC lies on a mapping of the tracking error that transforms the constrained behavior as defined in (3) into a significantly relaxed unconstrained problem. More specifically, we define:

$${ϵ}_i\left(t\right)=T\left({\xi }_i\left(t\right)\right),i=1,\dots ,n$$

(4)

where ${ϵ}_i\left(t\right)$ is the transformed error, $T:(-1,1)\to (-\infty ,\infty )$ is a strictly increasing, symmetric and bijective mapping, e.g., $T(★)=\frac{1}{2}\ln \left(\frac{1+★}{1-★}\right)$, and ${\xi }_i\left(t\right)=\frac{{\sigma }_i\left(e_i\left(t\right)\right)}{{\rho }_i\left(t\right)}$. To achieve the prescribed performance the following control signal [23] is used:

$${\tau }_i=-k{T}^{\prime }\left({\xi }_i\right){\rho }_i^{-1}\left(t\right)T\left({\xi }_i\right),k>0,\forall i=1,\dots ,n$$

(5)

where k is a positive gain and $T^{\prime }\left(\xi \right)$ the derivative of $T\left(\xi \right)$. The following proposition guarantees that no matter how large the upper bound of the transformed error ${ϵ}_i\left(t\right)$ is (which is affected by the model uncertainty), the performance specifications encapsulated in the corresponding performance function ${\rho }_i\left(t\right)$ are met.

Proposition 1 

([25]). The prescribed-performance control problem, as defined by (3), admits a solution if and only if the transformed error signals (4) can be kept bounded.

Proof. 

Given any finite initial condition ${[q_i\left(0\right),{\dot{q}}_i\left(0\right)]}^T$, we can always select ${\rho }_i\left(0\right)$ to satisfy (3) at $t=0$, hence ensuring that ${\xi }_i\left(0\right)\in (-1,1)$, and thus, the transformed system is initially well defined. Furthermore, if the transformed system is robustly stabilizable, there exists a continuous control law that guarantees that ${[q_i\left(t\right),{\dot{q}}_i\left(t\right)]}^T,{ϵ}_i\left(t\right)\in {\mathcal{L}}_{\infty }$ and as a consequence, there exist unknown constants ${ϵ}_{i_{lower}}$,${ϵ}_{i_{upper}}$, such that ${ϵ}_{i_{lower}}\le {ϵ}_i\left(t\right)\le {ϵ}_{i_{upper}},i=1\dots ,N\forall t\ge 0$. Since T is a smooth, strictly increasing function, its inverse exists, and thus, $T^{-1}\left({ϵ}_{i_{lower}}\right)\le {\xi }_i\left(t\right)\le T^{-1}\left({ϵ}_{i_{upper}}\right)\Rightarrow T^{-1}\left({ϵ}_{i_{lower}}\right){\rho }_i\left(t\right)\le {\sigma }_i\left(t\right)\le T^{-1}\left({ϵ}_{i_{upper}}\right)\forall t\ge 0$. Since system (2) belongs to the general class of nonlinear affine MIMO systems that are feedback linearizable, it can be easily verified that its transformed system is robustly stabilizable [25]. Therefore, the PPC problem admits a solution.    □

3.2. Optimal Control with Prescribed Performance

To induce optimality along with prescribed performance, we formulate the cost function of the optimal control problem as follows:

$$J={\int }_0^{\infty }\left({\alpha \parallel ϵ\left(t\right)\parallel }^2+\beta {\parallel \tau \left(t\right)-G\left({\overline{q}}_d\right)\parallel }^2\right)dt$$

(6)

where $ϵ={\left[\begin{array}{c}{ϵ}_1,\dots ,{ϵ}_n\end{array}\right]}^T$ denotes the vector of the transformed errors defined via (4) as ${ϵ}_i\left(t\right)=T\left(\frac{{\dot{q}}_i+\lambda (q_i-{\overline{q}}_{d_i})}{{\rho }_i\left(t\right)}\right),i=1,\dots ,n$. The constants $\alpha$ and $\beta$ are positive and regulate the trade-off between the state convergence and the energy of the input signal. Additionally, for an admissible control policy (i.e., a control policy that exhibits a finite cost value from any initial condition in the workspace) the transformed errors remain bounded since ${ϵ}_i\left(t\right)\in {\mathcal{L}}_2$ and thus the predefined performance specifications are met according to Proposition 1. Finally, notice that the term $G\left({\overline{q}}_d\right)$ is necessary in the input-related term in (6), so that ${\overline{q}}_d$ becomes an equilibrium and the integral cost is valid (otherwise the integral of the second term in the cost function (6) would grow unbounded as the input signal to keep the system at the desired position ${\overline{q}}_d$ would not vanish).

To find an admissible policy that minimizes (6) and guarantees predefined transient and steady state performance specifications, first, notice that (6) can be transformed in its differential form as $\dot{J}=-{\alpha \parallel ϵ\parallel }^2-\beta {\parallel \tau -G\left({\overline{q}}_d\right)\parallel }^2$. Thus, the equivalent HJB equation can be written as:

$$HJB\left(z\right)={\nabla }_q{J}^T\dot{q}+{\nabla }_{\rho }{J}^T\dot{\rho }+{\nabla }_{ϵ}{J}^T\dot{ϵ}+{\alpha \parallel ϵ\parallel }^2+\beta {\parallel \tau -G\left({\overline{q}}_d\right)\parallel }^2=0$$

(7)

for a stacked state-space vector $z={\left[q^T,{\rho }^T,{ϵ}^T\right]}^T$, where $\rho ={[{\rho }_1,\dots ,{\rho }_n]}^T$ denotes the vector of the performance functions, and ${\nabla }_{•}$ denotes the gradient with respect to each argument •. Applying the stationary condition in (7) after substituting the system dynamics (1) in $\dot{{ϵ}_i}$, the optimal control policy is calculated as:

$${\tau }^{★}(q,\rho ,ϵ)=-\frac{1}{2\beta }{M}^{-T}\left(q\right){\mathbf{T}}^{\prime }\mathrm{diag}{\left(\rho \right)}^{-1}{\nabla }_{ϵ}J+G\left({\overline{q}}_d\right),$$

(8)

where ${\mathbf{T}}^{\prime }$ denotes the diagonal matrix with the derivatives of the transformation function $\frac{dT(★)}{d★}$ for each error, respectively, which are yielded by $\dot{ϵ}$.

Now, we have to deal with two major obstacles. The first concerns the lack of knowledge of the system dynamics, which is heavily involved in both the HJB Equation (7) via $\dot{ϵ}$ as well as in the optimal control policy (8). Furthermore, notice that in order to implement (8) we need the gradient of the cost function. However, the HJB Equation (7) is a nontrivial partial differential equation to solve numerically (i.e., we cannot easily calculate ${\nabla }_{ϵ}J$ and apply it in the optimal control policy (8)). In the next section, both issues are rigorously addressed.

4. Methodology

To overcome the aforementioned hurdles, the first phase of our method is devoted to creating a sufficiently accurate approximation of the underlying open-loop dynamics by employing an artificial neural network. Subsequently, building upon the extracted knowledge of the system dynamics, we utilize a successive approximation strategy to solve the HJB equation.

4.1. Identification of System Dynamics

We adopt an iterative process, which aims at creating a progressively improving approximation of the unknown functions $f(q,\dot{q})$, $g\left(q\right)$, denoted as $\widehat{f}(q,\dot{q})$ and $\widehat{g}\left(q\right)$, respectively, at each iteration, until convergence is achieved, i.e., $|f(q,\dot{q})+g\left(q\right)\tau -\widehat{f}(q,\dot{q})-\widehat{g}\left(q\right)\tau |<\overline{ϵ}$, with $\overline{ϵ}$ being an arbitrarily small positive number. To acquire the data needed for the neural network estimation at each iteration and in order to guarantee that this data set is representative of the compact workspace ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}\subset {\mathbb{R}}^{2n}$, we form a reference trajectory by linking together multiple points located all over ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}$. To elaborate further on this, first, a set of N points $X={[X_1,\dots ,X_N]}^T$ is selected such that it covers ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}$. Then, we need to devise a closed path that traverses these points. For that purpose, we connect any pair of points $X_i\to X_j$ with a trajectory of minimum acceleration (see Chapter 3 in [26]), as follows:

$$\left[\begin{array}{c}{q}_d\left(t\right)\\ {\dot{q}}_d\left(t\right)\end{array}\right]=\left(\left[\begin{array}{cc}1& t\\ 0& 1\end{array}\right]\otimes {\mathbf{I}}_n\right)X_i-\left(\left[\begin{array}{cc}-\frac{t^3}{T^3}& -\frac{t^2(T-t)}{T^2}\\ \frac{6t(T-t)}{T^3}& \frac{t(2T-3t)}{T^2}\end{array}\right]\otimes {\mathbf{I}}_n\right)\left[X_j-\left(\left[\begin{array}{cc}1& T\\ 0& 1\end{array}\right]\otimes {\mathbf{I}}_n\right)X_i\right]$$

for all $t\in [0,T]$, where T denotes the transition time from $X_i$ to $X_j$. The path in every iteration is then created by a different random sequence of $X_i$, $i=1,\dots ,N$ (i.e., a permutation of X) to ensure that the final result is free of bias along a specific trajectory.

Subsequently, the aforementioned reference trajectory that covers the whole domain space will be tracked with predefined transient and steady-state performance using the PPC technique, as described in Section 3.1, with the position q and velocity $\dot{q}$ being measurable, according to Assumption 1. Hence, based on the collected data over the aforementioned series of reference trajectories, we shall approximate the unknown dynamics using a neural network structure $NN(q,\dot{q},\tau )$ that will fit the system dynamics as $f(q,\dot{q})+g\left(q\right)\tau \approx NN(q,\dot{q},\tau )$. In particular, we are interested in learning $f(q,\dot{q})$ and $g\left(q\right)$ separately, which correspond to the terms $-M{\left(q\right)}^{-1}(C(q,\dot{q})\dot{q}+G\left(q\right))$ and $M^{-1}\left(q\right)$, respectively. This can be straightforwardly accomplished by setting $\tau ={\mathbf{0}}_n$, where ${\mathbf{0}}_n$ denotes an n-dimensional vector of zeros, as follows:

$$NN(q,\dot{q},{\mathbf{0}}_n)\approx f(q,\dot{q}).$$

(9)

Then, by setting $\tau =c{\mathbf{1}}_n^{\left\{i\right\}}$, $i=1,\dots ,n$, where $c\ne 0$ is a constant number, and ${\mathbf{1}}_n^{\left\{i\right\}}$ denotes an n-dimensional vector with one in the ith element and zeros everywhere else, we obtain:

$$\frac{1}{c}{\left[NN(q,\dot{q},c{\mathbf{1}}_n^{\left\{i\right\}})-NN(q,\dot{q},{\mathbf{0}}_n)\right]}_{i=1,\dots ,n}\approx g(q,\dot{q}).$$

(10)

Finally, the gravity vector of the dynamic model that is employed in (8) can be easily accessed by the aforementioned structure following the property $G\left(q\right)=-g^{-1}\left(q\right)f(q,0)$.

Nevertheless, under Assumption 1, we do not have access to the acceleration $\ddot{q}$ in (2), and thus, the target values to train the neural network $NN(q,\dot{q},\tau )$ are not available. To remedy this issue, the following tracking differentiator [27] is employed:

$$\begin{array}{cc}\hfill \dot{z_1}& =z_2\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \dot{z_2}& =-k_{z_1}{R}^2(z_1-q)-k_{z_2}R(z_2-\dot{q})+NN(q,\dot{q},\tau )\hfill \end{array}$$

(12)

with positive gains $k_{z_1},k_{z_2},R$. When $R\to \infty$, then, based on [27], $z_1\to q$ and $z_2\to \dot{q}$, and consequently $\dot{z_2}\to \ddot{q}$, from which we may reconstruct the acceleration signal that will be employed for the neural network training. Notice that during the first round over the closed reference trajectory that traverses all points in the workspace ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}$, the NN structure is null (i.e., we have not initiated training since we have not collected the required data yet). Thus, it is activated after the first training stage (i.e., the first pass over all points in X). This means that in every new round (i.e., a new permutation of X), the extracted neural network originates from the previous iteration. After each round of training, we employ the knowledge we acquire for the dynamics not only to improve the acceleration estimation, but also to provide improved initial weights for the consecutive iterations. Consequently, after enough iterations (e.g., when the weights do not change above a threshold), an accurate model of the system dynamics is eventually acquired.

4.2. Solving the Hamilton–Jacobi–Bellman Equation

This subsection is dedicated to finding an admissible policy (8) that minimizes (6) and guarantees predefined transient and steady-state performance specifications. Towards this direction, a successive approximation strategy is adopted similarly to [21], where the approximation of the cost function converges uniformly to the optimal cost function.

In this respect, the solution to the HJB equation (i.e., the unknown cost function $J(q,\rho ,ϵ)$) is expanded over a set of basis functions with adjustable weights. The approximate solution assumed here is:

$$J_L(q,\rho ,ϵ)=\sum _{j=1}^L{w}_j^i{\sigma }_j(q,\rho ,ϵ)=w_L^T\sigma (q,\rho ,ϵ)$$

(13)

where $J_L(q,\rho ,ϵ)$ denotes the approximation of the unknown cost function, $J(q,\rho ,ϵ)$, ${\sigma }_{\mathbf{L}}={\left[\begin{array}{c}{\sigma }_1,\dots ,{\sigma }_L\end{array}\right]}^T$ includes polynomial regressor terms, and $w_L={\left[\begin{array}{c}{w}_1,\dots ,w_L\end{array}\right]}^T$ denotes the weight vector to be successively adjusted so that the residual error may be minimized, i.e., the best-fitted solution $J_L(q,\rho ,ϵ)$ to the HJB equation is extracted. Hence, if we substitute the trial solution into the HJB equation, the residual is formed as:

$$e_L(q,\rho ,ϵ)=w_L^T({\displaystyle {\sum }_{i=1}^n}{\nabla }_{q_i}{\sigma }^T\dot{q_i}+{\nabla }_{{\rho }_i}{\sigma }^T\dot{{\rho }_i}+{\nabla }_{{ϵ}_i}{\sigma }^T\dot{{ϵ}_i})+{\alpha \parallel ϵ\parallel }^2+\beta {\parallel \tau (q,\rho ,ϵ)-G\left({\overline{q}}_d\right)\parallel }^2=0$$

(14)

where

$$\begin{array}{c}\dot{q}=\mathrm{diag}\left(\rho \right)T^{-1}\left(ϵ\right)-\lambda (q-{\overline{q}}_d)\\ \dot{\rho }=-l\rho +l{\rho }_{\infty }\\ \dot{ϵ}=\mathrm{diag}{\left(\rho \right)}^{-1}\mathrm{diag}{\left(T^{-1}\left(ϵ\right)\right)}^{-1}(f(q,\dot{q})+g\left(q\right)\tau -T^{-1}\left(ϵ\right)(-l\rho +l{\rho }_{\infty }))\end{array}$$

In order to calculate the unknown weights $w_L$, the inner product of the residual and its derivative with respect to the weights is set to zero as follows:

$$\langle \frac{d{e}_L}{d{w}_L},e_L\rangle =0\iff$$

$$\begin{array}{c}{w}_L^T\langle \sum _{i=1}^n{\nabla }_{q_i}{\sigma }^T\dot{q_i}+\sum _{i=1}^n{\nabla }_{{\rho }_i}{\sigma }^T\dot{{\rho }_i}+\sum _{i=1}^n{\nabla }_{{ϵ}_i}{\sigma }^T\dot{{ϵ}_i},\\ \sum _{i=1}^n{\nabla }_{q_i}{\sigma }^T\dot{q_i}+\sum _{i=1}^n{\nabla }_{{\rho }_i}{\sigma }^T\dot{{\rho }_i}+\sum _{i=1}^n{\nabla }_{{ϵ}_i}{\sigma }^T\dot{{ϵ}_i}\rangle +\\ \langle \alpha \parallel ϵ{\parallel }^2+\beta \parallel \tau (q,\rho ,ϵ)-G\left(q_d\right){\parallel }^2,\\ \sum _{i=1}^n{\nabla }_{q_i}{\sigma }^T\dot{q_i}+\sum _{i=1}^n{\nabla }_{{\rho }_i}{\sigma }^T\dot{{\rho }_i}+\sum _{i=1}^n{\nabla }_{{ϵ}_i}{\sigma }^T\dot{{ϵ}_i}\rangle =0\end{array}$$

where the inner product between two vectors u and v is given by $\langle u,v\rangle ={\int }_{\mathcal{V}}uvdV$ over a domain $\mathcal{V}$. To solve the aforementioned problem with respect to $w_L$, a discretization of P points $(q^i,{\rho }^i,{ϵ}^i),i=1,\dots ,P$ is applied over a compact set ${\mathsf{\Omega }}_z\subset {\Re }^{3n}$ in order to obtain the following terms:

$$\begin{array}{c}X=\lfloor {\left(\sum _{i=1}^n{\nabla }_{q_i}{\sigma }^T\dot{q_i}+\sum _{i=1}^n{\nabla }_{{\rho }_i}{\sigma }^T\dot{{\rho }_i}+{\nabla }_{{ϵ}_i}{\sigma }^T\dot{{ϵ}_i}\right)}_{(q^1,{\rho }^1,{ϵ}^1)},\dots ,\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\sum _{i=1}^n{\nabla }_{q_i}{\sigma }^T\dot{q_i}+\sum _{i=1}^n{\nabla }_{{\rho }_i}{\sigma }^T\dot{{\rho }_i}+{\nabla }_{{ϵ}_i}{\sigma }^T\dot{{ϵ}_i}{)}_{(q^p,{\rho }^p,{ϵ}^p)}\rfloor \end{array}$$

(15)

$$\begin{array}{c}Y=\lfloor {\left({\alpha \parallel ϵ\parallel }^2+\beta {\parallel \tau (q,\rho ,ϵ)-G\left(q_d\right)\parallel }^2\right)}_{(q^1,{\rho }^1,{ϵ}^1)},\dots ,\hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\left({\alpha \parallel ϵ\parallel }^2+\beta {\parallel \tau (q,\rho ,ϵ)-G\left(q_d\right)\parallel }^2\right)}_{(q^p,{\rho }^p,{ϵ}^p)}\rfloor \end{array}$$

(16)

Consequently, the weights $w_L$ may be calculated using the least-squares method as follows:

$$w_L=-{\left(X^{T}X\right)}^{-1}\left(X^{T}Y\right)$$

(17)

and then be employed to update the optimal control policy in (8) by calculating the gradient of $J_L(q,\rho ,ϵ)$ along $ϵ$. However, owing to the fact that the approximation capabilities of polynomials hold locally over a compact set, when employing the optimal control policy (8) we need to ensure that the trajectories of $q,\rho ,ϵ$ evolve strictly within the compact set where the approximation of the cost function is valid. Owing to the decreasing property of the performance function, we have that $\rho \left(t\right)\le {\rho }_0\forall t\ge 0$. Consequently, if ${\rho }_0$ belongs to the compact set, so does the trajectory $\rho \left(t\right)$. Here, ${\mathsf{\Omega }}_z$ is chosen to be symmetric in each variable ${ϵ}_i$ about zero, i.e., ${ϵ}_i\in [-c,c]$. In addition, from the proof of Proposition 1, if $q\left(0\right)\in {\mathsf{\Omega }}_z$, then q stays trapped inside ${\mathsf{\Omega }}_z$ as well. Therefore, what remains is to guarantee that the transformed error stays inside ${\mathsf{\Omega }}_z$, which is achieved by imposing constraints on the weights $w_L$. Consider the n-dimensional cube:

$$M=\left\{V({ϵ}_1,{ϵ}_2,\dots ,{ϵ}_n)=max\{\parallel {ϵ}_1\parallel ,\parallel {ϵ}_2\parallel ,\dots ,\parallel {ϵ}_n\parallel \}=c\right\}$$

(18)

The transformed error stays trapped inside M if for the inner product of ${\dot{ϵ}}^T=[{\dot{ϵ}}_1\left(t\right),{\dot{ϵ}}_2\left(t\right),\dots ,{\dot{ϵ}}_N\left(t\right)]$ and $\nabla V\left(ϵ\right)$, it holds that ${\dot{ϵ}}^T\nabla V\left(ϵ\right)\le 0$. Essentially, this means that for each of the $2n$ facets of the n-dimensional cube, we have ${\dot{ϵ}}_N\le 0$ if ${ϵ}_N=c$, and ${\dot{ϵ}}_N\ge 0$ if ${ϵ}_N=-c$. Differentiating ${ϵ}_i$, we obtain:

$${\dot{ϵ}}_i=\frac{{ϵ}_i^{\prime }}{{\rho }_i}{\ddot{q}}_i+\frac{{ϵ}_i^{\prime }\lambda q_i{\rho }_i-{ϵ}_i^{\prime }[{\dot{q}}_i+\lambda (q_i-q_{d_i})]{\dot{\rho }}_i}{{\rho }_i^2}$$

(19)

where ${\ddot{q}}_i=f_i+g_i^T{\tau }^{*}$, and $f_i$ and $g_i^T$ denote the ith row of matrices f and g, respectively. Consequently, ${\dot{ϵ}}_i$ can be written as ${\dot{ϵ}}_i=A_i^T{w}_l+B_i$ where:

$$\begin{array}{cc}\hfill A_i^T& =-\frac{1}{2\beta }{g}_i^T{g}^T{\mathbf{T}}^{\prime }\mathrm{diag}{\rho }^{-1}{\nabla }_{{\sigma }_{ϵ}}^T\hfill \\ \hfill B_i& =\frac{{ϵ}_i^{\prime }}{{\rho }_i}(f_i+g_i^{T}G\left(q_d\right))+\frac{{ϵ}_i^{\prime }\lambda q_i{\rho }_i-{ϵ}_i^{\prime }[{\dot{q}}_i+\lambda (q_i-q_{d_i})]{\dot{\rho }}_i}{{\rho }_i^2}\hfill \end{array}$$

(20)

To guarantee safety [28], a discretization of M points is applied over each facet of the cube, for which:

$$\begin{array}{cc}\hfill {\tilde{A}}_i^{+}& =\left[A_i^T(q^1,{\rho }^1,{ϵ}^{1^{+}}),\dots ,A_i^T(q^M,{\rho }^M,{ϵ}^{M^{+}})\right]\hfill \\ \hfill {\tilde{A}}_i^{-}& =\left[A_i^T(q^1,{\rho }^1,{ϵ}^{1^{-}}),\dots ,A_i^T(q^M,{\rho }^M,{ϵ}^{M^{-}})\right]\hfill \\ \hfill {\tilde{B}}_i^{+}& =\left[B_i(q^1,{\rho }^1,{ϵ}^{1^{+}}),\dots ,B_i(q^M,{\rho }^M,{ϵ}^{M^{+}})\right]\hfill \\ \hfill {\tilde{B}}_i^{-}& =\left[B_i(q^1,{\rho }^1,{ϵ}^{1^{-}}),\dots ,B_i(q^M,{\rho }^M,{ϵ}^{M^{-}})\right]\hfill \end{array}$$

(21)

and ${ϵ}^{j^{-}}$ (${ϵ}^{j^{+}}$) denotes the jth sample point where ${ϵ}_i$ is fixed at ${ϵ}_i=-c$ (${ϵ}_i=c$). Summarizing, for each ${ϵ}_i$, the following inequality constraints must be satisfied:

$$\left[\begin{array}{c}\hfill {\tilde{A}}_i^{+}\\ \hfill -{\tilde{A}}_i^{-}\end{array}\right]w_L\le \left[\begin{array}{c}\hfill -{\tilde{B}}_i^{+}\\ \hfill {\tilde{B}}_i^{-}\end{array}\right]$$

(22)

Therefore, the weights $w_L$ may be obtained by solving the constrained least-squares problem:

$$\underset{w_L}{min}\frac{1}{2}{\parallel X{w}_L+Y\parallel }^2,\mathrm{s}.\mathrm{t}.\tilde{A}{w}_L\le \tilde{B}$$

(23)

where

$$\tilde{A}=\left[\begin{array}{c}\hfill {\tilde{A}}_1^{+}\\ \hfill -{\tilde{A}}_1^{-}\\ \hfill ⋮\\ \hfill {\tilde{A}}_N^{+}\\ \hfill -{\tilde{A}}_N^{-}\end{array}\right],\tilde{B}=\left[\begin{array}{c}\hfill -{\tilde{B}}_1^{+}\\ \hfill {\tilde{B}}_1^{-}\\ \hfill ⋮\\ \hfill -{\tilde{B}}_N^{+}\\ \hfill {\tilde{B}}_N^{-}\end{array}\right]$$

It should be noted that (23) can be solved as a quadratic optimization problem with linear constraints; thus, plenty of robust and computationally efficient methods to tackle it exist [29]. The aforementioned procedure that is applied iteratively to find an approximation of the optimal solution to the HJB equation is presented in Algorithm 1 and the results are summarized in the following Theorem.

Algorithm 1: Cost function approximation algorithm

1: Initialize based on the PPC technique the control policy: $\tau (q,\rho ,ϵ)=-k{T}^{\prime }\left(\xi \right){\rho }^{-1}ϵ+G\left({\overline{q}}_d\right)$ with $\xi =\frac{\dot{q}+\lambda (q-{\overline{q}}_d)}{\rho }$. 2: Select$(q^i,{\rho }^i,{ϵ}^i)$, $i=1,\dots ,P$ points over which the HJB will be fitted. 3: Repeat 4:     Calculate the terms X and Y from (15) and (16) over the data $(q^i,{\rho }^i,{ϵ}^i)$, $i=1,\dots ,P$. 5:     Calculate the terms $\tilde{A}$ and $\tilde{B}$ from (21) over the data $(q^j,{\rho }^j,{ϵ}^{j+})$, $j=1,\dots ,M$ and $(q^j,{\rho }^j,{ϵ}^{j-})$, $j=1,\dots ,M$. 6:     Find $w_L$ by solving the constrained least-squares problem (23). 7:     Update the control policy according to (8) employing the parameter vector $w_L$ in the approximation of ${\nabla }_{ϵ}J$. 8: until $w_L$ converges.

Theorem 1. 

Consider the Euler–Lagrange system (1) under Assumptions 1–3, as well as a compact set ${\mathsf{\Omega }}_{q,\dot{q},\tau }$ for which a sufficiently accurate neural network approximation $NN(q,\dot{q},\tau )$ of the system dynamics has been acquired. Then, for a compact set ${\mathsf{\Omega }}_z\subset {\Re }^{3n}$ where $z\triangleq {[q^T,{\rho }^T,{ϵ}^T]}^T$ and a linearly independent polynomial basis ${\left\{{\sigma }_j\right\}}_1^L$, Algorithm 1 converges to a near-optimal estimate $J_L(q,\rho ,ϵ)=w_L^T\sigma (q,\rho ,ϵ)$ of the minimum cost function, i.e., for any arbitrarily small constant $\tilde{\epsilon }$, there exists $L_0$ such that for any $L\ge L_0$:

$$\underset{z\in {\mathsf{\Omega }}_z}{sup}|J\left(z\right)-w_L^T\sigma \left(z\right)|<\tilde{\epsilon }$$

(24)

Proof. 

For a sufficiently accurate neural network approximation and by utilizing the admissible policy $u^{\left(0\right)}$ in the initialization of Algorithm 1, an initial, unique, least-squares solution $J_L^{\left(0\right)}$ for (8) can be acquired through the optimization problem (23). This solution is employed in the calculation of the control policy $u^{\left(1\right)}$, and by iterating the process, we obtain the successive approximation algorithm, which, along with the inequality constraints imposed, guarantees, following [30], that each new policy that is provided by the cost function of the previous one is stable and better than the previous one with respect to the metric (6), while also satisfying the prescribed performance specifications, since $ϵ\in {\mathcal{L}}_{\infty }$ and consequently remains bounded. Moreover, every trajectory $z\left(t\right)$ lies strictly within the set where the approximation capabilities of the polynomial approximation structure hold. Therefore, starting from an admissible policy, we always improve it until $u^{\left(i\right)}$ and $J_L^{\left(i\right)}$ converge to their optimal values. Finally, to show (24), notice that from Weierstrass’s approximation theorem, since $J\left(z\right)$ is $C^1$, it can be uniformly approximated as accurately as desired by a polynomial function within a compact set. As a consequence, no matter how small $\tilde{\epsilon }$ is chosen, there exists a polynomial basis ${\left\{{\sigma }_j\right\}}_1^L$ with $L\ge L_0\left(\tilde{\epsilon }\right)$ such that (24) holds. □

5. Simulation Results

In this section, we demonstrate the effectiveness of the proposed scheme via two simulated scenarios: a pendulum, illustrating the full capabilities of our method by performing a comparative study with another successive approximation strategy, as well as a two-degree-of-freedom robotic manipulator.

5.1. Case A: Pendulum

Consider a pendulum that obeys the following Euler–Lagrange dynamics

$$\ddot{q}=-\frac{g}{l}sin\left(q\right)-\frac{k}{m}\dot{q}+\frac{1}{m{l}^2}\tau$$

(25)

with q and $\dot{q}$ denoting the angular position and velocity, respectively, and $\tau$ denoting the applied torque that acts as the system’s input. In addition, m denotes the mass and l the length of the rod, with k being the friction coefficient and g the acceleration of gravity. The values that were adopted for the simulation are $m=5.2$ kg, $l=0.9$ m, $k=1$ kg/s, and $g=9.81$ m/s${}^2$. Despite its simplicity, several systems are modeled by equations similar to (25), rendering it of great practical importance.

Initially, a reference trajectory was designed so that it traversed (in random order) 51 points scattered over the domain set ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}=[-1,1]\times [-1,1]$. The gains of the PPC controller used to track it were $k=2$ and $\lambda =7$, with performance specifications dictated by ${\rho }_0=\pi /3$, ${\rho }_{\infty }=\pi /180$, and $l=1.5$. Finally, the gains of the tracking differentiator were chosen as $k_{z_1}=k_{z_2}=0.2$ and $R=100$. We also employed a neural network with a hidden layer containing eight neurons, to extract the system dynamics via the learning process.

The neural network’s training was carried out for 20 iterations (i.e., 20 permutations of the overall 51 points in the workspace), so that the approximation improved sufficiently. The training of the adopted neural network structure over the collected data during each iteration (i.e., after each one of the 20 reference trajectories) was conducted using the MATLAB toolbox employing the default Levenberg–Marquardt algorithm. The results of the approximation problem can be observed in Figure 2 for the corresponding terms of $f(q,\dot{q})$ and $g\left(q\right)$. It can be easily observed that through this process, we succeeded in obtaining a highly accurate (with an average relative approximation error less than $1\%$) estimate of the system dynamics, without relying on any prior knowledge.

In the second phase, we set a conventional PPC stabilizing controller, parametrized by $k=2$, $\lambda =2$, ${\rho }_0=\pi /3$, ${\rho }_{\infty }=\pi /180$ and $l=1.5$, as the initial admissible control policy. Our aim was to drive the system towards the fixed configuration ${\overline{q}}_d=0.5$, which is not a zero-input equilibrium of the system. The points used for the regression process were sampled from the set ${\mathsf{\Omega }}_{q,\rho ,ϵ}=\{\left[\begin{array}{cc}0& 1\end{array}\right]\times \left[\begin{array}{cc}{\rho }_{\infty }& 10{\rho }_{\infty }\end{array}\right]\}\times \left[\begin{array}{cc}-0.45& 0.45\end{array}\right]\}$. Please note that the values chosen for the PPC controller, as well as the set adopted for the regression process should be specifically picked to take into consideration the desired prescribed performance specifications, as well as the system’s region of operation, respectively. For the integral cost, the weights $\alpha =0.5$ and $\beta =0.5$ were chosen to specify an equal trade-off between the error convergence and the requested input energy. Moreover, $P=2500$ points were chosen within ${\mathsf{\Omega }}_{q,\rho ,ϵ}$, and the number of constraints was set to $M=1250$. The cost function was approximated by 32 polynomial basis functions, chosen as follows: $\sigma (q,\rho ,ϵ)={\left[T_{i_1}\left(q\right)T_{i_2}\left(\epsilon \right)T_{i_3}\left(\rho \right)\right]}_{i_1,i_2=\{1,\dots ,4\}\&i_3=\{1,2\}}$, where $T_i(·)$ denotes the Chebyshev polynomial of degree i. An important property of these polynomials is that they are orthogonal with respect to the inner product, rendering them ideal for performing polynomial regression.

The response of the system under the optimal policy was simulated for $3\mathrm{s}$ for five different initial conditions and is illustrated in Figure 3, where the ability of the algorithm to stabilize the system under strict performance specifications is demonstrated. The initial conditions for these simulations are included in

Table 1. Comparison between the initial PPC policy and the final optimal one.

Initial Condition $\left[\mathit{q}\right(0)$, $\mathit{\rho }\left(0\right)$, $\mathit{ϵ}\left(0\right)]$	Initial Policy’s Cost	Final Policy’s Cost	Final Policy’s Cost (Without Prescribed Performance)
$[0.73,0.148,0.45]$	$22.60$	$16.79$	$0.47$
$[0.2,0.165,0.25]$	$45.54$	$34.83$	$1.01$
$[0.56,0.174,-0.37]$	$3.21$	$1.59$	$0.05$
$[0.37,0.173,0.37]$	$6.63$	$1.4$	$0.21$
$[0.43,0.11,-0.13]$	$3.90$	$1.80$	$0.05$

, along with a comparison between the costs of the initial admissible PPC policy, the final one, and an optimal control policy obtained through a successive approximation strategy similar to [30]. The response of the system under the new optimal policy was simulated for $3\mathrm{s}$ for the same initial conditions and is illustrated in Figure 4, along with the control effort requested by it. While it is clear that this policy results in significantly less control effort and thus a decrease in cost, note that this comes at the expense of sacrificing prescribed performance, as it is dictated by the parameters of the adopted performance function. This becomes evident in Figure 4, where it can be seen that for the initial conditions $x_0^1$, the tracking error ${\sigma }_1\left(t\right)$ in our method stays within the region defined by the performance function, while the tracking error ${\sigma }_2\left(t\right)$ escapes the region, indicating the other policy’s trade-off of prescribed performance for optimality.

Regarding our method, it can be easily observed that the cost was successfully decreased, owing mainly to the reduction in the required control effort and despite the fact that we confined the system evolution within the set where the approximation capabilities of the polynomial structure held. Finally, Figure 5 shows the evolution of the transformed error. Notice that even though all initial conditions lie within ${\mathsf{\Omega }}_z$, if we obtain the weights $w_L$ through the solution of the unconstrained least squares problem (17), the trajectories of $ϵ$ escape the set ${\mathsf{\Omega }}_z$, and ${\tau }^{*}$ grows unbounded. However, by imposing a set of inequalities for $w_L$ as described in Section 4.2, the transformed error evolves strictly within ${\mathsf{\Omega }}_z$, where the polynomial approximation holds, and thus optimality along with prescribed performance specifications are met.

5.2. Case B: 2-DOF Robotic Manipulator

In this section, we present simulation results that demonstrate the effectiveness of our methodology in deriving a near-optimal control law, while taking into consideration prescribed performance specifications, for a two-degree-of-freedom robotic manipulator, as illustrated in Figure 6. The robotic manipulator obeys the following dynamic model

$$M\left(q\right)\ddot{q}+C(q,\dot{q})\dot{q}+G\left(q\right)=\tau ,$$

(26)

where $q={\left[q_1{q}_2\right]}^T$ and $\dot{q}={\left[{\dot{q}}_1{\dot{q}}_2\right]}^T$ denote the joint angular positions and velocities, respectively, $M\left(q\right)$ is a positive definite inertia matrix, $C(q,\dot{q})$ is the matrix that describes the Coriolis centrifugal phenomena, $G\left(q\right)$ is the vector describing the influence of gravity, and $\tau$ is the torque that acts as the system’s input.

More specifically, the inertia matrix is formulated as:

$$M=\left[\begin{array}{cc}{M}_{11}& M_{12}\\ M_{21}& M_{22}\end{array}\right]$$

where

$$\begin{array}{c}{M}_{11}=I_{Z_1}+I_{Z_2}+m_1\frac{l_1^2}{4}+m_2({l_1}^2+\frac{l_2^2}{4}+l_1{l}_2{c}_2)\\ M_{12}=M_{21}=I_{Z_2}+m_2(\frac{l_2^2}{4}+\frac{1}{2}{l}_1{l}_2{c}_2)\\ M_{22}=I_{Z_2}+m_2\frac{l_2^2}{4}\end{array}$$

In addition, the vector containing the Coriolis and centrifugal torques is defined as follows:

$$C(q,\dot{q})\dot{q}=\left[\begin{array}{cc}-c\dot{q_2}+k_1& -c(\dot{q_1}+\dot{q_2})\\ c\dot{q_1}& k_2\end{array}\right]\left[\begin{array}{c}\dot{q_1}\\ \dot{q_2}\end{array}\right]$$

with c being $c=\frac{1}{2}{m}_{1}g{l}_1{l}_2{s}_2$. Additionally, the gravity vector is given by

$$G\left(q\right)=\left[\begin{array}{c}\frac{1}{2}{m}_{1}g{l}_1{c}_1+m_{2}g(l_1{c}_1+\frac{1}{2}{l}_2{c}_{12})\\ \frac{1}{2}{m}_2{l}_{2}g{c}_{12}\end{array}\right]$$

(27)

and the terms $c_2,s_2$ and $c_{12}$ correspond to $cos\left(q_2\right),sin\left(q_2\right)$ and $cos(q_1+q_2)$, respectively. The values adopted for the simulation are depicted in

Table 2. Two-DOF robotic manipulator parameter values.

${\mathit{m}}_1$ (kg)	${\mathit{l}}_1$ (m)	${\mathit{I}}_{{\mathit{Z}}_1}(\mathbf{kg}·{\mathit{m}}^2)$	${\mathit{k}}_1$ (kg/s)	${\mathit{m}}_2$ (kg)	${\mathit{l}}_2$ (m)	${\mathit{I}}_{{\mathit{Z}}_2}(\mathbf{kg}·{\mathit{m}}^2)$	${\mathit{k}}_2$ (kg/s)	$\mathit{g}{(\mathit{m}/\mathit{s}}^2)$
3.2	0.5	0.96	1	2.0	0.4	0.81	1	9.81

, with $m_i$, $I_{zi}$ and $l_i$ denoting the mass, the moment of inertia and the length of link i, respectively, $k_i$ being the joint friction coefficient and g the acceleration of gravity.

In order to retrieve the manipulator’s dynamics, a reference trajectory was designed so that it traversed (in random order) 71 points scattered over the compact set ${\mathsf{\Omega }}_q\times {\mathsf{\Omega }}_{\dot{q}}=[-1,1]\times [-1,1]\times [-1,1]\times [-1,1]$. In order to track it, a PPC controller with gains $k=2$ and $\lambda =7$ was used, with performance specifications dictated by ${\rho }_0^1=\pi /3$, ${\rho }_{\infty }^1=\pi /180$, $l_1=1.5$, ${\rho }_0^2=\pi /3$, ${\rho }_{\infty }^2=\pi /180$ and $l_2=1.5$. In addition, the gains of the tracking differentiator were chosen as $k_{z_1}=k_{z_2}=0.2$ and $R={10}^5$. For the system identification phase, a shallow neural network of one hidden layer containing 12 neurons was utilized to extract the system dynamics via the learning process, and the simulation was carried out for 20 iterations so that a sufficiently accurate approximation was obtained. The results of the identification problem are depicted in Figure 7 for the corresponding terms of $f(q,\dot{q})=-M^{-1}\left(q\right)(C(q,\dot{q})\dot{q}+G\left(q\right))$ and Figure 8 for the corresponding terms of $g\left(q\right)=M^{-}1\left(q\right)$.

In the second phase, we set a conventional PPC stabilizing controller, parametrized by $k=2$, $\lambda =2$, ${\rho }_0^1={\rho }_0^2=\pi /3$, ${\rho }_{\infty }^1={\rho }_{\infty }^2=\pi /180$ and $l_1=l_2=1.5$, as the initial admissible control policy. Our aim was to drive the system towards the fixed configuration ${[{\overline{q}}_{1d},{\overline{q}}_{2d}]}^T={[\pi /6,-\pi /6]}^T$. The points used for the regression process were sampled from the set:

$$\begin{array}{c}{\mathsf{\Omega }}_{q,\rho ,ϵ}=\{\left[\begin{array}{cc}-0.5& 0.5\end{array}\right]\times \left[\begin{array}{cc}-0.45& 0.45\end{array}\right]\times \left[\begin{array}{cc}{\rho }_{\infty }^1& 10{\rho }_{\infty }^1\end{array}\right]\times \hfill \\ \hfill \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\left[\begin{array}{cc}-0.5& 0.5\end{array}\right]\times \left[\begin{array}{cc}-0.45& 0.45\end{array}\right]\times \left[\begin{array}{cc}{\rho }_{\infty }^2& 10{\rho }_{\infty }^2\end{array}\right]\}\end{array}$$

For the integral cost, the weights $\alpha =0.5$ and $\beta =0.5$ were chosen to specify an equal trade-off between the error convergence and the requested input energy. Moreover, $P=8000$ points were chosen within ${\mathsf{\Omega }}_{q,\rho ,ϵ}$, and the number of constraints was set to $M=1250$. The cost function was approximated by 64 polynomial basis functions, chosen as follows: $\sigma (q,\rho ,ϵ)={\left[T_{i_1}\left(q\right)T_{i_2}\left(\epsilon \right)T_{i_3}\left(\rho \right)\right]}_{i_1,i_2,i_3=\{1,\dots ,2\}}$, where $T_i(·)$ denotes the Chebyshev polynomial of degree i. The response of the system under the optimal policy was simulated for $3\mathrm{s}$ for four different initial conditions and is illustrated in Figure 9 and Figure 10, where the ability of the algorithm to stabilize the system under strict performance specifications is demonstrated. The initial conditions for these simulations are included in

Table 3. Comparison between the initial PPC policy and the final optimal one.

Initial Condition $[{\mathit{q}}_1\left(0\right)$, ${\mathit{\rho }}_1\left(0\right)$, ${\mathit{ϵ}}_1\left(0\right)]$, $[{\mathit{q}}_2\left(0\right)$, ${\mathit{\rho }}_2\left(0\right)$, ${\mathit{ϵ}}_2\left(0\right)]$	Initial Policy’s Cost	Final Policy’s Cost
$[0.84,0.17,0.159],[-0.12,0.108,-0.11]$	$5.24$	$1.18$
$[0.05,0.145,0.29],[-1.02,0.106,0.134]$	$7.24$	$4.16$
$[0.69,0.171,0.425],[-0.94,0.101,0.09]$	$5.69$	$1.01$
$[1.01,0.118,0.42],[-0.165,0.149,0.27]$	$5.66$	$1.39$

, along with a comparison between the costs of the initial admissible policy and the final optimal one. It is evident that the cost was successfully decreased, owing mainly to the reduction in the required control effort.

6. Conclusions

A near-optimal control policy with a predefined transient and steady-state response for uncertain Euler–Lagrange dynamics was proposed. The method consisted of a neural network identifier, capable of accurately extracting the unknown dynamics as well as an iterative process for solving the HJB equation in order to produce an optimal stabilizing control policy with prescribed performance. The aforementioned approach was demonstrated through extensive simulation studies on a pendulum and a two-degree-of-freedom robotic manipulator. Future work will focus on expanding this method so that it is applicable to other classes of systems, while taking into consideration input and state constraints.