Soft Real-Time Asynchronous Online Learning from Input–Output Data for UAV Model Reference Control Under Uncertain Dynamics and Faulty Actuation

Highlights

What are the main findings?

• Learning control for high-dimensional unknown multivariable systems is based on input–output data and takes place in a transformed virtual state-space representation that is completely observable, thus overcoming the dimensionality curse.
• Asynchronous learning exploiting multi-tasking, and smart actor–critic parameterization is a key aspect in implementing the adaptive learning control in fast real-time systems that are subject to parametric variations and actuator faults.

What are the implications of the main findings?

• The proposed model reference tracking control approach learns decoupled control starting from well-known and straightforward performance indicators like overshoot and rise time.
• Online adaptive asynchronous learning under permanent stabilizing control is ensured, while its effectiveness can be extended to more complex systems even beyond the complexity of quadrotor UAV double-integrator coupled three-channeled attitude control.

Abstract

An online off-policy asynchronous real-time model reference tracking control (OOART-MRTC) algorithm is proposed and validated for unmanned aerial vehicles (UAVs) characterized by faulty actuation and parametric uncertainty. The optimal control problem is posed based on approximate dynamic programming (ADP) and reinforcement learning (RL) theory, using a virtual state-space representation constructed exclusively on input–output true system data, which exploits the observability theory. OOART-MRTC learns control by interacting with the system, starting from an initial stabilizing controller derived from an approximate uncertain model. Learning convergence and stability under the proposed adaptive behavior are analyzed. Since the learning iterations cannot update within a sampling period, an asynchronous mechanism is proposed for updating the controller parameters, leveraging real-time control and multi-tasking. The complexity associated with the resulting high-dimensional system is solved by efficient linear parameterization and validated on a realistic case study where three coupled double integrators describe the UAV attitude control.

1. Introduction

In the control system domain, one of the most influential methods is the adaptive (or approximate) dynamic programming (ADP) (sometimes coined as neural dynamic programming; see [1,2], whose goal is learning optimal control under system model uncertainty [3,4,5,6,7,8,9,10,11]. Several mainstream ADP controls have evolved, like dual heuristic programming (DHP), heuristic dynamic programming (HDP), and global DHP (GDHP), to cope with learning either in model-free mode or in model-based mode; see, e.g., the works of [12,13,14,15,16]. The model-free setting leads to the family of action-dependent HDP (ADHDP) methods, prominently known as Q-learning in the reinforcement learning domain, which is more aligned to Artificial Intelligence.

In ADP, the rationale is that the Hamilton–Jacobi–Bellman (HJB) equation is hard or impossible to solve analytically for general nonlinear systems. Then, ADP proposes to use a so-called actor–critic architecture that employs (a) an actor that embodies the controller (sometimes called policy) and (b) a critic to approximate the state value function “V” or the more extended state action value function “Q”. The critic’s role is to appreciate the actor’s performance, and the family of Policy Iteration (PoIt) and Value Iteration (VI) algorithms was developed to solve the HJB equation forward in time. For the general unknown nonlinear systems, the actor and the critic are represented by neural network (NN) function approximators. A third algorithm called Policy Gradient (PG) can also solve the HJB, but commonly it performs backward in time in an iterative fashion and it is also less efficient than PoIt or VI. Both PoIt and VI come in several implementation flavors, like on- or off-policy, on- or off-line, adaptive or not, and incremental or using experience replay, based on buffers of transition samples. These transition samples model the underlying system’s state transition, ideally captured by a (non)linear state-space representation which in ADP is a deterministic expression of the more general Markov Decision Process (MDP).

Two factors are essential to the underlying system representation whose control is sought via ADP methods. The first and most important is the system state (sometimes called observation) which is unmeasurable for unknown system dynamics. The observability theory deeply rooted in classic control points out the solution to arriving at virtual, equivalent state-space representations, where the hidden true state is aliased by a sequence of input–output data samples that are readily measured from the system. This approach preserves the system input and output, as it only changes the internal state representation that makes the equivalent (or virtual) model a fully observable one, thus alleviating the environment partial observability issue. The second factor is the system input that serves a two-fold objective: control action and action–state space exploration. The action–state exploration is a general requirement for learning convergence of the ADP methods and it is even more critical with nonlinear systems; as with linear ones, exploration of a narrow action–state region can well generalize the optimal learned control solution towards global certificates.

The ADP methods naturally lend themselves to systems with modeling uncertainty and incomplete state measurements. Unmanned aerial vehicle (UAV) systems fall under this category due to their nonlinearity and high complexity and dimensionality. UAV control using ADP or RL methods has been tackled before either in linear or nonlinear settings; however, the specifics of uncertain models, real-time online learning capability, incomplete state measurement and stability preservation along the learning process have not been suitably approached before. In addition, the model reference control approach for UAVs, or for any kind of systems, is an essential instrument to synthesize the control performance specifications at a requirements level. It is therefore of interest to integrate such a feature in the aforementioned setting.

Early results on a 6DoF quadrotor UAV control with ADP were formulated in [17]. The learning takes place for a nonlinear model and uses NNs for the actor–critic. Although adaptive, the formulation misses the model reference setting and strong convergence and stability results. The work from [18] studied attitude and longitudinal motion of a quadrotor UAV based on a linearized model, in an adaptive control setting but without an actor–critic. Instead, classical LQR is combined with model reference adaptive control for a partially known model with uncertainty, and stability is assessed. The work’s innovation also includes faults on the control inputs actuation; however, all states are measured and the model-free design is not fully approached under partial observability. Stronger experimental evidence validated the approach. In another work, ref. [19] contributes a computationally-efficient method to store a Q-function generalization, a continuous action selection based on local Q-function approximation, and a combination of model identification and online learning for inner-loop control of a UAV system. A model reduction is performed to facilitate learning and overcome the curse of dimensionality. A reference tracking problem formulation was proposed; however, the model reference and convergence and stability analysis were not included. In [20], a novel RL for a thrust-vectoring controlled quadcopter was proposed, based on Proximal Policy Optimization (PPO). The reward was crafted to steer the flying robot towards its specified waypoint starting from its current location. The nonlinear dynamics complexity was used in the actor–critic; nevertheless, the model reference tracking, partial observability, and convergence and stability analysis were not research topics.

In [21], a quadratic programming neural dynamic controller is applied to UAVs for tracking control purposes. A complex numerical model was employed with a more conventional model-based design under uncertainty and validated in a thorough case study consisting of many flight modes. The observability issues and complete model-free design were not tackled. A deep RL (DRL) adaptive controller for an aerial robot was proposed in [22] and shown to be superior to traditional PID controllers; however, simulated environments do not pose the challenges associated with real-time interaction control. The authors of [23] relayed an improved tabular Q-learning for robust attitude control of a fixed-wing UAV, when affected by atmospheric disturbances, sensor noise, actuator faults, and other model uncertainty. Although the learning is adaptive, convergence and stability were not explicitly analyzed. In [24], a deep RL intelligent nonlinear controller for an experimental gliding UAV was validated using a modified DDPG algorithm. A sophisticated reward design combined with many simulation episodes leads to an optimal solution deemed superior to LQR and PPO. The results are missing convergence and stability analysis, although applied to a nonlinear problem model. A fully integrated RL approach combined with adaptive control was proposed in [25] for uncertain system control with reference model, in a real-time setting with stability analysis and actuator fault. However, full state measurement was considered even when the results were validated in a real-time UAV, and no partial environment observability was tackled. The authors of [26] proposed a robust optimal safe and stability certified control validated for UAV reference trajectory tracking under parametric uncertainty, under PPO RL. The real-time mechanisms are not analyzed, and the partial observability is not approached in this work. Ref. [27] proposed an optimal gain self-tuning approach for altitude, attitude, and longitudinal motion controllers of a 6DoF nonlinear drone, using the DRL mechanism combined with a custom reward. Theoretical stability is certified; however, the real-time interaction mechanism and the partial state observation are not considered, either.

One of the greater challenges with online real-time ADP and RL control concerns the interleaving of the update equations, data collection, and fixed sampling period control execution. In fact, this has been the major bottleneck against a more widespread adoption of the ADP methods. Even in the linear system case, the actor and critic updates may require working with many transition samples of high dimension to improve method conditioning, which makes the learning steps impossible to update within the system’s sampling period. This forces the actor and critic structure update to be performed in asynchronous mode [28], using specialized software mechanisms. The issue becomes even more critical with generalized NN-based actor–critic approaches when real-time interaction is required [29,30,31,32,33]. The problem size becomes larger with the state-action size, which itself grows when using a virtual state representation comprised of present and past IO system samples. The environment’s complete observability issue under real-time control, about how to build equivalent fully observable virtual states, has not been tackled systematically [34]. Another issue concerns the system stability preservation along the adaptive updates, which is necessary for safety insurance [35]. Addressing these two challenges may help with the adoption of ADP and RL methods in real-world systems and this idea drives the current contribution. Finally, in a model reference control context, the trajectory feasibility must account for the underlying dynamics, which certainly requires some qualitative system insight and consideration for several classic control rules.

The contributions in this work are enumerated as follows:

-

A theory for learning optimal control dealing with uncertain model dynamics and actuation faults, in a model reference tracking control approach, with equivalent virtual state-space representation where the virtual state is encoded as a moving window of input–output system samples.

-

The proposed reference model is different from those existing in the literature, as it originates from easily interpretable performance indicators like rise time and overshoot, which are widely adopted in the community.

-

Learning convergence of the proposed online off-policy asynchronous real-time model reference tracking control (OOART-MRTC) algorithm, with stability certification alongside learning, when starting from initially stabilizing control.

-

Validation of the OOART-MRTC on attitude control for a quadrotor UAV serves as a use case study, to show that a complex double-integrator with coupled channels transforms to a large virtual state-space representation consisting of tens of variables. Even under such dimensionality curse, the learning can still be effective, fast, convergent and stabilizing in online asynchronous mode.

Notation used: $R^m$ is the set of real-valued m-dimensional vectors which are column vectors by default. Upper-right “$T$” indicates matrix transpose. The norm operator ‖.‖ is implicitly Euclidean, over some vector or induced over some matrix. $I_n$ is the identity matrix with size $n\times n$.

The paper is organized as follows. The Section 2 presents the general controlled system dynamics converted to a virtual state-space representation, the tracked reference model, the reference model driving input, and the resulting extended state space system. Once set up, the model reference tracking optimal control problem is formed and the parameterized ADP-based solution is derived under the OOART-MRTC algorithm. It is still in this section that the learning convergence analysis is performed. The Section 3 is concerned with a complex results case study: a reference model tracking attitude control for a multivariable quadrotor UAV with an actuator fault and an uncertain model. A critical results discussion and interpretations are offered in the Section 4. The Section 5 concludes the findings.

2. Materials and Methods

2.1. The Controlled Uncertain Dynamic System

The original system’s state-space transition model with output equation is defined for the linear-time-invariant case as

$$\begin{array}{c}{x}_{k+1}=A_o{x}_k+B_o{u}_k,\\ y_k=C_o{x}_k+D_o{u}_k,\end{array}$$

(1)

where $x_k\in {\mathbb{D}}_x\subset {\mathbb{R}}^n$ is the state within its domain ${\mathbb{D}}_x$, $u_k\in {\mathbb{D}}_u\subset {\mathbb{R}}^m$ is control input within its domain ${\mathbb{D}}_{\mathrm{u}}$, $y_k\in {\mathbb{D}}_y\subset {\mathbb{R}}^p$ is the controlled output within its domain ${\mathbb{D}}_{\mathrm{y}}$. Subscript $k$ indexes the discrete time sample herein. Herein, the matrices $A_o, B_o, C_o, D_o$ are of appropriate dimensions. They are assumed generally unknown and they are being referred to as the true matrices. Their values could be expressed under additive or multiplicative uncertain formulation. The matrix $B_o$ could be expressed as $B_o=B_{nom}\cdot \Delta B\cdot \Lambda$, where $\Lambda$ models an actuation fault present in the control input $u_k$ whenever the value differs from $\Lambda =I_m$, and $\Delta B$ represents the deviation of the input matrix from its nominal value $B_{nom}$. This fault type described by multiplicative matrix $\Lambda$ is a form of partial effectiveness loss and not a complete malfunction. Similar arguments are possible for representing uncertainty about the remaining state matrices $A_o, C_o, D_o$, using their nominal counterparts. The described form (1) is more general, as it is able to capture time delays in the variables $u_k,x_k,y_k$ as well. This is provable by augmenting (1) with additional state components and it is a specific feature of discrete-time system representation.

Assumption 1.

The system described by (1) is both controllable and observable.

Following Assumption 1, an equivalent representation of (1), called a virtual system, is, in its state-space form

$$\begin{array}{c}{v}_{k+1}=A_v{v}_k+B_v{u}_k,\\ y_k=C_v{v}_k+D_v{u}_k,\end{array}$$

(2)

where $v_k\in {\mathbb{D}}_v\subset {\mathbb{R}}^v$ is called a virtual state vector. The innovation behind $v_k$ is that it consists of the input/output (IO) present and past values, i.e., $v_k≔ {\left[y_k^T,y_{k-1}^T,\dots ,y_{k-ny}^T,u_{k-1}^T,\dots ,u_{k-nu}^T\right]}^T$ with appropriate size of $v=\left(ny+1\right)\times p+nu\times m$, where $ny$ is the output order, $nu$ is the input order. Most notably, (1) and (2) share the same inputs and outputs. This means that a suitable controller for system (2) in fact uses the IO data from the system (1) and elaborates the control for (2), which in fact is equivalent to controlling the original system (1). From the IO behavior perspective, systems (1) and (2) are alike [34,36]. What is more appealing about (2) is that $v_k$ is a linear transform on $x_k$, which is captured by the next Theorem.

Theorem 1.

For an observable and controllable system (1), there exists a transformation matrix $\mathcal{T}$ which enables $x_k=\mathcal{T}{v}_k$.

Proof of Theorem 1.

See Appendix A. □

In the light of Theorem 1 previously introduced, it is well-known that for the integer size $v\ge v_{minimal}$ (or equivalently for orders $ny,nu$ greater than some minimal thresholds), more IO samples captured in $v_k$ do not bring more information about the unique state value $x_k$. The value of $v$ indirectly set by the values $ny,nu$ trades off the high dimensionality of $v_k$ with the inter-correlation between IO samples, i.e., it balances the exploration–exploitation factor. In practice, we start with $v$ corresponding to a minimal order known from historical analysis about the system (1) and increment it in unit steps, until there is no more gain in control performance.

2.2. The Reference Model to Be Tracked in Output and Matched in Dynamics

To force a desired control behavior for system (1) through its equivalent description (2), the most natural approach stemming from classic control is the reference model. This is, in fact, a supervised machine learning approach where one models the controller such that the closed loop matches a reference model’s dynamics [37]. For the single input–single output case, a reference model is commonly described by the linear transfer function $H\left(s\right)={\displaystyle \frac{y^m\left(s\right)}{r\left(s\right)}}={\displaystyle \frac{w_n^2}{s^2+2\xi w_{n}s+w_n^2}}{e}^{-s{T}_m}$ parameterized in the natural frequency $w_n$, the damping ration $\xi$, and the time delay $T_m$ with Laplace argument $s$. $y^m$(s) is the Laplace-domain reference model output, with $r\left(s\right)$ being the Laplace image of the reference input time signal. Generally, it is more intuitive to describe control performance in continuous-time and only afterwards discretize the transfer function to match the time-based description with systems (1) and (2) pertaining to the discrete-time representation. The parameter $w_n$ is tightly related to the rise time while $\xi$ determines the overshoot. Such relationships are found in families of characteristics or from tables. For a second order reference model, $\xi$ produces percent overshoot of $M_p=e^{-{\displaystyle \frac{\xi \pi }{\sqrt{1-{\xi }^2}}}}\times 100$[%], while the rise time is $t_r={\displaystyle \frac{\pi -\varphi }{w_n \sqrt{1-{\xi }^2}}}$ for $\varphi =\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\left({\displaystyle \frac{\sqrt{1-{\xi }^2}}{\xi }}\right)$.

In multivariable control, a transfer matrix reference model is commonly employed, which in most cases is also diagonal, in order to indirectly enforce control channel decoupling to the maximum extent possible, which greatly improves disturbance rejection resilience and debugging in the control performance. In model reference control, if we use a linear reference model for a nonlinear plant, a nonlinear controller is necessary if we want the closed loop to match the linear response of the reference model. A plain linear controller cannot achieve such a requirement. With a nonlinear controller, we aim for good closed-loop linearization. Such a behavior is very predictable, both in scale and in response, over wide operating ranges. This is a very desirable control systems feature. This kind of generalizability emerges from the linear systems superposition principle. A side note about the reference model: systems like (1) or (2) may also possess a non-minimum phase character, which is a kind of dynamics which should not be compensated for via the controller dynamics; hence, it must be accepted and forced as an explicit component in $H\left(s\right)$.

Eventually, $H\left(s\right)$ needs to be discretized and, for online ADP and RL control, it also needs to be transformed to a suitable state-space model. The final reference model state space would be described as

$$\begin{array}{c}{x}_{k+1}^m=A_m{x}_k^m+B_m{r}_k,\\ y_k^m=C_m{x}_k^m+D_m{r}_k,\end{array}$$

(3)

with $x_k^m$ the reference model’s state of appropriate and known size, $r_k,y_k^m\in {\Omega }_y$ being the reference inputs and reference model outputs, respectively, pinpointing that they must live in the same space as the original system’s outputs. The tuple ($A_m, B_m,C_m,D_m$) is one state-space realization out of infinitely many possible. This reference model is more flexible than most in the literature found in ADP and RL, since it stems from performance indicators which are familiar to most control system designers. The reference model output must be controllable and observable. For accuracy, we formally assume it.

Assumption 2.

The reference model (3) is both controllable and observable.

2.3. The Driving Reference Input Model

A linear reference input model is next employed with the dynamics

$$r_{k+1}=A_r{r}_k,$$

(4)

where $A_r$ is $p\times p$ and compatible with the number of controlled outputs. The reference input $r_k$ drives both the closed-loop control system (to be designed for (1), based on (2), as shown in the next subsections) and the reference model.

2.4. The Extended State-Space System

To arrive at a final state-space model that complies with the ADP and RL formulation, we define ${\chi }_k={\left[v_k^T,{\left(x_k^m\right)}^T,r_k^T\right]}^T$. Then the augmented system is

$$\begin{gathered} {\chi }_{k+1}=\left[\begin{array}{ccc}{A}_v& 0& 0\\ 0& A_m& B_m\\ 0& 0& A_r\end{array}\right]{\chi }_k+\left[\begin{array}{c}{B}_{\nu }\\ 0\\ 0\end{array}\right]u_k≔A{\chi }_k+B{u}_k \\ {y}_k=C_y {\chi }_k \\ {y}_k^m=C_{ym} {\chi }_k, \end{gathered}$$

(5)

For ${\chi }_k=\left[v_k^T,{\left(x_k^m\right)}^T,r_k^T\right]$ the size of the extended state vector results from the sizes of its independent components, while $C_y, C_{ym}$ are appropriate matrices extracting the original system’s outputs and the reference model outputs from ${\chi }_k$, respectively. This augmented system is next used for learning optimal control: it is fully state observable and controllable. The augmented system from (5) is fully observable and controllable by construction. The reference input $r_k$ from (4), although being a state within ${\chi }_k$ in (5) and not a traditional control input, is a user-set variable whose dynamics are selected beforehand. This $r_k$ drives the reference model, which is selected to be fully observable and controllable; otherwise, its output is not to be tracked. The virtual state space $v_{k+1}=A_v{v}_k+B_v{u}_k$ in (2) is fully observable since its state is built from IO measured data. Its controllability follows, as it has the same input and output as (1), which is assumed controllable. For this to happen, sufficient past IO samples must fill the shift buffers from $v_k$.

2.5. The Model Reference Tracking Cost in the Optimal ADP/RL Control

To induce a control learning behavior, the undiscounted infinite-horizon cost-to-go is

$$J\left({\chi }_k\right)={\sum }_{i=0}^{\infty }\rho ({\chi }_{k+i},u_{k+i}),$$

(6)

where $\rho ({\chi }_k,u_k)={‖y_k\left({\chi }_k\right)-y_k^m\left({\chi }_k\right)‖}^2+u_k^{T}R{u}_k$ captures the normed tracking error between the controlled system output and the reference model output, across all channels, and also penalizes the control effort. Generally, in the model reference tracking problem, no control penalty is used; hence, it is a little different from the classical LQR penalty. To solve the model reference tracking as an optimal control, we proceed as follows:

• Define the controller-dependent cost $J_{\mathfrak{C}} \left({\chi }_k\right)={\sum }_{i=0}^{\infty } \rho \left({\chi }_{k+i}\right)$ to make sure that, starting from every state ${\chi }_k$, the subsequent state transitions to $x_{k+1},x_{k+2},\dots$ are performed under the impact of the controller $u_k=\mathfrak{C}\left({\chi }_k\right)$ in its most general form. This controller form will be linear in the case of linear system, i.e., $u_k=\mathfrak{C}\left({\chi }_k\right)=K^T{\chi }_k$.
• To solve for the optimal control ${\mathfrak{C}}^{*}=\mathrm{a}\mathrm{r}\mathrm{g} \underset{\mathfrak{C}}{\min }{J}_{\mathfrak{C}} \left({\chi }_k\right)={\sum }_{i=0}^{\infty } \rho ({\chi }_{k+i})$, the ADP (RL) methods generally require the system model to be accurately known. To solve for the optimal control independently of the system model, introduce the extended cost $Q_{\mathfrak{C}} \left({\chi }_k,u_k\right)=\rho \left({\chi }_k,u_k\right)+{\sum }_{i=k+1}^{\infty } \rho \left({\chi }_i\right)$. This is the well-known Q-function, and it obeys the Bellman equation as

  $$Q_{\mathfrak{C}} \left({\chi }_k,u_k\right)={\rho }_i\left({\chi }_k,u_k\right)+Q_{\mathfrak{C}} \left({\chi }_{k+1},\mathfrak{C}\left({\chi }_{k+1}\right)\right)={\rho }_i\left({\chi }_k,u_k\right)+J_{\mathfrak{C}} \left({\chi }_{k+1}\right).$$

  (7)
• Smartly parameterize the Q-function and the controller using polynomials, or neural networks in the most general case.
• Simultaneously learn the optimal Q-function and the optimal controller in an off-policy style, using a dataset of transition samples collected from the system under any controller, most often an existing stabilizing one that is suboptimal and it is employed for good exploration. This dataset is called the Experience Replay Buffer (ERB).

For a linear system like (5), the classical LQR theory proves that the cost of operation under a linear controller is $J_{\mathfrak{C}} \left({\chi }_k\right)={\chi }_k^{T}P{x}_k$, quadratic in the state, under the controller $\mathfrak{C},$ with $P\succcurlyeq 0$ some symmetric positive definite matrix, and the optimal controller linear in the state, i.e., $u_k=K^T{\chi }_k$.

Proposition 1.

For the model reference tracking problem with the cost (6), the penalty complies with an LQR penalty formulation, having state and action penalty matrices $\overline{Q},R$, as in ${\rho }_k={\chi }_k^T \overline{Q}{\chi }_k+u_k^{T}R{u}_k$, where $\overline{Q}\succcurlyeq 0,R\succ 0$ are semipositive definite and positive definite, respectively.

Proof of Proposition 1.

It follows immediately since ${\rho }_k={‖y_k\left({\chi }_k\right)-y_k^m\left({\chi }_k\right)‖}^2+u_k^{T}R{u}_k={\left(y_k\left({\chi }_k\right)-y_k^m\left({\chi }_k\right)\right)}^T\left(y_k\left({\chi }_k\right)-y_k^m\left({\chi }_k\right)\right)+u_k^{T}R{u}_k={\chi }_k^T\left(C_y^T-C_{ym}^T\right)\left(C_y-C_{ym}\right){\chi }_k+u_k^{T}R{u}_k={\chi }_k^T \overline{Q}{\chi }_k+u_k^{T}R{u}_k$, where $\overline{Q}\succcurlyeq 0,R\succ 0$. □

Subsequently, it is straightforward to show that the extended Q-function cost is also quadratic in the argument $z_k={\left[{\chi }_k^T,u_k^T\right]}^T$, i.e., $Q_{\mathfrak{C}} \left({\chi }_k,u_k\right)=z_k^{T}H{z}_k$, with $H\ge 0$ some symmetric positive definite matrix, while the optimal control still remains linear in the state. The most important utility of the Q-function is that the control improvement is found directly from the condition ${\displaystyle \frac{d{Q}_{\mathfrak{C}}\left({\chi }_k,u_k\right)}{d{u}_k}}=0$, and it is analytically solvable with the mentioned parameterization.

A linear state feedback controller of the form $u_k=K^T{\chi }_k$ with gain matrix $K$ partitioned as $K^T=\left[K_1^T\right|K_2^T\left|K_3^T\right]$ renders the augmented closed-loop as

$${\chi }_{k+1}=\left[\begin{array}{ccc}{A}_v+B_v{K}_1^T& B_v{K}_2^T& B_v{K}_3^T\\ 0& A_m& B_m\\ 0& 0& A_r\end{array}\right]{\chi }_k=A_{cl}{\chi }_k.$$

(8)

The Q-function is linearly parameterized by retaining the unique up-to-second-degree polynomials resulting from the argument products. Let $Q_{\mathfrak{C}}\left(z_k\right)$ be expressed for $z_k={\left[{\chi }_k^T,u_k^T\right]}^T≔{\left[z_{k,1},z_{k,2},\dots ,z_{k,t}\right]}^T\in {\mathbb{R}}^t$ as a t-dimensional vector as

$$Q_{\mathfrak{C}}\left(z_k|\theta \right)={\theta }_1{z}_{k,1}^2+{\theta }_2{z}_{k,1}{z}_{k,2}+\dots +{\theta }_t{z}_{k,1}{z}_{k,t}+{\theta }_{t+1}{z}_{k,2}^2+{\theta }_{t+2}{z}_{k,2}{z}_{k,3}+\dots +{\theta }_{{\displaystyle \frac{t\left(t+1\right)}{2}}}{z}_{k,t}^2:={\phi }^T\left(z_k\right)\theta ,$$

(9)

where $\theta \in {\mathbb{R}}^{{\displaystyle \frac{t\left(t+1\right)}{2}}}≔{\left[{\theta }_1,\dots ,{\theta }_{{\displaystyle \frac{t\left(t+1\right)}{2}}}\right]}^T$ lumps all the parameters together and $\phi \left(z_k\right)\in {\mathbb{R}}^{{\displaystyle \frac{t\left(t+1\right)}{2}}}≔{\left[z_{k,1}^2,z_{k,1}{z}_{k,2},\dots ,z_{k,t}^2\right]}^T$ is a basis function vector called nonlinear embedding.

Remark 1.

It follows immediately from (9) that the Q-function takes the form $Q_{\mathfrak{C}}\left(z_k|\theta \right)=z_k^{T}H{z}_k$ with $H≔\left[\begin{array}{cccc}{\theta }_1& {\theta }_2/2& \dots & {\theta }_t/2\\ {\theta }_2/2& {\theta }_{t+1}& \dots & {\theta }_{2t-1}/2\\ ⋮& ⋮& \ddots & ⋮\\ {\theta }_t/2& {\theta }_{2t-1}/2& \dots & {\theta }_{{\displaystyle \frac{t\left(t+1\right)}{2}}}\end{array}\right]$ being a positive definite symmetric matrix, to be proven later.

Let the transition samples database in the ERB be $DS=\left\{\left({\chi }_k^{\left[i\right]},u_k^{\left[i\right]},{\chi }_{k+1}^{\left[i\right]}\right)\right\},i=1,\dots , N_D$. For the previous iteration controller $K_{i-1}$, with known previous iteration Q-function parameter ${\theta }_{i-1},$ we use a number $N_T$ of transition samples from the ERB, to write the following VI update in the Q-function parameter space

$$\left[\begin{array}{c}{\phi }^T\left({\chi }_k^{\left[1\right]},u_k^{\left[1\right]}\right)\\ {\phi }^T\left({\chi }_k^{\left[2\right]},u_k^{\left[2\right]}\right)\\ ⋮\\ {\phi }^T\left({\chi }_k^{\left[N_T\right]},u_k^{\left[N_T\right]}\right)\end{array}\right]{\theta }_i=\left[\begin{array}{c}{\rho }_i\left({\chi }_k^{\left[1\right]},u_k^{\left[1\right]}\right)+{\phi }^T\left({\chi }_{k+1}^{\left[1\right]},K_{i-1}^T{\chi }_{k+1}^{\left[1\right]}\right){\theta }_{i-1}\\ {\rho }_i\left({\chi }_k^{\left[2\right]},u_k^{\left[2\right]}\right)+{\phi }^T\left({\chi }_{k+1}^{\left[2\right]},K_{i-1}^T{\chi }_{k+1}^{\left[2\right]}\right){\theta }_{i-1}\\ ⋮\\ {\rho }_i\left({\chi }_k^{\left[N_T\right]},u_k^{\left[N_T\right]}\right)+{\phi }^T\left({\chi }_{k+1}^{\left[N_T\right]},K_{i-1}^T{\chi }_{k+1}^{\left[N_T\right]}\right){\theta }_{i-1}\end{array}\right],$$

(10)

which can be expressed as an overdetermined system $\tilde{M}{\theta }_i=\tilde{N}$ and be solved accordingly as ${\theta }_i={\left({\tilde{M}}^T\tilde{M}\right)}^{-1}{M}^T\tilde{N}$. In Value Iteration, we use parameterized notation $Q_i\left(x_k,u_k\right)={\phi }^T\left({\chi }_k,u_k\right){\theta }_i$ for the i-th iteration Q-function. In the right hand side of the VI update, $K_{i-1}=\underset{K}{\min } Q_{i-1}\left({\chi }_{k+1}, u_k\right)=\underset{K}{\min } {\phi }^T\left({\chi }_{k+1},K^T{\chi }_{k+1}\right){\theta }_{i-1}$.

Remark 2.

The major advantage of (10) is that it is model-free and independent of system matrices $\mathrm{A},\mathrm{B}$.

Remark 3.

Equation (10) is a policy estimate update and not a policy evaluation. If ${\theta }_i$ was used in the right-hand side, the available database $DS$ helps evaluating the Q-function of any arbitrary policy $K$ that fits into the expression of ${\phi }^T\left({\chi }_{k+1}^{\left[j\right]},K^T{\chi }_{k+1}^{\left[j\right]}\right)$.

Remark 4.

To solve VI policy estimate update (10) using ${\theta }_i={\left({\tilde{M}}^T\tilde{M}\right)}^{-1}{\tilde{M}}^T\tilde{N}$, the matrix $\left({\tilde{M}}^T\tilde{M}\right)$ needs to be invertible, requiring $\tilde{M}$ to be full column rank. This is achieved by sufficiently exploratory $u_k^{\left[j\right]}$, commonly obtained by noise injection, to decorrelate the tuples $\left({\chi }_k^{\left[j\right]},u_k^{\left[j\right]}\right)$ from each other. Accordingly, this decorrelates the expansions ${\phi }^T\left({\chi }_k^{\left[j\right]},u_k^{\left[j\right]}\right)$, which are the columns for $\tilde{M}$.

With ${\theta }_i$ found from (10), we proceed with updating the controller and finding $K_i$ from the condition ${\displaystyle \frac{d{Q}_{\mathfrak{C}}\left({\chi }_k,u_k|\theta \right)}{d{u}_k}}=0$. Let this controller be

$$K_i=L\left({\theta }_i\right),$$

(11)

where $L(.)$ is some analytically derived expression of ${\theta }_i$ resulting from the solution ${\displaystyle \frac{d{Q}_{\mathfrak{C}}\left({\chi }_k,u_k^{*}|\theta \right)}{d{u}_k}}=0$. It follows that the stationary point $u_k^{*}=K_i^T{\chi }_k$ is to be linear in the state. The analytical derivation is generally possible only for well-posed parameterizations such as the quadratic one used herein.

Example 1.

Suppose that ${\chi }_k={\left[{\chi }_{k,1},{\chi }_{k,2}\right]}^T$ and $u_k\in \mathbb{R}$. Then the nonlinear embedding vector is $\phi \left(z_k\right)={\left[{\left({\chi }_{k,1}\right)}^2,{\chi }_{k,1} \cdot {\chi }_{k,2}, {\chi }_{k,1}\cdot u_k, {\left({\chi }_{k,2}\right)}^2, {\chi }_{k,2}\cdot u_k, {\left(u_k\right)}^2\right]}^T\in {\mathbb{R}}^6$ and $Q\left(z_k|\theta \right)={\theta }_1{\chi }_{k,1}^2 +{\theta }_2{\chi }_{k,1} \cdot {\chi }_{k,2}+{\theta }_3 {\chi }_{k,1}\cdot u_k+ {\theta }_4{\chi }_{k,2}^2+ {\theta }_5{\chi }_{k,2}\cdot u_k,+ {\theta }_6{u}_k^2$. The solution to ${\displaystyle \frac{d{Q}_{\mathfrak{C}}\left({\chi }_k,u_k^{*}|\theta \right)}{d{u}_k}}=0$ is $u_k^{*}=-[{\displaystyle \frac{{\theta }_3}{{\theta }_6}}, {\displaystyle \frac{{\theta }_5}{{\theta }_6}}]\left[\begin{array}{c}{\chi }_{k,1}\\ {\chi }_{k,2}\end{array}\right]$, which confirms the linear dependence in the state, of the form $u_k=K^T{\chi }_k$.

Next, we propose the online off-policy asynchronous real-time algorithm for the model reference tracking control (OOART-MRTC) problem.

2.6. The Summarized Algorithm—OOART-MRTC

The online off-policy asynchronous real-time algorithm used for model reference trajectory tracking is given below in Algorithm 1:

Algorithm 1. The OOART-MRTC algorithm

1. Input parameters: $K_{-1},{\mathrm{\theta }}_0$ that characterizes $Q_0≔Q_{K_{-1}}$, the minibatch size $N_T$, orders $ny,nu$. 2. Reset the environment $x_0^m$, $r_0$, $v_0$ and fill the IO buffers with appropriate data. 3. With the initially stabilizing controller $K_{-1}$ in closed loop, collect transitions to fill in the ERB, using exploratory noise for good state-action coverage. 4. Set update iteration index $i=0$. 5. repeat (in the main thread, this is the step function with time-critical operations) 6.    read $y_k,y_k^m$. Push the newest $y_k$ to the outputs buffer. 7.    form the current sample sliding window virtual state $v_k^T≔ \left[y_k^T,y_{k-1}^T,\dots ,y_{k-ny}^T,u_{k-1}^T,\dots ,u_{k-nu}^T\right]$ from the IO buffers. 8.    construct the augmented state ${\chi }_k={\left[v_k^T,{\left(x_k^m\right)}^T,r_k^T\right]}^T$. 9.    calculate $u_k=K_i^T{\chi }_k$. 10.  send $u_k+{\eta }_k$ to the system, where ${\eta }_k$ is exploratory noise. 11.  compute the penalty ${\rho }_k$ at the current timestep. 12.  push the transition sample $\left({\chi }_{k-1},u_{k-1},{\chi }_k\right)$ in the ERB. 13.  push the latest $u_k$ to the inputs buffer. 14.  set ${\chi }_{k-1}\leftarrow {\chi }_k$. 15.  update the reference input based on the generative model $r_{k+1}=R\left(r_k\right)$. 16.  update the reference model state according to $x_{k+1}^m=A_r{x}_k^m+B_r{r}_k$. 17.  make $k=k+1$. 18.  sleep until the current timestep $\Delta T$ has elapsed. 19.  until experiment is stopped. 20.  21.  repeat (in a separate thread where the learning iterative updates take place) 22.  sample the ERB with random $N_T$ transitions. 23.  fill the matrices in (10) using the transition samples. 24.  if $i==0$, 25.   find ${\theta }_0$ for $K_{-1}$, using the policy evaluation variant of (10) with ${\theta }_0$ on both sides. 26.  else 27.  find ${\theta }_i$ based on $K_{i-1},{\theta }_{i-1}$ using Equation (10). 28.  find $K_i$ from ${\theta }_i$ using Equation (11). 29.  increment the iteration index $i=i+1$. 30.  until convergence or main thread is stopped

Specifics of the OOART-MRTC algorithm are discussed. The main computing thread first performs the time-critical operations (lines 6–10 in the OOART-MRTC algorithm), among the most important being outputs measurements, augmented state construction by moving a window over the IO data samples, control action computation, and sending to the environment, either under exploration mode (with noise) or without it. For real-time operation and interaction between the controller and the environment, resilient communication backbones are to be adopted, either low-level, like socket-based, or high-level, like Robot Operating System (ROS)-based [38]. Some less critical but still important management tasks are (lines 11–17 in the OOART-MRTC algorithm): the transition samples are collected in this thread also and pushed to the ERB, the reference input generative model and the reference model state-space are both updated for one timestep. The remaining time left from the sampling period $\Delta T$ is slept until the next interaction and control cycle begins.

In the parallel executing thread, the learning updates take place by sampling the ERB randomly for transitions and updating the critic and actor equations. Depending on the number of transitions, the time taken for these tasks may often exceed $\Delta T$; hence, the asynchronous update is imperative. Dedicated software mechanisms for preventing race conditions on reading and writing variables are required (like mutexes); see [37]. The locked variables are (1) the actor weights $K$ for reading, in order to make atomic inference on the actor in the control thread, without updating these weights from the actor–critic learning thread at the same time; (2) the ERB, for sampling in the actor–critic thread, to prevent race condition when updating it in the control main thread.

2.7. Learning Convergence Analysis for the OOART-MRTC

We formalize the convergence analysis by reparametrizing the VI updates and by setting some assumptions.

Assumption 3.

There exists an initial admissible controller $K_{-1}$ with cost-to-go $J_{K_{-1}}\left(\chi \right)={\chi }^T{P}_{-1}\chi$ with $P_{-1}=P_{-1}^T\succcurlyeq 0$ positive definite symmetric. Let ${\chi }_{+}=A\chi +Bu$ define a transition sample tuple $\left(\chi , u,{\chi }_{+}\right)$. Then the corresponding Q-function is $Q_0\left(\chi ,u\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+Q_0\left({\chi }_{+},K_{-1}{\chi }_{+}\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+J_{K_{-1}}\left({\chi }_{+}\right)=\rho (\chi , u)+{\chi }_{+}^T{P}_{-1}{\chi }_{+}$.

Define the unparameterized VI policy estimate update, which starts with $Q_0$ and uses the greedy updates, as

$$Q_i\left(\chi , u\right)=\rho \left(\chi , u\right)+\underset{u}{\min } Q_{i-1}({\chi }_{+},u).$$

(12)

Remark 5.

The value $Q_0$ is a true policy evaluation for $K_{-1}$ hence strongly related to $J_{K_{-1}}$, while the subsequent $Q_i$ will not represent true policy evaluations for the subsequent controllers $K_i$; hence, these updates will not be tightly related to costs-to-go, but only loose approximations. Moreover, the existence of $P_{-1}=P_{-1}^T\succcurlyeq 0$ defining the value function of using a stabilizing $K_{-1}$ is guaranteed.

Theorem 2.

The sequence $Q_i$ is monotonically decreasing as in $Q_i\left(\chi , u\right)\le Q_{i-1}\left(\chi , u\right)\le \dots \le Q_0(\chi , u)$ and $Q_i\ge 0$. Moreover, $Q_i\left(\chi ,u\right)={\left[\begin{array}{c}\chi \\ u\end{array}\right]}^T{H}_i\left[\begin{array}{c}\chi \\ u\end{array}\right]≔z^T{H}_{i}z$ preserves its quadratic form over the updates.

Proof of Theorem 2.

$Q_0\left(\chi ,u\right)=\rho \left(\chi , u\right)+{\chi }_{+}^T{P}_{-1}{\chi }_{+}\ge 0$ follows from $\overline{Q}\succcurlyeq 0,R\succ 0,P_{-1}\succcurlyeq 0$. Based on ${\chi }_{+}=A\chi +Bu$, we have $Q_0\left(\chi ,u\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+{\left(A\chi +Bu\right)}^T{P}_{-1}(A\chi +Bu)$. Moreover, $Q_0\left(\chi ,u\right)={\left[\begin{array}{c}\chi \\ u\end{array}\right]}^T{H}_0\left[\begin{array}{c}\chi \\ u\end{array}\right]≔z^T{H}_{0}z$ is quadratic, with $H_0=\left[\begin{array}{cc}Q+A^T{P}_{-1}A& B^T{P}_{-1}A\\ A^T{P}_{-1}B& R+B^T{P}_{-1}B\end{array}\right]\succcurlyeq 0$ and $H_0=H_0^T$. □

To find $Q_1$ from ${\mathrm{Q}}_0$ based on (12) written as $Q_1\left(\chi , u\right)=\rho \left(\chi , u\right)+\underset{u}{\min } Q_0({\chi }_{+},u)$, we compute $u^{*}=arg \underset{u}{\min } Q_0({\chi }_{+},u)$ from ${\displaystyle \frac{d{Q}_0\left({\chi }_{+},u\right)}{du}}=0$ and arrive at

$$u^{*}=-{\left(R+B^T{P}_{-1}B\right)}^{-1}{B}^T{P}_{-1}A{\chi }_{+}=K_0{\chi }_{+}.$$

(13)

Then

$$\underset{u}{\min } Q_0({\chi }_{+},u)=Q_0\left({\chi }_{+},K_0{\chi }_{+}\right)={\chi }_{+}^T \left(\overline{Q}+K_0^{T}R{K}_0\right){\chi }_{+}+{\chi }_{+}^T \left(A^T+K_0^T{B}^T\right)P_{-1}\left(A+B{K}_0\right){\chi }_{+}≔{\chi }_{+}^T{P}_0{\chi }_{+},$$

(14)

with

$$P_0=\overline{Q}+K_0^{T}R{K}_0+\left(A^T+K_0^T{B}^T\right)P_{-1}\left(A+B{K}_0\right)\succcurlyeq 0,$$

(15)

since $\overline{Q}\succcurlyeq 0,R\succ 0,P_{-1}\succcurlyeq 0$.

It follows that

$$Q_1\left(\chi , u\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+{\chi }_{+}^T{P}_0{\chi }_{+}\ge 0,$$

(16)

since $\overline{Q}\succcurlyeq 0,R\succ 0,P_0\succcurlyeq 0$ and also that $Q_1$ is quadratic in its argument by writing $Q_1\left(\chi ,u\right)={\left[\begin{array}{c}\chi \\ u\end{array}\right]}^T{H}_1\left[\begin{array}{c}\chi \\ u\end{array}\right]≔z^T{H}_{1}z$, with $H_1=\left[\begin{array}{cc}Q+A^T{P}_{0}A& B^T{P}_{0}A\\ A^T{P}_{0}B& R+B^T{P}_{0}B\end{array}\right]\succcurlyeq 0$ being positive definite symmetric.

It is also valid that

$$Q_1\left(\chi , u\right)=\rho \left(\chi , u\right)+Q_0\left({\chi }_{+},K_0{\chi }_{+}\right)\le \rho \left(\chi , u\right)+Q_0\left({\chi }_{+},K_{-1}{\chi }_{+}\right):=Q_0\left(\chi , u\right),$$

(17)

since $K_0$ is greedy w.r.t. $Q_0$ against any other controller, including $K_{-1}$.

To continue this rationale by induction, suppose that $Q_{i-1}\left(\chi , u\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+{\chi }_{+}^T{P}_{i-2}{\chi }_{+}\ge 0$ holds for $\overline{Q}\succcurlyeq 0,R\succ 0,P_{i-2}\succcurlyeq 0$. Then

$$\underset{u}{\min } Q_{i-1}({\chi }_{+},u)=Q_{i-1}\left({\chi }_{+},K_{i-1}{\chi }_{+}\right)={\chi }_{+}^T \left(\overline{Q}+K_{i-1}^{T}R{K}_{i-1}\right){\chi }_{+}+{\chi }_{+}^T \left(A^T+K_{i-1}^T{B}^T\right)P_{i-2}\left(A+B{K}_{i-1}\right){\chi }_{+}≔{\chi }_{+}^T{P}_{i-1}{\chi }_{+}\ge 0$$

(18)

must hold because $\overline{Q}\succcurlyeq 0,R\succ 0,P_{i-2}\succcurlyeq 0$. This means that $P_{i-1}=\overline{Q}+K_{i-1}^{T}R{K}_{i-1}+\left(A^T+K_{i-1}^T{B}^T\right)P_{i-2}\left(A+B{K}_{i-1}\right)\succcurlyeq 0$, and the VI update

$$Q_i\left(\chi , u\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+{\chi }_{+}^T{P}_{i-1}{\chi }_{+}\ge 0$$

(19)

must also hold, since $\overline{Q}\succcurlyeq 0,R\succ 0,P_{i-1}\succcurlyeq 0$. The previous expression also takes the form $Q_i\left(\chi ,u\right)={\left[\begin{array}{c}\chi \\ u\end{array}\right]}^T{H}_i\left[\begin{array}{c}\chi \\ u\end{array}\right]≔z^T{H}_{i}z$, proving that $Q_i$ is itself quadratic at any iteration $i$.

Assume again by induction that $Q_{i-1}\left(\chi , u\right)\le Q_{i-2}\left(\chi , u\right), \forall \chi , u$ and let

$$Q_i\left(\chi , u\right)=\rho \left(\chi , u\right)+\underset{u}{\min } Q_{i-1}\left({\chi }_{+},u\right),$$

(20)

$$Q_{i-1}\left(\chi , u\right)=\rho \left(\chi , u\right)+Q_{i-2}\left({\chi }_{+},K_{i-2}{\chi }_{+}\right),$$

(21)

where in the first equation above we used the greediness of $K_{i-1}$ w.r.t. $Q_{i-1}$ against any other controller, including $K_{i-2}$. By subtracting the second equation from the first, above, we obtain

$$Q_i\left(\chi , u\right)-Q_{i-1}\left(\chi , u\right)\le Q_{i-1}\left({\chi }_{+},K_{i-2}{\chi }_{+}\right)-Q_{i-2}\left({\chi }_{+},K_{i-2}{\chi }_{+}\right)\le 0,$$

(22)

because $Q_{i-1}\left({\chi }_{+},K_{i-2}{\chi }_{+}\right)\le Q_{i-2}\left({\chi }_{+},K_{i-2}{\chi }_{+}\right),\forall {\chi }_{+},$ which results in

$$Q_i\left(\chi , u\right)\le Q_{i-1}\left(\chi , u\right).$$

(23)

The above reasoning proves that $Q_i\left(\chi , u\right)\le Q_{i-1}\left(\chi , u\right)\le \dots \le Q_0(\chi , u)$. □

Theorem 3.

Every $K_i$ resulting as a byproduct of the VI update is stabilizing with every iteration.

Proof of Theorem 3.

Recall that ${\mathrm{K}}_{-1}$ is admissible and stabilizing. Notice that

$$Q_1\left(\chi , u\right)={\chi }^T \overline{Q}\chi +u^{T}Ru+{\chi }_{+}^T{P}_0{\chi }_{+}=\rho \left(\chi ,u\right)+{\chi }_{+}^T \left[\overline{Q}+K_0^{T}R{K}_0+\left(A^T+K_0^T{B}^T\right)P_{-1}\left(A+B{K}_0\right)\right]{\chi }_{+},$$

(24)

and also recall that

$$Q_0\left(\chi ,u\right)=\rho \left(\chi , u\right)+{\chi }_{+}^T{P}_{-1}{\chi }_{+}.$$

(25)

Compute $Q_1\left(\chi , u\right)-Q_0\left(\chi ,u\right)={\chi }_{+}^T \left[\overline{Q}+K_0^{T}R{K}_0+ \left(A^T+K_0^T{B}^T\right)P_{-1}\left(A+B{K}_0\right)-P_{-1}\right]{\chi }_{+}$ and since $Q_1\left(\chi , u\right)\le Q_0\left(\chi ,u\right)$ (proven by previous Theorem 2), it follows that

$$\overline{Q}+K_0^{T}R{K}_0+\left(A^T+K_0^T{B}^T\right)P_{-1}\left(A+B{K}_0\right)-P_{-1}\preccurlyeq 0$$

(26)

which is a discrete-time Lyapunov equation for the closed-loop system matrix $\overline{A}=A+B{K}_0$, where $\overline{Q}+K_0^{T}R{K}_0\succcurlyeq 0$. This implies that $K_0$ is stabilizing. □

By induction, suppose $Q_{i-1}\left(\chi ,u\right)=\rho (\chi , u)+{\chi }_{+}^T{P}_{i-2}{\chi }_{+}$ holds and that by the optimal VI update

$$Q_i\left(\chi , u\right)=\rho \left(\chi , u\right)+\underset{u}{\min } Q_{i-1}\left({\chi }_{+},u\right)$$

(27)

we have

$$Q_i\left(\chi , u\right)=\rho \left(\chi ,u\right)+{\chi }_{+}^T \left[\overline{Q}+K_{i-1}^{T}R{K}_{i-1}+\left(A^T+K_{i-1}^T{B}^T\right)P_{i-2}\left(A+B{K}_{i-1}\right)\right]{\chi }_{+}.$$

(28)

Then, since $Q_i\left(\chi , u\right)-Q_{i-1}\left(\chi ,u\right)\le 0$, it must follow that

$$\overline{Q}+K_{i-1}^{T}R{K}_{i-1}+\left(A^T+K_{i-1}^T{B}^T\right)P_{i-2}\left(A+B{K}_{i-1}\right)-P_{i-2}\preccurlyeq 0,$$

(29)

which is a Lyapunov equation in discrete time for the closed-loop system matrix $\overline{A}=A+B{K}_{i-1}$. Its fulfilment implies that $K_{i-1}$ is stabilizing, which concludes the proof. □

Corollary 1.

The OOART-MRTC VI-like updates converge to the optimal Q-function $Q^{*}(\chi , u)$ and to the optimal control.

Proof of Corollary 1.

Using Theorem 2 and Theorem 3 results, we have $0\le Q_i\left(\chi , u\right)\le Q_{i-1}\left(\chi , u\right)\le \dots \le Q_0(\chi , u)$ being monotonically decreasing and bounded above by $Q_0$ and below by 0, with $K_i$ stabilizing at every iteration. Then the sequence $Q_i$ must converge to a finite number, when $i\to \infty$, making $\underset{i\to \infty }{\lim } Q_i=Q^{*}$. Consequently, $\underset{i\to \infty }{\lim } K_i=K^{*}$, where $Q^{*}, K^{*}$ defines the fixed point of the VI updates for which it is true that

$$Q^{*}\left(\chi , u\right)=\rho \left(\chi , u\right)+Q^{*}({\chi }_{+},K^{*}{\chi }_{+}).$$

(30)

□

Remark 6.

The convergence analysis uses the system matrices $A,B$ only to uncover the problem structure; however, the system knowledge is not really needed in the OOART-MRTC VI-like updates.

3. Results: A Reference Model Tracking Attitude Control Case Study for a Multivariable Quadrotor UAV with an Actuator Fault and an Uncertain Model

A linearized quadrotor UAV attitude control model is

$$\begin{gathered} \left[\begin{array}{c}\dot{\varphi }\\ \dot{\theta }\\ \dot{\psi }\\ \ddot{\varphi }\\ \ddot{\theta }\\ \ddot{\psi }\end{array}\right]=\left[\begin{array}{cccccc}0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 1\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\end{array}\right]\left[\begin{array}{c}\varphi \\ \theta \\ \psi \\ \dot{\varphi }\\ \dot{\theta }\\ \dot{\psi }\end{array}\right]+\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\\ {\displaystyle \frac{L}{I_x}}& 0& -{\displaystyle \frac{L}{I_x}}& 0\\ 0& {\displaystyle \frac{L}{I_y}}& 0& -{\displaystyle \frac{L}{I_y}}\\ {\displaystyle \frac{1}{I_z}}& -{\displaystyle \frac{1}{I_z}}& {\displaystyle \frac{1}{I_z}}& -{\displaystyle \frac{1}{I_z}}\end{array}\right]\Lambda \left[\begin{array}{c}{u}_1\\ u_2\\ u_3\\ u_4\end{array}\right]\leftrightarrow {x_{k+1}=A}_{o\_CT}{x}_k+B_{o\_CT}\Lambda u_k, \\ \left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]=\left[\begin{array}{c}\varphi \\ \theta \\ \psi \end{array}\right]=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\end{array}\right]\left[\begin{array}{c}\varphi \\ \theta \\ \psi \\ \dot{\varphi }\\ \dot{\theta }\\ \dot{\psi }\end{array}\right]\leftrightarrow {y_k=C}_{o\_CT}{x}_k+D_{o\_CT}{u}_k, \end{gathered}$$

(31)

where $\varphi ,\theta ,\psi$ are the roll, pitch, and yaw attitude angle orientations of the body-fixed frame relative to the inertial frame, $I_x, I_y,I_z$ are the inertial moments of the quadrotor, $L$ is the rotor arm length relative to the mass center. Input variables $u_i,i=1,\dots ,4$ are the normalized collective propeller forces. Here, $\Lambda$ with the nominal value $\Lambda =I_4$ is the actuator health matrix that models a fault like the loss in actuation capacity, with positive diagonal elements within the interval ${\Lambda }_{\mathrm{i}\mathrm{i}}\in (0,1]$. The model is a linearized one around hover position and neglects the inner propeller dynamics [18]. In this assumption, it is noted that the dynamics of the attitude angles behave like coupled double integrators about each of the three controlled outputs. With a sampling period of $T_s=0.1$[s], the discretized model with zero-order hold becomes compliant with description (1), with appropriate size matrices $A_o, B_o, C_o, D_o$ (note the matrices ${A_{o\_CT}, B}_{o\_CT},C_{o\_CT},D_{o\_CT}$) in (31) are the continuous-time (CT) counterparts. The model is a four-input–three-output one with $m=4,p=3$.

3.1. Initial Stabilizing Controller Derivation

For the current case study, the considered nominal system parametric values are $L_{nom}=0.3$[m]$, I_{x,nom}=0.2 [\mathrm{k}\mathrm{g}\cdot {\mathrm{m}}^2], I_{y,nom}=0.2$[kg⋅m²]$, I_{z.nom}=0.4$[kg⋅m²]$, \Lambda =I_4$ similar to [25]. To these nominal values, the nominal CT matrices $A_{nom\_CT}=\left[\begin{array}{cccccc}0& 0& 0& 0.98& 0& 0\\ 0& 0& 0& 0& 0.98& 0\\ 0& 0& 0& 0& 0& 0.98\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0\end{array}\right], B_{nom\_CT}=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ 0& 0& 0& 0\\ {\displaystyle \frac{L_{nom}}{I_{x,nom}}}& 0& -{\displaystyle \frac{L_{nom}}{I_{x,nom}}}& 0\\ 0& {\displaystyle \frac{L_{nom}}{I_{y,nom}}}& 0& -{\displaystyle \frac{L_{nom}}{I_{y,nom}}}\\ {\displaystyle \frac{1}{I_{z,nom}}}& -{\displaystyle \frac{1}{I_{z,nom}}}& {\displaystyle \frac{1}{I_{z,nom}}}& -{\displaystyle \frac{1}{I_{z,nom}}}\end{array}\right]$ are used to discretize the state space model (31) and obtain the nominal discrete-time matrices $A_{nom},B_{nom}$. No uncertainty is considered about the output equation matrices $C_{nom}=C_o,D_{nom}=D_o$. The resulting nominal state space matrices are only used for searching an initial, stabilizing controller, with no focus on performance except stability certification.

For the sixth order nominal quadrotor model, six closed-loop poles equally spaced in the interval $[-1,-0.5]$ are allocated, let them be $p_i,i=1,\dots ,6$. Their discrete counterparts are $p_{di}=e^{T_s{p}_i}$. For the nominal system matrices $A_{nom},B_{nom}, C_{nom}$, we solve for $K_{1o}\in {\mathbb{R}}^{6\times 4}$ by placing the poles of $A_{nom}+B_{nom}{K}_{1o}^T$ at the desired locations $p_{di}$. For $ny=nu=r=2$, let the observability matrix from Theorem 1 be $\overline{\mathcal{O}}=[C_{nom}{A}_{nom};C_{nom}]$, the controllability matrix be $\mathcal{C}=\left[B_{nom}\right|A_{nom}{B}_{nom}]$, the matrix $\overline{N}=\left[\begin{array}{cc}{C}_{nom}{B}_{nom}& C_{nom}{A}_{nom}{B}_{nom}\\ 0& C_{nom}{B}_{nom}\end{array}\right]$, and the transform matrix be $\mathcal{T}=\left[A_{nom}^2 pinv\left(\overline{\mathcal{O}}{A}_{nom}\right)\left[I|0\right]\mathcal{C}-A_{nom}^2 pinv\left(\overline{\mathcal{O}}{A}_{nom}\right)\overline{N}\right]\in {\mathbb{R}}^{6\times 17}$. It follows that $A_v=pinv\left(\mathcal{T}\right)A_{nom}\mathcal{T},B_v=pinv\left(\mathcal{T}\right)B_{nom}, C_v=C_{nom}\mathcal{T}$. Note that the pole placement for MIMO systems requires controllability for proper control authority distribution over each input. Arbitrary pole locations may pose numerical challenges, both because there are infinite state feedback gains and owing to very close poles being mistaken for higher multiplicity. If pole placement is not numerically reliable, then the stabilization design should fall back to LQR instead of pole placement.

Next, we find a state-feedback gain in the equivalent transformed virtual state-space, using the transform matrix $\mathcal{T}$. Let this gain be $K_1={\mathcal{T}}^T{K}_{1o}\in {\mathbb{R}}^{17\times 4}$, which makes the virtual closed-loop matrix $A_v+B_v{K}_1^T\in {\mathbb{R}}^{17\times 17}$ a stable one.

From the output equation of the augmented system model (8), i.e., $y_k=C_{nom} x_k$, we express the steady-state conditions

$$\begin{array}{c}{y}_{\infty }=C_{nom}{x}_{\infty }=C_{nom}\mathcal{T}{v}_{\infty },\\ v_{\infty }=C_{nom}\mathcal{T}\left[\left(A_v+B_v{K}_1^T\right)v_{\infty }+B_v{K}_2^T{x}_{\infty }^m+B_v{K}_3^T{r}_{\infty }\right].\end{array}$$

(32)

To find the $K_3$ that assures unit gain from $r_{\infty }\to y_{\infty }$, we set $K_2=0$ and solve for $v_{\infty }={\left(I-C_{nom}\mathcal{T}\left(A_v+B_v{K}_1^T\right)\right)}^{-1}{B}_v{K}_3^T{r}_{\infty }$. Then, $y_{\infty }=I \cdot r_{\infty }$ for $I=C_{nom}\mathcal{T}{\left(I-C_{nom}\mathcal{T}\left(A_v+B_v{K}_1^T\right)\right)}^{-1}{B}_v{K}_3^T$, and $K_3$ is found.

The initial controller $K_{-1}^T:=\left[K_1^T\left|K_2^T\right|K_3^T\right]\in {\mathbb{R}}^{4\times 26}$ is used to stabilize the extended nominal state-space system (5), in order to collect the transition samples and fill the ERB. Note that the model reference dynamics part and the reference input model part from (5) are already stable; hence, the sole concern is to stabilize the virtual state-space dynamics (2).

3.2. The Data Collection Phase with the Uncertain Dynamics and with Actuator Fault

A total of 2500 transition samples are collected to fill the ERB, by interacting with the quadrotor system under the initial stabilizing controller $K_0$ under an exploratory setting. A fault in the actuator is simulated by making $\Lambda =diag(0.9, 0.9, 0.9, 0.9)$ and with the inertial parameters $I_x, I_y, I_z$ increased by 10% more than their nominal values. Hence, $B_{o\_CT}=B_{nom}\cdot \Delta B\cdot \Lambda$. The true state transition matrix $A_o$ is the discrete-time counterpart of the discretized system (31) with $A_{o\_CT}$. The interaction is next performed with a different system than the one used for stabilization.

The reference input is reset at $r_k=$0 on all three channels corresponding to roll, pitch, and yaw. Zero-mean normally distributed noise is added to the four control input channels at each sampling step time; its variance is 0.9. Every 50 samples, each reference switches to a uniform random amplitude value within [−1; 1]; however, we decorrelate the switching times of the reference inputs, in order to excite each channel independently and make it behave as a disturbance about the other control channels. This helps, furthermore, to achieve a good exploration. The collection phase results are displayed in Figure 1.

The reference inputs about each control channel also drive the reference model outputs. To impose a stable, desirable control performance, we employ an overdamped reference model of the form $H\left(s\right)={\displaystyle \frac{w_n^2}{s^2+2\xi w_{n}s+w_n^2}}$ with $w_n=1.5$[rad/s]$,\xi =0.9$ about each control channel (i.e., for each controlled angle $\varphi ,\theta ,\psi$), which is discretized for the same $T_s=0.1$[s]. Here, $\xi$ corresponds to percent overshoot of $M_p=e^{-{\displaystyle \frac{\xi \pi }{\sqrt{1-{\xi }^2}}}}\times 100$[%] = 0.15[%], while the rise time is $t_r={\displaystyle \frac{\pi -\varphi }{w_n \sqrt{1-{\xi }^2}}}=4.11$[s] for $\varphi =\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\left({\displaystyle \frac{\sqrt{1-{\xi }^2}}{\xi }}\right)$. The discretized reference model will be of the form (3), where $x_k^m\in {\mathbb{R}}^6,A_m\in {\mathbb{R}}^{6\times 6},B_m\in {\mathbb{R}}^{6\times 3},C_m\in {\mathbb{R}}^{3\times 6},D_m=0\in {\mathbb{R}}^{3\times 3}.$ More precisely, the continuous-time transfer matrix is therefore of the form

$$\left[\begin{array}{c}{y}_1^m\\ y_2^m\\ y_3^m\end{array}\right]=\left[\begin{array}{c}{\varphi }^m\\ {\theta }^m\\ {\psi }^m\end{array}\right]=\left[\begin{array}{ccc}{\displaystyle \frac{w_n^2}{s^2+2\xi w_{n}s+w_n^2}}& 0& 0\\ 0& {\displaystyle \frac{w_n^2}{s^2+2\xi w_{n}s+w_n^2}}& 0\\ 0& 0& {\displaystyle \frac{w_n^2}{s^2+2\xi w_{n}s+w_n^2}}\end{array}\right]\left[\begin{array}{c}{r}_{\varphi }\\ r_{\theta }\\ r_{\psi }\end{array}\right],$$

(33)

and then discretized accordingly at $T_s=0.1$[s]. Using the results of Theorem 1, the final and complete virtual state-space representation constructed for $ny=nu=2$ is of the form (5), with ${\chi }_k={\left[v_k^T,{\left(x_k^m\right)}^T,r_k^T\right]}^T\in {\mathbb{R}}^{26}$, including $v_k={\left[y_k^T,y_{k-1}^T,y_{k-2}^T,u_{k-1}^T,u_{k-2}^T\right]}^T\in {\mathbb{R}}^{17}$, $x_k^m\in {\mathbb{R}}^6,r_k\in {\mathbb{R}}^3$.

Remark 7.

The transition samples pertaining to the reference model can be generated offline (a posteriori) after the transition sample collection from the true quadrotor system, or they can be generated online in real-time along with the system trajectories. In our work, we choose to compute them online in real-time. Their offline generation is possible since the reference input trajectory $r_k$ is stored.

3.3. The Learning Stage

The learning process starts from the initial state-feedback gain $K_{-1}$ that was used for exploration and with ${\theta }_0$ being initialized with random uniform values in [0, 1]. The penalty function is ${\rho }_k={‖y_k\left({\chi }_k\right)-y_k^m\left({\chi }_k\right)‖}^2+u_k^{T}R{u}_k$ with $R=0.01\cdot I_4$ being diagonal and positive definite.

At each iteration of the OOART-MRTC algorithm, in a parallel processing thread, we extract a minibatch of 1200 transitions and solve for ${\theta }_1$ using Equation (10). Then $K_0$ is found based on Equation (11). This process is repeated until satisfactory convergence, i.e., until no more changes are observed in either $\theta$ or $K$. Solving such a large overdetermined system may exceed the allotted time available within a single sampling period. Thus, to make the computation feasible in online real-time, the algorithm learning updates are detached to a separate thread, according to the soft real-time software implementation principles exposed in [37]. Importantly, when extracting transition samples from the ERB, we filter out those samples which correspond to switching reference input values on any control channel. The reason is that the piecewise constant reference inputs comply with a generative model like $r_{k+1}=I_3{r}_k$ only when the references are constant, not when they switch.

In Figure 2, we observe the learning results after 300 timesteps corresponding to about 150 actor–critic iterative updates that take place asynchronously. The model reference tracking is very accurate across each channel. Decoupling of all three control channels is ensured, which fulfils the desired control objective for this very high-order quadrotor UAV system, for which each of the three control channels behaves as a double integrator.

The actor weights $K_i$ and critic weights ${\theta }_i$ throughout the learning iteration steps are visible in Figure 3. Both values of ${\theta }_i$ and the values of $K_i$ stabilize after only few iterations. The model reference tracking is accurately satisfied even long before the actor and critic weights stabilize, as is visible in the samples from Figure 2. The actor has $4\times 26=104$ parameters/weights and the critic has 465 parameters/weights.

The validated learning method based on OOART-MRTC relies strongly on the linearity assumption in the underlying system dynamics. The advantage is exposed clearly with the smart parameterization of the actor (controller) and the critic (extended Q-function). In the nonlinear dynamics case, neural networks are required [39,40,41,42]; however, such validation has already been performed for low-order systems [37]. Employing the virtual state representation leads to a very high-dimensional equivalent linear system which, although completely observable, comes with several challenges associated with the exploration. An efficient exploration implies that the sequences of inputs and outputs that form the augmented state vector must be decorrelated, which is only achievable with a sufficiently noisy excitation. While a good coverage of the state-action domain may not be easily attained in this case, the extrapolative capability that comes from the linear parameterization will make the optimality hold, at least in theory, at any scale around the equilibrium, no matter if the state transitions are collected in a confined smaller region, under persistent excitation. Hence, the clever parameterization compensates for the poorer exploration in this case.

3.4. Ablation Study 1: The Actor–Critic Update Time Relative to Environment Timestep

To monitor the actor–critic update time (ACUT) duration relative to the real-time sampling period of the environment ($T_s=0.1$[s] herein), the average ACUT duration is measured as a function of the batch size that is used to update the actor and critic weights. These batch sizes are $\{512,768,1024,1536\}$. The win-precise-time (v. 1.4.2) and time libraries are used under Windows^® and Python 3.10 for soft real-time environment stepping mechanisms and for function execution timing; the results are visible in Figure 4. Vectorized, batch-wise implementation with the NumPy library (v. 1.26.4) is used for code speed improvement when building the large-scale matrices and when solving the linear equation systems with the numpy.linalg.lstsq linear algebra package function.

Although we tested the learning with the batch sizes $\{512,768,1024,1536\}$, we did not observe stabilization issues or significant control actions and tracking error bursts in the initial learning phase. This is in spite of increasing the ACUT from $1.5T_s$ to about $2T_s$. This indicates that the learning is robust and bump-less.

3.5. Ablation Study 2: Using VI-like Instead of PoIt-like Style in the First Critic Weight Update

In the OOART-MRTC algorithm, in the asynchronous actor–critic update thread, the critic weights ${\theta }_0$ are found for $K_{-1}$ using Equation (10) with ${\theta }_0$ used on both sides. This corresponds to a PoIt-like critic update in the first step, and it is necessary, in order to evaluate the state-action Q-function of the initial stabilizing controller. The effects of this strategy shown with Figure 2 and Figure 3 reveal the strong stabilization of the learning, despite an increased actor–critic update time duration.

To disclose the importance of this first critic update step, using PoIt-like strategy, we instead change it to a VI-like update, where the critic ${\theta }_0$ is a randomly initialized vector within the unit magnitude uniform distribution. All subsequent critic updates are performed in VI-like style, which is known to converge, ultimately. However, a negative bursting effect is noticed, which is expected to alter the control stability, especially if the actor–critic update time duration increases. The learning behavior is shown in Figure 5 and Figure 6.

3.6. Ablation Study 3: Severe Actuator Loss and Performance Tracking Measurement

In the OOART-MRTC algorithm, we exacerbate the actuator loss, making $\Lambda =diag(0.9,0.8,0.5,0.0)$. This means that one control action corresponding to one rotor is zero, as the propeller is stuck. This way, the actuation is asymmetric and saturating on the faulty propeller. We redo the OOART-MRTC learning, with maximal batch size of 1536, after data is collected with the new severely affected actuator. The controller designed in the first place based on the nominal model delivers the tracking performance in Figure 7, with measured RMSE averaged on three rollouts being $226.29\times {10}^{-4}$.

After learning takes place, the tracking is visible in Figure 8, with measured RMSE averaged on three rollouts being $1.62\times {10}^{-4}$, which is two orders of magnitude smaller than with the initial stabilizing controller used for transition sample collection. The asymmetric nature of the control action is clearly visible as they try to compensate for one actuator loss out of four. The fourth control action $u_4$ “learns” to become zero since it has no effect (see its nonzero action in Figure 7; however, its effect is also null there). Importantly, the result may not be feasible in the real world, since the altitude may not be held by such severe loss. Herein, the control configuration of the linearized operation near steady state permits such a result.

4. Critical Discussions and Result Extensions

Regarding the transferability of the results towards other UAV types, the proposed framework is sufficiently general to accommodate other control configurations, since control necessities are ubiquitous. It is especially favorable towards the easily interpretable model reference tracking, and it can be extended to other performance requirements. Other potential control configurations include fixed-wing UAVs; ground-based unmanned vehicles (UGVs) like wheeled, tracked, legged/quadruped robots; unmanned surface vehicles (USVs); and unmanned underwater vehicles (UUVs).

Simulation experiments in this work were based on scenarios that favor real-world replication, to be performed near hovering point under prior stabilizing controllers, with step-like injected signals used as attitude setpoint excitations, leading to various quadrotor UAV attitude changes that generate the input–output data samples used for learning in the virtual embedded state-space representation. A dedicated flying space is required to allow for horizontal, constant-altitude motions. However, data collection can be relaxed near real operation missions, with injected attitude disturbances. These conditions are reproducible with commercial autopilots (e.g., ArduPilot, PX4), by hooking up within software, at the attitude controller setpoints module, in order to produce synthetic, disturbance-like data injections. Then, by reading the attitude angles from filtered gyroscopes via Extended Kalman Filter (EKF) attitude estimators, together with the controller output signal, the learning data is logged to a database from where transition can be sampled in asynchronous mode, for learning. This learning takes place in a difference thread; hence, it should not affect the real-timeliness constraints for the fast attitude control loops. This aspect was validated with soft real-time validation in the case study. The commonly used control architecture in commercial autopilots, i.e., the P-type dominantly used for attitude and the inner cascaded PID-type rate controllers, can be replaced by the proposed state-feedback controller, which is, in fact, an equivalent mathematical representation.

There are, nevertheless, a number of limitations and corner cases that require further investigation. The linearity assumption holds to a limited extent in practice. In the concrete case of UAVs, aggressive maneuvers requiring drastic attitude changes will exacerbate the nonlinearity, leading to control robustness loss and potential instability issues. Secondly, there are other potential issues that could infringe assumptions like observability, for example, sensor bias, packet losses, and delayed measurements. For the attitude control, delayed measurements are a lesser problem if the delay is fixed, as the model (2) can be further augmented with additional states. However, it is problematic if it is time-varying. Packet loss is not a rough issue, at least in the attitude controller case, since the attitude controller belongs to the critical low-level communication infrastructure and it is carefully designed in practice. Moreover, its real-timeliness constraints are not affected when the learning takes place asynchronously. In real autopilots, the sensor bias is estimated and compensated using EKF, by fusing many sensors like optical flow, IMU, the altimeter, LiDAR, and the camera. Therefore, the attitude angles are relatively unbiased, but noisy.

Another potential limitation is with the fault and uncertainty modeling used here, which does not cover all corner cases of time-varying, asymmetric, saturating, or stuck. The multiplicative uncertainty partially accounts for asymmetric and time-varying cases. Actuator jamming is captured by zeroed entries in $\Lambda$. This is a further research direction.

5. Conclusions

For the class of low-order system applications where an initial prior stabilizing controller is available, exploration can be done to an extent which allows for asynchronous online learning of the optimal controller. This is more advisable than synchronous online learning since the asynchronous approach is unconstrained to run within a sampling period; hence, it can solve the update equations by taking into account significantly more transition samples. Using many transition samples with each iteration update improves the conditioning of the underlying problem (an overdetermined linear equation system in this case). The equivalent state-space representation based on input–output samples is a major advantage since it does not require the true system state to be measured; however, the virtual system makes the learning environment fully observable, even if the equivalent state-space representation becomes significantly higher-order. The tracking problem is augmented and articulated by a reference model that imposes a desired behavior for the closed-loop system. Learning convergence of the proposed OOART-MRTC VI-based algorithm was analyzed under parametric uncertainty and actuation fault. Future research will strive to validate the proposed method on highly complex, nonlinear, and high-order experimental systems.