Model Predictive Regulation on Manifolds in Euclidean Space

Abstract

One of the crucial problems in control theory is the tracking of exogenous signals by controlled systems. In general, such exogenous signals are generated by exosystems. These tracking problems are formulated as optimal regulation problems for designing optimal tracking control laws. For such a class of optimal regulation problems, we derive a reduced set of novel Francis–Byrnes–Isidori partial differential equations that achieve output regulation asymptotically and are computationally efficient. Moreover, the optimal regulation for systems on Euclidean space is generalized to systems on manifolds. In the proposed technique, the system dynamics on manifolds is stably embedded into Euclidean space, and an optimal feedback control law is designed by employing well studied, output regulation techniques in Euclidean space. The proposed technique is demonstrated with two representative examples: The quadcopter tracking control and the rigid body tracking control. It is concluded from the numerical studies that the proposed technique achieves output regulation asymptotically in contrast to classical approaches.

1. Introduction

One of the fundamental problems in control theory is to regulate the output of the plant, which is known in the literature as model predictive regulation (MPR) [1]. Consider a controlled nonlinear system on a manifold M as

$$\dot{x}=f(x,u,w)$$

with an exosystem

$$\dot{w}=a\left(w\right)$$

and the system output

$$y=h(x,u,w),$$

where $x\in M,$$u\in {\mathbb{R}}^m,$$w\in {\mathbb{R}}^d,$ and $y\in {\mathbb{R}}^p.$ The optimal MPR problem is to determine a feedback control law $u=\kappa (x,w)$ that regulates the system output asymptotically to zero, i.e.,

$$\underset{t\to \infty }{lim}y\left(t\right)=0.$$

An optimal regulation control law is synthesized from an infinite horizon optimal control problem, i.e.,

$$\underset{u}{min}{\int }_0^{\infty }l(x\left(t\right),u\left(t\right),w\left(t\right))dt$$

subject to the controlled dynamics and a fixed initial condition $\left(x\left(0\right),w\left(0\right)\right)=(x_0,w_0),$ where the control Lagrangian, $l:M\times {\mathbb{R}}^m\times {\mathbb{R}}^d\to {\mathbb{R}}_{\ge 0}$, is sufficiently well-behaved and is zero if and only if $y=0$.

The output regulation problem for linear multivariate systems is formulated and its solvability conditions are derived, for the first time, by Francis in [2]. The regulation problem is further generalized for nonlinear systems [3] in which a feedforward control law is synthesized by solving the Francis–Byrnes–Isidori (FBI) partial differential equations (PDEs). The optimal output regulation problem is first studied by Krener [4], which proposes an optimal feedback control law synthesis in two steps: First, a feedforward control law is designed by solving the FBI PDEs using series solutions [5]. Second, using the feedforward control, the regulations problem is then translated to an optimal stabilization problem for designing an optimal feedback control law using the Al’brekht method [5]. In addition, Krener has developed a Nonlinear Systems Toolbox [6] for obtaining series solutions of the FBI PDEs and the Hamilton–Jacobi–Bellman (HJB) PDEs associated with the optimal stabilization problem [5].

Note that the FBI and the HJB PDEs are solved up to a polynomial approximation of a certain degree d, and therefore, the output regulation may not be achieved asymptotically if the output signal is not representable by a degree d polynomial. Due to such limitations of series solutions, output regulations may not be guaranteed for sophisticated output signals such as transcendental functions of the states. In this article, we address a class of output regulation problems in which the system states need to track signals generated by exosystems. For such output regulation problems, we derive novel FBI PDEs that are computationally efficient and achieve the output regulation asymptotically.

Another challenge is to extend the optimal output regulation problem to controlled systems on manifolds. A technique to deal with the output regulation problems on manifolds is to define equivalent output regulation problems on ambient Euclidean space and then employ well studied tools of optimal regulation on Euclidean space. A coherent way is to extend the system dynamics from the manifold to an ambient Euclidean space such that the manifold is a stable attractor of the extended system dynamics [7,8,9], and define the optimal regulation problem on that Euclidean space. An alternate approach is to define the regulation problem on local charts. This approach severely limits the applicability of the regulation problem due to local representation of the system dynamics. Moreover, the output regulation, under this approach, will only be limited to that particular local chart instead of the whole manifold.

In this article, we address two problems: First, we derive a reduced set of FBI PDEs for a class of output regulation problems on Euclidean space. We establish that, under the proposed approach, output regulation for a class of systems is achieved asymptotically by feedback control laws obtained via series solutions [5]. Second, we generalize the optimal regulation method on the Euclidean space to manifold via stable embedding of the controlled system dynamics on manifold into Euclidean spaces, and this optimal regulation problem was not addressed in [8]. Our proposed technique is then employed to the quadcopter tracking control and the rigid body attitude tracking system. It is observed from the numerical studies that the proposed approach provides better tracking performance as compared to the classical optimal regulation. Moreover, the controller built for the quadcopter in this paper is globally defined without any singularity, whereas the model predictive regulator designed in [10] is inherently locally defined since its use of Euler angles. This also demonstrates the superiority of our method. All the systems addressed in this article are continuous-time systems, in parallel to which there have been developments on model predictive regulation for discrete-time systems [1,11,12] and stochastic systems [13]. However, we do not treat either discrete-time or stochastic systems in this particle, leaving them as future research work.

This article unfolds as follows: Section 2 is devoted to a class of output regulation problems on Euclidean spaces and the derivation of the novel FBI PDEs. A generalized technique for the optimal output regulation of systems on manifolds is presented in Section 3. The proposed output regulation technique is well supported by the numerical studies conducted on representative examples of quadcopter tracking control and the rigid body attitude tracking system in Section 4. The concluding remarks and future directions are discussed in Section 5.

2. Optimal Output Regulation of Nonlinear Systems

Consider a nonlinear dynamic system

$$\begin{array}{cc}\hfill \dot{x}& =f(x,u,w),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill y& =h(x,u,w),\hfill \end{array}$$

(3)

with the following data:

(a)

plant state $x\in {\mathbb{R}}^n$, plant input $u\in {\mathbb{R}}^m$, exosystem state $w\in {\mathbb{R}}^d$ and system output $y\in {\mathbb{R}}^p$,

(b)

$f:{\mathbb{R}}^n\times {\mathbb{R}}^m\times {\mathbb{R}}^d\to {\mathbb{R}}^n$ is a map depicting the plant dynamics (1) on ${\mathbb{R}}^n$,

(c)

$a:{\mathbb{R}}^d\to {\mathbb{R}}^d$ is a map depicting the exosystem dynamics (2) on ${\mathbb{R}}^d$,

(d)

$h:{\mathbb{R}}^n\times {\mathbb{R}}^m\times {\mathbb{R}}^d\to {\mathbb{R}}^p$ accounts for the system output (3).

An output regulation problem is to find a control law

$$\begin{array}{cc}\hfill u:& {\mathbb{R}}^n\times {\mathbb{R}}^d\to {\mathbb{R}}^m\hfill \\ \hfill & (x,w)\mapsto u(x,w)\hfill \end{array}$$

(4)

that steers, for any set of initial conditions $(x_0,w_0)\in {\mathbb{R}}^n\times {\mathbb{R}}^d$, the system output of the nonlinear dynamics (1)–(3) asymptotically to zero, i.e.,

$$\underset{t\to \infty }{lim}y\left(t\right)=0.$$

Output regulation problems are very commonly formulated for disturbance rejection and reference tracking by the system. The exosystem dynamics is, in general, designed to generate a reference signal or a modeled disturbance signal. Let us consider a case of reference tracking in which the plant output is the output of the plant dynamics (1) that does not include the exosystem $\tilde{h}(x,u)\in {\mathbb{R}}^p$ for $(x,w)\in {\mathbb{R}}^n\times {\mathbb{R}}^m$, needs to track the reference signal, $q\left(w\right)\in {\mathbb{R}}^p$ for $w\in {\mathbb{R}}^d$, generated by the exosystem. Then, the system output

$$y=\tilde{h}(x,u)-q\left(w\right)$$

asymptotically converging to zero ensures that the plant output is asymptotically tracking the reference signal. In an identical manner, let us consider a case of disturbance rejection in which the plant output is stabilized asymptotically to zero while the plant dynamics (1) are subjected to the disturbance generated by the exosystem. Therefore, the system output

$$y=\tilde{h}(x,u)$$

is regulated to zero under the influence of the disturbance introduced in the plant dynamics.

Before we discuss necessary and sufficient conditions for the solvability of the output regulation problem, let us elucidate standard assumptions considered in the literature:

Assumption 1.

The following assumptions for the nonlinear dynamics (1)–(3) hold:

(a) 

The vector fields f and a and the map h are smooth.

(b) 

For the control input $u=0$, the system dynamic (1)–(3) has an equilibrium point $(x,w)=(0,0)$ such that the system output is zero, i.e., $h(0,0,0)=0$.

(c) 

The equilibrium exosystem state $w=0$ of the exosystem (2) is stable and there exists a neighborhood $W_0$ containing zero, such that every initial condition $w\left(0\right)\in W_0$ is Poisson stable.

(d) 

The linear approximation of the plant dynamics (1) is stabilizable at the equilibrium point $(x,u,w)=(0,0,0)$, i.e., the pair

$$A=\frac{\partial f}{\partial x}(0,0,0),B=\frac{\partial f}{\partial u}(0,0,0)$$

is stabilizable.

The output regulation problem with such a generality is difficult to solve in general. Therefore, the state feedback control law (4) is designed in an open neighborhood $\mathcal{O}\subset {\mathbb{R}}^n\times {\mathbb{R}}^d$ of the origin at 0, such that for any initial condition $\left(x\left(0\right),w\left(0\right)\right)=(x_0,w_0)\in \mathcal{O}$, the system output (3) of the dynamics (1)–(3) converges at zero. The solvability condition for the output regulation problem is established by the following theorem:

Theorem 1

([3]). Under Assumption 1, there exist a neighborhood $\mathcal{O}\subset {\mathbb{R}}^n\times {\mathbb{R}}^d$ of 0 and a $C^k(k\ge 2)$ state feedback, $u=u(x,w)\in {\mathbb{R}}^m$ for $(x,w)\in \mathcal{O}$, that asymptotically stabilizes the output of the system dynamics (1)–(3) to zero if and only if there exist $C^k$ mappings $x=\theta \left(w\right)$ with $\theta \left(0\right)=0$, and $u=\lambda \left(w\right)$ with $\lambda \left(0\right)=0$, both defined in a neighborhood $\mathcal{W}\subset {\mathbb{R}}^d$ of 0, such that

$$\begin{array}{cc}\hfill & \frac{\partial \theta }{\partial w}a\left(w\right)=f\left(\theta \left(w\right),\lambda \left(w\right),w\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & h\left(\theta \left(w\right),\lambda \left(w\right),w\right)=0.\hfill \end{array}$$

(6)

for all $w\in \mathcal{W}$.

The PDE (5) with the algebraic constraints (6) is known in the literature as the FBI equation [3,4]. As a consequence of Theorem 1, for any initial condition $x\left(0\right)=\theta \left(w\left(0\right)\right)$ with $w\left(0\right)\in \mathcal{W}$, the system dynamics (1)–(3) under the feedforward control $u=\lambda \left(w\right)$ leads to

$$y\left(t\right)=h\left(\theta \left(w\left(t\right)\right),\lambda \left(w\left(t\right)\right),w\left(t\right)\right)=0\mathrm{for}t\ge 0.$$

Thus, the feedback control law for the output regulation can be designed as

$$\begin{array}{c}\hfill u(x,w)=\lambda \left(w\right)+\kappa \left(x-\theta \left(w\right),w\right),\end{array}$$

(7)

where the feedback term $\kappa$ with $\kappa \left(0\right)=0$ is derived to make the so-called output regulation manifold

$${\mathcal{M}}_{\mathcal{R}}=\left\{\left.(x,w)\in {\mathbb{R}}^n\times {\mathbb{R}}^d\right|x-\theta \left(w\right)=0\right\}$$

a stable attractor.

The problem of synthesizing optimal feedback control laws for output regulation is first proposed by Krener [4,14], and that is generalized to model predictive regulations [15]. The feedback control law (7) using Krener’s method is designed in two steps:

(i)

The feedforward control $u=\lambda \left(w\right)$ and the output regulation manifold ${\mathcal{M}}_{\mathcal{R}}$ are designed by solving the FBI Equations (5) and (6).

(ii)

For the feedback $\kappa$, the nonlinear dynamics (1)–(3) is defined in the new coordinates

$$z=x-\theta \left(w\right),v=u-\lambda \left(w\right)$$

as

$$\begin{array}{cc}\hfill & \dot{z}=f\left(z+\theta \left(w\right),v+\lambda \left(w\right),w\right)-\frac{\partial \theta }{\partial w}a\left(w\right),\hfill \\ \hfill & \dot{w}=a\left(w\right).\hfill \end{array}$$

(8)

Under these new coordinate changes, the output regulation problem (4) is posed as an optimal stabilization problem for asymptotic stabilization of the dynamics (8) to zero as

$$\begin{array}{cc}\hfill \underset{v}{min}& {\int }_0^{\infty }l\left(z\left(t\right),v\left(t\right)\right)dt\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \left\{\begin{array}{c}\mathrm{system}\mathrm{dynamics}\left(8\right),\hfill \\ \left(z\left(0\right),w\left(0\right)\right)=(z_0,w_0),\hfill \end{array}\right.\hfill \end{array}$$

(9)

where $(z_0,w_0)$ is fixed and the smooth control Lagrangian

$$l:{\mathbb{R}}^n\times {\mathbb{R}}^m\to {\mathbb{R}}_{\ge 0},(z,v)\mapsto l(z,v)$$

(10)

satisfies $l(z,v)=0$ if and only if $(z,v)=(0,0)$. Then, the feedback term $\kappa$ in (7) is the feedback control law v obtained by solving the optimal control problem (9), i.e.,

$$v=\kappa (z,w)=\kappa \left(x-\theta \left(w\right),w\right).$$

Remark 1.

Note that the PDE (5) along with the algebraic constraints (6) is often solved approximately via finite series solutions [4,5]. Assume that the solution $(\theta ,\lambda )$ of the PDE is approximated by polynomials of degree r of the form

$$\begin{array}{c}\hfill {\theta }^{\left(r\right)}\left(w\right)=\sum _{i=1}^r{\theta }^{\left[i\right]}\left(w\right),\\ \hfill {\lambda }^{\left(r\right)}\left(w\right)=\sum _{i=1}^r{\lambda }^{\left[i\right]}\left(w\right),\end{array}$$

(11)

where ${\gamma }^{\left[i\right]}\left(\alpha \right)$ denotes a polynomial homogeneous of degree i in α. Then, under the change of coordinates

$$z=x-{\theta }^{\left(r\right)}\left(w\right)\mathrm{and}v=u-{\lambda }^{\left(r\right)}\left(w\right),$$

the optimal stabilization problem (9) leads to the following feedback control law

$$\begin{array}{c}\hfill {\overline{u}}^{\left(r\right)}(x,w)={\lambda }^{\left(r\right)}\left(w\right)+\kappa \left(x-{\theta }^{\left(r\right)}\left(w\right),w\right)\end{array}$$

(12)

that in turn ensures that the state-action pair $(x,u)$ converges asymptotically to $\left({\theta }^{\left(r\right)}\left(w\right),{\lambda }^{\left(r\right)}\left(w\right)\right)$. It is worth noting that $h\left({\theta }^{\left(r\right)}\left(w\right),{\lambda }^{\left(r\right)}\left(w\right),w\right)$ may not be zero due to the approximation of the feedforward control $u=\lambda \left(w\right)$ and the output regulation manifold ${\mathcal{M}}_{\mathcal{R}}$. Therefore, the series solution (12) does not guarantee asymptotic convergence of the system output (3) to zero. However, the output approximation error

$$e_y\left(w\right)=h\left(\theta \left(w\right),\lambda \left(w\right),w\right)-h\left({\theta }^{\left(r\right)}\left(w\right),{\lambda }^{\left(r\right)}\left(w\right),w\right)$$

is of order $O{\left(w\right)}^{r+1}$ [4], Theorem 4.2.

Equipped with a sufficient understanding of output regulation, let us design a feedback law for a class of nonlinear systems that leads to an asymptotic convergence of the system output to zero.

2.1. Problem Statement

Consider a nonlinear system

$$\begin{array}{cc}\hfill {\dot{x}}_1& =f_1(x_1,x_2,u,w),\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill {\dot{x}}_2& =f_2(x_1,x_2,u,w),\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill y& =x_1-\tilde{h}\left(w\right),\hfill \end{array}$$

(16)

where $x=(x_1,x_2)\in {\mathbb{R}}^p\times {\mathbb{R}}^{n-p}$ is the plant state with vector field $f=(f_1,f_2)$ governing the plant dynamics, $w\in {\mathbb{R}}^d$ is the exosystem state with vector field a governing the exosystem dynamics and $y\in {\mathbb{R}}^p$ is the system output.

Assumption 2.

Assumption 1 holds for the system dynamics (13)–(16) with the choice of $x=(x_1,x_2),$$f=(f_1,f_2)$ and $h(x,u,w)=x_1-\tilde{h}\left(w\right)$ that brings the dynamics (13)–(16) to the standard form (1)–(3).

Note that the system dynamics (13)–(16) is in standard form, and therefore, Theorem 1, leads to the following necessary and sufficient condition for the solvability of the output regulation problem for the system dynamics (13)–(16):

Theorem 2.

Under Assumption 2, there exist a neighborhood $\mathcal{O}\subset {\mathbb{R}}^n\times {\mathbb{R}}^d$ of 0 and a $C^k(k\ge 2)$ state feedback

$$u:\mathcal{O}\to {\mathbb{R}}^m,(x_1,x_2,w)\mapsto u(x_1,x_2,w)$$

that asymptotically stabilizes the output of the system dynamics (13)–(16) to zero if and only if there exist $C^k$ mappings $x_2=\tilde{\theta }\left(w\right)$ with $\tilde{\theta }\left(0\right)=0$, and $u=\lambda \left(w\right)$ with $\lambda \left(0\right)=0$, both defined in a neighborhood $\mathcal{W}\subset {\mathbb{R}}^d$ of 0, satisfying the conditions

$$\begin{array}{cc}\hfill & \frac{\partial \tilde{\theta }}{\partial w}a\left(w\right)=f_2(\tilde{h}\left(w\right),\tilde{\theta }\left(w\right),\lambda \left(w\right),w),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & \frac{\partial \tilde{h}}{\partial w}a\left(w\right)-f_1(\tilde{h}\left(w\right),\tilde{\theta }\left(w\right),\lambda \left(w\right),w)=0.\hfill \end{array}$$

(18)

Proof. 

We know that the dynamics (13)–(16) with the choice of $x=(x_1,x_2),f=(f_1,f_2)$ and $h(x,u,w)=x_1-\tilde{h}\left(w\right)$ is in standard form (1)–(3). Hence, applying Theorem 1 to the dynamics (13)–(16) gives: There exists a neighborhood $\mathcal{O}\subset {\mathbb{R}}^n\times {\mathbb{R}}^d$ of 0 and a $C^k(k\ge 2)$ state feedback

$$u:\mathcal{O}\to {\mathbb{R}}^m,(x_1,x_2,w)\mapsto u(x_1,x_2,w)$$

that asymptotically stabilizes the output of the system dynamics (13)–(16) to zero if and only if there exist $C^k$ mappings $(x_1,x_2)=({\theta }_1\left(w\right),\tilde{\theta }\left(w\right))$ with $({\theta }_1\left(0\right),\tilde{\theta }\left(0\right))=0$, and $u=\lambda \left(w\right)$ with $\lambda \left(0\right)=0$, both defined in a neighborhood $\mathcal{W}\subset {\mathbb{R}}^d$ of 0, satisfying the conditions

$$\begin{array}{cc}\hfill & \frac{\partial {\theta }_1}{\partial w}a\left(w\right)=f_1({\theta }_1\left(w\right),\tilde{\theta }\left(w\right),\lambda \left(w\right),w),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \frac{\partial \tilde{\theta }}{\partial w}a\left(w\right)=f_2({\theta }_1\left(w\right),\tilde{\theta }\left(w\right),\lambda \left(w\right),w),\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & {\theta }_1\left(w\right)-\tilde{h}\left(w\right)=0.\hfill \end{array}$$

(21)

The algebraic constraint (21) is satisfied if and only if ${\theta }_1\left(w\right)=\tilde{h}\left(w\right)$. Therefore, substituting ${\theta }_1=\tilde{h}$ in (19) leads to (18) and (20) leads to (17). This proves the assertion. □

Remark 2.

Note that the PDE (17) and (18) with algebraic constraint is in the same form as (5) and (6); however, the dimension of the PDE (17) and (18) is reduced. Therefore, the reduced order PDE (17) and (18) is computationally efficient.

We now turn to designing an optimal feedback control law using Krener’s method that locally regulates the system output (16) of the dynamics (13)–(16) asymptotically to zero.

First, a feedforward control law is designed by solving the FBI Equations (17) and (18) using HJB series solutions [4,5]. Let the series solution of the FBI Equations (17) and (18) be given by

$$\begin{array}{c}\hfill x_2={\tilde{\theta }}^{\left(r\right)}\left(w\right),\mathrm{and}u={\lambda }^{\left(r\right)}\left(w\right),\end{array}$$

(22)

where ${\gamma }^{\left(r\right)}\left(w\right)$ is a homogeneous polynomial in w up to degree r.

Second, the error dynamics is defined, under the change of coordinates

$$y=x_1-\tilde{h}\left(w\right),z=x_2-{\tilde{\theta }}^{\left(r\right)}\left(w\right),\mathrm{and}v=u-{\lambda }^{\left(r\right)}\left(w\right),$$

as

$$\begin{array}{cc}\hfill \dot{y}& =f_1\left(y+\tilde{h}\left(w\right),z+{\tilde{\theta }}^{\left(r\right)}\left(w\right),{\lambda }^{\left(r\right)}\left(w\right),w\right)-\frac{\partial \tilde{h}}{\partial w}a\left(w\right),\hfill \\ \hfill \dot{z}& =f_2\left(y+\tilde{h}\left(w\right),z+{\tilde{\theta }}^{\left(r\right)}\left(w\right),{\lambda }^{\left(r\right)}\left(w\right),w\right)-\frac{\partial {\tilde{\theta }}^{\left(r\right)}}{\partial w}a\left(w\right),\hfill \\ \hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

(23)

and the output regulation problem is translated to a stabilization problem as

$$\begin{array}{cc}\hfill \underset{v}{min}& {\int }_0^{\infty }l\left(y\left(t\right),z\left(t\right),v\left(t\right)\right)dt\hfill \\ \hfill \mathrm{subject}\mathrm{to}& \left\{\begin{array}{c}\mathrm{system}\mathrm{dynamics}\left(23\right),\hfill \\ \left(y\left(0\right),z\left(0\right),w\left(0\right)\right)=(y_0,z_0,w_0),\hfill \end{array}\right.\hfill \end{array}$$

(24)

where $(y_0,z_0,w_0)$ is fixed and the smooth control Lagrangian

$$l:{\mathbb{R}}^p\times {\mathbb{R}}^{n-p}\times {\mathbb{R}}^m\to {\mathbb{R}}_{\ge 0},(y,z,v)\mapsto l(y,z,v)$$

(25)

satisfies $l(y,z,v)=0$ if and only if $(y,z,v)=0$. The infinite horizon optimal control problem (24) is solved using Al’brekht’s method and the feedback control law

$$v=\kappa (y,z)=\kappa \left(x_1-\tilde{h}\left(w\right),x_2-{\tilde{\theta }}^{\left(r\right)}\left(w\right)\right)$$

is designed that locally stabilizes $(y,z)$ asymptotically to zero [4], Theorem 4.2. Therefore, the optimal feedback control

$$\begin{array}{c}\hfill {\tilde{u}}^{\left(r\right)}(x,w)={\lambda }^{\left(r\right)}\left(w\right)+\kappa \left(x_1-\tilde{h}\left(w\right),x_2-{\tilde{\theta }}^{\left(r\right)}\left(w\right)\right)\end{array}$$

(26)

locally regulates the output of the dynamics (13)–(16) to zero asymptotically. The system output y converges asymptotically to zero due to the fact that the PDE series solutions (22) do not affect the output regulation manifold

$${\mathcal{M}}_{\mathcal{R}}=\left\{(x_1,x_2,w)\in {\mathbb{R}}^n\times {\mathbb{R}}^d\left|x_1-\tilde{h}\left(w\right)=0\right.\right\}.$$

2.2. Computational Complexity

The feedback regulation problem for the system (13)–(16) is solved in two ways. A feedback control law is obtained by solving one of the FBI (5) and (6) and the FBI (17) and (18). As the dimension of the PDE in FBI (17) and (18) is reduced by p as compared to the FBI (5) and (6), it leads to a significant reduction in computation time. On the other hand, the regulation manifold of the system (13)–(16) is explicitly known and therefore, the feedback regulation law obtained by FBI (5) and (6) is more accurate as compared to FBI (17) and (18). A series solution of degree r of the FBI (5) requires the solving of a linear system of order $O\left((n+m)d^j\right)$ recursively for each degree $j=1,\dots ,r$. Therefore, the computation time for solving the FBI (17) and (18) using series solutions up to degree r is of order $O({(n+m)}^3{d}^{3r})$ and for the FBI (5) and (6) is of the order $O({(n-p+m)}^3{d}^{3r}).$ It can be concluded from the computation time analysis that there will be a significant reduction in computation time when the degree of the approximate series solution is large.

Let us now generalize the output regulation problem to manifolds. We know that many robotics and aerospace systems evolve on manifolds. The optimal stabilization theory developed by Krener [4] cannot be directly applied to the system evolving on manifolds. An intuitive way is to extended the system to the ambient Euclidean space and design the controller in that ambient space; however, such extensions may not preserve the stabilizability of the linearized system, which is one key assumption for the FBI Equations (5) and (6). This hurdle is circumvented by stably embedding the system dynamics into the ambient Euclidean space [8].

3. Output Regulation on Manifolds

Consider a class of nonlinear systems on a manifold $M\subset {\mathbb{R}}^n$

$$\begin{array}{cc}\hfill \dot{x}& =f(x,u,w),\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill y& =h(x,u,w),\hfill \end{array}$$

(29)

where plant state $x\in M$, plant input $u\in {\mathbb{R}}^m$, exosystem state $w\in {\mathbb{R}}^d$ and system output $y\in N$ such that the manifold $N\subset {\mathbb{R}}^n$ is embedded in ${\mathbb{R}}^p$ with $p\le n$.

The output regulation problem on the manifold is solved by stably embedding the system dynamics (27)–(29) to an appropriate Euclidean space such that the linearized system in the ambient Euclidean space is stabilizable. We would like to stress on the fact that the stabilizability of the linearized dynamics is one of the key assumptions for existence of an output regulating feedback control law; see Assumption 1.

A stabilizable extension of the dynamics (27)–(29) on the ambient Euclidean space ${\mathbb{R}}^n$ is conducted in two steps [8]:

• The plant dynamics (27) is extended to ${\mathbb{R}}^n$ and the system output (29) is extended on ${\mathbb{R}}^p$ as

  $$\begin{array}{cc}\hfill \dot{x}& =f_e(x,u,w),x\in {\mathbb{R}}^n,u\in {\mathbb{R}}^m,w\in {\mathbb{R}}^d\hfill \end{array}$$

  (30)

  $$\begin{array}{cc}\hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

  (31)

  $$\begin{array}{cc}\hfill y& =h_e(x,u,w),y\in {\mathbb{R}}^p\hfill \end{array}$$

  (32)

  such that $f_e(x,u)=f(x,u)$ and $h_e(x,u,w)=h(x,u,w)$ for all $(x,u,w)\in M\times {\mathbb{R}}^m\times {\mathbb{R}}^d$. As the extended plant dynamics (30) is identical to (27) on M, the manifold M is an invariant subset of ${\mathbb{R}}^n$, i.e., for initial conditions $\left(x\left(0\right),w\left(0\right)\right)\in M\times {\mathbb{R}}^d$, system trajectories of the dynamics (30)–(32) satisfy

  $$\left(x\left(t\right),w\left(t\right)\right)\in M\times {\mathbb{R}}^d\mathrm{for}\mathrm{all}t.$$
• Add a drift term to the extended plant dynamics (30) such that it is stabilizable in the transversal direction to M in ${\mathbb{R}}^n.$ Suppose there exists a function $V:U\subset {\mathbb{R}}^n\to {\mathbb{R}}_{\ge 0}$ on open neighborhood U of M in ${\mathbb{R}}^n$ such that

  $$M=V^{-1}\left(0\right),$$

  and

  $$\nabla V\left(x\right)·f_e(x,u,w)=0\mathrm{for}\mathrm{all}(x,u,w)\in U\times {\mathbb{R}}^m\times {\mathbb{R}}^d.$$
  Therefore, the extended plant dynamics (30) is stably extended and that leads to the following linearly stabilizable extension of (27)–(29) on $U\times {\mathbb{R}}^m\times {\mathbb{R}}^d$:

  $$\begin{array}{cc}\hfill \dot{x}& =\tilde{f}(x,u,w):=f_e(x,u,w)-\alpha \nabla V\left(x\right),\hfill \end{array}$$

  (33)

  $$\begin{array}{cc}\hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

  (34)

  $$\begin{array}{cc}\hfill y& =h_e(x,u,w),\hfill \end{array}$$

  (35)

  where $\alpha >0$. Here, instead of the number $\alpha >0$, one can more generally use an $n\times n$ positive definite symmetric matrix-valued function. A detailed discussion on the transversal stability of M in the stably extended dynamics (33)–(35) may be found in [8].

The system dynamics (33)–(35) is defined in Euclidean space and therefore, Krener’s method for designing feedback control for the output regulation problem is directly applicable without any modification.

For the sake of clarity, let us consider an example of a single axis rotation of a rigid body. The state space of the dynamics is $\mathrm{SO}\left(2\right)\times \mathbb{R}$ where $\mathrm{SO}\left(2\right),$ (the set of $2\times 2$ orthonormal matrices with determinant 1,) accounts for the attitude of the rigid body and the angular velocity of the body about the rotation axis lies in $\mathbb{R}$. The manifold $\mathrm{SO}\left(2\right)$ is a Lie group and the set $\mathfrak{so}\left(2\right)$, (the set of $2\times 2$ real skew-symmetric matrices,) is its Lie algebra. The attitude dynamics for single axis rotation of the rigid body is given by

$$\begin{array}{cc}\hfill \dot{R}& =R\widehat{\mathsf{\Omega }},\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill J\dot{\mathsf{\Omega }}& =\tau ,\hfill \end{array}$$

(37)

where $(R,\mathsf{\Omega })\in \mathrm{SO}\left(2\right)\times \mathbb{R}$ with R determines the attitude of the rigid body and $\mathsf{\Omega }$ determines the angular velocity of the rigid body, $J\in {\mathbb{R}}_{\ge 0}$ is the moment of inertia, $\tau \in \mathbb{R}$ is the torque applied along the axis of rotation, and the hat map $\wedge :\mathbb{R}\to \mathfrak{so}\left(2\right)$ is the vector space isomorphism defined as follows: for $\beta \in \mathbb{R}$

$$\widehat{\beta }=\left[\begin{array}{cc}0& -\beta \\ \beta & 0\end{array}\right].$$

Note that the manifold $\mathrm{SO}\left(2\right)$ is embedded in ${\mathbb{R}}^{2\times 2}$ and therefore, the system dynamics (36) and (37) is naturally extended to the ambient space ${\mathbb{R}}^{2\times 2}\times \mathbb{R}$. However, such natural extensions may not guarantee the stabilization of its linearized dynamics around an equilibrium point of interest. Let us define a stable extension of the dynamics (36) and (37) in a neighborhood

$${\mathrm{GL}}^{+}\left(2\right)=\left\{X\in {\mathbb{R}}^{2\times 2}\left|\det \left(X\right)>0\right.\right\}$$

of $\mathrm{SO}\left(2\right).$ To this end, let us define a Lyapunov-like function, $V:{\mathrm{GL}}^{+}\left(2\right)\times \mathbb{R}\to {\mathbb{R}}_{\ge 0}$ by

$$V(R,\mathsf{\Omega })=\frac{1}{4}{\parallel R^{\top }R-I\parallel }^2$$

for $(R,\mathsf{\Omega })\in {\mathrm{GL}}^{+}\left(2\right)\times \mathbb{R}$ with the usual Euclidean norm $\parallel ·\parallel$ on ${\mathbb{R}}^{2\times 2}$, which satisfies

$$V^{-1}\left(0\right)=\mathrm{SO}\left(2\right),\mathrm{and}{\nabla }_{R}V·\left(R\widehat{\mathsf{\Omega }}\right)=0.$$

It leads to a stable extension of the dynamics (36) and (37) on ${\mathbb{R}}^{2\times 2}\times \mathbb{R}$ as

$$\begin{array}{cc}\hfill \dot{R}& =R\widehat{\mathsf{\Omega }}-\alpha R(R^{\top }R-I),\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill J\dot{\mathsf{\Omega }}& =\tau ,\hfill \end{array}$$

(39)

where $\alpha >0.$ Let us consider an output regulation problem on the manifold $\mathrm{SO}\left(2\right)$, where the exosystem

$$\dot{w}=a\left(w\right)$$

generates attitude signals, $\tilde{h}\left(w\right)\in \mathrm{SO}\left(2\right)$ with $w\in {\mathbb{R}}^d$, for the dynamics (36) and (37) to track. The system dynamics with an exosystem for the output regulation is defined as

$$\begin{array}{cc}\hfill \dot{R}& =R\widehat{\mathsf{\Omega }}-\alpha R(R^{\top }R-I),\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill J\dot{\mathsf{\Omega }}& =\tau ,\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill \dot{w}& =a\left(w\right),\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill y& =R-\tilde{h}\left(w\right).\hfill \end{array}$$

(43)

The dynamics (40)–(43) are defined in Euclidean space and therefore, the Krener’s method [4] for optimal regulation is readily applied to find a feedforward and feedback control law. Using Theorem 2, the feedforward control law, $\tau =\lambda \left(w\right)$ with $w\in {\mathbb{R}}^d$, which makes the manifold $\mathsf{\Omega }=\tilde{\theta }\left(w\right)$ invariant, needs to satisfy the following FBI equations

$$\begin{array}{cc}\hfill & \frac{\partial \tilde{h}}{\partial w}a\left(w\right)=\tilde{h}\left(w\right)\widehat{\tilde{\theta }\left(w\right)}\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill & J\frac{\partial \tilde{\theta }}{\partial w}a\left(w\right)=\lambda \left(w\right).\hfill \end{array}$$

(45)

Remark 3.

Note that the FBI (44) and (45) is a PDE in $\mathbb{R}$ with algebraic constraints in ${\mathbb{R}}^{2\times 2}$ in contrast to the FBI obtained using Theorem 1 that is a PDE in ${\mathbb{R}}^{2\times 2}\times \mathbb{R}$ with algebraic constraints in ${\mathbb{R}}^{2\times 2}$. This simple example demonstrates that the PDE dimension is reduced to a large extent and it contributes to fast computation.

Remark 4.

Embedding $\mathit{SO}\left(2\right)$ to ${\mathbb{R}}^{2\times 2}$ increases the dimension of the state space by 3; however, one can identify $\mathit{SO}\left(2\right)$ with the unit circle and embed the unit circle in ${\mathbb{R}}^2$ (the ambient space) which only increases the dimension of the state space by 1.

Remark 5.

Note that the output regulation technique by Krener [4] does not incorporate state and control constraints. For output regulation of the safety critical systems where state and control constraints are crucial to consider at the controller design stage, a model predictive control approach is proposed by Krener [15]. The model predictive control approach is directly extended to manifolds by stably extending the system dynamics to an ambient Euclidean space.

4. Simulation Results

Let us solve the output regulation problem for the bi-directional quadcopter [16] and the rigid body attitude motion with Krener’s Matlab-based Nonlinear Systems Toolbox [6]. We demonstrate with the quadcopter example that fairly complex problems can be handled using this approach.

4.1. Quadcopter

A bidirectional quadcopter is an unmanned aerial vehicle that is fitted with four rotors to generate bidirectional (upward and downward) thrust and a torque to orient the quadcopter. The system dynamics for the quadcopter is given by

$$\begin{array}{cc}\hfill \dot{R}& =R\widehat{\mathsf{\Omega }},\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill \mathbb{I}\dot{\mathsf{\Omega }}& =\widehat{\mathbb{I}\mathsf{\Omega }}\mathsf{\Omega }+\tau ,\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill m\ddot{x}& =-mg{e}_3+R{e}_{3}f\hfill \end{array}$$

(48)

where $R\in \mathrm{SO}\left(3\right)$ (the set of $3\times 3$ rotation matrices) denotes the attitude, $\mathsf{\Omega }\in {\mathbb{R}}^3$ denotes the body angular velocity, and $x\in {\mathbb{R}}^3$ defines the position of the quadcopter. The control inputs are f and $\tau$, where $f\in \mathbb{R}$ accounts for the upward thrust generated by the rotors, and $\tau \in {\mathbb{R}}^3$ is the torque applied on the body. The parameter m is the quadcopter mass, $e_3=(0,0,1),$ and $\mathbb{I}$ is the $3\times 3$ moment of inertia matrix. The hat map $\wedge :{\mathbb{R}}^3\to \mathfrak{so}\left(3\right)$, where $\mathfrak{so}\left(3\right)$ denotes the set of $3\times 3$ skew symmetric matrices, is a vector space isomorphism defined as $\widehat{x}y=x\times y$ for all $x,y\in {\mathbb{R}}^3$.

Consider a position tracking problem in which the quadcopter traces a path

$$\begin{array}{c}\hfill \tilde{h}\left(w\right)=\left(\begin{array}{c}{w}_2+2w_1{w}_2\\ w_1-2\left(w_1^2-w_2^2\right)\\ -3w_2+4w_2^3\end{array}\right)\in {\mathbb{R}}^3\end{array}$$

(49)

that is generated by an exosystem

$$\begin{array}{c}\hfill \dot{w}=aw,\end{array}$$

(50)

where

$$a=\left(\begin{array}{cc}0& -0.1\\ 0.1& 0\end{array}\right).$$

The tracking problem is to regulate the output

$$\begin{array}{c}\hfill y=x-\tilde{h}\left(w\right)\end{array}$$

(51)

to zero that is subject to the quadcopter dynamics (46)–(48) and the exosystem dynamics (50). Note that the output regulation problem for the quadcopter dynamics (46)–(48) with the exosystem (50) and the output (51) is in the standard form. Therefore, Krener’s method extended to manifolds, as described in Section 3, is employed to design a feedback control law for regulating the output asymptotically to zero.

In order to apply Krener’s method, let us first stably extend the quadcopter dynamics (46)–(48) to Euclidean space in an identical manner as in (38) and (39). The stably extended dynamics with the exosystem and output is defined by

$$\begin{array}{cc}\hfill \dot{R}& =R\widehat{\mathsf{\Omega }}-\alpha R(R^{\top }R-I),\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill \mathbb{I}\dot{\mathsf{\Omega }}& =\widehat{\mathbb{I}\mathsf{\Omega }}\mathsf{\Omega }+\tau ,\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill \dot{x}& =v,\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill \dot{v}& =-g{e}_3+\frac{f}{m}R{e}_3,\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill \dot{w}& =aw,\hfill \end{array}$$

(56)

$$\begin{array}{cc}\hfill y& =x-\tilde{h}\left(w\right),\hfill \end{array}$$

(57)

where $R\in {\mathbb{R}}^{3\times 3},\mathsf{\Omega }\in {\mathbb{R}}^3,x\in {\mathbb{R}}^3,v\in {\mathbb{R}}^3,w\in {\mathbb{R}}^2,$ and $y\in {\mathbb{R}}^3.$ The dynamics (52)–(57) is in standard form and therefore, a feedback control law (12) up to degree r is given by

$$\begin{array}{c}\hfill {\overline{u}}^{\left(r\right)}=\left(\begin{array}{c}{\overline{\tau }}^{\left(r\right)}\\ {\overline{f}}^{\left(r\right)}\end{array}\right)={\lambda }^{\left(r\right)}\left(w\right)+\kappa \left({\overline{z}}^{\left(r\right)},w\right)\end{array}$$

(58)

where the feedforward ${\lambda }^{\left(r\right)}$ and the stabilizing manifold ${\overline{z}}^{\left(r\right)}=\left(R,\mathsf{\Omega },x,v\right)-{\theta }^{\left(r\right)}\left(w\right)$ are computed by solving the FBI (5) and (6) and the feedback $\kappa$ is computed by solving the stabilization problem (9) using Al’brekht’s method. On the other hand, in our technique, the feedback control (26) up to degree r is given by

$$\begin{array}{c}\hfill {\tilde{u}}^{\left(r\right)}=\left(\begin{array}{c}{\tilde{\tau }}^{\left(r\right)}\\ {\tilde{f}}^{\left(r\right)}\end{array}\right)={\tilde{\lambda }}^{\left(r\right)}\left(w\right)+\kappa \left(x-\tilde{h}\left(w\right),{\tilde{z}}^{\left(r\right)},w\right)\end{array}$$

(59)

where the feedforward ${\tilde{\lambda }}^{\left(r\right)}$ and the stabilizing manifold ${\tilde{z}}^{\left(r\right)}=\left(R,\mathsf{\Omega },v\right)-{\tilde{\theta }}^{\left(r\right)}\left(w\right)$ are computed by solving the FBI (17) and (18) and the feedback $\kappa$ is computed by solving the stabilization problem (24) using Al’brekht’s method. Let $\overline{x}\left\{r\right\}$ and $\tilde{x}\left\{r\right\}$ be the positions that are traced by the quadcopter (46)–(48) under the feedback ${\overline{u}}^{\left(r\right)}$ and ${\tilde{u}}^{\left(r\right)}$, respectively. Then, the corresponding tracking errors are given by

$$\overline{y}\left\{r\right\}=\overline{x}\left\{r\right\}-\tilde{h}\left(w\right)\mathrm{and}\tilde{y}\left\{r\right\}=\tilde{x}\left\{r\right\}-\tilde{h}\left(w\right).$$

The following parameters have been considered for the numerical experiments:

$$\begin{array}{cc}\hfill & m=0.468\mathrm{kg},g=9.81{\mathrm{ms}}^2,\mathbb{I}=23\times {10}^{-4}\mathrm{diag}(1,1,2)\hfill \\ \hfill & R\left(0\right)=I_{3\times 3},\mathsf{\Omega }\left(0\right)=(0,0,0),x\left(0\right)=(0.2,0.04,0.3)\hfill \\ \hfill & v\left(0\right)=(0,0,0),w\left(0\right)=(0.3,0.2).\hfill \end{array}$$

We can infer from the phase portrait in Figure 1 that the position trajectory $\tilde{x}\left\{r\right\}$ that is traced under the control law ${\tilde{u}}^{\left(r\right)}$ is tracing the exosystem trajectory $\tilde{h}$ more effectively as compared to the position trajectory $\overline{x}\left\{r\right\}$ that is traced under the control law ${\overline{u}}^{\left(r\right)}.$ The tracking errors $\tilde{y}\left\{r\right\}$ eventually converge to zero as shown in Figure 2, but $\overline{y}\left\{r\right\}$ does not converges to zero in Figure 3, which supports the claim that the tracking performance of the control law ${\tilde{u}}^{\left(2\right)}$ is better than ${\overline{u}}^{\left(2\right)}$. As shown in Figure 4, the optimal tracking control law ${\tilde{u}}^{\left(2\right)}$ shows that the quadcopter rotors will produce a negative thrust to catch up the exosystem trajectory and then torque and thrust eventually go to zero as the tracking error $\tilde{y}\left\{r\right\}$ goes to zero.

Figure 1. Phase portrait of quadcopter position.

Figure 2. Tracking error with state feedback ${\tilde{u}}^{\left(2\right)}$.

Figure 3. Tracking error with state feedback ${\overline{u}}^{\left(2\right)}$.

Figure 4. Optimal tracking control law ${\tilde{u}}^{\left(2\right)}$.

Remark 6.

Note that the reference $\tilde{h}\left(w\right)$ generated by the exosystem (50) is a cubic polynomial in w, and therefore, the feedback controls ${\tilde{u}}^{\left(3\right)}$ and ${\overline{u}}^{\left(3\right)}$ are identical. However, the feedback control ${\tilde{u}}^{\left(2\right)}$ provides good performance and is computationally less intensive as compared to the feedback control ${\overline{u}}^{\left(3\right)}$.

4.2. Rigid Body Attitude Control

A rigid body attitude dynamics is given by

$$\begin{array}{cc}\hfill \dot{R}& =R\widehat{\mathsf{\Omega }},\hfill \\ \hfill \mathbb{J}\dot{\mathsf{\Omega }}& =\widehat{\mathbb{J}\mathsf{\Omega }}\mathsf{\Omega }+\tau ,\hfill \end{array}$$

where $R\in \mathrm{SO}\left(3\right)$ denotes the attitude, $\mathsf{\Omega }\in {\mathbb{R}}^3$ denotes the body angular velocity, the control input $\tau \in {\mathbb{R}}^3$ is the torque applied on the body, and $\mathbb{J}$ is the $3\times 3$ moment of inertia matrix. The hat map $\wedge :{\mathbb{R}}^3\to \mathfrak{so}\left(3\right)$ is the vector space isomorphism mentioned for the quadcopter example.

Consider a rigid body attitude tracking problem in which the rigid body is tracking an attitude profile,

$$\begin{array}{c}\hfill \tilde{h}\left(w\right)=exp\left({\widehat{e}}_s{w}_1\right)\in {\mathbb{R}}^3\end{array}$$

(60)

where $e_s={(1,1,1)}^{\top }$ and $w=(w_1,w_2)\in {\mathbb{R}}^2$ that is generated by an exosystem on ${\mathbb{R}}^2$ defined by

$$\begin{array}{c}\hfill \dot{w}=aw\end{array}$$

(61)

with

$$a=\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right).$$

The output regulation problem for the rigid body tracking problem is to regulate the system output

$$\begin{array}{c}\hfill y=R-\tilde{h}\left(w\right)\end{array}$$

to zero asymptotically. We adopt an identical procedure, as in case of quadcopter control, to derive the feedback control ${\tilde{\tau }}^{\left(r\right)}$ and ${\overline{\tau }}^{\left(r\right)}$ as discussed in (58) and (59), respectively, and the corresponding tracking errors are given by $\tilde{y}\left\{r\right\}$ and $\overline{y}\left\{r\right\}$. Note that the rigid body tracking case is different from the quadcopter tracking in the sense that we are tracking a transcendental function (60). Therefore, the feedback control ${\overline{\tau }}^{\left(r\right)}$ cannot achieve regulation asymptotically for any r; however, the feedback control ${\tilde{\tau }}^{\left(r\right)}$ achieves regulation asymptotically for $r=2$; see Figure 5.

Figure 5. Tracking errors with three different controllers for the rigid body attitude control.

5. Conclusions and Future Works

This article presents a technique of designing an optimal feedback controller that achieves regulation asymptotically for a class of controlled systems. Moreover, we have generalized the optimal regulation problems on Euclidean spaces to manifolds with the embedding technique, and demonstrated its applicability by designing optimal tracking feedback control laws for the bi-directional quadcopter system and the rigid body control system. As a future work, we plan to investigate the case of model predictive regulation on Lie groups in the framework developed in this paper. We also plan to apply this framework to reinforcement learning.