Physical Reinforcement Learning with Integral Temporal Difference Error for Constrained Robots

Abstract

The paradigm of reinforcement learning (RL) refers to agents that learn iteratively through continuous interactions with their environment. However, when the value function is unknown, a neural network is used, which is typically encoded into an unknown temporal difference equation. When RL is implemented in physical systems, explicit convergence and stability analyses are required to guarantee the worst-case operations for any trial, even when the initial conditions are set to zero. In this paper, physical RL (p-RL) refers to the application of RL in dynamical systems that interact with their environments, such as robot manipulators in contact tasks and humanoid robots in cooperation or interaction tasks. Unfortunately, most p-RL schemes lack stability properties, which can even be dangerous for specific robot applications, such as those involving contact (constrained) tasks or interaction tasks. Considering an unknown and disturbed DAE2 robot, in this paper a p-RL approach is developed to guaranteeing robust stability throughout a continuous-time-adaptive actor–critic, with local exponential convergence of force–position tracking error. The novel adaptive mechanisms lead to robustness, while an integral sliding mode enforces tracking. Simulations are presented and discussed to show our proposal’s effectiveness, and some final remarks are addressed concerning the structural aspects.

1. Introduction

Reinforcement learning (RL) has emerged as an alternative decision-making learning technique that issues a reinforcement to compute the action, or control, based on recollection of the discount reward encoded throughout the performance evaluation; a distinct feature of RL versus control is precisely that it evaluates the performance each time and iteratively corrects the action to improve it, leading to a task being learned. Unfortunately, for us roboticists, it was developed and substantiated for the tuple of state–action, with the state s being the output and the action $\pi$ the input of a system $\Sigma$—however, without an explicit structure of $\Sigma$. Thus, to deploy RL in the physical layer, e.g., a robot, a lot of effort is required to guarantee that the physical structural properties of all of the components are met, with concerns arising about the safety of its deployment in a real-world application [1]. In addition, due to the iterative realm of these schemes, the safety requirements are exacerbated, leading to sim-to-real schemes to transfer RL schemes to the physical realm. In addition, when a rigid-body robot engages in contact with a rigid environment, high-frequency contact is of concern due to the high energy transfer index. Fortunately, the robotics community has proposed RL schemes based on a Lagrangian structure for $\Sigma$ and derived RL motor learning, ready to meet such structures based on passivity-based and model-free adaptive schemes [2]. In the latter type of scheme, ordinary differential equations (ODEs) model the Lagrangian robot for free-motion tasks, and safety issues are handled since the passivity is preserved; uncertainties and unmodeled dynamics are considered; smooth action is synthesized; and no trial-and-error is involved, but explicit energy-based stability conditions are established. In this context, the energy-based stability and adaptive mechanism that substitutes pre-training sets apart this approach of RL for robots as a machine learning scheme since the learning paradigm of a black-box interactive process of exploration and exploitation is circumvented using a structure energy-based adaptation that guides reward recollection without pre-training, quite convenient for physical world applications. In this paper, we extended such an approach to constrained motion modeled using a differential algebraic system of equations (DAE2) to study position–force learning tasks.

Most robotic contact tasks require precise control of the normal force exerted on a surface while following a desired end-effector trajectory on the tangent plane. Active force–position control considers the DAE2’s constrained dynamics, wherein the derivative of the momentum represents the Lagrangian multiplier (contact force) to be controlled. Most schemes circumvent the DAE2 formulation, using ODE (unconstrained motion) dynamical models, where no state is related to the contact force [3]. Popular schemes that use the latter have proliferated, among them impedance control [4,5] and stiffness control, which cannot guarantee position–force convergence. On the other hand, the DAE2 formulation leverages orthogonalized subspaces to model orthogonal operators from the holonomic constraint, thereby projecting force and velocity errors into orthogonal complements. Then, control is designed to enforce force and position convergence simultaneously and independently [6], followed by model-based schemes [7]. Additionally, to compensate for unknown robot dynamics, even Lipschitz disturbances, a method called neurocontrol is proposed where neural networks, NNs, are used as approximator functions or to compensate for unknown dynamics. Some approaches to contact tasks based on NNs are presented in [8,9,10]. In recent years, the application of reinforcement learning (RL) to dynamical systems has attracted attention due to its ability to learn through interaction by encoding different behaviors [11]. Crucial elements of RL are the instrumentation of a “reward” and its “effort” (the value function), which qualify a robot’s task performance accordingly. Depending on such an evaluation, it issues a reinforcement to the “action”, which leads to an improved performance while minimizing the value function.

Actor–critic is a technique used to instrument RL based on a critic NN used to approximate the value function of the temporal difference error equation. In contrast, the actor NN approximates inverse dynamics to yield the action [12]. In control schemes, the actor NN is coupled with a damping term to stabilize the robot. This scheme has gained popularity due to its ability to handle uncertainties while gradually improving the performance to facilitate the successful completion of robotic tasks for free motion, such as in [13,14], to name a few. For contact/interaction tasks, see [15,16,17,18]; notably, in [19], an admittance controller computes the desired force based on RL, leading to a stable force error. Similarly, in [15], adaptive dynamic programming based on an LQR is explored, with bounded errors. In [16], an impedance-based advantage actor–critic (A2C) deals with constraint uncertainties, while [17] uses an impedance-based natural gradient actor–critic (NAC), with experiments. Additionally, [18] presents a model-based AC for optimal impedance tuning, and it is used for cooperation with multi-robots in [20,21], even for humanoid robots [22] without a stability analysis. Although variants of the AC approach have shown effectiveness for contact/interaction tasks, the literature has overlooked the force–position convergence, which is critical due to interactions that occur at a high frequency (due to rigid-body contact), in particular when the robot is constrained by a rigid environment, i.e., a holonomic constraint that yields a DAE2 model.

An early review [23] and recent work [24] on deep RL for manipulation have pointed out the need to explicitly address certain challenges in constrained robot dynamics resulting from the physical interaction, which we refer to as physical RL (p-RL). This acronym aims to distinguish it from RL schemes for unconstrained robots; the underlying foundation of p-RL is based on sound physical interaction principles or contact tasks rather than simply software apps (s-RL). That is, in p-RL, the physical layer of a robotic system cannot withstand hundreds, or thousands, or millions, of training or pre-training tests, as are typically required nowadays in RL, due to the risk of compromising the robot’s structural integrity. Thus, p-RL schemes need to be substantiated through a model-based stability analysis to produce an actionable (physically speaking) action command. Actionable p-RL implies control that is smooth, causal, model-free, and bounded, without pre-training requirements.

In this manuscript, we propose actionable p-RL for a DAE2 robot, showing position–force convergence with a stability analysis. A continuous-time AC scheme deploys RL, extending the methodology [25] from unconstrained to constrained robot dynamics. Hereby, fast and robust adaptation for the critic is obtained by handling robust convergence of the so-called integral temporal difference (ITD) error. At the same time, a passivity-based actor is proposed over orthogonalized error manifolds, where robust convergence is guaranteed based on chatterless integral sliding modes.

2. Preliminaries

2.1. Robot Dynamics and the Problem Statement

Let us take a robot manipulator with n degrees of freedom whose end-effector is constrained to evolve onto a rigid surface, modeled by the following differential algebraic equation (DAE index 2, or DAE-2)

$$H\left(q\right)\ddot{q}+C(q,\dot{q})\dot{q}+{\tau }_{fr}+g\left(q\right)=\tau +J_{\phi }^T\left(q\right)\lambda +{\tau }_d,$$

(1)

$$s.t.\phi \left(q\right)=0,$$

(2)

where $q\in {\mathbb{R}}^n$ stands for the generalized position and $\dot{q}\in {\mathbb{R}}^n$ the generalized velocity, $H\left(q\right)\in {\mathbb{R}}^{n\times n}$ is the inertia matrix, $C(q,\dot{q})\in {\mathbb{R}}^{n\times n}$ is the Coriolis matrix, $g\left(q\right)\in {\mathbb{R}}^n$ is the gravity vector, $J_{\phi }\in {\mathbb{R}}^{l\times n}$ is the constrained Jacobian, $\lambda \in {\mathbb{R}}^1$ stands for the contact force in operational space, and ${\tau }_{fr}$ represents affine friction and ${\tau }_d$ a Lipschitz disturbance. Notice that the rigid surface is modeled by the implicit equation $\phi \left(q\right)=0\in R^l$, for $l=1$ independent contact point. Additionally, the DAE system is of index 2 since the input appears in the second derivative of the constraint, and its state is $\xi =(q,\dot{q},p=\int \lambda )$, where $q\in {\mathbb{R}}^{n-r},\dot{q}\in {\mathbb{R}}^{n-r}$, and the momentum $p\in {\mathbb{R}}^1$.

2.2. The Actor–Critic Learning Control Problem

Leveraging the functional $Y_r=H\left(q\right){\ddot{q}}_r+C(q,\dot{q}){\dot{q}}_r+g\left(q\right)+{\tau }_{fr}$, where ${\dot{q}}_r$ is a nominal velocity reference to the shape orthogonalized error coordinate system $S_r=\dot{q}-{\dot{q}}_r$, [26], then the open-loop error equation is defined as

$$H\left(q\right){\dot{S}}_r+C(q,\dot{q})S_r+B{S}_r=\tau +{\tau }_d+J_{\phi }^T\lambda -Y_r.$$

(3)

Thus, the problem statement is to design a model-free p-RL control mechanism for a DAE2 robot (1) and (2) in order to generate a smooth action policy $\tau$ such that the state $\xi =(q,\dot{q},p)\to {\xi }_d=(q_d,{\dot{q}}_d,p_d=\int {\lambda }_d)$ without any previous training or learning and any knowledge of the robot dynamics, $Y_r$. The use of the term “model-free” in this work refers to a lack of knowledge of the robot’s dynamic model, but the kinematics are assumed to be known.

3. Learning Actor–Critic Design

In the AC algorithm proposed here, the critic learning process is based on a stability-guaranteed integral temporal difference error. On the other hand, the actor synthesizes a partial action, playing the role as an approximator of the inverse dynamics. In this way, the proposed controller is composed of the action plus a stabilizer term, typically a PID-like term.

In the following sections, it is shown how stability is ensured in the NN-critic using Lyapunov stability theory. After that, the adaptation laws for the NN-actor are designed to guarantee closed-loop stability.

3.1. The Value Function and the Temporal Difference Error

Let the value function in continuous time, representing the cumulative rewards, given as [27],

$$R={\int }_{t_0}^{\infty }{e}^{-{\displaystyle \frac{m-t}{\psi }}}r\left(m\right)dm,$$

(4)

where r represents the instant reward, and $\psi$ corresponds to a discount factor with $t_0\le m\le \infty$. In continuous time, the value function is driven by the temporal difference (TD) error [11], such that differentiating (4), we have

$$\dot{R}={\displaystyle \frac{1}{\psi }}R-r\left(t\right).$$

(5)

In cases where the value function R is not exactly known, the learning goal is to guarantee that

$$\delta =\dot{R}-{\displaystyle \frac{1}{\psi }}R+r=0.$$

(6)

Expressing (6) as a lower-order equation $\gamma =\int \delta ={\int }_{t_0}^{t_f}\delta \left(\tau \right)d\tau$, the integral temporal difference (ITD) error arises as

$$\gamma =R-{\displaystyle \frac{1}{\psi }}\int R+\int r.$$

(7)

Notice that if $\gamma \to 0$, so does (6), and henceforth (5) applies. The foundations of this chain of implications lead to the learning rationale encoded by the unknown value function (4).

Remark 1.

The motivation behind the introduction of the ITD error is to drive the learning and stability properties of the critic by γ, specifically the quasi-sliding mode condition. Additionally, the inclusion of the ITD eliminates the need for $\ddot{R}$ in the stability analysis. In the next section, the use of the TD error and the ITD as tools to design critic adaptive weights is explained.

3.2. The Critic Neural Network

Assuming that the smooth value function R of (4) is unknown, yet it is Lipschitz, then it can be approximated by a neural network. To this end, a large vector of the ideal weights $W_c^{\ast }$ and a basis $Z_c^{\ast }\left(X_c^{\ast }\right)$ exist for a set of ideal inputs $X_c^{\ast }$ such that $R=W_c^{\ast T}{Z}_c^{\ast }$. Now, considering a smaller set of weights $W_c\subset W_c^{\ast }$ and basis $Z_c\subset Z_c^{\ast }(\ast )$, a small reconstruction error ${ϵ}_R$ appears that approximates the value function, i.e.,

$$R=W_c^T{Z}_c+{ϵ}_R.$$

(8)

The linear parametrization (8) suggests that a subset of adaptive weights ${\widehat{W}}_c\in W_c$ and a basis exist such that the value function R can be approximated as

$$\widehat{R}={\widehat{W}}_c^T{Z}_c,$$

(9)

where ${\widehat{W}}_c\in {\mathbb{R}}^{p\times 1}$ are adaptive weights; $Z_c=\sigma \left({\chi }_c\right)$ is the bipolar sigmoid function, with the scaled input ${\chi }_c=V_c^T{X}_c$, with $V_c$ a constant matrix; and $X_c$ is the input vector to the neural network. Using (9) and its derivative, the neural approximation of $\delta$ becomes

$$\begin{array}{cc}\hfill \widehat{\delta }& =\dot{\widehat{R}}-{\displaystyle \frac{1}{\psi }}\widehat{R}+r\hfill \\ \hfill & ={\dot{\widehat{W}}}_c^T{Z}_c+{\widehat{W}}_c^T{\dot{Z}}_c-{\displaystyle \frac{1}{\psi }}{\widehat{W}}_c^T{Z}_c+r,\hfill \end{array}$$

(10)

where $\widehat{\delta }\left(t_0\right)\ne 0$ in general. Note that using the NN parametrization (9) in Equation (7), we obtain $\dot{\widehat{\gamma }}=\widehat{\delta }$. The challenge now becomes proposing an adaptive law ${\dot{\widehat{W}}}_c$ such that $\widehat{\delta }$ converges, meaning that the learning process is achieved within a small approximation error.

Proposition 1

(Adaptive Critic Weight Design). Let the adaptation law for the critic’s weights be defined as follows:

$${\dot{\widehat{W}}}_c=-K_{\omega }\mathrm{sgn}({\widehat{W}}_c)-K\mathrm{sgn}(\widehat{\gamma }){\displaystyle \frac{Z_c}{Z_c^T{Z}_c}},$$

(11)

where $K_w$, K are positive gains. Then, $\widehat{\gamma }\to {ϵ}_{\gamma }$ for ${ϵ}_{\gamma }$ a constant, which leads to $\dot{\widehat{\gamma }}=0\to$ $\widehat{\delta }=0$. Finally, due to the reconstruction error, we obtain $\delta \to {ϵ}_{\delta }$.

Proof. 

See Appendix A.1. □

3.3. The Actor Neural Network

The actor network has the role of approximating the inverse dynamics parametrized by the functional $Y_r$. Then, analogously constant weights $W_a$ and the basis $Z_a$ exist such that

$$Y_r=W_a^T{Z}_a+{ϵ}_a,$$

(12)

where ${ϵ}_a$ is the neural reconstruction error. Henceforth, an actor neural network exists that approximates (12) as follows

$${\widehat{Y}}_r={\widehat{W}}_a^T{Z}_a,$$

(13)

where ${\widehat{W}}_a\in {\mathbb{R}}^{h\times n}$ are adaptive weights, and $Z_a=\sigma \left({\chi }_a\right)$ is the bipolar sigmoid function, with the input ${\chi }_a=V_a^T{X}_a$, where $V_a$ are constant weights, and $X_a$ its input vector. The adaptation law ${\dot{\widehat{W}}}_a$ still needs to be designed.

Proposition 2

(Adaptive Actor Weight Design). Let the following adaptive law update the actor weights:

$${\dot{\widehat{W}}}_a=-{\Gamma }_a{Z}_a{S}_r^T-{\Gamma }_a{\widehat{W}}_a{\left(\widehat{\gamma }r\right)}^2,$$

(14)

where ${\Gamma }_a\in {\mathbb{R}}_{+}^{h\times h}$ is the feedback gain, and ${\left(\widehat{\gamma }r\right)}^2$ constitutes the reinforcement. Then, weights ${\widehat{W}}_a$ are bounded.

Proof. 

See Appendix A.2. □

3.4. Model-Free Actor–Critic Control Design

Assuming that $Y_r$ is unknown and approximated by the NN-actor, let the controller be given as

$$\tau =-K_d{S}_r+{\widehat{W}}_a^T{Z}_a+J_{\phi }^T\left(-{\lambda }_d-{\dot{S}}_{dF}+K_F\tanh \left(\mu S_{qF}\right)+\eta S_{vF}\right),$$

(15)

where $S_r=\dot{q}-{\dot{q}}_r$ is an orthogonalized extended velocity error arising when the nominal reference ${\dot{q}}_r$ is

$${\dot{q}}_r=Q\left(q\right)\left(\dot{q}-\alpha \Delta q+S_{dq}-K_q\int \mathrm{sgn}\left(S_{qq}\right)\right)+J_{\phi }^{+}\left(q\right)\beta S_{vF},$$

(16)

where $Q\left(q\right)=I-J_{\phi }^{+}\left(q\right)J_{\phi }\left(q\right)$ is the orthogonal projection of $J_{\phi }$, $J_{\phi }^{+}$ is the right pseudoinverse, $\Delta q=q-q_d$ is the position error, $q_d\in {\mathcal{C}}^2$ is the desired position, and $\beta$ is a positive definite gain. Thus, the extended velocity error $S_r$ can be rewritten as follows:

$$S_r=Q\left(q\right)S_{vq}-J_{\phi }^{+}\left(q\right)\beta S_{vF},$$

(17)

where

$$S_{vq}=S_{qq}+K_q\int \mathrm{sgn}\left(S_{qq}\right)\mathrm{position} \mathrm{error} \mathrm{manifold},$$

(18)

$$S_{vF}=S_{qF}+K_F\int \mathrm{sgn}\left(S_{qF}\right)\mathrm{force} \mathrm{error} \mathrm{manifold},$$

(19)

for $S_{qq}=S_q-S_{dq}$, $S_q=\Delta \dot{q}+\alpha \Delta q$, $S_{dq}=S_q\left(t_0\right)e^{-k_{1}t}$, $S_{qF}=S_F-S_{dF}$, $S_F=\int \Delta \lambda$, $S_{dF}=S_F\left(t_0\right)e^{-k_{2}t}$, and $\Delta \lambda =\lambda -{\lambda }_d$, where ${\lambda }_d$ is the desired contact force, and $\alpha$, $k_1$, $k_2$, $K_q$, $K_F$, $\beta$ are positive definite feedback gains. Using (3) and (15), we obtain the following closed-loop error equation:

$$\begin{array}{cc}\hfill H\left(q\right){\dot{S}}_r=& -C(q,\dot{q})S_r-B{S}_r+{\tau }_d-K_d{S}_r+{\widehat{W}}_a^T{Z}_a-Y_r\hfill \\ \hfill & +J_{\phi }^T({\dot{S}}_{vF}+\eta S_{vF})+K_F{J}_{\phi }^T{ϵ}_z,\hfill \end{array}$$

(20)

where ${ϵ}_z=(\tanh \left(\mu S_{qF}\right)-\mathrm{sgn}\left(S_{qF}\right))$. Notice that the state of the closed-loop error equation is complemented with (10), (11) and (14). We are now ready to announce the main result.

Theorem 1.

Consider the constrained robot dynamics (1) in the closed loop with the proposed actor–critic learning controller (15), which leads to the system of Equations (10), (11), (14) and (20). Then, from Propositions 1 and 2 and selecting $K_d$, $K_q$, $K_F$ large enough, the integral sliding mode at $S_{qq}=0$ and $S_{qF}=0$ is enforced. Thus, the local exponential convergence of the tracking errors $\Delta q$, $\Delta \dot{q}$, and $\Delta \lambda$ is guaranteed, and $\widehat{\delta }\to 0$ without any knowledge of the DAE2 dynamics.

Proof. 

See Appendix A.2. □

4. Simulations

Simulations are carried out in Matlab Simulink using the integrator ode4, with a fixed step of $1\times {10}^{-4}$. The simulation consists of a 3-dof robot in contact with a rigid plane parallel to the $x-y$ plane, see Figure 1, at a height of $z_d=0.15$ m. The desired trajectory onto the plane $x-y$ is a circle with the radius $a=0.15$ centered at $(x_c,y_c)=(0.5,0)$; then, the desired Cartesian path is $x_d=x_c+a\cos \left(wt\right)$ and $y_d=y_c+a\sin \left(wt\right)$, with the frequency $w=1$; the desired force is ${\lambda }_d=25+2\sin \left(t\right)$ N.

The robot parameters are the following: link lengths $l_1=0.5$, $l_2=0.5$, and $l_3=0.35$ m; link masses $m_1=7.2$, $m_2=5$, and $m_3=1.9$ kg; distances to the center of mass $l_{c1}=0.25$, $l_{c2}=0.19$, and $l_{c3}=0.12$; and inertias $I_1=0.15$, $I_2=0.02$, and $I_3=0.016$. The initial conditions are $q\left(t_0\right)={(0,0,-\pi /2)}^T rad$, $\dot{q}\left(t_0\right)={(0,0,0)}^T rad/s$, corresponding to being in contact with the plane.

The reward is designed to continuously supervise the tracking performance. Thus, a weighted summation of the position and velocity errors is considered, without weighting the integral of the tracking errors, regardless of its orthogonal projection. Then, $r=\Delta q^{T}P\Delta q+\Delta {\dot{q}}^{T}A\Delta \dot{q}+{(\int \Delta \lambda )}^2$, with $P=diag\left(0.9\right)\in {\mathbb{R}}^{n\times n}$, $A=diag\left(0.1\right)\in {\mathbb{R}}^{n\times n}$ so that the position is greatly weighted given that the robot is constrained in its position. The adaptation gains are selected as $K_w=7$, $K=50$, ${\Gamma }_a=I\times 1500$ and the feedback gains are selected as $K_d=diag\left(5010050\right)$, $\beta =1$, $\alpha =diag\left(333\right)$, $K_q=diag\left(111\right)$, $K_F=1.15$, $k_1=50$, $k_2=10$, $\eta =2000$.

Results

In Figure 2, we can see that the integral temporal difference error converges close to zero and remains constant; consequently, the temporal difference error, a derivative of the ITD error, converges to zero. This means that the adaptive laws for the NN-critic guarantee the convergence of the TD error. The integral temporal difference error dynamics facilitate motor learning through reward recollection; the larger the ITD error or the reward, the greater the reinforcement to improve the task performance. This is independent of how the reward policy is designed, including Pavlov’s negative rewards [28].

The joint position and velocity errors are shown in Figure 3, which converge approximately after 1 s, while Figure 4a shows the convergence of the force error in approximately 0.02 s, considerably faster than the position and velocity errors, shown in Figure 4b. Extended velocity errors are shown in Figure 5a, which remain bounded. Additionally, Figure 5b and Figure 6a show how the invariant manifolds for position-velocity and force, respectively, remain near zero. In Figure 6b, the performance of the smooth control signals is shown. Finally, Figure 7 shows how the actor–critic weights remain bounded, as is expected according to the theory. The convergence of the tracking errors stems from the underlying integral sliding mode, notably without any chattering in the controller, which contrasts with the existing literature, which lacks a similar result. Chattering refers to a very high-frequency control component that puts the structural integrity of the robot, let alone its constrained motion, in danger. It is unclear how to achieve smooth and robust control and convergence using another technique.

To compare the performance of the proposed approach, a comparative study is presented under the same assumptions as those for the model-free DAE robot, including the parameters and simulation conditions. Specifically, it is used as a force–position neurocontroller for a DAE-2 robot. For the comparative study, the following performance metrics are used: $ITA{E}_{pos}={\int }_{t_0}^{t}t{∥\Delta q∥}^2$ represents the accumulation of the position error over time; $ITA{E}_{vel}={\int }_{t_0}^{t}t{∥\Delta \dot{q}∥}^2$ represents the accumulation of the velocity error over time; $IAC={\int }_{t_0}^t∥\tau ∥$ and $IACV={\int }_{t_0}^t∥{\displaystyle \frac{d}{dt}}\tau ∥$ represent the control effort index and the smoothness of the control signal index, respectively.

Table 1. Metrics obtained from our proposal and the neurocontroller for contact tasks.

Control Law	Performance Metrics
	${\mathit{ITAE}}_{\mathit{pos}}$	${\mathit{ITAE}}_{\mathit{vel}}$	$\mathit{IAC}$	$\mathit{IACV}$
Our Proposal	4.796 × 10−4	9.194 × 10−2	258.8	142.9
Neurocontroller	7.142 × 10−4	9.246 × 10−2	259	151.6

presents the numerical values obtained for each case, indicating that our proposal yields smaller values, representing a better performance than that of the neurocontroller.

5. Conclusions

A novel model-free AC controller for contact tasks, which yields motor learning in DAE2 robots, is presented. This scheme is based on a method called p-RL, which refers to the application of RL to dynamical systems that interact with their environments. The proposed p-RL is instrumented through an AC architecture for robotic contact tasks, instrumenting novel adaptive laws. It is worth mentioning that the stability involves the nominal DAE2 dynamics; however, the AC-based controller scheme is free of model dynamics.

Update laws are designed to satisfy Lyapunov stability rather than being concerned with parametric variations, as in gradient descent. In this way, local minima are avoided, resulting in stronger stability properties in the time domain of the complete state of the closed-loop error equation.

A simulation for a 3D robot constrained by a rigid plane is presented, demonstrating tracking with smooth control actions, without requiring knowledge of the disturbed dynamics. An experimental evaluation is underway, including a comparison with various reward policies, as well as asymptotic model-free controllers that guarantee tracking, such as neurocontrol and integral sliding modes. Finally, we have employed a general formulation of DAE2 robots, which enables a straightforward extension to multirobot tasks, such as cooperative manipulators and robotic hand manipulation, particularly in critical contact tasks where continuous oversight of the performance is required.

Safety concerns arise when attempting to deploy an RL scheme (obtained under the premise of an abstract system $\Sigma$) in a physical system. Lyapunov-based schemes are a straightforward methodology to deal with, as devised more than two decades ago in the sound paper [29], or, for instance, in synthesizing control Lyapunov functions from barrier Lyapunov functions or Lyapunov stability, as in our case.

Unlike the traditional RL schemes reported in the literature, the proposed p-RL does not require any training or learning in the actor or the critic. Although sim-to-real transfer may be recommended to reduce high real-world trial costs, it is not a fundamental step for the proposed approach. That is, the process of transferring knowledge or policies learned in a simulated environment to the real world is not necessary. Only the process of tuning the gains is required to satisfy the stability conditions, where physics-based simulators can be used to improve the transferability.

Last but not least, model-free force–position control for DAE2 robots was solved over three decades ago, without chattering [7]. Therefore, the question arises as to what contribution is obtained by introducing motor learning. When motor learning is deployed, tracking is commensurate with the performance evaluation, not only asymptotic tracking. This latter notion enforces tracking as $t\to \infty$, whereas motor learning supervises the performance at each instant, not in the limit, at infinity.