Adaptive 3D Visual Servoing of a Scara Robot Manipulator with Unknown Dynamic and Vision System Parameters

Abstract

In the present work, we develop an adaptive dynamic controller based on monocular vision for the tracking of objects with a three-degrees of freedom (DOF) Scara robot manipulator. The main characteristic of the proposed control scheme is that it considers the robot dynamics, the depth of the moving object, and the mounting of the fixed camera to be unknown. The design of the control algorithm is based on an adaptive kinematic visual servo controller whose objective is the tracking of moving objects even with uncertainties in the parameters of the camera and its mounting. The design also includes a dynamic controller in cascade with the former one whose objective is to compensate the dynamics of the manipulator by generating the final control actions to the robot even with uncertainties in the parameters of its dynamic model. Using Lyapunov’s theory, we analyze the two proposed adaptive controllers for stability properties, and, through simulations, the performance of the complete control scheme is shown.

1. Introduction

Currently, research in the field of industrial robot control is focused on adding external sensors combined with advanced control strategies so that robots can work in unknown or semi-structured environments, thus, increasing the field of applications. Among the most widely used external sensors, vision sensors provide rich information about the working space. For this reason, vision-based control systems applied to robotics have been extensively studied in recent years.

Servo-visual control systems can be classified into image- or position-based systems according to how the control errors are defined, and in handheld or fixed camera system depending on the location of the vision camera with respect to the robot [1]. Furthermore, these control systems can be dynamic [2] or kinematic [3] depending on whether or not the dynamics of the robot are considered in its design.

Although the parameters of the models of these control systems can be obtained with sufficient precision through calibrations or identification techniques, there will always be uncertainties due to assembly errors, variations in the load handled, etc. To deal with these uncertainties, various adaptive robot controllers have been proposed. In [4], uncertainties in the kinematic parameters of a handheld camera system were considered, without demonstrations of the stability of the system. In [2], only the uncertainties in the parameters of the vision system were dealt with, and local stability results were presented. On the other hand, in [5], a precise knowledge of the kinematics of the robot and the vision system was assumed, considering only uncertainties in the robot’s dynamics.

The authors in [6,7,8] presented adaptive controllers with uncertainties in the vision system with a proof of global convergence to zero of the control errors for positioning tasks and only with bounded errors for the following tasks. These works did not consider the dynamics of the robot. Other works, such as [9,10], considered uncertainties in both the camera and robot parameters for a fixed camera system: the first with the assumption of needing an explicit measurement of the speed of the end of the robot in the image, and the second with a similar approach to the first but that avoided this measurement.

Both works dealt separately with the problem of adapting the parameters of the camera and the robot with a cascade structure and with a complete stability test of the system. Although the design of the adaptive robot controller followed a classical structure, it was based on a model with torque and force inputs, which is not the case with a real industrial robot. The design of the servo-visual controller was complex, and the simulation results did not effectively show a convergence to zero of the control errors. In the above mentioned works, the controllers were designed for planar or 2-D robots.

Currently, few works have considered adaptive servo-visual control for a 3-D robot. In [11], an adaptive dynamic controller designed with backstepping techniques was presented that considered uncertainties in both the camera and robot parameters in a unified structure and allowed the tracking of objects in an asymptotic way. However, this was achieved due to the use of two fixed cameras mounted on perpendicular planes. In [12], the author proposed an adaptive kinematic controller based on linearization techniques using a fixed camera. He considered two decoupled systems for the controller design, one for depth control and one for 2-D displacement control.

In [13], an adaptive visual servo controller for trajectory tracking was presented using a calibrated Kinect camera, which acted as a dense stereo sensor, and a controller based only on the inverse kinematic model of the manipulator with a position-based control law. The authors did not provide a stability analysis of the proposed system, and the performance of the controller was verified through experimental results considering only the Root Mean Squared Error (RMSE) of the position of the robot end effector as representative of the controller’s precision.

In [14], an adaptive control approach was presented considering the unknown robot’s kinematics and dynamics. The system used a calibrated camera to identify and calculate the Cartesian position of the robot’s operating end, on which an unknown tool was mounted, in order to estimate the 3-D dimensional information of the tool through the kinematic observer. The adaptive controller was of the free–model type combined with a kinematic observer. The stability of the closed-loop system was demonstrated by Lyapunov but was strongly conditioned to the convergence of the kinematic observer, which necessarily required persistently exciting trajectories to converge. The performance was shown through simulations.

In the work [15], an adaptive controller for trajectory tracking by a 3-degrees of freedom (DOF) manipulator robot with a fixed camera configuration was presented, considering both the parameters of the camera as well as the dynamics of the manipulator and the dynamics of its electric actuators as unknown. They proposed a control system with two control laws based on the backstepping technique and with speed measurement in the image plane. The first control law set the armature current of the motors as an auxiliary control variable, and another control law was given to generate the voltage references to the motors as a final control action.

For the adaptation of the different parameters, eight adaptive laws were required, and they demonstrated the stability of the proposed closed–loop system proposed by Lyapunov’s theory, assuming that the estimated parameters did not cause a singularity in the estimation of the independent-depth image Jacobian matrix. The simulation results only showed the convergence to zero of the position and speed control errors in the image for a single circular path in the image plane, and they did not show the convergence of the estimated parameters or the auxiliary control variable and the articular positions of the manipulator during the task.

A similar approach was presented in [16]; however, they avoided measuring the velocity in the image plane by incorporating a kinematic observer in cascade with the adaptive controller. They showed that the image space tracking errors converged to zero using a depth-dependent quasi-Lyapunov function plus the Lyapunov-like standard function and the asymptotic convergence of the observation errors in the image space. However their simulation results only showed the convergence to zero of the position control errors for a single circular path in the image plane. Similar to the previous work, they did not show the convergence of the estimated parameters, and, in addition, their results showed that the depth estimation error did not converge.

In [17], a research work prior to the current one was presented, in which only a planar robot with two degrees of freedom was considered and where the unknown depth of the target was constant. In the present work, we propose an adaptive control system consisting of two cascaded controllers to control a 3-D robot. The first is an adaptive image based kinematic visual servo controller in a fixed camera setup, the aim of which is that the robot follows a desired 3-D Cartesian trajectory even without knowing the depth and relative orientation between the end of the robot and the camera.

The second controller is an adaptive dynamic controller, with joint velocity reference inputs from the first controller, which compensates for the dynamics of the manipulator even with uncertainties in its dynamic parameters. The designed control system considers the real dynamics of a Scara 3-D industrial manipulator robot, and the ultimately boundedness of the control errors of the entire system is demonstrated using Lyapunov’s theory. The performance is shown through representative simulations.

2. Robot Model

The kinematic and dynamic model of a Scara-type robot manipulator [18] is presented as follows.

2.1. Kinematic Model

The kinematic model of a Scara Bosch SR800 [18] manipulator with 3 degrees of freedom (DOF), can be written as:

$${\mathbf{x}}^w\left(\mathbf{q}\right)={\mathbf{t}}_r^w=\left[\begin{array}{c}{x}_r^w\\ y_r^w\\ z_r^w\end{array}\right]=\left[\begin{array}{c}{l}_{2}cos(q_1+q_2)+l_{1}cos\left(q_1\right)\\ l_{2}sen(q_1+q_2)+l_{1}sen\left(q_1\right)\\ h+q_3\end{array}\right],$$

(1)

where ${\mathbf{t}}_r^w$ is the position of the end of the robot in the inertial frame $\langle w\rangle$; $l_1$ and $l_2$ are the lengths of the first two links; h is the maximum height of the operating end, and $\mathbf{q}={[q_1,q_2,q_3]}^T$ is the vector of joint positions, see Figure 1. The transformation between the robot frame $\langle r\rangle$ and the inertial frame $\langle w\rangle$ is given by the vector ${\mathbf{t}}_r^w$ and the rotation matrix ${\mathbf{R}}_r^w$ equal to the identity matrix due to a mechanical compensation of the robot’s orientation:

$${\mathbf{g}}_r^w=\left[\begin{array}{cc}{\mathbf{R}}_r^w& {\mathbf{t}}_r^w\\ \mathbf{0}& 1\end{array}\right]=\left[\begin{array}{cc}\mathbf{I}& {\mathbf{t}}_r^w\\ \mathbf{0}& 1\end{array}\right].$$

(2)

2.2. Dynamic Model

The 3-DOF dynamic model of the Bosch SR800 robot can be written as [18]

$$\mathbf{M}\left(\mathbf{q}\right)\ddot{\mathbf{q}}+\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}}={\dot{\mathbf{q}}}_{ref},$$

(3)

where $\mathbf{M}\left(\mathbf{q}\right)$ is the inertia matrix, $\mathbf{C}(\mathbf{q},\dot{\mathbf{q}})$ is the matrix of centripetal and Coriolis forces, $\dot{\mathbf{q}}$ and $\ddot{\mathbf{q}}$ are the vectors of joint velocities and accelerations, and ${\dot{\mathbf{q}}}_{ref}$ is the vector of reference speeds applied to the robot as a control action to its internal servos.

The model (3) can be rewritten in parameterized form as:

$${\dot{\mathbf{q}}}_{ref}=\mathsf{\Phi }(\mathbf{q},\dot{\mathbf{q}},\ddot{\mathbf{q}}){\mathbf{X}}_{inert}$$

(4)

where $\mathsf{\Phi }(\mathbf{q},\dot{\mathbf{q}},\ddot{\mathbf{q}})$ is the regression matrix whose elements are functions of $\mathbf{q}$, $\dot{\mathbf{q}}$, and $\ddot{\mathbf{q}}$; and ${\mathbf{X}}_{inert}$ is the vector of dynamic and actuator’s parameters of the Bosch SR800 robot, whose identified value is ${\mathbf{X}}_{inert}={[1.73,0.19,0.09,0.10,0.92,0.009]}^T$.

3. Simulated Experimental Platform and Vision System

To validate the proposed control system, we used the model of an experimental platform, which is a real SCARA Bosch SR-800 industrial robot of 4 DOF, available in our workplace and which represents a common robotic structure referred to in many works [19].

The camera used in the experiments was a Drangonfly Express IEEE1394b camera. This camera is capable of acquiring and transmitting color images through the bus IEEE-1394b at a speed of 200 fps (frames per seconds) and with a resolution of 640 × 480 px (pixels). The camera should be mounted at a certain height over the manipulator robot in such a way that it allows for capturing the entire workspace. The images captured by the camera were processed in real time using functions from the OpenCV library to extract two characteristic points fixed to the end effector of the robot, which were used as image features in the visual servoing algorithm as explained below.

The modeling of the vision system provides the equations that relate the position of the end of the robot $({\mathbf{x}}^w={\mathbf{t}}_r^w)$ and the fixed distance $d_x^w$ to a second point, displaced on the x axis of the robot frame $\langle r\rangle$, in 3-D space with its corresponding projection on the image plane. Figure 1 shows the frames associated to the manipulator $\langle r\rangle$, the camera $\langle c\rangle$, and the 3-D space $\langle w\rangle$ and $\langle w_2\rangle$, to which the poses of the robot and the camera, respectively, are defined. The transformations between the inertial frames $\langle w\rangle$ and $\langle w_2\rangle$, and between the frames $\langle c\rangle$ and $\langle w_2\rangle$, are given by:

$${\mathbf{g}}_{w_2}^w=\left[\begin{array}{cc}\mathbf{I}& {\mathbf{t}}_{w_2}^w\\ \mathbf{0}& 1\end{array}\right]{\mathbf{g}}_c^{w_2}=\left[\begin{array}{cc}{\mathbf{R}}_c^{w_2}& \mathbf{0}\\ \mathbf{0}& 1\end{array}\right],$$

(5)

where ${\mathbf{t}}_{w_2}^w={[x_{w_2}^w,y_{w_2}^w,z_{w_2}^w]}^T$ is the position vector (generally it is unknown) from $\langle w_2\rangle$ expressed in $\langle w\rangle$ and ${\mathbf{R}}_c^{w_2}={\mathbf{R}}_z\left(\theta \right){\mathbf{R}}_{ideal}={\left({\mathbf{R}}_c^{w_2}\right)}^{-1}={\mathbf{R}}_{w_2}^c$ with:

$${\mathbf{R}}_z\left(\theta \right)=\left[\begin{array}{ccc}cos\left(\theta \right)& -sen\left(\theta \right)& 0\\ sen\left(\theta \right)& cos\left(\theta \right)& 0\\ 0& 0& 1\end{array}\right],{\mathbf{R}}_{ideal}=\left[\begin{array}{ccc}0& -1& 0\\ -1& 0& 0\\ 0& 0& -1\end{array}\right].$$

(6)

The 3-D point that represents the position of the robot is mapped to the point ${\mathbf{x}}_1^I$ of the camera image plane using:

$${\lambda }_z{\overline{\mathbf{x}}}_1^I=\underset{\mathsf{\Pi }}{\underbrace{\mathbf{K}{\mathsf{\Pi }}_o{\mathbf{g}}_{w_2}^c}}\underset{{\overline{\mathbf{x}}}^{w_2}}{\underbrace{{\mathbf{g}}_w^{w_2}{\overline{\mathbf{x}}}^w}},$$

(7)

where ${\overline{\mathbf{x}}}_1^I$ is the point ${\mathbf{x}}_1^I$ expressed in homogeneous coordinates, $\mathbf{K}$ and ${\mathsf{\Pi }}_0$ are the matrices of intrinsic parameters of the camera and of perspective projection, ${\mathbf{x}}^{w_2}={\mathbf{t}}_r^w-{\mathbf{t}}_{w_2}^w$ is the robot position expressed in $\langle w_2\rangle$, and ${\lambda }_z$ is an unknown scale factor. Operating algebraically, the expression (7) can be rewritten in non-homogeneous coordinates as:

$${\mathbf{x}}_1^I={\alpha }^{\prime }{\mathbf{R}}_z\left(\theta \right){\mathbf{x}}^{w_2^{\prime }}+{\mathbf{o}}^I,$$

(8)

where ${\mathbf{x}}^{w_2^{\prime }}={\mathbf{R}}_{ideal}({\mathbf{t}}_r^w-{\mathbf{t}}_{w_2}^w)$ and ${\alpha }^{\prime }=\frac{\alpha }{z_{w_2}^w-(h+q_3)}$ is the unknown depth of the operating end in the image frame $\langle I\rangle$, $\alpha$ is the scale factor, and ${\mathbf{o}}^I$ is the coordinate vector of the center of the image (parameters of $\mathbf{K}$). Similarly, the second point ${\mathbf{x}}^w+\mathbf{d}$ on the end of the robot with $\mathbf{d}={[d_x^w,0,0,1]}^T$ is mapped to the point ${\mathbf{x}}_2^I$:

$${\lambda }_z{\overline{\mathbf{x}}}_2^I={\lambda }_z{\overline{\mathbf{x}}}_1^I+\mathbf{K}{\mathbf{R}}_{w_2}^c\mathbf{d}.$$

(9)

The distance between the two points in the image plane is given by $d^I={\alpha }^{\prime }{d}_x^w$. Then, the vector of image features is defined as:

$${\mathbf{x}}^I=\left[\begin{array}{c}{x}_1\\ y_1\\ d^I\end{array}\right]=\left[\begin{array}{c}{\mathbf{x}}_1^I\\ d^I\end{array}\right]=\left[\begin{array}{c}{\alpha }^{\prime }{\mathbf{R}}_z\left(\theta \right){\mathbf{x}}^{w_2^{\prime }}+{\mathbf{o}}^I\\ {\alpha }^{\prime }{d}_x^w\end{array}\right],$$

(10)

and its time derivative is given by:

$${\dot{\mathbf{x}}}^I=\alpha {\mathbf{R}}_z\left(\theta \right){\mathbf{R}}_{ideal}\mathbf{J}(\mathbf{q},{\mathbf{t}}_{w_2}^w,d_x^w)\dot{\mathbf{q}}=\alpha {\mathbf{R}}_z\left(\theta \right)\mathbf{u}=\mathbf{D}\mathbf{u},$$

(11)

where $\mathbf{u}={\mathbf{R}}_{ideal}\mathbf{J}(\mathbf{q},{\mathbf{t}}_{w_2}^w,d_x^w)\dot{\mathbf{q}}$, $\mathbf{J}(\mathbf{q},{\mathbf{t}}_{w_2}^w,d_x^w)$ is the Jacobian of the robot, and $\mathbf{D}=\alpha {\mathbf{R}}_z\left(\theta \right)$ represents the generally unknown parameters of the vision system.

4. Adaptive Kinematic Servovisual Controller

The inverse model of the expression (11) is given by:

$$\mathbf{u}={\mathbf{D}}^{-1}{\dot{\mathbf{x}}}^I=\left[\begin{array}{ccc}{p}_1& p_2& 0\\ -p_2& p_1& 0\\ 0& 0& p_3\end{array}\right]\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{y}}_1\\ {\dot{d}}^I\end{array}\right],$$

(12)

where $p_1=cos\left(\theta \right)/\alpha$, $p_2=sen\left(\theta \right)/\alpha$ and $p_3=1/\alpha$.

${\mathbf{x}}^T{\mathbf{D}}^{-1}\mathbf{x}>0$ for all $\mathbf{x}\ne \mathbf{0}$ if $\left|\theta \right|<\frac{\pi }{2}$. Equation (12) can be expressed in the following ways:

$$\mathbf{u}=\left[\begin{array}{ccc}{p}_1& 0& 0\\ 0& p_1& 0\\ 0& 0& p_3\end{array}\right]\left[\begin{array}{c}{\dot{x}}_1\\ {\dot{y}}_1\\ {\dot{d}}^I\end{array}\right]+\left[\begin{array}{c}{p}_2{\dot{y}}_1\\ -p_2{\dot{x}}_1\\ 0\end{array}\right]=\mathbf{P}{\dot{\mathbf{x}}}^I+\mathbf{\eta },$$

(13)

$$\mathbf{u}=\left[\begin{array}{ccc}{\dot{x}}_1& {\dot{y}}_1& 0\\ {\dot{y}}_1& -{\dot{x}}_1& 0\\ 0& 0& {\dot{d}}^I\end{array}\right]\left[\begin{array}{c}{p}_1\\ p_2\\ p_3\end{array}\right]=\mathbf{\varphi }\mathbf{p}.$$

(14)

Defining the control errors as ${\tilde{\mathbf{x}}}^I={\mathbf{x}}^I-{\mathbf{x}}_d^I$, where ${\mathbf{x}}_d^I$ is the desired visual characteristic in the image plane. Two different control laws can be proposed as described below.

4.1. Control Law with Measurement of ${\dot{\mathbf{x}}}^I$

The following adaptive control law is proposed with measurement of the speed of the image characteristics:

$${\mathbf{u}}_c=\widehat{\mathbf{P}}\left[\begin{array}{c}{\mathbf{\rho }}_1\\ {\mathbf{\rho }}_2\\ {\mathbf{\rho }}_3\end{array}\right]+\left[\begin{array}{c}{\widehat{p}}_2{\dot{y}}_1\\ -{\widehat{p}}_2{\dot{x}}_1\\ 0\end{array}\right]=\left[\begin{array}{ccc}{\mathbf{\rho }}_1& {\dot{y}}_1& 0\\ {\mathbf{\rho }}_2& -{\dot{x}}_1& 0\\ 0& 0& {\mathbf{\rho }}_3\end{array}\right]\left[\begin{array}{c}{\widehat{p}}_1\\ {\widehat{p}}_2\\ {\widehat{p}}_3\end{array}\right]={\mathbf{\varphi }}_c\widehat{\mathbf{p}},$$

(15)

where ${\mathbf{u}}_c={\mathbf{R}}_{ideal}\mathbf{J}(\mathbf{q},{\mathbf{t}}_{w_2}^w,d_x^w){\dot{\mathbf{q}}}_{ref}$ represents the robot’s control action in Cartesian coordinates and ${\dot{\mathbf{q}}}_{ref}$ the joint velocity command sent to the robot. The vector $\widehat{\mathbf{p}}$ represents the estimated parameters of the vision system for which the vector $\tilde{\mathbf{p}}=\widehat{\mathbf{p}}-\mathbf{p}$ was considered as the parameter error; the matrix ${\mathbf{\varphi }}_c$ is composed with the elements of the vector ${\dot{\mathbf{x}}}^I$ and the vector $\mathbf{\rho }$ whose expression is:

$$\mathbf{\rho }=\left[\begin{array}{c}{\mathbf{\rho }}_1\\ {\mathbf{\rho }}_2\\ {\mathbf{\rho }}_3\end{array}\right]={\dot{\mathbf{x}}}_d^I-\lambda {\tilde{\mathbf{x}}}^I.$$

(16)

where $\lambda >0$ represents the gain of the controller.

Controller Analysis

Assuming perfect velocity tracking ($\mathbf{u}={\mathbf{u}}_c$), that is, $\dot{\mathbf{q}}={\dot{\mathbf{q}}}_{ref}$, from the expressions (13) and (15), the closed-loop equation of the system is, thus, obtained:

$$\mathbf{P}{\dot{\mathbf{x}}}^I+\mathbf{\eta }={\mathbf{\varphi }}_c\widehat{\mathbf{p}}={\mathbf{\varphi }}_c(\mathbf{p}+\tilde{\mathbf{p}})=\mathbf{P}\mathbf{\rho }+\mathbf{\eta }+{\mathbf{\varphi }}_c\tilde{\mathbf{p}}$$

$$\mathbf{P}({\dot{\tilde{\mathbf{x}}}}^I+\mathbf{\lambda }{\tilde{\mathbf{x}}}^I)={\mathbf{\varphi }}_c\tilde{\mathbf{p}}.$$

(17)

Then, the following Lyapunov candidate function is proposed:

$$V=\frac{1}{2}{\tilde{\mathbf{x}}}^{I^T}\mathbf{P}{\tilde{\mathbf{x}}}^I+\frac{1}{2}{\tilde{\mathbf{p}}}^T\mathbf{\gamma }\tilde{\mathbf{p}},$$

(18)

whose time derivative is:

$$\dot{V}=-\lambda {\tilde{\mathbf{x}}}^{I^T}\mathbf{P}{\tilde{\mathbf{x}}}^I+{\tilde{\mathbf{x}}}^{I^T}{\mathbf{\varphi }}_c\tilde{\mathbf{p}}+{\tilde{\mathbf{p}}}^T\mathbf{\gamma }\dot{\tilde{\mathbf{p}}}.$$

(19)

This defines the adapting law:

$$\dot{\tilde{\mathbf{p}}}=-{\mathbf{\gamma }}^{-1}{\mathbf{\varphi }}_c^T{\tilde{\mathbf{x}}}^I,$$

(20)

where $\mathbf{\gamma }>\mathbf{0}$ is an adaptation gain matrix. Replacing (20) in the expression (19), we obtain:

$$\dot{V}=-\lambda {\tilde{\mathbf{x}}}^{I^T}\mathbf{P}{\tilde{\mathbf{x}}}^I\le 0.$$

(21)

Therefore, ${\tilde{\mathbf{x}}}^I\in L_{\infty }$ and $\tilde{\mathbf{p}}\in L_{\infty }$. Integrating Equation (21), it can be proven that ${\tilde{\mathbf{x}}}^I\in L_2$. From the expression (17), it is proven that ${\dot{\tilde{\mathbf{x}}}}^I\in L_{\infty }$. Then, by Barbalat’s lemma, we conclude that ${\tilde{\mathbf{x}}}^I\left(t\right)\to \mathbf{0}$ with $t\to \infty$, thus, achieving the control objective.

4.2. Control Law without Measurement of ${\dot{\mathbf{x}}}^I$

The following proposed adaptive control law does not require measurement of the speed of image features:

$${\mathbf{u}}_c=\left[\begin{array}{ccc}{\rho }_1& {\rho }_2& 0\\ {\rho }_2& -{\rho }_1& 0\\ 0& 0& {\rho }_3\end{array}\right]\left[\begin{array}{c}{\widehat{p}}_1\\ {\widehat{p}}_2\\ {\widehat{p}}_3\end{array}\right]={\mathbf{\varphi }}_c\widehat{\mathbf{p}},$$

(22)

where ${\mathbf{\varphi }}_c$ is composed with the elements of the vector $\mathbf{\rho }$ given by the expression (16).

4.2.1. Controller Analysis

Assuming perfect velocity tracking ($\mathbf{u}={\mathbf{u}}_c$), from the expressions (12) and (22), the closed-loop equation of the system is obtained:

$${\mathbf{D}}^{-1}{\dot{\mathbf{x}}}^I={\mathbf{\varphi }}_c\widehat{\mathbf{p}}={\mathbf{\varphi }}_c(\mathbf{p}+\tilde{\mathbf{p}})={\mathbf{D}}^{-1}\mathbf{\rho }+{\mathbf{\varphi }}_c\tilde{\mathbf{p}}$$

$${\mathbf{D}}^{-1}({\dot{\tilde{\mathbf{x}}}}^I+\lambda {\tilde{\mathbf{x}}}^I)={\mathbf{\varphi }}_c\tilde{\mathbf{p}}.$$

(23)

Then, the following Lyapunov candidate function is proposed:

$$V={\tilde{\mathbf{x}}}^{I^T}{\mathbf{D}}^{-1}{\tilde{\mathbf{x}}}^I+\frac{1}{2}{\tilde{\mathbf{p}}}^T\mathbf{\gamma }\tilde{\mathbf{p}},$$

(24)

whose time derivative is:

$$\begin{array}{cc}\hfill \dot{V}& ={\tilde{\mathbf{x}}}^{I^T}{\mathbf{D}}^{-1}{\dot{\tilde{\mathbf{x}}}}^I+{\dot{\tilde{\mathbf{x}}}}^{I^T}{\mathbf{D}}^{-1}{\tilde{\mathbf{x}}}^I+{\tilde{\mathbf{p}}}^T\mathbf{\gamma }\dot{\tilde{\mathbf{p}}}\hfill \\ \hfill & =-2\lambda {\tilde{\mathbf{x}}}^{I^T}{\mathbf{D}}^{-1}{\tilde{\mathbf{x}}}^I+{\alpha }^2{\tilde{\mathbf{x}}}^{I^T}{\mathbf{D}}^{-2^T}{\mathbf{\varphi }}_c\tilde{\mathbf{p}}\hfill \end{array},$$

(25)

where the expression (20) is used as the adapting law. From (25), outside the ball

$${\parallel {\tilde{\mathbf{x}}}^I\parallel }_{\infty }\le \frac{2a\left|{\dot{\mathbf{x}}}_{d_{max}}^I\right|}{\lambda }\frac{\parallel \tilde{\mathbf{p}}\parallel }{1-2a\parallel \tilde{\mathbf{p}}\parallel }=\mathbf{\eta },$$

(26)

we verified that $\dot{V}<0$. Then, the control error is finally bounded by the value of $\mathbf{\eta }$.

4.2.2. Remarks

• For positioning $\left|{\dot{\mathbf{x}}}_{d_{max}}^I\right|=0$; therefore, ${\tilde{\mathbf{x}}}^I\left(t\right)\to \mathbf{0}$ with $t\to \infty$
• For trajectories that are persistently exciting, it can be shown that $\parallel \tilde{\mathbf{p}}\left(t\right)\parallel \to 0$ with $t\to \infty$, and therefore ${\tilde{\mathbf{x}}}^I\left(t\right)\to \mathbf{0}$ with $t\to \infty$.

Then, under these conditions, we proved the same as for the controller with measurement of ${\dot{\mathbf{x}}}^I$, that ${\tilde{\mathbf{x}}}^I\left(t\right)\to \mathbf{0}$ with $t\to \infty$, thus, achieving the control goal.

5. Dynamic Compensation Design

This section ignores the perfect velocity tracking assumption considering a velocity tracking error ($\mathbf{u}={\mathbf{u}}_c+\tilde{\mathbf{u}}$) due to the dynamics of the robot. Under this condition, the closed-loop Equation (23) now results in:

$${\mathbf{D}}^{-1}({\dot{\tilde{\mathbf{x}}}}^I+\lambda {\tilde{\mathbf{x}}}^I)={\mathbf{\varphi }}_c\tilde{\mathbf{p}}+\tilde{\mathbf{u}},$$

(27)

and the time derivative of (24) is:

$$\dot{V}=-\lambda {\tilde{\mathbf{x}}}^{I^T}{\mathbf{D}}^{-1}{\tilde{\mathbf{x}}}^I+(1+\frac{1}{\alpha }){\tilde{\mathbf{x}}}^{I^T}\tilde{\mathbf{u}}+\frac{1}{\alpha }{\tilde{\mathbf{x}}}^{I^T}{\mathbf{D}}^2{\mathbf{\varphi }}_c\tilde{\mathbf{p}},$$

(28)

From (28), outside the ball,

$${\parallel {\tilde{\mathbf{x}}}^I\parallel }_{\infty }\le \frac{1}{\lambda }\frac{\left|{\dot{\mathbf{x}}}_{d_{max}}^I\right|\parallel \tilde{\mathbf{p}}\parallel +1.5\parallel \tilde{\mathbf{u}}\parallel }{p_1-\parallel \tilde{\mathbf{p}}\parallel }=\mathbf{\eta },$$

(29)

it is verified that $\dot{V}<0$. Then, the control error is finally bounded by the value of $\eta$. Note that $\tilde{\mathbf{u}}\left(t\right)$ does not necessarily converge to zero, since, by including the robot dynamics, the convergence of $\tilde{\mathbf{p}}\left(t\right)\to 0$ is not always achieved as an attempt is made to identify a different structure for which the kinematic controller was designed. As a consequence, the control error increases.

To solve the degradation of the kinematic control, a cascaded adaptive dynamic controller is proposed that makes the robot reach the reference speed provided by the kinematic controller, again restoring the good performance of the control system, see Figure 2. Defining the speed control error as $\dot{\tilde{\mathbf{q}}}=\dot{\mathbf{q}}-{\dot{\mathbf{q}}}_d$, the following control law is proposed:

$${\dot{\mathbf{q}}}_{ref}=\widehat{\mathbf{M}}\mathbf{\nu }+\widehat{\mathbf{C}}{\dot{\mathbf{q}}}_d={\mathbf{\varphi }}_d{\widehat{\mathbf{X}}}_{inerc},$$

(30)

where $\mathbf{\nu }={\ddot{\mathbf{q}}}_d-\mathbf{K}\dot{\tilde{\mathbf{q}}}$, $\mathbf{K}$ is a positive definite gain matrix, ${\widehat{\mathbf{X}}}_{inerc}$ represents the estimated robot parameters, ${\tilde{\mathbf{X}}}_{inerc}={\widehat{\mathbf{X}}}_{inerc}-{\mathbf{X}}_{inerc}$ is the parameter error vector, and $\widehat{\mathbf{M}}$ and $\widehat{\mathbf{C}}$ are the matrices of inertia and Coriolis torques calculated with the estimated parameters. Replacing ${\dot{\mathbf{q}}}_{ref}$ in the dynamic model (3), we obtain the closed-loop equation of the system:

$$\mathbf{M}\ddot{\mathbf{q}}+\mathbf{C}\dot{\mathbf{q}}={\mathbf{\varphi }}_d{\widehat{\mathbf{X}}}_{inerc}={\mathbf{\varphi }}_d{\mathbf{X}}_{inerc}+{\mathbf{\varphi }}_d{\tilde{\mathbf{X}}}_{inerc}$$

$$\mathbf{M}(\ddot{\tilde{\mathbf{q}}}+\mathbf{K}\dot{\tilde{\mathbf{q}}})+\mathbf{C}\dot{\tilde{\mathbf{q}}}={\mathbf{\varphi }}_d{\tilde{\mathbf{X}}}_{inerc}.$$

(31)

We consider the following positive definite function:

$$V=\frac{1}{2}{\dot{\tilde{\mathbf{q}}}}^T\mathbf{M}\dot{\tilde{\mathbf{q}}}+\frac{1}{2}{\tilde{\mathbf{X}}}_{inerc}^T{\mathbf{\gamma }}_{dyn}{\tilde{\mathbf{X}}}_{inerc},$$

(32)

and its time derivative in the trajectories of the system:

$$\begin{array}{cc}\hfill \dot{V}& ={\dot{\tilde{\mathbf{q}}}}^T\mathbf{M}\ddot{\tilde{\mathbf{q}}}+\frac{1}{2}{\dot{\tilde{\mathbf{q}}}}^T\dot{\mathbf{M}}\dot{\tilde{\mathbf{q}}}+{\tilde{\mathbf{X}}}_{inerc}^T{\mathbf{\gamma }}_{dyn}{\dot{\tilde{\mathbf{X}}}}_{inerc}\hfill \\ \hfill & =-\mathbf{K}{\dot{\tilde{\mathbf{q}}}}^T\mathbf{M}\dot{\tilde{\mathbf{q}}}+{\dot{\tilde{\mathbf{q}}}}^T(\frac{\dot{\mathbf{M}}}{2}-\mathbf{C})\dot{\tilde{\mathbf{q}}}+{\tilde{\mathbf{X}}}_{inerc}^T({\mathbf{\varphi }}_d\dot{\tilde{\mathbf{q}}}+{\mathbf{\gamma }}_{dyn}{\dot{\tilde{\mathbf{X}}}}_{inerc})\hfill \end{array},$$

(33)

where the term $(\frac{\dot{\mathbf{M}}}{2}-\mathbf{C})$ is zero, since $\mathbf{C}$ is the antisymmetric matrix calculated with the Christoffel terms. Defining, as adaptation law,

$${\dot{\tilde{\mathbf{X}}}}_{inerc}=-{\mathbf{\gamma }}_{dyn}^{-1}{\mathbf{\varphi }}_d^T\dot{\tilde{\mathbf{q}}},$$

(34)

and replacing it in expression (33), we obtain:

$$\dot{V}=-\mathbf{K}{\dot{\tilde{\mathbf{q}}}}^T\mathbf{M}\dot{\tilde{\mathbf{q}}}\le 0,$$

(35)

and therefore $\dot{\tilde{\mathbf{q}}}$ and ${\tilde{\mathbf{X}}}_{inerc}$$\in L_{\infty }$. Furthermore, by integrating $\dot{V}$ over $[0,T]$, it can be shown that $\dot{\tilde{\mathbf{q}}}\in L_2$. From expression (31), it is proven that $\ddot{\tilde{\mathbf{q}}}\in L_{\infty }$. Then, by Barbalat’s lemma, we conclude that $\dot{\tilde{\mathbf{q}}}\left(t\right)\to \mathbf{0}$ with $t\to \infty$, thus, achieving the control objective.

As proven above, the result $\dot{\tilde{\mathbf{q}}}\left(t\right)\to \mathbf{0}$ with $t\to \infty$ implies that:

$$\tilde{\mathbf{u}}\left(t\right)={\mathbf{R}}_{ideal}\mathbf{J}(\mathbf{q},{\mathbf{t}}_{w_2}^w,d_x^w)\dot{\tilde{\mathbf{q}}}\left(t\right)\to \mathbf{0}\mathrm{with}t\to \infty$$

(36)

Then, going back to Equation (29) and introducing the convergence condition on $\tilde{\mathbf{u}}$ (36), the error bound conditions of Equation (26) and, therefore, the stability conditions previously obtained for the kinematic controller are asymptotically recovered even in the presence of unknown robot dynamics.

6. Simulations

In this section, we show simulation experiments that can be considered realistic and whose results are very close to those that would be obtained with the real robot. This is because the model used in these simulations is an identified model of the real robot, which represents the dynamics of both the rigid body and that of the electric motors and reduction gears in its actuators. A complete study of this model, the reduction of the total set of dynamic parameters to a minimum base parameters set that considers only the dominant and identifiable dynamics, and its subsequent expansion to model and identify the dynamics of its actuators is found in [18].

To verify the performance of the proposed control system, realistic simulations were carried out for a positioning task and for a trajectory following task using the identified kinematic and dynamic model corresponding to the Bosch SR800 SCARA industrial robot. The parameters of the vision system were $\theta =10$ and $\alpha =820$ px/mm, and errors of $20\%$ and $10\%$ were considered, respectively.

Figure 3 shows the evolution of the image characteristics for a positioning task, starting from rest at the position ${\mathbf{x}}^w\left(0\right)={[0.63,0.22,0.35]}^T\left[m\right]$ and reaching the desired position ${\mathbf{x}}_d^w={[0.31,-0.61,0.45]}^T\left[m\right]$. The kinematic controller was applied first to the kinematic model of the robot, and then its dynamics were incorporated. The gains set to the values shown in

Table 1. The gains for the positioning task.

Kinematic Model	Dynamic Model
$\mathbf{\lambda }=\left[\begin{array}{ccc}10& 0& 0\\ 0& 10& 0\\ 0& 0& 10\end{array}\right]$	$\mathbf{\lambda }=\left[\begin{array}{ccc}15& 0& 0\\ 0& 15& 0\\ 0& 0& 0.0004\end{array}\right]$
$\mathbf{\gamma }={10}^5\left[\begin{array}{ccc}800& 0& 0\\ 0& 0.02& 0\\ 0& 0& 0.08\end{array}\right]$	$\mathbf{\gamma }=2\left[\begin{array}{ccc}{10}^{10}& 0& 0\\ 0& {10}^3& 0\\ 0& 0& 1\end{array}\right]$

. Figure 4 and Figure 5 show the norm of the control error and the convergence of the vision system parameters; observing that, in both cases, the control error converged close to zero as indicated in the remarks of Section 4.

On the other hand, Figure 6 shows the image feature vector ${\mathbf{x}}^I$ for the following task, starting from the position ${\mathbf{x}}^w\left(0\right)={[0.523,0.278,0.423]}^T\left[m\right]$ and following a circular spiral reference ${\mathbf{x}}_d^w={[0.15cos24tcost+0.540,0.15sin24tcost-0.046,0.1sint+0.423]}^T\left[m\right]$. The servo-visual controller was applied to a kinematically modeled robot, then the robot dynamics were incorporated, and later this dynamic was compensated with the adaptive controller, considering a $50\%$ error in the robot parameters. The gains used in these three cases are shown in

Table 2. The gains for the following task.

Kinematic Model	Dynamic Model	Compensation Dynamic Model
$\mathbf{\lambda }=\left[\begin{array}{ccc}200& 0& 0\\ 0& 400& 0\\ 0& 0& 20\end{array}\right]$	$\mathbf{\lambda }=\left[\begin{array}{ccc}100& 0& 0\\ 0& 300& 0\\ 0& 0& 10\end{array}\right]$	$\mathbf{\lambda }=\left[\begin{array}{ccc}60& 0& 0\\ 0& 610& 0\\ 0& 0& 20\end{array}\right]$
$\mathbf{\gamma }={10}^5\left[\begin{array}{ccc}200& 0& 0\\ 0& 200& 0\\ 0& 0& 0.008\end{array}\right]$	$\mathbf{\gamma }={10}^9\left[\begin{array}{ccc}100& 0& 0\\ 0& 10& 0\\ 0& 0& 0.002\end{array}\right]$	$\mathbf{\gamma }={10}^5\left[\begin{array}{ccc}400& 0& 0\\ 0& 400& 0\\ 0& 0& 2\end{array}\right]$

. Figure 7 and Figure 8 show the norm of the ${\tilde{\mathbf{x}}}^I$ vector and the convergence of the vision system parameters.

Figure 9 and Figure 10 show the norm of the speed control error of the adaptive dynamic controller and the convergence of the dynamic parameters of the robot, respectively.

7. Discussion

In Section 4, the design of two new adaptive servo-visual kinematic controllers for trajectory tracking with a 3-DOF manipulator robot was presented. The main contribution of these new control schemes is their simple design, their image-based control law, and their unique adaptation law, unlike previous works with complex controllers as in [15,16]. From a practical point of view, these do not require the measurement of speed in the image plane.

Finally, these schemes represent a generalization of previous work [17] to the case of 3-D movement by the manipulator. This makes it possible to consider not only the intrinsic and extrinsic parameters of the vision system as unknown but also the depth of the objective, which can be time-variant and, as shown in Section 3, can be estimated with an appropriate selection of the image characteristics.

In the simulations, the tracking task and the target points in the positioning task were chosen to show, in both cases, the performance of the controller without speed measurement in the case where the depth of the target is time-variant. It was also demonstrated by Lyapunov’s theory that the scheme that required a speed measurement of the image characteristics achieved an asymptotic convergence of the control errors in both the positioning and tracking tasks.

On the other hand, the scheme that did not require such measurement achieved an asymptotic convergence only in positioning tasks as shown by the simulation results of Figure 4 and in following tasks in which the trajectories to follow sufficiently excited the dynamics of the system with the spiral trajectory whose result is shown in Figure 7. However, in this last scheme, the stability analysis showed that, even in trajectories that were not persistently exciting, the control errors always remained bounded. In a previous work [17], it was shown that, for the case of 2-D motion where the depth of the target was constant, the controllers always reached the control target even on non-exciting trajectories, such as a ramp type.

Figure 5 shows that the control actions generated by the kinematic controller, regardless of whether they applied to an idealized robot modeled only with its kinematics or to a real robot modeled with its exact dynamic model, the estimated parameters always converged on the positioning tasks, although not to the true values. This shows that, for these tasks, the performance of the kinematic controller is sufficient, and dynamics compensation is not required.

However, in high-speed tracking tasks that excite the manipulator dynamics, such as those in Figure 6, it can be seen in Figure 7 how the control error in the image plane converged asymptotically to zero when the control actions generated by the kinematic controller were applied to an idealized robot modeled only with its kinematics, and the estimated parameters converged to their true values as shown in Figure 8. However, when these actions were applied to a real robot modeled with its exact dynamic model, the performance was very poor since it was attempting to control a system with a different structure than that for which the controller was designed, generating undesirable high frequency movements, like those shown in Figure 8 and Figure 9, and even the estimated parameters may not converge as can be seen in Figure 8.

Figure 7 and Figure 9 show that the performance of the kinematic controller is practically recovered when the manipulator dynamics were compensated, limiting the control errors as indicated by the stability test of the dynamic compensator in Section 5, even with unknown manipulator parameters. Figure 10 shows that most of the parameters converged to their true value and others remained bounded very close to them. Furthermore, Figure 8 shows how the convergence of the vision system parameters was also recovered, although not to the true values as in the ideal kinematic situation.

8. Conclusions

An adaptive 3-D kinematic servo visual controller for positioning and trajectory tracking was designed for a Scara robot manipulator and its stability based on Lyapunov’s theory was proven. Simulation experiments showed that, for positioning, the control objective was always reached regardless of the manipulator dynamics. On the other hand, for following tasks with generally unknown robot dynamics, we observed that the kinematic control kept errors limited; however, its performance degraded with its aplication to an ideal robot without dynamics.

However, the cascaded adaptive dynamic controller efficiently compensated for the unknown dynamics of the manipulator, and the final performance approximated that of the kinematic control even though the estimated robot parameters did not converge to their true values as shown by the simulation results and the stability proof based in Lyapunov. Work is currently underway to fine-tune an experimental system in order to take this research to the experimentation phase.