Uncalibrated Adaptive Visual Servoing of Robotic Manipulators with Uncertainties in Kinematics and Dynamics

Abstract

In the study, we propose a novel adaptive visual servoing control scheme for robotic manipulators with kinematic and dynamic uncertainties, where the camera used is uncalibrated, which implies that its intrinsic and extrinsic parameters are unavailable for measurement. For our scheme, a depth-independent composite Jacobian matrix is constructed to make visual parameters and robotic physical parameters appear linearly in a parametrized uniform form so that an adaptive algorithm can be developed to estimate their values. With the raised adaptive algorithm, the potential singularity of the Jacobian matrix can be well circumvented by updating estimated parameters in an appropriate tiny range of actual values. With our scheme, the asymptotic convergence of the image tracking error to zero is established successfully, in addition to the signal boundedness of the closed-loop system. The effectiveness of the proposed scheme is confirmed by simulation results based on a 6-DOF PUMA manipulator.

1. Introduction

The control scheme of robotic manipulators is always a hot topic in research for its wide application in various industries [1,2,3,4,5,6,7,8]. With the development of information technology, visual sensors are widely used in the control system of the robotic manipulator to deal with complex and volatile tasks. In [1], the traditional visual feedback control approach is proposed for an assembling robotic manipulator. Specifically, the traditional visual feedback system adopts a look-and-move approach, which separates the image process from the controller of a manipulator to make the control structure hierarchical [5]. The visual servo originated from [4], which is a vision-based controller constructed to drive the motor of a manipulator directly. Lots of visual servoing control approach for a robotic manipulator have been proposed, such as [6,9,10,11,12,13], in recent decades.

To categorize diverse visual servoing architectures, many taxonomies (the number of cameras, the configuration of vision systems, and the space where error functions are defined) are proposed [6]. Generally, the visual servoing control can be categorized into subsequent two approaches according to the definition of its error function. One is the Position-Based Visual Servoing (PBVS) control scheme, whose error function is defined in a task space by the difference between the current position (estimated by visual information) and the desired position of a manipulator [11]. The control accuracy of the PBVS control scheme is subject to the precise estimation of a robotic position, which is dependent on the calibration of the parameters in the visual servoing system. The other is the Image-Based Visual Servoing (IBVS) control scheme which directly utilizes image information to construct the error function in image space [12]. Both of the PBVS and the IBVS traditional control methods need to calibrate the system, including a vision system and a robotic manipulator system. Additionally, the performance of the visual servoing system is directly influenced by the calibration accuracy. However, parametric calibration is a tedious and highly costly task. Even a precise calibrated system would change in some harsh environments. To address this problem, the techniques of uncalibrated visual servoing for a robotic manipulator have become a popular research topic in the field of visual servoing [9,10].

The key idea of the so-called uncalibrated visual servoing method is designing a controller of a robotic manipulator by the image errors for ensuring that the system error converges to an acceptable range without the pre-calibrated system parameters (the extrinsic/intrinsic parameters of the used camera and the robotic parameters). Since the IBVS methods are more robust for the error of the control model rather than the PBVS methods substantial efforts (concerning IBVS methods) in addressing the uncertainties due to unknown or unprecise parameters have been made [13,14,15,16,17]. Generally, the design idea of an uncalibrated IBVS method features the estimation of the composite Jacobian matrix (the image matrix [14]) utilized to design the control law. The composite Jacobian matrix relates the velocity of the end-effector of a robotic manipulator in joint space to the velocity of the image features in image space. An online composite Jacobian matrix estimator based on a dynamic Kalman filter algorithm is developed in [14]. In [15,16,17], the approaches based on recursive least squares are proposed to estimate the composite Jacobian matrix online. Furthermore, Ref. [13] presents a fuzzy learning approach for an IBVS system. The above approaches are all developed based on kinematics and feature the following points: (i) a numerical method is adopted to estimate the whole Jacobian matrix without the explicit expression of a Jacobian matrix; (ii) the uncertainties in the dynamics of a robotic manipulator are neglected. However, the performance and stability of visual servoing are affected by the non-linear forces of a robotic manipulator in high-speed motion. Therefore, a vast amount of research about an IBVS method considering the nonlinear force of the robotic dynamics has been invested [18,19,20,21,22,23,24,25,26,27].

Compared to the kinematics-based control scheme, the dynamics-based control scheme mainly adopts the analytical method to figure out the linear form of unknown parameters in the explicit expression of a Jacobian matrix and then adopts the adaptive algorithm to estimate them. In [19], an approximate Jacobian control method for a planar robotic manipulator with kinematic and dynamic uncertainties is proposed. With visual servoing control for a planar robotic manipulator, the depth of the feature point relative to the camera frame is constant. However, it is worth mentioning that the depth information of an image feature appearing in the explicit Jacobian matrix is unknown and unmeasurable, and thus dealing with the uncertainties caused by the unknown depth information is a challenging problem in the dynamics-based control scheme. The direct scheme to tackle the unknown depth information is utilizing a stereo camera [20,28], which aggravates the computation complexity. Additionally, an adaptive updating algorithm for the depth information of image features is proposed in [21]. However, it is available only under the slow time-varying constraint of the depth of information. Thus, a depth-independent interaction matrix is originally presented in [22] to address the constraint of the slow time-varying for the depth information, which enables the interaction matrix parametrized by the camera parameters. Subsequently, a dynamic trajectory tracking control for uncalibrated visual servoing is extended in [23]. However, the methods in [22,23] are limited by the prior knowledge of robotic kinematics, assuming that the robotic Jacobian matrix is available for measurement. With un-precise kinematic parameters, a depth-independent image Jacobian matrix is utilized to map the image errors onto the robotic joint space in [24], where unknown parameters are estimated by an adaptive algorithm online. The work in [25] generalizes the control method in [24] from a point feature to a generalized image feature. However, the precise world coordinates of feature points adopted in [24,25] are unmeasurable in a practical case. Easy access to all the parameters and data in a controller is desired for an uncalibrated visual servoing control method. Moreover, the depth and the Jacobian matrix are individually estimated in [26,27]. Meanwhile, the uncertainties in dynamics caused by unknown robotic physical parameters, i.e., the gravitational torque of robotic manipulators, are neglected in [25,27]. In short, the kinematic and dynamic uncertainties caused by unknown parameters of the visual servoing are not or partially discussed in the above methods.

Prompted by the above studies and expanding from our previous work [29], in this paper, an IBVS scheme is presented to handle the kinematic and dynamic uncertainties (caused by the unknown parameters of the camera and the robotic manipulator) under the uncalibrated environment. The raised controller drives a robotic manipulator with uncertainties, where the feature points are marked, ensuring the projection of those points converges to the desired area in the image plane. One of the major problems that obstruct the application of the method in [22] in absence of the parameters of a robotic manipulator is the fact that the robotic Jacobian matrix can not be parameterized as a linear form together with the interaction matrix. The other problem is the gravitational torque for an uncalibrated robotic manipulator is unknown, which yields uncertainties in dynamics. To solve the former problem, a depth-independent composite Jacobian matrix and a depth Jacobian matrix are constructed, both of which can be parametrized in a uniform linear parametrized form. Therefore, the unknown parameters, including the extrinsic/intrinsic visual parameters and the robotic physical parameters, can be updated by an adaptive algorithm. However, since the physical parameters of manipulators are estimated by an adaptive algorithm, the non-singularity of the robotic Jacobian matrix or even the composite Jacobian matrix cannot be guaranteed, which leads to difficulties in the convergence of the image errors. To ensure asymptotic convergence, a novel adaptive algorithm using a combination of the Slotine-Li algorithm [30] with the parameter projection algorithm [31] is proposed. The aim of the introduction of the parameter projection algorithm is to ensure the estimated parameters converge to an appropriate tiny neighborhood of their actual values such that the adaptive composite Jacobian is non-singular. Meanwhile, to solve the latter problem, the adaptive algorithm we proposed is applied to compensate for the gravitational torque of a robotic manipulator. Since the estimated error of the robotic gravitational torque directly relates to the convergence of the image error, the parameter projection algorithm ensures that the adaptively estimated gravitational torque is always within an appropriate range of its true value. In this sense, the image error caused by the adaptive estimation of the gravitational torque can also be reduced to some extent. In summary, the study of this work has the following contributions and novelties:

• A depth-independent composite Jacobian matrix is constructed, with which the unknown visual parameters and robotic physical parameters can be well arranged in a uniform linear form so that their uncertainties can be addressed by an adaptive estimation approach. Different from the existing works on the adaptive uncalibrated visual servoing control, e.g., [26,27], the unknown parameters of the visual servoing are updated by one adaptive law, which is capable of reducing the number of adaptive laws greatly.
• In a whole uncalibrated environment, an adaptive visual servoing controller is proposed for robotic manipulators, considering the uncertainties in kinematics and dynamics comprehensively, which means all the data and parameters utilized in the controller can be obtained easily. Particularly, robotic physical parameters adopted in our raised controller can be obtained by an ocular estimation instead of a precise measuring in [22,23]. In addition, different form the related methods [25,27], the one raised in this paper takes the uncertain dynamics caused by the gravitational parameters into account and compensates for the gravity of robotic manipulators.
• Apart from these, a novel adaptive estimation algorithm is raised, which is capable of avoiding the possible singularity of the estimated Jacobian matrix, and most importantly ensuring the asymptotic convergence of the image error. Compared with the related work in [22,23,24], the cartesian coordinates of feature points, acquired difficultly in a practical operation, are avoided in the raised adaptive algorithm, and the gravitational torque of robotic manipulators can be compensated well ensuring the image error asymptotically converges to zero better.

Finally, the asymptotic convergence of image errors is proved by the Lyapunov method. The effectiveness of the raised visual servoing control approach is confirmed by simulation based on a 6 degree-of-freedoms (DOFs) PUMA manipulator.

Notation 1. 

For clarity, we introduce here the rule for choosing notations in the paper, and definitions of some symbols. The bold upper and lower case letters are used to denote matrices and vectors, and the symbol $\widehat{}$ above or a matrix, vector, or scalar is used to denote an adaptive estimate of the counterpart. Besides, the notation ${\mathbf{0}}_{m\times n}$ is adopted to denote a zero matrix with m rows and n columns.

2. Background and Problem Statement

Consider the eye-to-hand visual control system presented in Figure 1, for which a pinhole camera is fixed to monitor the feature point marked on the end-effector of a robotic manipulator. For simplicity, and bringing our attention to the studied visual control problem well, the visual servoing system is assumed to satisfy the so-called field-of-view constraint (FOV) that the feature point marked on the end-effector is always within the field of view of the used camera. To establish the mathematical model of the studied visual control system, three coordinate frames are introduced: (i) the robotic base coordinate frame $F_B$; (ii) the vision coordinate frame $F_C$; (iii) the image coordinate frame $F_I$. In general, the visual servo system is composed of two subsystems, as presented below.

2.1. Perspective Projection Model

The process of the projection of a point from Euclidean space to image space amounts to transforming its coordinate from $F_B$ to $F_I$. As shown in Figure 1, let us consider a feature point $\mathit{P}$ marked on the end-effector of a manipulator, whose coordinate mapping relationship between $F_B$ and $F_C$ can be given by

$$\left(\begin{array}{c}{\mathit{P}}^C\\ 1\end{array}\right)=\mathit{T}\left(\begin{array}{c}{\mathit{P}}^B\\ 1\end{array}\right),$$

(1)

where the homogeneous transform matrix, $\mathit{T}\in {\mathbb{R}}^{4\times 4}$, represents the relationship of the rotation and translation between $F_B$ and $F_C$. Consider the projection of $\mathit{P}$ on the image plane labeled by $\mathit{p}$, whose homogeneous image coordinate is given by $\mathit{y}={\left(\begin{array}{cc}{\mathit{p}}^{\mathrm{T}}& 1\end{array}\right)}^{\mathrm{T}}$. Furthermore, the coordinate mapping relationship between $F_I$ and $F_C$ can be represented by the perspective projection model as follows

$$\begin{array}{c}\hfill \mathit{y}=\frac{1}{z^c}\mathbf{\Omega }\left(\begin{array}{c}{\mathit{P}}^C\\ 1\end{array}\right),\end{array}$$

(2)

where $z^c$ represents the depth of the feature point marked on the end-effector of a manipulator relative to the origin of the vision frame $F_C$, the matrix $\mathbf{\Omega }\in {\mathbb{R}}^{3\times 4}$ is composed of the intrinsic parameter of a camera, whose form can be represented as

$$\begin{array}{c}\hfill \mathbf{\Omega }=\left(\begin{array}{cccc}{\beta }_x& {\beta }_x{\varphi }_1& u_0& 0\\ 0& {\beta }_y{\varphi }_2& v_0& 0\\ 0& 0& 1& 0\end{array}\right),\end{array}$$

(3)

where ${\beta }_x$ and ${\beta }_y$ are the normalized focal lengths with respect to the $U_I$ and $V_I$ axes of the image coordinate frame $F_I$, respectively, and the ${\varphi }_i,i=1,2$ are the parameters determined by the nonlinear distortion of a camera. As shown in Figure 1, the $Z_C$ axis of $F_C$ is coincident with the camera optical axis and meanwhile intersects the image plane vertically at the principle point of a camera $P_p=(u_0,v_0)$. It should be noted that the matrices determined by the extrinsic and intrinsic parameters of the camera model are constant and unknown. To estimate them with an adaptive approach, a perspective projection matrix $\mathbf{\Xi }\in {\mathbb{R}}^{3\times 4}$ is defined as follows

$$\begin{array}{c}\hfill \mathbf{\Xi }=\mathbf{\Omega }\mathit{T}=\left(\begin{array}{ccc}{\xi }_{11}& \cdots & {\xi }_{14}\\ ⋮& \ddots & ⋮\\ {\xi }_{31}& \cdots & {\xi }_{34}\end{array}\right).\end{array}$$

(4)

It then follows from (1), (2), and (4) that

$$\begin{array}{c}\hfill \mathit{y}=\frac{1}{z^c}\mathbf{\Xi }\left(\begin{array}{c}{\mathit{P}}^B\\ 1\end{array}\right)=\frac{1}{z^c}\mathbf{\Xi }\mathit{x},\end{array}$$

(5)

where $\mathit{x}\in {\mathbb{R}}^{4\times 1}$ is the homogeneous coordinate of $\mathit{P}$ with respect to $F_B$, and the depth $z^c$ in (5) have the following form:

$$\begin{array}{c}\hfill z^c={\mathbf{\xi }}_3^{\mathrm{T}}\mathit{x}.\end{array}$$

(6)

Here ${\mathbf{\xi }}_3^T$ represents the third row of $\mathbf{\Xi }$. Observing from the perspective projection model in (5) and (6), the image coordinate, $\mathit{y}$, is the output of the system and the homogeneous coordinate, $\mathit{x}$, is the input of the model which can be accessed indirectly. Thus, the precision of the perspective projection model can be ensured when the perspective projection matrix, $\mathbf{\Xi }$, is precise and accessible.

Remark 1. 

For a control model with some uncertainties due to the unknown parameters such as (5), it is commonly practiced to utilize the adaptive algorithm to construct an estimator of the unknown parameters in a controller [32,33]. Once we obtain the mapping model of the visual servo shown in Figure 1, it seems that the most reasonable control scheme is to incorporate the adaptive algorithm into the general control approaches of the robotic manipulator. However, $z^c$, the depth of $\mathit{P}$ relative to the optical point of the camera, is absent in single-camera vision. Furthermore, the depth information is inversely proportional to the perspective projection matrix so it is hard to estimate coupled with other unknown visual parameters directly. Therefore, the form of the depth in (6) is beneficial to the estimation of the unknown visual parameters in absence of the depth information.

2.2. Kinematics and Dynamics Model of Robotic Manipulator

For the robotic manipulator, the task of its kinematics analysis is to handle the mapping relationship between the work space and the joint space, and the task of its dynamics analysis is to construct the bridge from the desired joint positions/velocities to the desired joint torque. Thus, this section will is divided into two parts: one is the kinematics model, and one is the dynamics model.

2.2.1. Kinematics Model

Differentiating the nonlinear mapping between the work space and joint space $\mathit{x}=f\left(\mathit{q}\right)$ in time, we obtain the well-known forward kinematics equation as follows

$$\begin{array}{c}\hfill \dot{\mathit{x}}=\mathit{J}\left(\mathit{q}\right)\dot{\mathit{q}},\end{array}$$

(7)

where $\mathit{q}\in {\mathbb{R}}^{n\times 1}$ represents the joint position of the n-degree-of-freedom manipulator, and $\mathit{J}\left(\mathit{q}\right)\in {\mathbb{R}}^{n\times n}$ denotes the robotic Jacobian matrix. It is more desirable to obtain the relationship of the joint-space velocities relative to the image-space one rather than relative to the work-space one. Thus, differentiating (5) in time, we obtain the mapping from the work-space velocities to the image-space velocities as follows

$$\begin{array}{cc}\hfill \frac{d}{dt}\mathit{y}=& \frac{1}{z^c}\left(\mathbf{\Xi }\dot{\mathit{x}}-\mathit{y}{\dot{z}}^c\right)\hfill \\ \hfill =& \frac{1}{z^c}\mathbf{\Lambda }\left(\mathit{y}\right)\dot{\mathit{x}},\hfill \end{array}$$

(8)

where $\mathbf{\Lambda }\left(\mathit{y}\right)\in {\mathbb{R}}^{3\times 4}$, named the depth-independent interaction matrix [22], can be represented as

$$\begin{array}{c}\hfill \mathbf{\Lambda }\left(\mathit{y}\right)=\mathbf{\Xi }-\mathit{y}{\mathbf{\xi }}_3^{\mathrm{T}}=\left(\begin{array}{c}{\mathbf{\xi }}_1^{\mathrm{T}}-u{\mathbf{\xi }}_3^{\mathrm{T}}\\ {\mathbf{\xi }}_2^{\mathrm{T}}-v{\mathbf{\xi }}_3^{\mathrm{T}}\\ {\mathbf{0}}_{1\times 4}\end{array}\right).\end{array}$$

(9)

Substituting (7) into (8) yields that

$$\begin{array}{c}\hfill \frac{d}{dt}\mathit{y}=\frac{1}{z^c}\mathbf{\Lambda }\left(\mathit{y}\right)\mathit{J}\left(\mathit{q}\right)\dot{\mathit{q}}.\end{array}$$

(10)

As the kinematics parameters of a robotic manipulator are unknown in an uncalibrated environment, the Jacobian matrix $\mathit{J}\left(\mathit{q}\right)$, including the kinematics parameters and the state variables of the manipulator, is required to be parametrized and then estimated in the control scheme. To facilitate the parametrization, construct an auxiliary matrix, i.e.,

$$\begin{array}{cc}\hfill {\mathbf{\Lambda }}_J(\mathit{q},\mathit{y})=& \mathbf{\Lambda }\left(\mathit{y}\right)\mathit{J}\left(\mathit{q}\right)\hfill \\ \hfill =& \left(\begin{array}{ccc}{\mathbf{\xi }}_1^{\mathrm{T}}{\mathit{j}}_1-u{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_1& {\mathbf{\xi }}_1^{\mathrm{T}}{\mathit{j}}_2-u{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_2& {\mathbf{\xi }}_1^{\mathrm{T}}{\mathit{j}}_3-u{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_3\\ {\mathbf{\xi }}_2^{\mathrm{T}}{\mathit{j}}_1-v{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_1& {\mathbf{\xi }}_2^{\mathrm{T}}{\mathit{j}}_2-v{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_2& {\mathbf{\xi }}_2^{\mathrm{T}}{\mathit{j}}_3-v{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_3\\ 0& 0& 0\end{array}\right),\hfill \end{array}$$

(11)

where ${\mathit{j}}_i$ denotes the ith column of $\mathit{J}\left(\mathit{q}\right)$ and ${\mathbf{\Lambda }}_J(\mathit{q},\mathit{y})$ is different with the composite Jacobian matrix $\frac{1}{z^c}{\mathbf{\Lambda }}_J(\mathit{q},\mathit{y})$ from Equation (10) in depth, which is named the depth-independent composite Jacobian matrix.

Remark 2. 

For the depth-independent interaction matrix frame proposed in [22], the Jacobian matrix is utilized in the controller as prior knowledge so that only the components of Ξ need to be estimated. Since the components of Ξ appear linearly in $\mathbf{\Lambda }\left(\mathit{y}\right)$, it is easy to parametrize $\mathbf{\Lambda }\left(\mathit{y}\right)$. However, the above linear case of the estimated parameters does not appear in parametrization when the estimated parameters involve the unknown kinematics parameters. Furthermore, it is extremely complicated to estimate the matrix $\mathbf{\Lambda }\left(\mathit{y}\right)$ and the Jacobian matrix separately by the adaptive algorithm because they include the unknown parameters and system state variables simultaneously. Therefore, introduce the depth-independent composite Jacobian matrix such as (11) to couple the unknown visual parameters with the unknown kinematics parameters and then extract them from the state variables.

Similarly, differentiating (6) in time, and incorporating with (7) yields that

$$\begin{array}{c}\hfill {\dot{z}}^c={\mathbf{\xi }}_3^{\mathrm{T}}\mathit{J}\left(\mathit{q}\right)\dot{\mathit{q}}.\end{array}$$

(12)

Similar to the (10), the unknown kinematics parameters in Jacobian matrix $\mathit{J}\left(\mathit{q}\right)$ coupled with the visual parameters in the perspective projection matrix are proportional to the velocity of the depth in (12). For the sake of the formulation of the uniform parametrized form relative to ${\mathbf{\Lambda }}_J(\mathit{q},\mathit{y})$, the (12) can be given the following form by

$$\begin{array}{c}\hfill {\dot{z}}^c={\mathbf{\lambda }}_j\left(\mathit{q}\right)\dot{\mathit{q}},\end{array}$$

(13)

where ${\mathbf{\lambda }}_j\left(\mathit{q}\right)$ is defined as

$$\begin{array}{c}\hfill {\mathbf{\lambda }}_j\left(\mathit{q}\right)={\mathbf{\xi }}_3^{\mathrm{T}}\mathit{J}\left(\mathit{q}\right)=\left(\begin{array}{ccc}{\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_1& {\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_2& {\mathbf{\xi }}_3^{\mathrm{T}}{\mathit{j}}_3\end{array}\right).\end{array}$$

(14)

Remark 3. 

As ${\mathbf{\lambda }}_j$ shown in (13) reflects the influence of the depth transform velocity of the feature point by the joint-space velocity of a manipulator, we call it the depth Jacobian matrix. The depth Jacobian matrix in (14) is also dependent on the unknown visual and robotic kinematics parameters, which are required to be estimated in the controller together with the depth-independent composite Jacobian matrix ${\mathbf{\Lambda }}_J$ in (11). For the sake of the simplicity of the parametrization, we construct uniform product forms by the components of the perspective matrix and the Jacobian matrix so that all of the components of ${\mathbf{\Lambda }}_J$ and ${\mathbf{\lambda }}_j$ are linear. Moreover, the product forms can be parametrized by the same coupled parameters shown in Proposition 1, it follows that the matrix, mentioned above, can be represented as the linear form of the united parameters finally.

Based on the remark above, we have the following properties.

Proposition 1. 

For any product of ${\mathbf{\xi }}_k^T$ and ${\mathit{J}}_l$, we have

$$\begin{array}{c}\hfill {\mathbf{\xi }}_k^{\mathrm{T}}{\mathit{j}}_l={\mathbf{\eta }}_{kl}^{\mathrm{T}}\mathbf{\varphi },\end{array}$$

(15)

where ${\mathbf{\eta }}_{kl}^{\mathrm{T}}$ is independent of the unknown parameters including the physical parameters of the manipulator and the parameters of the visual system, but the parametric vector $\mathbf{\varphi }$ contains all of them.

Proposition 2. 

Given any vector $\mathbf{\varrho }\in {\mathbb{R}}^{4\times 1}$, ${\mathbf{\Lambda }}_J(·)\mathbf{\varrho }$ and ${\mathbf{\lambda }}_j(·)\mathbf{\varrho }$ can be parameterized as follows

$$\begin{array}{cc}\hfill {\mathbf{\Lambda }}_J(·)\mathbf{\varrho }=& {\mathit{\Psi }}_1(\mathbf{\varrho },·)\mathbf{\varphi },\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\mathbf{\lambda }}_j(·)\mathbf{\varrho }=& {\mathit{\Psi }}_2(\mathbf{\varrho },·)\mathbf{\varphi },\hfill \end{array}$$

(17)

where ${\mathit{\Psi }}_i(\mathbf{\varrho },·)$ ($i=1,2$) are regression matrices.

It is worth mentioning that all of the unknown parameters in ${\mathbf{\Lambda }}_J(·,·)$ and ${\mathbf{\lambda }}_j(·)$ are arranged in the parametric vector $\mathbf{\varphi }$, which dimension is shaped by the number of those unknown parameters.

2.2.2. Dynamics Model

The dynamics model of a robotic manipulator generally has the following form [30]:

$$\begin{array}{c}\hfill \mathit{M}\left(\mathit{q}\right)\ddot{\mathit{q}}+\frac{1}{2}\dot{\mathit{M}}\left(\mathit{q}\right)\dot{\mathit{q}}+\mathit{C}(\dot{\mathit{q}},\mathit{q})\dot{\mathit{q}}+\mathit{g}\left(\mathit{q}\right)=\mathbf{\tau }.\end{array}$$

(18)

Here $\mathit{M}\left(\mathit{q}\right)$ and $\mathit{C}(\dot{\mathit{q}},\mathit{q})$ represent the Inertia and the Coriolis/Centrifugal force, respectively, and for any given vector $\mathit{r}$,

$$\begin{array}{c}\hfill {\mathit{r}}^{\mathrm{T}}\mathit{C}(\mathit{q},\dot{\mathit{q}})\mathit{r}=0,\end{array}$$

(19)

which means that $\mathit{C}(·,·)$ is a skew-symmetric matrix. The vector $\mathbf{\tau }\in {\mathbb{R}}^{n\times 1}$ in (18) represents the robotic joint input torque, which is also the output of the designed controller. The vector $\mathit{g}\left(\mathit{q}\right)$ in (18) denotes the joint torque for compensating the robotic gravitational force, which relates to the physical parameters and the positions of the robotic manipulator, and means it is unmeasurable in the uncalibrated visual control.

Proposition 3. 

For the gravitational vector $\mathit{g}\left(\mathit{q}\right)$, we have

$$\begin{array}{c}\hfill \mathit{g}\left(\mathit{q}\right)={\Psi }_g\left(\mathit{q}\right)\mathbf{\phi },\end{array}$$

(20)

where ${\Psi }_g\left(\mathit{q}\right)$ is a regression matrix, which does not contain any physical parameters of a robotic manipulator, but the vector $\mathbf{\phi }$ involves all of the uncertain parameters.

Remark 4. 

Generally, the dynamics parameter matrices in (18), including $\mathit{M}\left(\mathit{q}\right)$, $\mathit{g}\left(\mathit{q}\right)$ and $\mathit{C}(\mathit{q},\dot{\mathit{q}})$, require to be estimated in the case without precise calibration. However, introducing their estimator in the controller increases the calculation burden of the system because they are not independent. With the adaptive algorithm of the robotic manipulator proposed by Slotine-Li [30], the computation of the controller can be simplified by the effective exploitation of the structure of manipulator dynamics. From the property of the skew-symmetric matrix $\mathbf{C}(\mathit{q},\dot{\mathit{q}})$ in (19), it implies the derivative of the kinetic energy, ${\dot{\mathit{q}}}^{\mathrm{T}}\mathit{M}\left(\mathit{q}\right)\dot{\mathit{q}}$, is identically equal to the power input, such as

$$\begin{array}{c}\hfill \frac{1}{2}\frac{d}{dt}{\dot{\mathit{q}}}^{\mathrm{T}}\mathit{M}\left(\mathit{q}\right)\dot{\mathit{q}}={\dot{\mathit{q}}}^{\mathrm{T}}[\mathbf{\tau }-\mathit{g}\left(\mathit{q}\right)].\end{array}$$

Therefore we only need to introduce the estimator of $\mathit{g}\left(\mathit{q}\right)$ rather than the inertia and Centripetal/Coriolis terms into the controller for the sake of computational simplicity.

2.3. Problem Formulation

Based on the background knowledge above, we next state our control problem formally.

Problem 1. 

Given the desired image coordinate ${\mathit{y}}^{*}$, designing a control law for the output torques, $\mathbf{\tau }$ in (18), and an adaptive updating law for the unknown parameters in absence of the precise calibration of the vision subsystem and the manipulator subsystem, such that the image coordinate of the projection of the feature point marked on the robotic end-effector, $\mathit{y}$, converges to the desired image coordinate after an initial adaptation process.

Remark 5. 

Although plenty of research has been conducted on the related fields, such as [18,19,20,21,22,23,24,25,26,27], it is still challenging to perform a visual servoing task with kinematic and dynamic uncertainties, even a simplest setpoint control. For existing uncalibrated IBVS schemes, the controllers generally are constructed by the estimated Jacobian matrix. However, the controller design of their methods will suffer from some more complicated problems when the uncertainties of robotic manipulators are involved, such as the absence of the depth of the feature point shown in Remark 1, the parametrization of the unknown coupled parameters shown in Remarks 2 and 3 and the estimation of the dynamics parameters shown in Remark 4. Furthermore, the singularity of the estimated image Jacobian matrix also ought to be taken into account due to the characteristic of the adaptive algorithm. Meanwhile, it should mention that the work to prove the asymptotic convergence to zero of the image errors is difficult too.

3. Uncalibrated Adaptive Visual Control Approach and Stability Analysis

To address the uncertain effects due to the lacking of precise calibration of the unknown parameters, an uncalibrated adaptive visual control approach is presented newly for the IBVS manipulator systems. Motivated by the study in [22,24,26], the design idea of our control approach has the following novelties: first, let visual and robotic physical parameters be arranged in a uniform linear form, which ensures these parameters in the closed-loop dynamics can be estimated by one adaptive law, reducing the number of adaptive algorithms; second, a new adaptive algorithm is raised, where estimated parameters are updated in an appropriate tiny range of their actual values so that the potential singularity of the estimated Jacobian matrix can be circumvented without any pre-selected trajectory point, and most importantly guarantees the asymptotic convergence of image errors.

3.1. Controller Design

The desired homogeneous image coordinate of the feature point marked on the robotic end-effector is denoted by ${\mathit{y}}^{*}$, and the image error $\mathit{e}$ is defined as follows

$$\begin{array}{c}\hfill \mathit{e}=\mathit{y}-{\mathit{y}}^{*}.\end{array}$$

(21)

Here the last component of $\mathit{e}\in {\mathbb{R}}^{3\times 1}$ is always zero since $\mathit{y}=[u,v,1]$ and ${\mathit{y}}^{*}=[u_d,v_d,1]$, where $u_d$ and $v_d$ are constants. Applying the adaptive parameters, the raised controller can be presented as follows

$$\begin{array}{c}\hfill \mathbf{\tau }=\widehat{\mathit{g}}\left(\mathit{q}\right)-{\mathit{K}}_1\dot{\mathit{q}}-\left({\widehat{\mathbf{\Lambda }}}_J^{\mathrm{T}}(\mathit{q},\mathit{y})+\frac{1}{2}{\widehat{\mathbf{\lambda }}}_j^{\mathrm{T}}\left(\mathit{q}\right){\mathit{e}}^{\mathrm{T}}\right)\mathit{B}\mathit{e},\end{array}$$

(22)

where $\mathit{B}\in {\mathbb{R}}^{3\times 3}$ is a position positive-definite gain matrix and the related term (the last term of the right side in (22)) implies the image error feedback. Here ${\mathit{K}}_1\dot{\mathit{q}}$ represents the joint velocity feedback, where ${\mathit{K}}_1\in {\mathbb{R}}^{n\times n}$ is a velocity positive-definite gain matrix. Although three estimators are introduced in the controller above (22), i.e., $\widehat{\mathit{g}}$ (18), ${\widehat{\mathbf{\Lambda }}}_J^{\mathrm{T}}$ (11) and ${\widehat{\mathbf{\lambda }}}_j^{\mathrm{T}}$ (13), the last two estimators do not need to be updated, respectively. It is crucial to accomplish the parametrization of the unknown parameters before designing the adaptive laws.

Firstly, the closed-loop dynamics follows from (22) and (18) that:

$$\begin{array}{cc}\mathit{M}\left(\mathit{q}\right)\ddot{\mathit{q}}+\left(\frac{1}{2}\dot{\mathit{M}}\left(\mathit{q}\right)+\mathit{C}(\mathit{q},\dot{\mathit{q}})\right)\dot{\mathit{q}}=\hfill & \\ & \hfill \left({\widehat{\mathbf{\Lambda }}}_J^{\mathrm{T}}(\mathit{y},\mathit{q})+\frac{1}{2}{\widehat{\mathbf{\lambda }}}_j^{\mathrm{T}}\left(\mathit{q}\right){\mathit{e}}^{\mathrm{T}}\right)\mathit{B}\mathit{e}-{\mathit{K}}_1\dot{\mathit{q}}+\tilde{\mathit{g}}\left(\mathit{q}\right),\end{array}$$

(23)

where $\tilde{\mathit{g}}\left(\mathit{q}\right)=\widehat{\mathit{g}}\left(\mathit{q}\right)-\mathit{g}\left(\mathit{q}\right)$ denotes the estimated error of the gravitational force. It then follows from Proposition 3 that:

$$\begin{array}{c}\hfill \tilde{\mathit{g}}\left(\mathit{q}\right)={\mathbf{{\rm Y}}}_d\left(\mathit{q}\right)\tilde{\mathbf{\phi }},\end{array}$$

(24)

where $\tilde{\mathbf{\phi }}=\widehat{\mathbf{\phi }}-\mathbf{\phi }$ represents the estimated error of robotic physical parameters.

Subsequently, according to Proposition 2, it follows

$$\begin{array}{c}\hfill -{\left[{\tilde{\mathbf{\Lambda }}}_J(\mathit{q},\mathit{y})+\frac{1}{2}\left(\mathit{e}{\tilde{\mathbf{\lambda }}}_j\left(\mathit{q}\right)\right)\right]}^{\mathrm{T}}\mathit{B}\mathit{e}={\mathbf{{\rm Y}}}_k(\mathit{q},\mathit{y})\tilde{\mathbf{\varphi }},\end{array}$$

(25)

where $\widetilde{\left(·\right)}$ denotes the estimated error of $\left(·\right)$.

Remark 6. 

The intention of developing an uncalibrated visual servoing control approach is to alleviate the impact of lacking precise measurements on the bias of the results, which by no means desires to complete the task without any prior knowledge. Thus, the selection of the initial values of unknown parameters is crucial for the convergence of task errors before implementing adaptive estimation. A too-large initial bias may cause the deterioration of control performance and even the system collapse. Compared to traditional methods, one of the advantages of the presented method is that highly servoing performance can be guaranteed even in the absence of accurate measurement.

3.2. Parameter Estimation

An adaptive algorithm is presented to update the above parameters in (24) and (25), which newly combines the Slotine-Li algorithm [30] with the parameter projection algorithm [31].

Compared with the precise calibration of $\mathbf{\varphi }$ and $\mathbf{\phi }$, the prior knowledge regarding a certain range ${\mathit{R}}^n$ where they are situated is easier to obtain, which means that an appropriate range can be found to restrict the parametric estimation in a practical case. It follows that the convex compact sets ${\mathbf{\Omega }}_{\mathit{k}}$ and ${\mathbf{\Omega }}_{\mathit{d}}$ where the unknown parameters are bounded, i.e.,

$$\begin{array}{c}\hfill {\mathbf{\Omega }}_{\mathit{k}}=\left\{{\widehat{\varphi }}_i\left(0\right)\in \widehat{\mathbf{\varphi }}\left(0\right)|{\underline{\varphi }}_i<{\widehat{\varphi }}_i\left(0\right)<{\overline{\varphi }}_i\right\},\end{array}$$

(26)

$$\begin{array}{c}\hfill {\mathbf{\Omega }}_{\mathit{d}}=\left\{{\widehat{\phi }}_i\left(0\right)\in \widehat{\mathbf{\phi }}\left(0\right)|{\underline{\phi }}_i<{\widehat{\phi }}_i\left(0\right)<{\overline{\phi }}_i\right\},\end{array}$$

(27)

where $\underline{(·)}$ and $\overline{(·)}$ are the lower and upper bounds of the counterpart, respectively. It is desirable for the estimated parameters to always situate in these sets, which means $\widehat{\mathbf{\phi }}\left(t\right)\subset {\mathbf{\Omega }}_{\mathit{d}}$ and $\widehat{\mathbf{\varphi }}\left(t\right)\subset {\mathbf{\Omega }}_{\mathit{k}}$ when $t\ge 0$. To this end, the parameter vectors, $\widehat{\mathbf{\varphi }}$ and $\widehat{\mathbf{\phi }}$, are updated by the following adaptive update laws

$$\begin{array}{cc}\hfill \dot{\widehat{\mathbf{\varphi }}}=& -{\mathbf{\Gamma }}_k^{-1}{\mathbf{{\rm Y}}}_k^{\mathrm{T}}(\mathit{q},\mathit{y})\dot{\mathit{q}}+{\mathit{f}}_k,\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill \dot{\widehat{\mathbf{\phi }}}=& -{\mathbf{\Gamma }}_d^{-1}{\mathbf{{\rm Y}}}_d^{\mathrm{T}}\left(\mathit{q}\right)\dot{\mathit{q}}+{\mathit{f}}_d,\hfill \end{array}$$

(29)

where ${\mathbf{\Gamma }}_{(·)}$ is a positive-definite and diagonal matrix and ${\mathit{f}}_{(·)}$ is a correction vector, whose component $f_{{(·)}_i}$ is constructed in the following form:

$$\begin{array}{cc}\hfill f_{k_i}=& \left\{\begin{array}{cc}0\hfill & \mathrm{if}{\underline{\varphi }}_i<{\widehat{\varphi }}_i<{\overline{\varphi }}_i\hfill \\ \hfill & \mathrm{or}{\widehat{\varphi }}_i={\underline{\varphi }}_i\mathrm{and}{h}_{ki}\ge 0\hfill \\ \hfill & \mathrm{or}{\widehat{\varphi }}_i={\overline{\varphi }}_i\mathrm{and}{h}_{ki}\le 0\hfill \\ -h_{ki}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill f_{d_i}=& \left\{\begin{array}{cc}0\hfill & \mathrm{if}{\underline{\phi }}_i<{\widehat{\phi }}_i<{\overline{\phi }}_i\hfill \\ \hfill & \mathrm{or}{\widehat{\phi }}_i={\underline{\phi }}_i\mathrm{and}{h}_{di}\ge 0\hfill \\ \hfill & \mathrm{or}{\widehat{\phi }}_i={\overline{\phi }}_i\mathrm{and}{h}_{di}\le 0,\hfill \\ -h_{di}\hfill & \mathrm{otherwise},\hfill \end{array}\right.\hfill \end{array}$$

(31)

where $h_{(·)i}$ is the components of ${\mathit{h}}_{(·)}$, which is defined as follows

$$\begin{array}{cc}\hfill {\mathit{h}}_k=& -{\mathbf{\Gamma }}_k^{-1}{\mathbf{{\rm Y}}}_k^{\mathrm{T}}(\mathit{q},\mathit{y})\dot{\mathit{q}},\hfill \end{array}$$

(32)

$$\begin{array}{cc}\hfill {\mathit{h}}_d=& -{\mathbf{\Gamma }}_d^{-1}{\mathbf{{\rm Y}}}_d^{\mathrm{T}}\left(\mathit{q}\right)\dot{\mathit{q}}.\hfill \end{array}$$

(33)

Remark 7. 

The difficult convergence of estimated parameters is a well-known and challenging issue in adaptive control. Since that issue is generally harmless to the control objective, it can be ignored in the controller design. However, the Jacobian matrix estimated by an adaptive algorithm has a risk of the singular in the uncalibrated visual servo tasks due to the above difficulty, which may degrade the degree of freedom of robotic manipulators and eventually lead to the collapse of systems. Considering that the Jacobian matrix derived from the actual parameters is generally non-singular, it is reasonable to assume that the estimation in a range around its real value can also meet the non-singular requirement of the Jacobian matrix. To this end, we design a novel estimation algorithm synthesized by the technique of parameter projection in (28), which ensures the update of estimated parameters in an appropriate region and thus avoids the potential singularity of the estimated Jacobian matrix.

3.3. Stability Analysis

With the raised control approach, the stability analysis of the uncalibrated visual servoing system is given.

Theorem 1. 

With the unknown parameters in the controller (22) adjusted by adaptive algorithm (28)–(33), the steady-state tracking performance, i.e., ${lim}_{t\to \infty }\mathit{e}\left(t\right)=0$ can be guaranteed, in addition to the signal boundedness of the closed-loop visual servoing system.

Proof. 

Consider the Lyapunov function

$$\begin{array}{cc}\hfill V=& \frac{1}{2}{z}^c{\mathit{e}}^{\mathrm{T}}\mathit{B}\mathit{e}+\frac{1}{2}{\dot{\mathit{q}}}^{\mathrm{T}}\mathit{M}\left(\mathit{q}\right)\dot{\mathit{q}}\hfill \\ \hfill +& \frac{1}{2}{\tilde{\mathbf{\varphi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_k\tilde{\mathbf{\varphi }}+\frac{1}{2}{\tilde{\mathbf{\phi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_d\tilde{\mathbf{\phi }}.\hfill \end{array}$$

(34)

Differentiating (34) in time, it follows

$$\begin{array}{cc}\hfill \dot{V}=& z^c{\dot{\mathit{e}}}^T\mathit{B}\mathit{e}+\frac{1}{2}{\dot{z}}^c{\mathit{e}}^T\mathit{B}\mathit{e}+{\tilde{\mathbf{\varphi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_k\dot{\tilde{\mathbf{\varphi }}}\hfill \\ \hfill +& {\dot{\mathit{q}}}^{\mathrm{T}}[\mathbf{\tau }-\mathit{g}\left(\mathit{q}\right)]+{\tilde{\mathbf{\phi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_d\dot{\tilde{\mathbf{\phi }}},\hfill \end{array}$$

(35)

which combined with (10), (11) and (13) results that

$$\begin{array}{cc}\hfill \dot{V}=& {\dot{\mathit{q}}}^T\left({\mathbf{\Lambda }}_J^T(\mathit{q},\mathit{y})+\frac{1}{2}{\left(\mathit{e}{\mathbf{\lambda }}_j\left(\mathit{q}\right)\right)}^T\right)\mathit{B}\mathit{e}+{\tilde{\mathbf{\varphi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_k\dot{\tilde{\mathbf{\varphi }}}\hfill \\ \hfill +& {\dot{\mathit{q}}}^{\mathrm{T}}[\mathbf{\tau }-\mathit{g}\left(\mathit{q}\right)]+{\tilde{\mathbf{\phi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_d\dot{\tilde{\mathbf{\phi }}}.\hfill \end{array}$$

(36)

Applying the raised controller (22) into above (36) and then incorporating with (25) and (24), we have:

$$\begin{array}{cc}\hfill \dot{V}=& -{\dot{\mathit{q}}}^{\mathrm{T}}{\mathit{K}}_1\dot{\mathit{q}}\hfill \\ \hfill & +{\dot{\mathit{q}}}^{\mathrm{T}}{\mathbf{{\rm Y}}}_d\left(\mathit{q}\right)\tilde{\mathbf{\phi }}+{\tilde{\mathbf{\phi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_d\dot{\tilde{\mathbf{\phi }}}\hfill \\ \hfill & +{\dot{\mathit{q}}}^{\mathrm{T}}{\mathbf{{\rm Y}}}_k(\mathit{y},\mathit{q})\tilde{\mathbf{\varphi }}+{\tilde{\mathbf{\varphi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_k\dot{\tilde{\mathbf{\varphi }}},\hfill \end{array}$$

(37)

which follows from (28)–(33) that

$$\begin{array}{cc}\hfill \dot{V}=& -{\dot{\mathit{q}}}^{\mathrm{T}}{\mathit{K}}_1\dot{\mathit{q}}+{\tilde{\mathbf{\phi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_d{\mathit{f}}_d+{\tilde{\mathbf{\varphi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_k{\mathit{f}}_k.\hfill \end{array}$$

(38)

Since the definition of ${\mathit{f}}_{(·)}$ in (30) and (31), the analysis of (38) can be divided in the following two cases: first, when the estimated parameter is within the prescribed range ${\mathbf{\Omega }}_{(·)}$ or tends to enter that range, $f_{(·)i}=0$, which implies ${\tilde{\phi }}_i^T{\Gamma }_{dii}{f}_{di}=0$ and ${\tilde{\varphi }}_i^T{\Gamma }_{kii}{f}_{ki}=0$; second, when the estimated parameter is at the boundary of the prescribed range ${\mathbf{\Omega }}_{(·)}$ and tends to leave that range, $f_{(·)i}=-h_{(·)i}$, which means ${\tilde{\phi }}_i^T{\Gamma }_{dii}{f}_{di}=-{\tilde{\phi }}_i^T{\Gamma }_{dii}{h}_{di}\le 0$ and ${\tilde{\varphi }}_i^T{\Gamma }_{kii}{f}_{ki}=-{\tilde{\varphi }}_i^T{\Gamma }_{kii}{h}_{ki}\le 0$, where ${\tilde{\phi }}_i^T{h}_{di}\ge 0$ and ${\tilde{\varphi }}_i^T{h}_{ki}\ge 0$. Thus, it follows that

$$\begin{array}{c}\hfill {\tilde{\mathbf{\phi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_d{\mathit{f}}_d+{\tilde{\mathbf{\varphi }}}^{\mathrm{T}}{\mathbf{\Gamma }}_k{\mathit{f}}_k\le 0,\end{array}$$

(39)

which implies that $\dot{V}$ is negative semi-definite, i.e.,

$$\begin{array}{c}\hfill \dot{V}\le -{\dot{\mathit{q}}}^{\mathrm{T}}{\mathit{K}}_1\dot{\mathit{q}}\le 0.\end{array}$$

(40)

The $\dot{V}\le 0$ accompanied by $V\ge 0$ conveys the stability of the visual servoing system and the boundedness of V, which in turn means that the variables, i.e., $\tilde{\mathbf{\phi }}$, $\mathit{e}$, $\tilde{\mathbf{\varphi }}$, and $\dot{\mathit{q}}$ in (34), are bounded. It follows from (28)–(33) that $\dot{\widehat{\mathbf{\varphi }}}$ and $\dot{\widehat{\mathbf{\phi }}}$ are bounded, which, incorporated with (23), implies that $\ddot{q}$ is bounded. Furthermore, the boundedness of $\dot{q}$ and $\ddot{q}$ conveys information about the uniform continuity of $\dot{q}$, which follows from Barbalat’s lemma that

$$\begin{array}{c}\hfill \underset{t\to \infty }{lim}\dot{\mathit{q}}\left(t\right)=0.\end{array}$$

(41)

The visual servoing system tends to an equilibrium point as $\dot{\mathit{q}}\to 0$, where (23) can be rewritten as follows

$$\begin{array}{c}\hfill \underset{\mathbf{D}(\widehat{\mathbf{\varphi }},\mathit{y})}{\underbrace{\left({\widehat{\mathbf{\Lambda }}}_J^{\mathrm{T}}(\mathit{q},\mathit{y})+\frac{1}{2}{\widehat{\mathbf{\lambda }}}_j^{\mathrm{T}}\left(\mathit{q}\right){\mathit{e}}^{\mathrm{T}}\right)}}\mathit{B}\mathit{e}=0.\end{array}$$

To demonstrate $\mathit{e}\to 0$, an auxiliary matrix ${\mathit{D}}^{\mathrm{T}}\mathit{D}$ is constructed as follows

$$\begin{array}{c}\hfill {\mathit{D}}^{\mathrm{T}}\mathit{D}=\left(\begin{array}{cc}\mathbf{\Theta }& {\mathbf{0}}_{2\times 1}\\ {\mathbf{0}}_{1\times 2}& 0\end{array}\right),\end{array}$$

from which it can be derived that $Rank\left({\mathit{D}}^{\mathrm{T}}\mathit{D}\right)=Rank\left(\mathit{D}\right)$. Since $\mathbf{\Theta }$ is nonsingular, incorporated with $Rank\left({\mathit{D}}^{\mathrm{T}}\mathit{D}\right)=2$, it follows that the image tracking error $\mathit{e}$ converges to zero asymptotically as time goes to infinity. In fact, the conditions of the singularity of $\mathbf{\Theta }$ are rigorous and even impossible for practical engineering, i.e.,

$$\begin{array}{c}\hfill D_{11}\times D_{22}-D_{12}\times D_{21}=0,\\ \hfill D_{11}\times D_{23}-D_{13}\times D_{21}=0,\\ \hfill D_{12}\times D_{23}-D_{13}\times D_{22}=0,\end{array}$$

which must be held simultaneously. Additionally, the estimated parameters are limited in an appropriate range around their practical value under the raised adaptive algorithm, which implies $\mathbf{\Theta }$ can be viewed as invertible, and thus, in conclusion, $\mathit{e}\to 0$. □

4. Simulation

As shown in Figure 2, an 6 DOFs industrial manipulator, Puma 560, is utilized to give a simulation verification of the proposed method. Conventionally, the description of the orientations and the poses of serial links are introduced to the Denavit-Hartenberg parameters represent method (D−H) which indicates the physical features of the manipulator with four parameters. The D−H represents the parameters of the manipulator as shown in

Table 1. D−H Represent Parameters.

Joint	Angle (Rad)	Offset (m)	Length (m)	Twist (Rad)
1	$q_1$	0	0	$\frac{1}{2}\pi$
2	$q_2$	0	0.4318	0
3	$q_3$	0.15005	0.0203	$-\frac{1}{2}\pi$
4	$q_3$	0.4318	0	$\frac{1}{2}\pi$
5	$q_3$	0	0	$-\frac{1}{2}\pi$
6	$q_3$	0	0	0

. The ith non-zero link offset and link length are represented by $d_i$ and $a_i$, respectively. Meanwhile, the mass of the serial links is listed as follows, $m_1=0$, $m_2=17.4$ and $m_3=4.8$, $m_4=0.82$, $m_5=0.34$ and $m_6=0.09$. Since the computing complexity of the proposed scheme is determined by the number of unknown parameters, only the uncertainties caused by the non-zero parameters are taken into account.

The position/orientation of the camera are unrestricted if the FOV constraint is satisfied, the parameters of which are given by

$$\begin{array}{c}\hfill \mathbf{\Xi }=\left(\begin{array}{cccc}750& 256& 443.4& 1186.5\\ 0& -393.5& -818.4& 743.6\\ 0& 0.5& -0.9& 2.4\end{array}\right).\end{array}$$

(42)

The parameter vector $\mathbf{\varphi }=\left[\begin{array}{ccc}{\mathbf{\xi }}_3^{\mathrm{T}}& {\mathbf{\xi }}_2^{\mathrm{T}}& {\mathbf{\xi }}_1^{\mathrm{T}}\end{array}\right]\otimes \left[\begin{array}{cccc}{d}_4& d_3& a_3& a_2\end{array}\right]$, where ⊗ is a Kronecker product operator, and the actual value of which is shown as follows, [−0.374, −0.130, −0.018, −0.374, 0.216, 0.075, 0.01, 0.216, 0, 0, 0, 0, −353.387, −122.802, −16.614, −353.387, −169.922, −59.048, −7.988, −169.922, 0, 0, 0, 0, −191.462, −66.533, −9.001, −191.462, 110.541, 38.413, 5.197, 110.542, 323.850, 112.538, 15.225, 323.85]. Additionally, the parameter vector $\mathbf{\phi }$ is given by [0.09, 0.82, 4.8, 17.4, 0.039, 0.147, 0.354, 0.002, 0.007, 0.017, 0.097, 0.039, 0.147, 0.354, 2.073, 7.513]. For operationalizing the adaptive law, let $\overline{\mathbf{\varphi }}=1.5\mathbf{\varphi }$, $\underline{\mathbf{\varphi }}=0.75\mathbf{\varphi }$, $\widehat{\mathbf{\varphi }}\left(0\right)=0.85\mathbf{\varphi }$, and so does $\mathbf{\phi }$. The control frame in matlab simulink is shown in Figure 3, where the controller input, namely desired image coordinate, is set as ${\mathit{y}}^{*}={[600,300]}^T$.

Observing the torque in Figure 4, the $\mathbf{\tau }$ is bounded and convergent. Furthermore, the angles $\mathit{q}$ of the robotic joint, shown in Figure 5, have a convergent performance as well. As we can see the convergence to zero of the angular velocity $\dot{\mathit{q}}$ of the robotic joint in Figure 6 demonstrates the validity of the deduction in (41). This declares that there is indeed an equilibrium point in our control system, in other words, the visual seroving system incorporated proposed control scheme is stable.

Since the precise measurement of the model parameters of the robotic manipulator and the visual camera is almost inaccessible in practical implementation, the uncertainties caused by those parameters in kinematics and dynamics will be the key factor of the accuracy of the control target. However, the following results will show us the effectiveness of the proposed adaptive algorithm which utilize the approximate model parameter to estimate the unknown torques and even to help the controller to achieve the control target. The estimation of the $\mathbf{\varphi }$ and $\mathbf{\phi }$ can be seen in Figure 7 and Figure 8, respectively. It is more exciting to see in Figure 9 that the estimated gravitational torque in the controller has a good performance in following its actual gravitational torque, which can direct improve the accuracy of the image tracking.

The 3−D space trajectory of the point marked on the end-effector of the robotic manipulator is depicted in Figure 10, correspondingly its plane trajectory on the image plane is illustrated in Figure 11. Observing the trajectory in Figure 10 and Figure 11, we can find that the feature point is successfully converged to the desired point on image space.

Note that the symbol □ in Figure 10 represents the final position of the end-effector, not the desired one in task space, which means that the end-effector is not necessarily driven to the desired location in task space while the image tracking error converges to zero. Although some existing studies [22,23,24,25] propose a method to regulate the spacial position of the end-effector by a sufficient pre-selected point on a 3−D trajectory, it is an impractical task for an uncalibrated visual servoing control. The consistent convergence of image and spatial position is a challenging problem for the uncalibrated IBVS control scheme with monocular vision.

Furthermore, the image tracking error converges asymptotically toward zero, as shown in Figure 12, which demonstrates the conclusion in Theorem 1 that the adaptive controller can ensure the steady-state tracking performance of the visual servoing system even in presence of uncertainties in kinematics and dynamics.

Remark 8. 

Despite the desired control performance that can be achieved for a simulated 6−DOFs manipulator, the raised theoretical control scheme can have some restrictions in real applications. From the simulation study, we find that the number of adaptive laws will increase significantly with the degrees of freedom, so a computational burden may be caused in the applications involving rather high degrees of freedom, which can have destroying effects on closed-loop system performance. Besides, the proposed setpoint control scheme may be insufficient to meet the performance requirements of real applications due to the absence of constraints on the posture and trajectory of manipulators.

Remark 9. 

When implementing the proposed control scheme on practical manipulators, some additional issues could be encountered, e.g.,

• Nonlinear dynamics. Although the dynamics of manipulators have been considered in the proposed method, there are still some unmodeled dynamic factors ignored, such as the Coulomb friction, the viscous friction, and the motor dynamics, which may deteriorate system performance and even lead to system collapse in a practical scenario.
• Nonlinear constraint of actuators. The nonlinear constraints widely exist in many physical actuators, which may lead to errors in the practical results and even instability of the system. To address it, the nonlinear constraints of actuators, such as the backlash, dead zone, saturation, and hysteresis, should be compensated in practical applications. Furthermore, the unknown actuator failures should also be considered regarding real-world visual servo tasks, which would lead to catastrophic results once a failure occurs.
• Noise and disturbances. The image-based visual servoing control method is sensitive to noise and disturbances in the image measurements, which can affect the accuracy and stability of the control system. In our research, the image noise and corresponding image processing techniques do not account for this, but they should be considered in practical applications. Additionally, the variations in lighting, occlusions, and other environmental factors should be considered too, which may significantly impact the quality of visual feedback.
• Real-time performance. Visual servoing systems may suffer from latency issues, which may be caused by communication and sampling, thus achieving the real-time performance of manipulators remains a challenge for real-world scenarios.

These issues mentioned above may be regarded as our next study in the future.

5. Conclusions

In our study, an uncalibrated adaptive visual servoing control scheme is proposed novelly. The kinematic and dynamic uncertainties caused by unknown parameters (including extrinsic/intrinsic parameters of a camera and physical parameters of a robotic manipulator) are addressed in this paper. Specifically, a depth-independent composite Jacobian matrix is used to parametrize the visual parameters combined with the robotic parameters in a linear form. The proposed adaptive algorithm is applied to update the estimated parameters in order to guarantee that the image tracking error asymptotically converges to zero. At present, however, our raised control scheme is only studied in theory, and the illustration of its effectiveness is mainly by means of the stability analysis with rigorous proof, and the simulation technique. To verify the proposed control scheme experimentally, there are still some issues, as mentioned in Remark 9, to be addressed, which may be regarded as our next study in the future.