Mobile Robot Control Based on 3D Visual Servoing: A New Approach Combining Pose Estimation by Neural Network and Differential Flatness

Abstract

This paper deals with 3D visual servoing applied to mobile robots in the presence of measurement disturbances, caused in particular by target occlusion. We propose a new approach based on the flatness concept. In 3D visual servoing, the task is performed out of image coordinate space and targets may leave the camera field of view during navigation (servoing). Forced to navigate blindly during one or more periods of time, the robot will use our new open-loop control algorithm inspired by the flatness concept. The 3D visual servoing method is performed using robot pose estimation. This estimation generally contains some errors. The exact position of the robot is therefore not guaranteed, and robust feedback control is necessary to reject these errors in the input. To solve this problem, we propose a new pose estimation method that uses neural networks. We reduce the complexity of the architecture of the neural networks used (the number of variables to estimate) by proving that the location and the orientation of the robot can be ensured by using a single point in the image coordinate space for mobile robots with two degrees of freedom. To show the efficiency of the proposed algorithm, we use the RVCTOOLS MATLAB toolbox.

1. Introduction

There are two main approaches in visual servoing: Position-Based Control (PBC) (or 3D visual servoing) [1,2] and Image-Based Control (IBC) (or 2D visual servoing) [3]. The error function in PBC is computed in the Cartesian coordinate system, while in IBC, this error is measured in the image coordinate system. In this paper, we are interested in 3D visual servoing in the case of a mobile robot equipped with a perspective camera. The generated control law must minimize the error between the instantaneous position of the robot and a desired position. These two positions are calculated (estimated) from the observations of the camera. Features are extracted from images. An ideal model of the object (target) is generated and used to determine its position. The camera/robot position is deduced according to the intrinsic and extrinsic parameters of the camera, namely, through pose estimation. Several works have attempted to solve the pose estimation problem [1,2,3,4,5]. In general, proposed methods exploit the 2D or 3D object (target) model and the camera’s intrinsic parameters. They exploit various types of visual information such as points [6], rights [7], etc. We can consider these methods as static methods or approaches. Other attempts of pose estimation explore the dynamic aspect of the whole servoing problem. They use 3D reconstruction by dynamic vision techniques; they allow one to estimate the object model or to determine the camera’s location based on the 2D or 3D movement of the robot [8,9]. However, regardless of the pose reconstruction method used, pose estimation is very sensitive to measurement noise and visual sensor calibration errors. This generates, after the convergence of servo control (when the steady state is reached), a bias between the position of the robot and the object of interest to be reached. It is, in fact, the major problem of 3D visual servoing. To deal with this problem, we perform the pose estimation by using neural networks. Generally, to solve the problem of pose estimation, we need at least three points in the image [1] (six neurons in the input layer), which can generate an excessive number of neurons in the hidden layer. To reduce this number, the concept of differential flatness is used: we proved in [10] that, in the case of a two-degrees-of-freedom mobile robot, the variation in a single point in the picture coordinate space can be used to ensure the robot’s position and orientation in relation to the target.

In PBC, the camera trajectory is directly controlled in Cartesian coordinate space. There are, however, some issues, including the lack of control over the image space, which means that the object may leave the field of view of the camera during navigation, as well as the pose estimation algorithm generally causing errors [11]. Using the flatness concept [12,13,14,15,16,17], we propose a new approach to solve these problems. While servoing, the object leaves the camera’s field of view. At this moment, we propose implementing a blind motion of the robot by using an open-loop control strategy that relies on the flatness principle. Robust feedback control is required to reject the error pose estimation in the input. The path planning problem is, in general, difficult to solve due to the need to solve the differential system equation. The flatness principle can be used to fix this problem without approximation or to solve the differential system equation. In fact, the flatness property guarantees the existence of a flat output, allowing all system variables to be parameterized as a function of a finite number of derivatives [18]. We use the RVCTOOLS MATLAB toolbox to demonstrate the efficiency of the proposed algorithm.

This paper is organized as follows: Section 2 focuses on the 3D visual servoing problem. Section 3 introduces the flatness control theory. The neural network architecture used to estimate the robot’s pose is described in Section 4. Section 5 presents a dynamic model of the mobile robot and the integration of the flatness control technique into this configuration. Section 6 and Section 7 show robot path planning and robot path tracking. Finally, experimental results are presented in Section 8 using the RVCTOOLS MATLAB toolbox.

2. Three-Dimensional Visual Servoing Problem

In the case of 3D visual servoing, the function to be minimized is expressed in the Cartesian coordinate system. The location (position and orientation) of the camera denoted as r is deduced from the image data. The main task of 3D visual servoing is to reach the desired situation denoted by $r\ast$. The operating principle of 3D visual servoing is described in Figure 1. The advantage of a 3D measurement is to be able to specify the trajectory of the camera in the space where it is most easily described, which also makes it possible to ensure a feasible trajectory of the camera in space and especially in the presence of obstacles.

One of the major problems in 3D visual servoing is determining the relative pose and orientation of the observed object or target. In a formal way, the pose estimation is achieved by matching image pixels with space points: 2D–3D correspondence. This estimation is carried out by using variables deduced from points, regions or curves.

To obtain “good” pose estimation, different filters are generally used to estimate the translation and rotation parameters. It should be noted here that regardless of the pose reconstruction method used, the depth estimate is highly sensitive to measurement noise and errors caused by the calibration of the visual sensor. After convergence of the servo control and once the steady state is reached, we have a bias between the position of the robot and the object of interest to be reached. Another major problem with 3D visual servoing is that there is no control in the image plane. The target may leave the field of view during navigation. We propose to solve these two problems using the concept of differential flatness.

3. The Flat System

Consider the following differential equation for a nonlinear system:

$$\dot{x}=f(x,u),$$

(1)

where the state vector is $x\in I{R}^n$ and the input vector is $u\in I{R}^m$. The considered system is a differentially flat system if and only if a variable $z\in I{R}^m$ exists. It takes the following form:

$$z=h(x,u,\dot{u},\dots ,u^{(r)}).$$

(2)

Therefore, the system state inputs are given by:

$$x=A(z,\dot{z},\dots ,z^{(\alpha )}),$$

(3)

$$u=B(z,\dot{z},\dots ,z^{(\alpha +1)}),$$

(4)

where α is an integer. z is the flat output of the considered system called the endogenous variable. It enables the parameterization of any variable in the system. The real output is written using the flat output as:

$$y=C(z,\dot{z},\dots ,z^{(\sigma )}),$$

(5)

where $\sigma$ is an integer. This allows for computing system trajectories utilizing output trajectory definition without resolving any differential equations.

3.1. Mobile Robot Model

The model of the mobile robot that has been considered can be seen in Figure 2.

L: axis length;

R: radius of each wheel;

${\dot{\phi }}_1,{\dot{\phi }}_2$: wheels’ velocities;

$P(x_R,y_R):$ robot position;

$\theta$: robot’s orientation angle;

$R_r$: robot coordinate system;

$R_c$: camera coordinate system.

The kinematic model for the considered mobile robot can be written as follows (without taking the sliding phenomenon into account):

$$\{\begin{array}{l}{\dot{x}}_R=u_1\cos (\theta )\\ {\dot{y}}_R=u_1\sin (\theta )\\ \dot{\theta }=u_2\end{array}$$

(6)

where:

$$\left(\begin{array}{l}{u}_1\\ u_2\end{array}\right)=\left(\begin{array}{cc}\frac{R}{2}& \frac{R}{2}\\ \frac{-R}{L}& \frac{R}{L}\end{array}\right)\left(\begin{array}{l}{\omega }_1\\ {\omega }_2\end{array}\right)\mathrm{and}{\dot{\phi }}_1={\omega }_{1}R,{\dot{\phi }}_2={\omega }_{2}R.$$

${\omega }_1$ and ${\omega }_2$ are the rotation speeds of the wheels.

3.2. Pose Estimation

To express the relationship between the position and the orientation of the robot with respect to the target object (the pose), we will use the principle of 2D visual servoing. Two-dimensional visual servoing utilizes the relationship between the camera velocity $\left(V\right)$ and the time variation in the visual information ($\dot{S}$). This relationship is expressed by Equation (7)

$$\dot{S}=L_{s}V,$$

(7)

where $L_s$ is the interaction matrix, also noted by the image Jacobian which links the camera’s instantaneous velocity $V$ to the time variation in the visual information $S$. If we note:

$$e\left(t\right)=S\left(t\right)-S^{\ast }\left(t\right),$$

(8)

using (7) and (8), it is possible to obtain the relationship between the camera velocity and the time variation in the error between $S$ and the desired visual information $S^{\ast }$ as:

$$\dot{e}=L_{s}V,$$

(9)

where $L_s$ is the interaction matrix (image Jacobian), defined as:

$$L_s=\left[\begin{array}{cccccc}-\frac{1}{z}& 0& \frac{x}{Z}& xy& -\left(1+x^2\right)& y\\ 0& -\frac{1}{Z}& \frac{y}{Z}& 1+y^2& -xy& -x\end{array}\right],$$

(10)

where $x$ and $y$ are feature coordinates expressed in the 2D image space and given by:

$$\{\begin{array}{c}x=\frac{X}{Z}=\frac{\left(u-c_u\right)}{f·\alpha }\\ y=\frac{Y}{Z}=\frac{\left(v-c_v\right)}{f}\end{array}$$

(11)

where $\mathit{X}=\left(X,Y,Z\right)$ is the 3D point coordinates in the camera coordinate system; $m=\left(u,v\right)$, expressed in pixels, provides the coordinates of the image point; $a=\left(c_u,c_v,f,\alpha \right)$ is the camera intrinsic parameters set, of which $c_u$ and $c_v$ are the principal point coordinates; $f$ is the focal distance; and $\alpha$ is the ratio of the pixel dimensions.

In this paper, we consider $S=\left(x,y\right)$ to be the coordinates of the point in the image plane coordinate system. We test if $S=\left(x,y\right)$ is a flat output of the considered system. According to Equation (7), we have:

$$V=L_s^{+}\dot{S},$$

(12)

where $V={\left({\dot{x}}_R,{\dot{y}}_R,0,0,0,\dot{\theta }\right)}^T$ is the velocity of the camera, and $L_s^{+}$ is the pseudo-inverse of the interaction matrix. $V={\left({\dot{x}}_R,{\dot{y}}_R,0,0,0,\dot{\theta }\right)}^T$ can be expressed using $S$ and its derivative. Thus, we can write:

$$V=h\left(S,\dot{S}\right),$$

(13)

where $h$ is a nonlinear function ${ℝ}^2\times {ℝ}^2\to {ℝ}^6$ and can be expressed analytically. $\left({\dot{x}}_R,{\dot{y}}_R,\dot{\theta }\right)$ can be also expressed according to $S$ and its derivative. Hence, we can write:

$$\theta =arctan\left(\frac{{\dot{y}}_R}{{\dot{x}}_R}\right)=f_1\left(S,\dot{S}\right),$$

(14)

$$u_1=\sqrt{{\dot{x}}_R^2+{\dot{y}}_R^2}=f_2\left(S,\dot{S}\right),$$

(15)

$$u_2=\frac{{\ddot{y}}_R{\dot{x}}_R-{\dot{y}}_R{\ddot{x}}_R}{{\dot{x}}_R^2+{\dot{y}}_R^2}=f_3\left(S,\dot{S}\right),$$

(16)

where $f_1,f_2,f_3$ are three nonlinear functions ${ℝ}^2\times {ℝ}^2\to {ℝ}^6$, which can be deduced from (13). Finally, $S=\left(x,y\right)$ are flat outputs of the considered system. We conclude that the location and the orientation of the robot can be deduced using the coordinates of a single point in the image [10].

4. Neural Network Architecture

The use of neural networks considerably reduces the complexity of the visual servoing algorithm during its real-time implementation since the system recovers its new position thanks to previously carried out offline learning. According to the theorem announced in [10], we will only consider a single point in the initial image at $t$ and its correspondent in the image taken at $t+\Delta t$, as shown in Figure 3. $P\left(x\left(t\right),y\left(t\right)\right)$: coordinates of $P$ in the image coordinate system at $t$. $Pf\left(xx,yy\right)$: coordinates of $P$ in the image coordinate system at $t+1$ (after Translation $T$ and Rotation $R$) with: $xx=x\left(t+1\right)=x\left(t\right)+{\Delta }_x\left(t\right)$ and $yy=y\left(t+1\right)=y\left(t\right)+{\Delta }_y\left(t\right)$. $\left({\Delta }_x\left(t\right),{\Delta }_y\left(t\right)\right)$ being the variation of $P$ coordinates in the image coordinate system considered at coordinates in the image coordinate system considered at $t$. Finally, $\Delta t_{xR}$, $\Delta t_{yR}$ and $\Delta R_{zR}$ are, respectively, variations of displacement according $x_R,y_R$ and rotation according $z_R$ of the robot in the Cartesian coordinate system. Figure 4 shows the proposed neural network architecture. Figure 5 shows a superposition of two images: the first with a small square representing the target observed by the camera when the robot is in its initial position ($S\left(t\right)$) and the second with a big square representing the target observed by the camera when the robot is in its final position (desired visual information $S^{\ast }\left(t\right)$). Our proposed algorithm must generate a control law that allows the robot to “see” the target (the square) as it is presented by the large square from an initial view represented by the small square. The blue point is the only point considered to perform the pose estimation. The circles represent trajectories of different points in the image coordinate system. Figure 6, Figure 7 and Figure 8 show very high concordance between the actual and the estimated translations and rotation pose parameters.

5. Control Law Design

The objective of the command is to ensure an asymptotic pursuit of a desired trajectory noted $\left(x^{\ast },y^{\ast }\right)$ capable of rejecting some additive disturbances, beginning first of all by demonstrating that the point $P$ of coordinates $P\left(x\left(t\right),y\left(t\right)\right)$ can be considered as a flat output of the studied system. Indeed, we express all state and control variables in terms of $x$ and $y$ coordinates and their derivatives. Let:

$$\theta =\arctan \left(\frac{\dot{y}}{\dot{x}}\right)$$

(17)

$$u_1=\sqrt{{\dot{x}}^2+{\dot{y}}^2}$$

(18)

$$u_2=\frac{\ddot{y}\dot{x}-\dot{y}\ddot{x}}{{\dot{x}}^2+{\dot{y}}^2}$$

(19)

Since a non-reversible relation between flat inputs and outputs is obtained, we introduce an auxiliary input which is the derivative with respect to the time of command $u_1$, i.e.,

$${\dot{u}}_1=\frac{\dot{x}\ddot{x}+\dot{y}\ddot{y}}{\sqrt{{\dot{x}}^2+{\dot{y}}^2}}$$

(20)

This auxiliary input makes the relationship between the input vector and the flat output vector reversible. This can be written in the following form:

$$\left(\begin{array}{c}{\dot{u}}_1\\ u_2\end{array}\right)=\left(\begin{array}{cc}\frac{\dot{x}}{\sqrt{{\dot{x}}^2+{\dot{y}}^2}}& \frac{\dot{y}}{\sqrt{{\dot{x}}^2+{\dot{y}}^2}}\\ \frac{-\dot{y}}{{\dot{x}}^2+{\dot{y}}^2}& \frac{\dot{x}}{{\dot{x}}^2+{\dot{y}}^2}\end{array}\right)\left(\begin{array}{c}\ddot{x}\\ \ddot{y}\end{array}\right)$$

(21)

In this case, we use exact linearization based on the concept of differential flatness presented by [18]. Thus, we replace $\dot{x}$ and $\dot{y}$ by their respective desired values ${\dot{x}}^{\ast }$ and ${\dot{y}}^{\ast }$. The linearization process gives the following expression:

$$\left(\begin{array}{c}{\dot{u}}_1\\ u_2\end{array}\right)=\left(\begin{array}{cc}\frac{{\dot{x}}^{\ast }}{\sqrt{{\dot{x}}^2+{\dot{y}}^2}}& \frac{{\dot{y}}^{\ast }}{\sqrt{{\dot{x}}^2+{\dot{y}}^2}}\\ \frac{-{\dot{y}}^{\ast }}{{\dot{x}}^{\ast 2}+{\dot{y}}^{\ast 2}}& \frac{{\dot{x}}^{\ast }}{{\dot{x}}^{\ast 2}+{\dot{y}}^{\ast 2}}\end{array}\right)\left(\begin{array}{c}\ddot{x}\\ \ddot{y}\end{array}\right)$$

(22)

The resulting linearized system is equivalent to a system which has two chains of integration of the following form:

$$\{\begin{array}{c}\ddot{x}={\vartheta }_x\\ \ddot{y}={\vartheta }_y\end{array}$$

(23)

where ${\vartheta }_x$ and ${\vartheta }_y$ are two auxiliary control inputs to be specified and which ensure the asymptotic pursuit of the desired trajectory.

6. Path Planning

Suppose we want to move the robot from its starting position $\left(x_i^{\ast },y_i^{\ast }\right)$ at $t_i$ to a final position $\left(x_f^{\ast },y_f^{\ast }\right)$ at $t_f$ and that it must avoid an obstacle by passing through a critical position. Let us take for example the point with coordinates $\left(\frac{x_f^{\ast }+x_i^{\ast }}{2},2y_f^{\ast }-y_i^{\ast }\right)$, which represents the maximum point of the curve between $y_i^{\ast }$ and $y_f^{\ast }$. The proposed trajectory can then be described by Figure 9.

The desired trajectory $y^{\ast }\left(x^{\ast }\right)$ must therefore satisfy the following constraints:

$$\begin{gathered} y^{\ast }\left(x_i^{\ast }\right)=y_i^{\ast }, \\ {y}^{\ast }\left(x_f^{\ast }\right)=y_f^{\ast }, \\ {y}^{\ast }\left(\frac{x_f^{\ast }+x_i^{\ast }}{2}\right)=2y_f^{\ast }-y_i^{\ast }, \\ \frac{d{y}^{\ast }}{d{x}^{\ast }}\left(\frac{x_f^{\ast }+x_i^{\ast }}{2}\right)=0; \\ \frac{d^2{y}^{\ast }}{d^2{x}^{\ast }}\left(\frac{x_f^{\ast }+x_i^{\ast }}{2}\right)<0 \end{gathered}$$

(24)

As an example, we take the polynomial equation in $x^{\ast }$ given by Equation (25), which satisfies the constraints cited in (24):

$$y^{\ast }\left(x^{\ast }\right)=y_i^{\ast }+\left(y_f^{\ast }-y_i^{\ast }\right)\left(\frac{x^{\ast }-x_i^{\ast }}{x_f^{\ast }-x_i^{\ast }}\right)\left(9-12\left(\frac{x^{\ast }-x_i^{\ast }}{x_f^{\ast }-x_i^{\ast }}\right)+4{\left(\frac{x^{\ast }-x_i^{\ast }}{x_f^{\ast }-x_i^{\ast }}\right)}^2\right)$$

(25)

The construction of the variation of $x^{\ast }\left(t\right)$ must now satisfy the following boundary conditions:

$$\begin{gathered} x^{\ast }\left(t_i\right)=x_i^{\ast },{\dot{x}}^{\ast }\left(t_i\right)=0,\cdots ,x^{\ast \left(5\right)}\left(t_i\right)=0 \\ {x}^{\ast }\left(t_f\right)=x_f^{\ast },{\dot{x}}^{\ast }\left(t_f\right)=0,\cdots ,x^{\ast \left(5\right)}\left(t_f\right)=0 \end{gathered}$$

This results in the following polynomial of degree 11:

$$x^{\ast }\left(t\right)=x_i^{\ast }+\left(x_f^{\ast }-x_i^{\ast }\right){\sigma }^6\left(t\right)\left(462-1980\sigma \left(t\right)+3465{\sigma }^2\left(t\right)-3080{\sigma }^3\left(t\right)+1386{\sigma }^4\left(t\right)-252{\sigma }^5\left(t\right)\left(t\right)\right)$$

(26)

where:

$$\sigma \left(t\right)=\frac{t-t_i}{t_f-t_i}$$

(27)

7. Path Tracking

The endogenous dynamic loop calculated in Section 5 allows us to define the new input as follows:

$${\vartheta }_x=-\left(\frac{k_2^x{s}^2+k_1^{x}s+k_0^x}{s\left(s+k_3^x\right)}\right)\left(x-x^{\ast }\right)+{\ddot{x}}^{\ast }$$

(28)

$${\vartheta }_y=-\left(\frac{k_2^y{s}^2+k_1^{y}s+k_0^y}{s\left(s+k_3^y\right)}\right)\left(y-y^{\ast }\right)+{\ddot{y}}^{\ast }$$

(29)

where $k_i^x,k_i^y\in ℝ, i=0, 1, 2$. $k_i^x,k_i^y$ are chosen so that the polynomials of Equations (30) and (31) are Hurwitz polynomials:

$$P_x\left(s\right)=s^4+k_3^x{s}^3+k_2^x{s}^2+k_1^{x}s+k_0^x=0$$

(30)

$$P_y\left(s\right)=s^4+k_3^y{s}^3+k_2^y{s}^2+k_1^{y}s+k_0^y=0$$

(31)

ensuring an asymptotic pursuit of the desired trajectory $\left(x^{\ast },y^{\ast }\right)$.

In Section 5, Section 6 and Section 7, we assume that we have values of the coordinates of the point $P$. In a visual servo, the only sensor used is the visual sensor. In order to determine the values of the coordinates of this point, we will use a perspective camera embedded in the robot.

When moving the robot, two cases are possible: The first is that the object does not leave the field of view of the camera during the servoing. Estimating the robot’s situation is then possible by using images from the camera and developing a pose estimation algorithm. The visual servo control diagram is given in Figure 10. The second possible case is when the object leaves the field of view of the camera. In this case, we cannot estimate the situation (pose) of the robot from the acquired images and we will therefore consider a blind movement of the robot thanks to an open-loop control described by Figure 11. Once the object returns to the field of view of the camera, the command developed in the first case is then reconsidered.

8. Experimental Results

In order to show the efficiency of the proposed algorithm, we consider two examples of simulation illustrating the two possible cases: Case 1, where the object does not leave the field of view of the camera during the servoing, and Case 2, where the object can leave the field of view of the camera during the servoing. The simulations are carried out using the RVCTOOLS MATLAB toolbox.

The created platform includes a mobile robot, equipped with a perspective camera, wishing to reach an object identified by four points. The learning phase necessary for the estimation of the pose was carried out by the neural network of Figure 4. The number of neurons of the input layer is 2, 20 for the hidden layer and 3 for the output layer.

8.1. Case 1

In this first case, we deal with the planning and pursuit of a given desired trajectory in the Cartesian plane. During the passage of the robot equipped with a perspective camera, from an initial situation to a final situation, the target object does not leave the field of vision during the enslavement. The problem posed is then described in Figure 12. The desired trajectories are described by Equations (25) and (26). The coordinates of the initial position in the Cartesian coordinate system are given by: $\left(x_i,y_i,{\theta }_i\right)=\left(0 \mathrm{m},0 \mathrm{m},0.5404 \mathrm{rad}\right)$. The coordinates of the four corresponding points in the image plane are given by: $P_I=\left(\begin{array}{cc}402& 402\\ 408& 605\end{array} \begin{array}{cc}602& 602\\ 605& 408\end{array}\right)$. The final position in the Cartesian coordinate system is: $\left(x_f,y_f,{\theta }_f\right)=\left(1.5 \mathrm{m},0.1 \mathrm{m},0 \mathrm{rad}\right)$. The coordinates of the four corresponding points in the image plane are given by: $P_f=\left(\begin{array}{cc}291& 291\\ 384& 652\end{array} \begin{array}{cc}577& 577\\ 652& 384\end{array}\right)$. Pole placement is fixed by the following parameters: $k_3^x=3$, $k_2^x=8$, $k_1^x=3$, $k_0^x=8$, $k_3^y=3$, $k_2^y=8$, $k_1^y=3$, $k_0^y=8$. Figure 13 presents the evolution of the trajectory of the robot in the image plane. It is like a superposition of successive frames taken by the robot’s camera. It is clear that the object does not leave the field of view during navigation. Figure 14 presents the evolution of the trajectory of the robot in the Cartesian frame. It is clear that the tracking problem is solved through the proposed algorithm. Figure 15 and Figure 16 respectively represent the variations of the two commands to be applied to the robot. The proposed control laws make it possible to give physically realizable values. This result is obtained thanks to the pole placement technique, which imposes the dynamics of the closed-loop system.

8.2. Case 2

We consider in this second example the case where the object can leave the field of view during its displacement from the initial position to the final position. The problem is described in Figure 17. The numerical values of the parameters of the studied process are practically the same as those in Case 1, with the exception of the model of the trajectory $y^{\ast }\left(x^{\ast }\right)$, which must satisfy the four following conditions:

$$y^{\ast }\left(x_i^{\ast }\right)=y_i^{\ast },y^{\ast }\left(x_f^{\ast }\right)=y_f^{\ast },y^{\ast }\left(\frac{x_f^{\ast }+x_i^{\ast }}{2}\right)=4y_f^{\ast }-y_i^{\ast }\mathrm{and}\frac{d{y}^{\ast }}{d{x}^{\ast }}\left(\frac{x_f^{\ast }+x_i^{\ast }}{2}\right)=0$$

Figure 18 represents the evolution of the trajectory of the robot in the image frame. It is clear that the object leaves the field of view during the enslavement. Figure 19 represents the evolution of the trajectory of the robot in the Cartesian frame. Compared with Figure 14, we can easily see how the robot corrects its trajectory after missing the target and succeeds in reaching its desired trajectory; this is obtained thanks to an open-loop control based on flatness control. Figure 20 and Figure 21 respectively represent the variations of the two commands applied to the robot. The absence of the target followed by its recovery by the camera installed on the robot are visible in the fluctuations in the control law u₂ between the time frames 5 s and 8 s. It should be noted that the switching between the two control approaches (in closed loop and in open loop) is made between the instants $t=5.5 \mathrm{s}$ and $t=7.5 \mathrm{s}$. It should also be noted that the switchover does not cause any sudden changes in the operating states of the actuators, and consequently, no oscillation in the responses of the system is noticed.

9. Conclusions

In this paper, we proposed the planning and tracking of a trajectory for 3D visual servoing based on the flatness concept. We noted that 3D visual servoing presents two major problems: the estimation of the pose and the absence of control in the image plane. Regarding the problem of pose estimation, we used a single point in the image plane to determine the pose of the robot. A simplified architecture of a neural network that has only two inputs was implemented. The results of the pose estimation are very satisfactory. The second problem occurs when the object (target) leaves the field of view of the camera during the servoing. In this case, we proposed a new approach based on the concept of differential flatness. Indeed, as soon as the object leaves the field of vision of the camera, a blind movement of the robot using an open-loop control is considered. Since the pose estimation algorithms are based on a full observation of the target, this observation can fail due to occlusion phenomena and generate disturbances. A robust control was then proposed in order to reject these disturbances. The simulation results proved the effectiveness of the proposed method. Indeed, when the object leaves the field of view, the robot is able to reach its desired trajectory.