Visual Servo-Based Real-Time Eye Tracking by Delta Robot

Abstract

This work presents and validates an eye-tracking-based visual system for driving the delta robot. A delta robot is tracked by image processing based on vision servo control. The vision servo program is developed in C++ to perform image processing-based object detection. For image processing, Haar classifier-based methods are used. Finally, image processing and motion controller movements are integrated into one system to perform the visual servo-based motion of the end effector of the delta robot. Experiments are performed to validate the proposed method from the perspective of image processing. Moreover, this paper validates the kinematic analysis, which is vital for obtaining 3D information on the end-effector of the delta robot. The presented model can be implemented in eye clinics to facilitate ophthalmologists by replacing manual eye-checking equipment with automatic, unattended, computerized eye checkups.

1. Introduction

The delta robot has gained great attention in both the industry and academia as its mechanical structure provides many advantages including light weight, fast movement speed, accurate positioning, high accuracy and low cost [1]. These characteristics make it suitable for a variety of applications, i.e., pick and place, mechanical and electrical assembly tasks, CNC machines, 3D printing, medical equipment, pharmaceuticals and others [2,3,4,5,6,7,8,9]. Generally, a parallel delta robot is a three-degree-of-freedom (DOF) parallel robot that consists of active and passive arms that are kinematics chains. The active link is connected to a servo actuator and the passive link is connected to the active link and base plate (a fixture for attaching the end-effector); the whole body represents a continuous motion [10]. Its unique structure provides several advantages including high stiffness, high accuracy, precision, and low moving inertia [4], which has also gained researchers’ attention in terms of medical and pharmaceutical applications [11,12,13,14,15,16,17]. Apart from the operating room, these robots are also employed in clinical settings where they are used to automate manual, repetitive, and high-volume tasks to facilitate doctors and caregivers so that they can focus on their strategic tasks. There are several clinical applications in which robots are employed, among which one of the important applications is the examination of the eye, where an ophthalmologist deals with different types of diseases like near-sightedness, far-sightedness, cataracts, glaucoma, molecular degeneration, etc. During the examination, ophthalmologists use different types of equipment such as auto-refractometers to set the face position in front of the machine to obtain an accurate and precise reading of the eyes. Thus, for proper examination, doctors have to physically change face position according to the machine requirement or communicate with the patient to move the face or head up, down, left, or right, which is repetitive and time-consuming. By considering the growing demand for delta robots in the medical field, this work presents a three-degree freedom (3DOF) delta robot based on a vision-based control technique that can be used in eye clinics to facilitate ophthalmologists during examination, i.e., right placement of the eyes/face of the patient. This research is carried out to design a delta robot that can move along X, Y, and Z coordinates, from position 1 to position 2 in the Cartesian coordinate system. The combination of the constrained motion of the three arms connecting the traveling plate to the base plate ensues in a resulting three-degree-of-freedom (DOF) delta robot. It is designed to give the position values to the delta robot, and then the motors of the robot will move to the position according to the given values. Numerous algorithms have been developed to estimate the position, speed, and orientation, track specific objects, and detect obstacles according to the requirements of the application [18,19,20,21,22,23,24,25,26]. This work is based on the vision control technique for eye tracking. Generally, vision-based control depends on the data acquired from a visual sensor (for example a camera) to track a specific object and to control the motion of a robot accordingly [27]. The use of vision control techniques can make the robot work more efficiently due to its improved adaptability to the external environment. Herein the model is developed to detect the face first by the camera, and then to detect the eyes. Afterward, the position of the eyes is sent to the controller, and hence the controller gives a command to the delta robot; as a result, the delta robot end-effector will move to the given position (eye position). During this process, if the subject starts changing its position in any direction, the camera will detect the change in position of the face and eyes and a new position will be generated by the camera. For this change in position, the controller will again send a command to the robot to obtain the new location of the eyes. So, this is not only precise eye detection but also real-time eye tracking, which is a suitable option to be employed in eye clinics because, during diagnosis and examination, the ophthalmologist must physically and verbally interact with patients about their correct face positioning to obtain precise eye readings.

However, further improvement of the model can be achieved in eye tracking, false detection of faces and eyes, and precise position tracking.

2. Robot Architecture

2.1. Design of the Delta Parallel Structure

Herein, the delta model is structured to move along XYZ and is designed to reach the operation at three degrees of freedom to achieve a better position. The robot consists of actuators (three motors), a base plate, an upper robot arm, a lower robot arm (forearm), and a traveling plate TCP. In this work, the analysis of the delta robot’s three degrees of freedom is designed and structured mechanically with three motors mounted on the fixed frame of the hardware structure. The structure is formed with one fixed base plate on which motors are mounted and the end effector, which is moveable by linking the upper arm and forearm with the motor’s fixed plate. The model is driven by a Galil controller. The Galil motion controller controls three motors simultaneously mounted in the frame structure of the delta robot at an angle of 120 degrees apart. The motors attached to the mechanical structure are connected with the Galil controller via connecting wires and communication lines; simultaneously, the Galil motion controller is connected to a PC through Galil suite (0.4.8.734) software installed on the computer. After completing hardware and connections, the motion controller is assigned programming code developed for the delta robot in C++; the resulting code will be downloaded to the Galil controller, and hence the software will drive three motors according to the C++ program. Assigning programming code to the microcontroller, the controller will be able to control three motors according to the program. The delta parallel structure is designed in SolidWorks, and the main mechanical design indexes are shown below (

Table 1. Dimensions of delta parallel structure.

Translational Workspace	Rotation Range	Linear Motion Speed	Angular Motion Speed	Feedback Force	Torque
120 × 100 mm	±60	8 mm/s	10 r/min	20 N	0.3 N·m

).

The architecture of the delta parallel robot is demonstrated in Figure 1, which is mainly composed of a fixed disc, three moving primary arms, and three moving secondary arms connected with the moving end of the delta parallel structure. Due to the limitation of parallel quadrangles from the arm structure, the moving platform can only carry out three-dimensional translational motion.

2.2. Kinematics Analysis

The motion of the end-effector of the delta mechanism is driven by three driving components at the static platform. The angle of the end-effector can be calculated by the angle of rotation of three driving components, and the position of the end-effector can be known. The forward and inverse kinematics for obtaining the position of the delta robot are presented as follows [28,29].

2.2.1. Forward Kinematics

For forward kinematics, the angle of each driving component is $({\theta }_1,{\theta }_2,{\theta }_3)$ to compute $O\prime (X_d,Y_d,Z_d)$. The analytical expression of the positive solution is obtained by using the vector loop equation; the most suitable result can be calculated as

$$E_i{P}_i=OO\prime +O{P}_i-O{B}_i-B_i{E}_i$$

(1)

With known vectors, the three equations are as follows:

$$\{X_d+l_1\cos ({\theta }_i)\cos ({\eta }_i)-[R-r]\mathrm{c}\mathrm{o}{\mathrm{s}}^2({\eta }_i)\}+\{Y_d+l_1\cos ({\theta }_i)\sin ({\eta }_i)-[R-r]\}+\{Z_d+l_1\mathrm{s}\mathrm{i}{\mathrm{n}}^2({\theta }_i)\sin ({\theta }_i)\}$$

(2)

where i = 1, 2, 3.

After simplification,

$$X_i=-[r\cos ({\theta }_i)-R\cos ({\theta }_i)=l_1\cos ({\eta }_i)\cos ({\theta }_i)]$$

(3)

$$Y_i=-[r\cos ({\theta }_i)-R\cos ({\theta }_i)=l_1\cos ({\eta }_i)\sin ({\theta }_i)]$$

(4)

$$Z_i=l_1\sin ({\theta }_i)$$

(5)

By simplification, we get the solution of the end effector as follows:

$$X_d=m_1+n_1{z}_1$$

(6)

$$Y_d=m_2+n_2{z}_d$$

(7)

$$Z_d={\displaystyle \frac{\left(-F+\sqrt{F^2-4E{G}^2}\right)}{2E}}$$

(8)

The solution of the end effector (Equations (6)–(8)) is the forward kinematics of the delta parallel robot in terms of XYZ coordinates, as illustrated in Figure 2.

2.2.2. Inverse Kinematics

For the basic operating mechanism, the solution for inverse kinematics is used to find the three control variables for the motors after the coordination of the central point of the moving platform related to the static coordinate frame has been decided. It is important to solve the inverse kinematics; thus, the end-effector can move to the desired position. A geometrical representation of the delta robot is illustrated in Figure 3.

The indirect kinematics determines the active joint angles ${\mathsf{\varphi }}_i$ based on a given position $\left(X_p,Y_p,Z_p\right)$ of the end effector. By using position vectors in coordination frames, we get

$$x^2+{[(R+L_b\sin {\theta }_3-r)+y]}^2+{(L_b\cos {\theta }_3+z)}^2=L_a^2$$

(9)

After a series of simplifications, we can get a quadratic equation with one unknown,

$$K_i{t}_i^2+U_i{t}_i+V_i=0$$

(10)

where $(i=1,2,3)$,

$$t_i=\tan ({\displaystyle \frac{1}{2}}{\theta }_i)$$

(11)

$$\left[\begin{array}{l}{K}_i\\ U_i\\ V_i\end{array}\right]=\left[\begin{array}{l}{\displaystyle \frac{L_a^2-L_b^2-x^2-y^2-z^2-{\left(R-r\right)}^2+\left(R-r\right)\left(\sqrt{3}x+y\right)}{L_b}}+2z\\ -2\left[2\left(R-r\right)-\sqrt{3}x-y\right]\\ {\displaystyle \frac{L_a^2-L_b^2-x^2-y^2-z^2-{\left(R-r\right)}^2+\left(R-r\right)\left(\sqrt{3}x+y\right)}{L_b}}-2z\end{array}\right]$$

(12)

$K_i,U_i,V_i$ are known; then,

$$t_i={\displaystyle \frac{-U_i\pm \sqrt{U_i^2-4K_i{V}_i}}{2K_i}}(i=1,2,3)$$

(13)

During the delta robot operation, after setting the pose of the moving frame, the input angle to each motor can be calculated by

$${\theta }_i=2\arctan (t_i)$$

(14)

The inverse kinematics of the delta parallel robot in terms of ${\theta }_i$ is displayed in Figure 4. The angle of the driving component can be obtained by the upper form, but there may be a combination of eight different solutions, which does not exist. The uniqueness of the solution must be eliminated by adding certain constraints, and the most suitable solution is obtained. The delta parallel mechanism is only small when the bar is out of the way, and the rod is not easily interfered with. Therefore, the elimination strategy in this paper is that the driving angle of each driving rod is the most medial.

2.3. Verification of Kinematics Solution

To make the subsequent analysis process more accurate and reliable, the actual three-dimensional model is used to verify the calculation method of the positive and negative solutions. Firstly calibration was performed. The upper arm was put parallel with the base plate to set angles of rotation of the motors. This process was repeated for three upper arms to set three motor angles to achieve calibration at the initial stage. Afterward, the end-effector was set to be positioned at the center, parallel to the base plate, by moving the end-effector relative to the base fixed plate. This process of calibration is necessary to obtain accurate simulation results and hence compare them with real-time values. The verification result of the forward kinematics solution is demonstrated in Figure 5.

The Matlab function input angle is the three-joint angle of the actual reading of SolidWorks 3D, and the correctness of the forward kinematic solution is verified by comparing the input coordinates of these two methods. In order to achieve further accuracy in calculation, a random set of readings was given to both Matlab (R2018a) and SolidWorks (2018), and the calculated error is shown in

Table 2. SolidWorks and Matlab software computing results.

Serial Number (sr)	Example			SolidWorks 3D (mm)			Matlab (mm)			Error
	${\mathit{\theta }}_1$	${\mathit{\theta }}_2$	${\mathit{\theta }}_3$	X	Y	Z	X	Y	Z	Δ
1	−30	−30	−30	0	0	124.14	0	0	124.14	0
2	7.99	0	8.44	2.43	3.31	151.41	2.44	3.31	151.41	0.01
3	59.98	74.16	23.6	−19	−103.8	277.6	−18.97	−103.8	277.63	0.121
4	19.1	−14	−12	−43.57	2.56	160.74	−43.58	2.57	160.74	0.014
5	−6.2	23.86	6.6	33.54	−24.6	183.42	33.54	−24.59	183.42	0.01
6	67.03	61.91	18.8	−62	−82.25	270.4	−62.09	−82.24	270.42	0.017
7	3.33	−0.63	41.1	30.92	64.38	191.82	30.93	64.39	191.82	0.014
8	24.84	46.44	43.2	44.71	−6.89	261.39	44.71	−6.9	261.39	0.01
9	76.39	75.4	75.6	−2.3	0.49	342.81	−2.03	0.48	342.81	0.01
10	24.17	44.86	47.3	49	5.23	262.1	49.02	5.24	262.14	0.014
11	22.61	63.6	18	43.4	−90.2	235	43.38	−90.2	235.03	0.01
12	37.03	73.61	25.6	36.1	−102.5	259.5	36.1	−102.5	259.53	0.01
13	53.82	26.47	49.8	−37	45.79	271.8	−36.95	45.8	271.8	0.014
14	27.53	61.65	68.3	87	16.46	284.6	87.03	16.45	284.63	0.017

. The coordinates in the calculated position are in good agreement with a maximum error of no more than 0.1mm.

3. System Implementation

In this work, the model is designed with one fixed base plate on which motors are mounted and an end-effector that is moveable by linking the upper arm and forearm with the motor’s fixed plate. The model is driven by a Galil controller which controls three motors simultaneously that are mounted in the frame structure of the delta robot at an angle of 120 degrees apart. The attached motors are connected to the Galil controller via connecting wires and communication lines; simultaneously, the Galil motion controller is connected to a PC through the Galil suite software installed on the computer. C++ programming code is developed for the delta robot to drive and control three motors. In the first part of the model, the subject of the delta robot with a three-axis exchange delta parallel mechanism observes translational motion with three degrees of freedom (DOF) along the XYZ Cartesian coordinate system.

In the second part of the modeling, the desired position of the robot is taken by the camera, which will send the position to the robot by the Galil motion controller camera associated with the delta parallel mechanism structure. The mechanism is based on face detection first, after which the program will jump to detecting the eyes of the object. After the successful detection of eyes, the program finds the eye position according to the camera resolution (that will be the frame of reference to calculate XYZ parameters), resulting in the end-effector of the delta robot moving to drive to the detected eyes.

For control technique, this work used position tracking methodology by employing the Galil controller, which records the position of the actuators by encoders. If the target given to the program is reached, the system collects continuous information from the encoder and hence records the path trajectory. The controller will calculate and record the acceleration, deceleration, and speed parameters. The kinematics for the delta robot is developed in C++ code. According to that kinematics, we have to input XYZ to obtain $({\theta }_1,{\theta }_2,{\theta }_3)$ for each motor. The positioning values of the three motors are recorded as per the calculation of the kinematics.

4. Results and Discussion

4.1. Position Tracking of Delta Robot

Inverse kinematic analysis-based C++ code is developed using Microsoft Visual Studio 15.5. The position XYZ is given to the controller as an input to obtain $({\theta }_1,{\theta }_2,{\theta }_3)$ as output. Figure 6 demonstrates the motion of the end-effector of the delta robot, which is the angular output of the Galil motion controller; however, the positional input for the controller is given by the camera.

4.2. Eye Tracking

For this work, eyes are taken as a target for object detection using image processing. The code uses the Haar cascade classifier technique for eye detection. In this technique, firstly an image frame is taken by the camera and a classifier is applied to it for face detection and then eye detection. If the result is true in both cases, then the position of eyes is measured and sent to the Galil motion controller as an input to the kinematic algorithm, resulting in the movement of the end-effector of the robot to the detected eye position. But if the result is false or the eyes go out of the frame, the Haar cascade classifier receives a false result and the image window will pause until the eyes come into the range. The output in 2D position before implementing image processing on the controller and delta robot is demonstrated in Figure 7a. In Figure 7b, we validate eye tracking after implementing the image processing-based technique, where the detailed face detection, eye detection and tracking process can be observed. As shown in the figure, the object detection using a camera features real-time tracking of eyes for the delta robot. By using a camera, the relative position of the eyes is recorded and sent to the Galil motion controller, which saves the recorded values sent by the encoder of each motor.

4.3. Experimental Validation

The position taken by the camera is an input to the delta robot; this position is encoded in terms of XYZ and decoded in terms of $({\theta }_1,{\theta }_2,{\theta }_3)$. For verification, this angular information of each motor is recorded by the encoder to the motion controller, and then the same data is analyzed in MATLAB to observe the real-time curve of the XYZ position simultaneously. The real-time position tracking of the delta robot along the X, Y and Z axes is verified in Figure 8.

4.4. Dual Operating Characteristics

This model is designed to be operated in two modes, i.e., active and passive mode. In active mode, position tracking of the eyes is automatically performed by the camera. However, the passive mode is especially used in the design parameters of the model: if tracking is not achieved by a camera, then the passive mode can be used to operate the delta robot manually. Manual operation of the delta robot means tracking the delta robot by giving the position to the end-effector with a keyboard. In the medical domain, higher accuracy and precision are required to use a machine on a patient, so herein two different interfaces for control are used; i.e., one is by the use of a camera through which real-time eye tracking can be achieved, and another is by the use of a keyboard to input the position of the end-effector of the delta robot manually.

5. Conclusions

This paper presents a delta robot for real-time continuous eye tracking by using the Galil motion controller, which can be used to facilitate ophthalmologists during the eye-checking process. The delta robot consists of three actuators for moving its end effector and is designed to move along X, Y, and Z coordinates, from position 1 to position 2 in the Cartesian coordinate system. The C++ vision servo program is designed to carry out object detection through image processing using Haar classifiers. Finally, the visual servo-based motion of the delta robot’s end-effector is accomplished by combining motion controller movements and image processing into a single system. Experiments are conducted to assess the effectiveness of the developed model in terms of image processing. Furthermore, the kinematic analysis, which is essential for acquiring three-dimensional data on the delta robot, is validated in this paper. The model is developed to have dual operating modes; one is active mode and another is passive mode. In active mode, position tracking is performed by a camera, i.e., automatic tracking; however in passive mode, position tracking is achieved by a keyboard, i.e., manual tracking. This model can be implemented to facilitate eye specialists by replacing manual eye-checking equipment with automatic, unattended, computerized eye checkups by the use of an auto-refractometer.