# Hand–Eye Calibration Using a Tablet Computer

## Abstract

**:**

## 1. Introduction

## 2. Related Works

## 3. Proposed Method

#### 3.1. Overview

#### 3.2. Coordinate System and Homogeneous Transformation Matrix (HTM)

#### 3.3. Transformation from ${\mathsf{\Sigma}}_{t}$ to ${\mathsf{\Sigma}}_{b}$

#### 3.4. Representation by DH Method

#### 3.5. Parameters to Be Optimized

#### 3.6. Two-Stage Optimization

#### 3.6.1. First Optimization

#### 3.6.2. Second Optimization

## 4. Experiment

#### 4.1. Used Robot and Devices

#### 4.2. Data Creation

#### 4.3. Set Values for Known Parameters

#### 4.4. Setup for DE

## 5. Results and Consideration

#### 5.1. First-Stage Optimization

#### 5.2. Second-Stage Optimization

## 6. Conclusions

## Funding

## Conflicts of Interest

## References

- Lin, W.; Liang, P.; Luo, G.; Zhao, Z.; Zhang, C. Research of Online Hand-Eye Calibration Method Based on ChArUco Board. Sensors
**2022**, 119, 3805. [Google Scholar] - Hua, J.; Zeng, L. Hand-Eye Calibration Algorithm Based on an Optimized Neural Network. Actuators
**2021**, 10, 85. [Google Scholar] [CrossRef] - Enebuse, I.; Foo, M.; Ibrahim, B.S.K.K.; Ahmed, H.; Supmak, F.; Eyobu, O.S. A Comparative Review of Hand-Eye Calibration Techniques for Vision Guided Robots. IEEE Access
**2021**, 9, 113143–113155. [Google Scholar] [CrossRef] - Jiang, J.; Luo, X.; Luo, Q.; Qiao, L.; Li, M. An Overview of Hand-Eye Calibration. Int. J. Adv. Manuf. Technol.
**2022**, 22, 77–97. [Google Scholar] - Shiu, Y.C.; Ahmad, S. Calibration of wrist-mounted robotic sensors by solving homogeneous transform equations of the form AX = XB. IEEE Trans. Robot. Autom.
**1989**, 5, 16–29. [Google Scholar] - Motai, Y.; Kosaka, A. Hand-Eye Calibration Applied to Viewpoint Selection for Robotic Vision. IEEE Trans. Ind. Electron.
**2008**, 55, 3731–3741. [Google Scholar] - Tsai, R.Y.; Lenz, R.K. A new technique for fully autonomous and efficient 3D robotics hand/eye calibration. IEEE Trans. Robot. Autom.
**1989**, 5, 345–358. [Google Scholar] - Wang, C.C. Extrinsic calibration of a vision sensor mounted on a robot. IEEE Trans. Robot. Autom.
**1992**, 8, 161–175. [Google Scholar] - Park, F.C.; Martin, B.J. Robot sensor calibration: Solving AX = XB on the Euclidean group. IEEE Trans. Robot. Autom.
**1994**, 10, 717–721. [Google Scholar] [CrossRef] - Ma, S.D. A self-calibration technique for active vision systems. IEEE Trans. Robot. Autom.
**1996**, 12, 114–120. [Google Scholar] - Daniilidis, K. Hand-Eye Calibration Using Dual Quaternions. Int. J. Robot. Res.
**1999**, 18, 286–298. [Google Scholar] - Horaud, R.; Dornaika, F. Hand-Eye Calibration. Int. J. Robot. Res.
**1995**, 14, 195–210. [Google Scholar] [CrossRef] - Andreff, N.; Horaud, R.; Espiau, B. Robot Hand-Eye Calibration using Structure from Motion. Int. J. Robot. Res.
**2001**, 20, 228–248. [Google Scholar] - Zhao, Z. Hand-eye calibration using convex optimization. In Proceedings of the IEEE International Conference on Robotics and Automation, Shanghai, China, 9–13 May 2011; pp. 2947–2952. [Google Scholar]
- Heller, J.; Havlena, M.; Pajdla, T. Globally Optimal Hand-Eye Calibration Using Branch-and-Bound. IEEE Trans. Pattern Anal. Mach. Intell.
**2016**, 38, 1027–1033. [Google Scholar] [PubMed] - Zhuang, H.; Roth, Z.; Sudhakar, R. Simultaneous robot/world and tool/flange calibration by solving homogeneous transformation equations of the form AX = YB. IEEE Trans. Robot. Autom.
**1994**, 10, 549–554. [Google Scholar] [CrossRef] - Mišeikis, J.; Glatte, K.; Elle, O.J.; Torresen, J. Automatic Calibration of a Robot Manipulator and Multi 3D Camera System. In Proceedings of the IEEE/SICE International Symposium on System Integration, Sapporo, Japan, 13–15 December 2016; pp. 735–741. [Google Scholar]
- Koide, K.; Menegatti, E. General Hand-Eye Calibration Based on Reprojection Error Minimization. IEEE Robot. Autom. Lett.
**2019**, 4, 1021–1028. [Google Scholar] [CrossRef] - Projection in Intel RealSense SDK 2.0. Available online: https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20 (accessed on 19 January 2023).
- DENSO ROBOT USER MANUALS. Available online: http://eidtech.dyndns-at-work.com/support/RC8_Manual/005929.html (accessed on 19 January 2023).
- Lynch, K.M.; Park, F.C. Modern Robotics: Mechanics, Planning, and Control; Cambridge University Press: Cambridge, UK, 2017. [Google Scholar]
- Das, S.; Suganthan, P.N. Differential Evolution: A Survey of the State-of-the-Art. IEEE Trans. Evol. Comput.
**2011**, 15, 4–31. [Google Scholar]

**Figure 3.**(

**a**) Positions of landmarks in the image coordinate system; (

**b**) positions of the hand in the robot-based coordinate system.

**Figure 4.**Coordinate systems. Red, green, and blue arrows represent x, y, and z axes, respectively. DENSO VP-6242 robot [20] is used.

**Figure 6.**(

**a**) Transferred landmarks from ${\mathsf{\Sigma}}_{i}$ to ${\mathsf{\Sigma}}_{b}$ and ${\mathsf{\Sigma}}_{t}$ to ${\mathsf{\Sigma}}_{b}$ using optimized parameters, respectively. (

**b**) Output hand positions using Equation (8) with optimized parameters and compensated ground truth using Equation (20).

**Figure 8.**(

**a**) Captured image of the tablet display by RGB-D camera; (

**b**) binarized image to detect displayed black dots.

Equation Number | Known | Unknown (Six-DoF HTM) | Unknown (DH Method) |
---|---|---|---|

(1) | ${x}_{h},{y}_{h},{z}_{h}$ | ||

(2) | ${x}_{c},{y}_{c}$, ${\alpha}_{c}$, ${\beta}_{c}$, ${\gamma}_{c}$ | ||

(3) | ${f}_{x}$, ${f}_{y}$, ${c}_{x}$, ${c}_{y}$, ${z}_{m}^{c}$ | ||

(6) | ${\theta}_{h}$ | ${x}_{p}$, ${y}_{p}$ | ${x}_{p}$, ${y}_{p}$ |

(11) | ${\alpha}_{t}$, ${\beta}_{t}$, ${\gamma}_{t}$, ${x}_{t}$, ${y}_{t}$ | ||

(13) | ${\alpha}_{c}$, ${\beta}_{c}$, ${\gamma}_{c}$ | ||

(14) | ${\theta}_{1}^{c}$, ${a}_{1}^{c}$ | ||

(15) | ${\alpha}_{t}$, ${\beta}_{t}$, ${\gamma}_{t}$ | ||

(16) | ${\theta}_{1}^{t}$, ${a}_{1}^{t}$ |

m | $({\mathit{u}}_{\mathit{m}},{\mathit{v}}_{\mathit{m}})$ in ${\mathbf{\Sigma}}_{\mathit{i}}$ [px] | ${\mathit{z}}_{\mathit{m}}^{\mathit{c}}$ in ${\mathbf{\Sigma}}_{\mathit{c}}$ [mm] | $({\mathit{x}}_{\mathit{m}}^{\mathit{r}},{\mathit{y}}_{\mathit{m}}^{\mathit{r}})$ in ${\mathbf{\Sigma}}_{\mathit{b}}$ [mm] | $(\mathbf{\Delta}{\mathit{x}}_{\mathit{m}}^{\mathit{r}},\mathbf{\Delta}{\mathit{y}}_{\mathit{m}}^{\mathit{r}})$ in ${\mathbf{\Sigma}}_{\mathit{t}}$ [px] |
---|---|---|---|---|

0 | (119, 105) | 168.2 | (289, 1) | (−3, 1) |

1 | (339, 108) | 168.6 | (347, 8) | (−2, 3) |

2 | (558, 109) | 167.7 | (403, 14) | (−1, 5) |

3 | (556, 246) | 167.5 | (395, −20) | (−1, 4) |

4 | (338, 244) | 169.2 | (327, −17) | (−3, −1) |

5 | (119, 242) | 168.7 | (257, −16) | (3, 3) |

6 | (118, 379) | 168.6 | (249, −59) | (2, 0) |

7 | (337, 379) | 170.2 | (305, −67) | (−1, 3) |

8 | (554, 381) | 168.9 | (363, −77) | (1, 2) |

Parameter | Value |
---|---|

$({x}_{h},{y}_{h},{z}_{h})$ | (320, −70, 290) |

$({f}_{x},{f}_{y})$ | (617.7, 617.7) |

$({c}_{x},{c}_{y})$ | (316.5, 242.3) |

${\theta}_{h}$ | $22.5\times m$ |

Hyperparameter | Value |
---|---|

N | 10,000 |

G | 10,000 |

CR | 0.9 |

F | 0.5 |

Crossover strategy | Binomial crossover |

Mutation strategy | DE/rand/1 |

Parameter | ${x}_{c}$ | ${y}_{c}$ | ${\alpha}_{c}$ | ${\beta}_{c}$ | ${\gamma}_{c}$ | ||

Search range | [−20, 20] | [20, 50] | [−30, 30] | [−30, 30] | [−30, 30] | ||

Parameter | ${x}_{p}$ | ${y}_{p}$ | ${\alpha}_{t}$ | ${\beta}_{t}$ | ${\gamma}_{t}$ | ${x}_{t}$ | ${y}_{t}$ |

Search range | [−20, 20] | [−40, −10] | [−30, 30] | [−30, 30] | [−30, 30] | [100, 200] | [0, 100] |

Parameter | ${\theta}_{1}^{c}$ | ${a}_{1}^{c}$ | ${\alpha}_{t}$ | ${\beta}_{t}$ | ${\gamma}_{t}$ | ||

Search range | [−30, 30] | [20, 100] | [−30, 30] | [−30, 30] | [−30, 30] | ||

Parameter | ${\theta}_{1}^{t}$ | ${a}_{1}^{t}$ | ${\alpha}_{t}$ | ${\beta}_{t}$ | ${\gamma}_{t}$ | ${x}_{p}$ | ${y}_{p}$ |

Search range | [0, 90] | [80, 200] | [−45, 45] | [−30, 30] | [−30, 30] | [−20, 20] | [−40, −10] |

Seed Number | |||||
---|---|---|---|---|---|

1 | 2 | 3 | 4 | 5 | |

${F}^{1\mathrm{st}}$ | 1.68 | 1.68 | 1.68 | 1.68 | 1.68 |

${f}_{1}^{1\mathrm{st}}$ | 0.64 | 0.64 | 0.64 | 0.64 | 0.64 |

${f}_{2}^{1\mathrm{st}}$ | 1.04 | 1.04 | 1.04 | 1.04 | 1.04 |

${x}_{c}$ | −2.09 | −2.09 | −2.09 | −2.09 | −2.09 |

${y}_{c}$ | 33.99 | 33.99 | 33.99 | 33.99 | 33.99 |

${\alpha}_{c}$ | −0.46 | −0.46 | −0.46 | −0.46 | −0.46 |

${\beta}_{c}$ | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 |

${\gamma}_{c}$ | 0.72 | 0.72 | 0.72 | 0.72 | 0.72 |

${x}_{p}$ | 193.33 | 193.33 | 193.33 | 193.33 | 193.33 |

${y}_{p}$ | 31.62 | 31.62 | 31.62 | 31.62 | 31.62 |

${\alpha}_{t}$ | 1.51 | 1.51 | −1.51 | −1.51 | −1.51 |

${\beta}_{t}$ | −11.93 | −11.93 | 11.93 | 11.93 | 11.93 |

${\gamma}_{t}$ | −1.26 | −1.26 | −1.26 | −1.26 | −1.26 |

${x}_{t}$ | −0.46 | −0.46 | −0.46 | −0.46 | −0.46 |

${y}_{t}$ | −22.27 | −22.27 | −22.27 | −22.27 | −22.27 |

Seed Number | |||||
---|---|---|---|---|---|

1 | 2 | 3 | 4 | 5 | |

${F}^{1\mathrm{st}}$ | 7.10 | 7.10 | 7.10 | 7.10 | 7.10 |

${f}_{1}^{1\mathrm{st}}$ | 4.86 | 4.86 | 4.86 | 4.86 | 4.86 |

${f}_{2}^{1\mathrm{st}}$ | 2.24 | 2.24 | 2.24 | 2.24 | 2.24 |

${\theta}_{1}^{c}$ | 11.79 | 5.67 | 1.99 | 2.45 | 13.22 |

${a}_{1}^{c}$ | 91.02 | 91.02 | 91.02 | 91.02 | 91.02 |

${\alpha}_{c}$ | 10.25 | 10.25 | 10.25 | 10.25 | 10.25 |

${\beta}_{c}$ | −0.51 | −0.51 | −0.51 | −0.51 | −0.51 |

${\gamma}_{c}$ | −13.07 | −6.95 | −3.27 | −3.73 | −14.50 |

${\theta}_{1}^{t}$ | 20.18 | 2.57 | 3.34 | 6.11 | 0.45 |

${a}_{1}^{t}$ | 94.79 | 94.79 | 94.79 | 94.79 | 94.79 |

${\alpha}_{t}$ | 35.09 | 35.09 | 35.09 | 35.09 | −35.09 |

${\beta}_{t}$ | −5.33 | −5.33 | −5.33 | −5.33 | 5.33 |

${\gamma}_{t}$ | −26.08 | −8.48 | −9.28 | −12.01 | −6.36 |

${x}_{p}$ | −0.86 | −0.86 | −0.86 | −0.86 | −0.86 |

${y}_{p}$ | −24.49 | −24.49 | −24.49 | −24.49 | −24.49 |

Trial Number | ||||||
---|---|---|---|---|---|---|

Mean Touching Error | 1 | 2 | 3 | 4 | 5 | Average |

in px | 6.47 | 6.80 | 6.62 | 6.38 | 6.64 | 6.58 |

in mm | 1.23 | 1.29 | 1.26 | 1.21 | 1.26 | 1.25 |

Seed Number | 1 | 2 | 3 | 4 | 5 |
---|---|---|---|---|---|

${F}^{2\mathrm{nd}}$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |

${f}_{1}^{2\mathrm{nd}}$ | 0.19 | 0.19 | 0.19 | 0.19 | 0.19 |

${f}_{2}^{2\mathrm{nd}}$ | 0.82 | 0.82 | 0.82 | 0.82 | 0.82 |

${X}_{1}$ | −0.11 | −0.11 | −0.11 | −0.11 | −0.11 |

${Y}_{1}$ | 0 | 0 | 0 | 0 | 0 |

${S}_{1}^{x}$ | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |

${S}_{1}^{y}$ | 1.02 | 1.02 | 1.02 | 1.02 | 1.02 |

${\theta}_{n}$ | 0.03 | 0.03 | 0.03 | 0.03 | 0.03 |

${X}_{2}$ | 0.06 | 0.06 | 0.06 | 0.06 | 0.06 |

${Y}_{2}$ | −0.11 | −0.11 | −0.11 | −0.11 | −0.11 |

${S}_{2}^{x}$ | 1.02 | 1.02 | 1.02 | 1.02 | 1.02 |

${S}_{2}^{y}$ | 0.99 | 0.99 | 0.99 | 0.99 | 0.99 |

${\theta}_{2}$ | −0.03 | −0.03 | −0.03 | −0.03 | −0.03 |

Seed Number | 1 | 2 | 3 | 4 | 5 |
---|---|---|---|---|---|

${F}^{2\mathrm{nd}}$ | 4.27 | 4.27 | 4.27 | 4.27 | 4.27 |

${f}_{1}^{2\mathrm{nd}}$ | 1.73 | 1.73 | 1.73 | 1.73 | 1.73 |

${f}_{2}^{2\mathrm{nd}}$ | 2.54 | 2.54 | 2.54 | 2.54 | 2.54 |

${X}_{1}$ | −0.18 | −0.18 | −0.18 | −0.18 | −0.18 |

${Y}_{1}$ | −0.28 | −0.28 | −0.28 | −0.28 | −0.28 |

${S}_{1}^{x}$ | 1.03 | 1.03 | 1.03 | 1.03 | 1.03 |

${S}_{1}^{y}$ | 0.89 | 0.89 | 0.89 | 0.89 | 0.89 |

${\theta}_{n}$ | 3.77 | 3.77 | 3.77 | 3.77 | 3.77 |

${X}_{2}$ | 0.41 | 0.41 | 0.41 | 0.41 | 0.41 |

${Y}_{2}$ | 2.58 | 2.58 | 2.58 | 2.58 | 2.58 |

${S}_{2}^{x}$ | 0.98 | 0.98 | 0.98 | 0.98 | 0.98 |

${S}_{2}^{y}$ | 1.14 | 1.14 | 1.14 | 1.14 | 1.14 |

${\theta}_{2}$ | −4.51 | −4.51 | −4.51 | −4.51 | −4.51 |

Trial Number | ||||||
---|---|---|---|---|---|---|

Mean Touching Error | 1 | 2 | 3 | 4 | 5 | Average |

in px | 5.14 | 5.37 | 5.18 | 5.39 | 5.39 | 5.30 |

in mm | 0.98 | 1.02 | 0.99 | 1.03 | 1.03 | 1.01 |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the author. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Sato, J.
Hand–Eye Calibration Using a Tablet Computer. *Math. Comput. Appl.* **2023**, *28*, 22.
https://doi.org/10.3390/mca28010022

**AMA Style**

Sato J.
Hand–Eye Calibration Using a Tablet Computer. *Mathematical and Computational Applications*. 2023; 28(1):22.
https://doi.org/10.3390/mca28010022

**Chicago/Turabian Style**

Sato, Junya.
2023. "Hand–Eye Calibration Using a Tablet Computer" *Mathematical and Computational Applications* 28, no. 1: 22.
https://doi.org/10.3390/mca28010022