Teleoperation of High-Speed Robot Hand with High-Speed Finger Position Recognition and High-Accuracy Grasp Type Estimation
Abstract
:1. Introduction
- Low speed: The sampling rate is course, and the gain of the robot controller becomes small, resulting in low responsiveness.
- Low responsiveness: The latency from the human motion to the robot motion is long, making it difficult to remotely operate the robot. Furthermore, the system cannot respond to rapid and random human motion.
- Integration of a machine learning technique and high-speed image processing;
- High-speed finger tracking using the integrated image processing;
- High-accuracy grasp type estimation;
- Real-time teleoperation of a high-speed robot hand system;
- Evaluation of the developed teleoperation system.
2. Experimental System
2.1. High-Speed Vision System
2.2. High-Speed Robot Hand
2.3. Real-Time Controller
3. Grasp Type Estimation Based on High-Speed Finger Position Recognition
- Acquisition of the image by the high-speed camera:First, images can be captured by the high-speed camera at 1000 fps.
- Estimation of finger position by CNN and finger tracking by high-speed image processing:The CNN and finger tracking are executed on the images. The calculation process of the CNN is run at 100 Hz, and finger tracking is run at 1000 Hz; the results of the CNN are interpolated by using the results of finger tracking. As a result, the finger positions are recognized at 1000 Hz.
- Estimation of grasp type by decision tree classifier:Based on the finger positions, grasp type estimation is performed by using a decision tree classifier.
- Grasping motion of the high-speed robot hand:According to the estimated grasp type, the high-speed robot hand is controlled to grasp the object.
3.1. High-Speed Finger Position Recognition with CNN
3.1.1. Estimation of Finger Position by CNN
- Input: an array of 128 × 128 × 1;
- Output: 12 values;
- Alternating layers: six Convolution layers and six Max Pooling layers;
- Dropout layer placed before the output layer;
- The filter size of the Convolution layers was 3 × 3, the number of filters 32, and the stride 1;
- The pool size of Max Pooling was 2 × 2.
3.1.2. Finger Tracking by High-Speed Image Processing
- n-th frame: CNNIn the n-th frame, let an estimated fingertip position obtained by the CNN be . Using Equation (2) below, the image is binarized, the ROI with the center position is extracted, and the center of the fingertip in the ROI is assumed to be (Figure 5a). In the image binarization, the original image and the binarized image are and , respectively. Furthermore, the threshold of the image binarization is set at .The image moment is represented by (Equation (3)), and the center position () of the fingertip in the ROI is :The value of is substituted for the fingertip position in the n-th frame.
- ()-th frame: Finger trackingAfter binarizing the image in the ()-th frame , the ROI with the center position is extracted, and the center of the fingertip in the ROI is assumed to be (Figure 5b). At that time, let the finger position in the ()-th frame be , calculated by the following equation:
Algorithm 1 Finger tracking with low latency |
|
- The characteristics of the two modes are summarized below:
- Algorithm 1. Low-latency mode:
- As the result of the CNN, which is used for finger tracking, the latest result is utilized. The advantage is that the latency is reduced because no time is required to wait for the CNN results. The disadvantage is that if the CNN processing is delayed, the tracking process will be based on the CNN results for distant frames, which will reduce the accuracy.
- Algorithm A1. High-accuracy mode:
- By fixing the interval T of the number of frames at which the CNN is executed, the process of updating the estimate by CNN is performed at fixed intervals. The advantage is that hand tracking is based on frequently acquired CNNs, which improves accuracy. The disadvantage is that when the CNN processing is delayed, the latency increases because there is a waiting time for updating the CNN results before the tracking process starts.
3.2. Grasp Type Estimation
3.2.1. Estimation of Grasp Type by Decision Tree
3.2.2. Grasping Motion of High-Speed Robot Hand
4. Experiments and Evaluations
4.1. Finger Position Recognition
4.1.1. Preparation for Experiment
4.1.2. Experiment—1-A
4.1.3. Results
4.1.4. Discussion
4.2. Grasp Type Estimation
4.2.1. Preparation for the Experiment
4.2.2. Experiment—1-B
4.2.3. Result
4.2.4. Discussion
4.3. Teleoperated Grasp by Robot Hand
4.3.1. Experiment—2
4.3.2. Results
4.3.3. Discussion
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
Appendix A
Algorithm A1 Finger tracking with high accuracy |
|
References
- Sheridan, T.B. Telerobotics, Automation and Human Supervisory Control; The MIT Press: Cambridge, MA, USA, 1992. [Google Scholar]
- Tachi, S. TeTelecommunication, Teleimmersion and Telexistence; Ohmsha: Tokyo, Japan, 2003. [Google Scholar]
- Tachi, S. TeTelecommunication, Teleimmersion and Telexistence II; Ohmsha: Tokyo, Japan, 2005. [Google Scholar]
- Cui, J.; Tosunoglu, S.; Roberts, R.; Moore, C.; Repperger, D.W. A review of teleoperation system control. In Proceedings of the Florida Conference on Recent Advances in Robotics, Ft. Lauderdale, FL, USA, 8–9 May 2003; pp. 1–12. [Google Scholar]
- Tadakuma, R.; Asahara, Y.; Kajimoto, H.; Kawakami, N.; Tachi, S. Development of anthropomorphic multi-DOF master-slave arm for mutual telexistence. IEEE Trans. Vis. Comput. Graph. 2005, 11, 626–636. [Google Scholar] [CrossRef] [PubMed]
- Angelika, P.; Einenkel, S.; Buss, M. Multi-fingered telemanipulation—Mapping of a human hand to a three finger gripper. In Proceedings of the 17th IEEE International Symposium on Robot and Human Interactive Communication, Munich, Germany, 1–3 August 2008; pp. 465–470. [Google Scholar]
- Kofman, J.; Wu, X.; Luu, T.J. Teleoperation of a robot manipulator using a vision-based human-robot interface. IEEE Trans. Ind. Electron. 2005, 52, 1206–1219. [Google Scholar] [CrossRef]
- Chen, N.; Chew, C.-M.; Tee, K.P.; Han, B.S. Human-aided robotic grasping. In Proceedings of the 21st IEEE International Symposium on Robot and Human Interactive Communication, Paris, France, 9–13 September 2012; pp. 75–80. [Google Scholar]
- Hu, C.; Meng, M.Q.; Liu, P.X.; Wang, X. Visual gesture recognition for human-machine interface of robot teleoperation. In Proceedings of the 2003 IEEE/RSJ International Conference Intelligent Robots and Systems, Las Vegas, NV, USA, 37–31 October 2003; pp. 1560–1565. [Google Scholar]
- Cela, A.; Yebes, J.J.; Arroyo, R.; Bergasa, L.M.; Barea, R.; López, E. Complete Low-Cost Implementation of a Teleoperated Control System for a Humanoid Robot. Sensors 2013, 13, 1385–1401. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Yang, K.; Bergasa, L.M.; Romera, E.; Huang, X.; Wang, K. Predicting Polarization Beyond Semantics for Wearable Robotics. In Proceedings of the 2018 IEEE-RAS 18th International Conference on Humanoid Robots, Beijing, China, 6–9 November 2018; pp. 96–103. [Google Scholar]
- Zhang, Z. Microsoft Kinect Sensor and Its Effect. IEEE Multimed. 2012, 19, 4–10. [Google Scholar] [CrossRef] [Green Version]
- Weichert, F.; Bachmann, D.; Rudak, B.; Fisseler, D. Analysis of the Accuracy and Robustness of the Leap Motion Controller. IEEE Sens. J. 2013, 13, 6380–6393. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Lien, J.; Gillian, N.; Karagozler, M.E.; Amihood, P.; Schwesig, C.; Olson, E.; Raja, H.; Poupyrev, I. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Trans. Graph 2016, 35, 1–19. [Google Scholar] [CrossRef] [Green Version]
- Zhang, F.; Bazarevsky, V.; Vakunov, A.; Tkachenka, A.; Sung, G.; Chang, C.; Grundmann, M. MediaPipe hands: On-device real-time hand tracking. arXiv 2020, arXiv:2006.10214. [Google Scholar]
- Tamaki, E.; Miyake, T.; Rekimoto, J. A Robust and Accurate 3D Hand Posture Estimation Method for Interactive Systems. Trans. Inf. Process. Soc. Jpn. 2010, 51, 229–239. [Google Scholar]
- Premaratne, P. Human Computer Interaction Using Hand Gestures; Springer: Berlin/Heidelberg, Germany, 2014. [Google Scholar]
- Ankit, C. Robust Hand Gesture Recognition for Robotic Hand Control; Springer: Berlin/Heidelberg, Germany, 2018. [Google Scholar]
- Hoshino, K.; Tamaki, E.; Tanimoto, T. Copycat hand—Robot hand generating imitative behavior. In Proceedings of the IECON 2007—33rd Annual Conference of the IEEE Industrial Electronics Society, Taipei, Taiwan, 5–8 November 2007; pp. 2876–2881. [Google Scholar]
- Griffin, W.; Findley, R.; Turner, M.; Cutkosky, M. Calibration and mapping of a human hand for dexterous telemanipulation. In Proceedings of the ASME IMECE Symp. Haptic Interfaces for VirtualEnvironments and Teleoperator Systeme, Orlando, FL, USA, 5–10 November 2000; pp. 1–8. [Google Scholar]
- Meeker, C.; Haas-Heger, M.; Ciocarlie, M. A Continuous Teleoperation Subspace with Empirical and Algorithmic Mapping Algorithms for Non-Anthropomorphic Hands. arXiv 2019, arXiv:1911.09565v1. [Google Scholar]
- Saen, M.; Ito, K.; Osada, K. Action-intention-based grasp control with fine finger-force adjustment using combined optical-mechanical tactile sensor. IEEE Sens. J. 2014, 14, 4026–4033. [Google Scholar] [CrossRef]
- Niwa, M.; Iizuka, H.; Ando, H.; Maeda, T. Tumori Control: Manipulate Robot with detection and transmission of an Archetype of Behavioral Intention. Trans. Virtual Real. Soc. Jpn. 2012, 17, 3–10. [Google Scholar]
- Fallahinia, N.; Mascaro, S.A. Comparison of Constrained and Unconstrained Human Grasp Forces Using Fingernail Imaging and Visual Servoing. In Proceedings of the IEEE Conference on Robotics and Automation 2020 (ICRA 2020) Camera-Ready, Paris, France, 31 May–31 August 2020; pp. 2668–2674. [Google Scholar]
- Anvari, M.; Broderick, T.; Stein, H.; Chapman, T.; Ghodoussi, M.; Birch, D.W.; Mckinley, C.; Trudeau, P.; Dutta, S.; Goldsmith, C.H. The impact of latency on surgical precision and task completion during robotic-assisted remote telepresence surgery. Comput. Aided Surg. 2005, 10, 93–99. [Google Scholar] [CrossRef] [PubMed]
- Lum, M.J.H.; Rosen, J.; King, H.; Friedman, D.C.W.; Lendvay, T.S.; Wright, A.S.; Sinanan, M.N.; Hannaford, B. Teleoperation in surgical robotics—Network latency effects on surgical performance. In Proceedings of the 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Minneapolis, MI, USA, 3–6 September 2009; pp. 6860–6863. [Google Scholar]
- Katsuki, Y.; Yamakawa, Y.; Watanabe, Y.; Ishikawa, M. Development of fast-response master-slave system using high-speed non-contact 3D sensing and high-speed robot hand. In Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 28 September–2 October 2015; pp. 1236–1241. [Google Scholar]
- Yamakawa, Y.; Katsuki, Y.; Watanabe, Y.; Ishikawa, M. Development of a High-Speed, Low-Latency Telemanipulated Robot Hand System. Robotics 2021, 10, 41. [Google Scholar] [CrossRef]
- Ito, K.; Sueishi, T.; Yamakawa, Y.; Ishikawa, M. Tracking and recognition of a human hand in dynamic motion for Janken (rock-paper-scissors) robot. In Proceedings of the 2016 IEEE International Conference on Automation Science and Engineering (CASE), Fort Worth, TX, USA, 21–25 August 2016; pp. 891–896. [Google Scholar]
- Available online: http://www.hfr.iis.u-tokyo.ac.jp/research/Janken/index-e.html (accessed on 20 April 2022).
- Namiki, A.; Imai, Y.; Ishikawa, M.; Kaneko, M. Development of a high-speed multifingered hand system and its application to catching. In Proceedings of the 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems, Las Vegas, NV, USA, 27 October–1 November 2003; pp. 2666–2671. [Google Scholar]
- The Keras Blog—Building Powerful Image Classification Models Using Very Little Data. Available online: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html (accessed on 20 April 2022).
- Neural Networks for Machine Learning Lecture 6a Overview of Mini-Batch Gradient Descent. Available online: https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf (accessed on 20 April 2022).
- Canziani, A.; Paszke, A.; Culurciello, E. An Analysis of Deep Neural Network Models for Practical Applications. arXiv 2017, arXiv:1605.07678. [Google Scholar]
- Yoshida, K.; Yamakawa, Y. 2D Position Estimation of Finger Joints with High Spatio-Temporal Resolution Using a High-speed Vision. In Proceedings of the 21nd Conference of the Systems Integration Division of the Society of Instrument and Control Engineers (SI2020), Fukuoka, Japan, 8–10 September 2020; pp. 1439–1440. (In Japanese). [Google Scholar]
- Huang, S.; Shinya, K.; Bergstrom, N.; Yamakawa, Y.; Yamazaki, T.; Ishikawa, M. Dynamic compensation robot with a new high-speed vision system for flexible manufacturing. Int. J. Adv. Manuf. Technol. 2018, 95, 4523–4533. [Google Scholar] [CrossRef]
- Wong, T. Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognit. 2015, 48, 2839–2846. [Google Scholar] [CrossRef]
- Available online: http://www.hfr.iis.u-tokyo.ac.jp/research/Teleoperation/index-e.html (accessed on 20 April 2022).
Evaluation Index | Conventional Method | Proposed Method |
---|---|---|
Comfortable operation | Contact | Non-contact |
Application to various robots | Motion mapping | Intention extraction |
Fine-motion recognition | Low-speed | High-speed |
Finger | MSE | |
---|---|---|
with Finger Tracking/Pixel | without Finger Tracking/Pixel | |
Index | 0.95 | 1.03 |
Middle | 0.97 | 1.24 |
Ring | 1.19 | 1.60 |
Pinky | 1.18 | 1.47 |
Thumb | 0.99 | 1.02 |
Average | 1.06 | 1.27 |
Finger | Joint | Time/ms |
---|---|---|
Middle finger | root | 25 |
top | 25 | |
root | 26 | |
Left and right thumbs | top | 36 |
rotation around palm | 18 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yamakawa, Y.; Yoshida, K. Teleoperation of High-Speed Robot Hand with High-Speed Finger Position Recognition and High-Accuracy Grasp Type Estimation. Sensors 2022, 22, 3777. https://doi.org/10.3390/s22103777
Yamakawa Y, Yoshida K. Teleoperation of High-Speed Robot Hand with High-Speed Finger Position Recognition and High-Accuracy Grasp Type Estimation. Sensors. 2022; 22(10):3777. https://doi.org/10.3390/s22103777
Chicago/Turabian StyleYamakawa, Yuji, and Koki Yoshida. 2022. "Teleoperation of High-Speed Robot Hand with High-Speed Finger Position Recognition and High-Accuracy Grasp Type Estimation" Sensors 22, no. 10: 3777. https://doi.org/10.3390/s22103777