Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System
Abstract
:1. Introduction
2. Materials and Methods
2.1. Teleoperation System Overview
2.1.1. Robot and User Communication
2.1.2. MR Subspace
2.2. Velocity-Centric Motion Mapping
2.3. Experiments
2.3.1. Hypothesis
2.3.2. Experimental Setup
2.3.3. Experimental Procedure
2.4. Analyses
3. Results and Discussion
3.1. Objective Measures
3.2. Subjective Measures
4. Conclusions and Future Work
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Su, H.; Qi, W.; Yang, C.; Sandoval, J.; Ferrigno, G.; de Momi, E. Deep Neural Network Approach in Robot Tool Dynamics Identification for Bilateral Teleoperation. IEEE Robot. Autom. Lett. 2020, 5, 2943–2949. [Google Scholar] [CrossRef]
- Li, S.; Rameshwar, R.; Votta, A.M.; Onal, C.D. Intuitive Control of a Robotic Arm and Hand System with Pneumatic Haptic Feedback. IEEE Robot. Autom. Lett. 2019, 4, 4424–4430. [Google Scholar] [CrossRef]
- Conte, D.; Leamy, S.; Furukawa, T. Design and Map-Based Teleoperation of a Robot for Disinfection of COVID-19 in Complex Indoor Environments. In Proceedings of the 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics, SSRR 2020, Abu Dhabi, United Arab Emirates, 4–6 November 2020; pp. 276–282. [Google Scholar]
- Yang, G.; Lv, H.; Zhang, Z.; Yang, L.; Deng, J.; You, S.; Du, J.; Yang, H. Keep Healthcare Workers Safe: Application of Teleoperated Robot in Isolation Ward for COVID-19 Prevention and Control. Chin. J. Mech. Eng. 2020, 33, 47. [Google Scholar] [CrossRef]
- Li, C.; Gu, X.; Xiao, X.; Lim, C.M.; Duan, X.; Ren, H. A Flexible Transoral Robot Towards COVID-19 Swab Sampling. Front. Robot. AI 2021, 8, 51. [Google Scholar] [CrossRef]
- Chen, Y.; Wang, Q.; Chi, C.; Wang, C.; Gao, Q.; Zhang, H.; Li, Z.; Mu, Z.; Xu, R.; Sun, Z.; et al. A Collaborative Robot for COVID-19 Oropharyngeal Swabbing. Robot. Auton. Syst. 2022, 148, 103917. [Google Scholar] [CrossRef] [PubMed]
- Zhou, J.; Chen, W.; Cheng, S.S.; Xue, L.; Tong, M.C.F.; Liu, Y. Bio-Inspired Soft (BIS) Hand for Tele-Operated COVID-19 Oropharyngeal (OP) Swab Sampling. In Proceedings of the 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), Sanya, China, 27–31 December 2021; pp. 80–86. [Google Scholar]
- Yiming, L.; Chunki, Y.; Zhen, S.; Ya, H.; Kuanming, Y.; Tszhung, W.; Jingkun, Z.; Ling, Z.; Xingcan, H.; Khazaee, N.S.; et al. Electronic Skin as Wireless Human-Machine Interfaces for Robotic VR. Sci. Adv. 2022, 8, eabl6700. [Google Scholar] [CrossRef]
- Feizi, N.; Tavakoli, M.; Patel, R.V.; Atashzar, S.F. Robotics and AI for Teleoperation, Tele-Assessment, and Tele-Training for Surgery in the Era of COVID-19: Existing Challenges, and Future Vision. Front. Robot. AI 2021, 8, 16. [Google Scholar] [CrossRef]
- Li, C.; Wang, T.; Hu, L.; Tang, P.; Wang, L.; Zhang, L.; Guo, N.; Tan, Y. A Novel Master-Slave Teleoperation Robot System for Diaphyseal Fracture Reduction: A Preliminary Study. Comput. Assist. Surg. 2016, 21, 163–168. [Google Scholar] [CrossRef] [Green Version]
- Wang, Z.; Chen, Z.; Liang, B. Fixed-Time Velocity Reconstruction Scheme for Space Teleoperation Systems: Exp Barrier Lyapunov Function Approach. Acta Astronaut. 2019, 157, 92–101. [Google Scholar] [CrossRef]
- Shen, Y.; Guo, D.; Long, F.; Mateos, L.A.; Ding, H.; Xiu, Z.; Hellman, R.B.; King, A.; Chen, S.; Zhang, C.; et al. Robots under COVID-19 Pandemic: A Comprehensive Survey. IEEE Access 2021, 9, 1590–1615. [Google Scholar] [CrossRef]
- Lee, K.H.; Pruks, V.; Ryu, J.H. Development of Shared Autonomy and Virtual Guidance Generation System for Human Interactive Teleoperation. In Proceedings of the 2017 14th International Conference on Ubiquitous Robots and Ambient Intelligence, URAI 2017, Jeju, Korea, 28 June–1 July 2017; pp. 457–461. [Google Scholar]
- Song, K.T.; Jiang, S.Y.; Lin, M.H. Interactive Teleoperation of a Mobile Manipulator Using a Shared-Control Approach. IEEE Trans. Hum.-Mach. Syst. 2016, 46, 834–845. [Google Scholar] [CrossRef]
- Gao, L.; Xu, Z.; Huang, W.; Song, A. Design and Application of Experimental Platform for Interactive Teleoperation Robot. Dongnan Daxue Xuebao (Ziran Kexue Ban)/J. Southeast Univ. (Nat. Sci. Ed.) 2004, 34, 775–779. [Google Scholar]
- Saeidi, H.; Wagner, J.R.; Wang, Y. A Mixed-Initiative Haptic Teleoperation Strategy for Mobile Robotic Systems Based on Bidirectional Computational Trust Analysis. IEEE Trans. Robot. 2017, 33, 1500–1507. [Google Scholar] [CrossRef]
- Solanes, J.E.; Muñoz, A.; Gracia, L.; Martí, A.; Girbés-Juan, V.; Tornero, J. Teleoperation of Industrial Robot Manipulators Based on Augmented Reality. Int. J. Adv. Manuf. Technol. 2020, 111, 1077–1097. [Google Scholar] [CrossRef]
- Navarro, F.; Fdez, J.; Garzón, M.; Roldán, J.J.; Barrientos, A. Integrating 3D Reconstruction and Virtual Reality: A New Approach for Immersive Teleoperation. In Advances in Intelligent Systems and Computing; Springer: Berlin/Heidelberg, Germany, 2018; Volume 694, pp. 606–616. [Google Scholar]
- Lipton, J.I.; Fay, A.J.; Rus, D. Baxter’s Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing. IEEE Robot. Autom. Lett. 2018, 3, 179–186. [Google Scholar] [CrossRef] [Green Version]
- Dinh, T.Q.; Yoon, J.I.; Marco, J.; Jennings, P.; Ahn, K.K.; Ha, C. Sensorless Force Feedback Joystick Control for Teleoperation of Construction Equipment. Int. J. Precis. Eng. Manuf. 2017, 18, 955–969. [Google Scholar] [CrossRef] [Green Version]
- Truong, D.Q.; Truong, B.N.M.; Trung, N.T.; Nahian, S.A.; Ahn, K.K. Force Reflecting Joystick Control for Applications to Bilateral Teleoperation in Construction Machinery. Int. J. Precis. Eng. Manuf. 2017, 18, 301–315. [Google Scholar] [CrossRef]
- Nakanishi, J.; Itadera, S.; Aoyama, T.; Hasegawa, Y. Towards the Development of an Intuitive Teleoperation System for Human Support Robot Using a VR Device. Adv. Robot. 2020, 34, 1239–1253. [Google Scholar] [CrossRef]
- Meeker, C.; Rasmussen, T.; Ciocarlie, M. Intuitive Hand Teleoperation by Novice Operators Using a Continuous Teleoperation Subspace. In Proceedings of the IEEE International Conference on Robotics and Automation, Brisbane, Australia, 21–25 May 2018; pp. 5821–5827. [Google Scholar]
- Ellis, S.R.; Adelstein, B.D.; Welch, R.B. Kinesthetic Compensation for Misalignment of Teleoperator Controls through Cross-Modal Transfer of Movement Coordinates. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2002, 46, 1551–1555. [Google Scholar] [CrossRef]
- Li, G.; Caponetto, F.; del Bianco, E.; Katsageorgiou, V.; Sarakoglou, I.; Tsagarakis, N.G. Incomplete Orientation Mapping for Teleoperation with One DoF Master-Slave Asymmetry. IEEE Robot. Autom. Lett. 2020, 5, 5167–5174. [Google Scholar] [CrossRef]
- Bejczy, B.; Bozyil, R.; Vaiekauskas, E.; Petersen, S.B.K.; Bogh, S.; Hjorth, S.S.; Hansen, E.B. Mixed Reality Interface for Improving Mobile Manipulator Teleoperation in Contamination Critical Applications. Procedia Manuf. 2020, 51, 620–626. [Google Scholar] [CrossRef]
- Triantafyllidis, E.; McGreavy, C.; Gu, J.; Li, Z. Study of Multimodal Interfaces and the Improvements on Teleoperation. IEEE Access 2020, 8, 78213–78227. [Google Scholar] [CrossRef]
- Yew, A.W.W.; Ong, S.K.; Nee, A.Y.C. Immersive Augmented Reality Environment for the Teleoperation of Maintenance Robots. In Procedia CIRP; Elsevier: Amsterdam, The Netherlands, 2017; Volume 61, pp. 305–310. [Google Scholar]
- Komatsu, R.; Fujii, H.; Tamura, Y.; Yamashita, A.; Asama, H. Free Viewpoint Image Generation System Using Fisheye Cameras and a Laser Rangefinder for Indoor Robot Teleoperation. ROBOMECH J. 2020, 7, 15. [Google Scholar] [CrossRef]
- Ribeiro, L.G.; Suominen, O.J.; Durmush, A.; Peltonen, S.; Morales, E.R.; Gotchev, A. Retro-Reflective-Marker-Aided Target Pose Estimation in a Safety-Critical Environment. Appl. Sci. 2021, 11, 3. [Google Scholar] [CrossRef]
- Illing, B.; Westhoven, M.; Gaspers, B.; Smets, N.; Bruggemann, B.; Mathew, T. Evaluation of Immersive Teleoperation Systems Using Standardized Tasks and Measurements. In Proceedings of the 29th IEEE International Conference on Robot and Human Interactive Communication, RO-MAN 2020, Naples, Italy, 31 August–4 September 2020; pp. 278–285. [Google Scholar]
- Marques, B.; Teixeira, A.; Silva, S.; Alves, J.; Dias, P.; Santos, B.S. A Critical Analysis on Remote Collaboration Mediated by Augmented Reality: Making a Case for Improved Characterization and Evaluation of the Collaborative Process. Comput. Gr. 2022, 102, 619–633. [Google Scholar] [CrossRef]
- Marques, B.; Silva, S.S.; Alves, J.; Araujo, T.; Dias, P.M.; Santos, B.S. A Conceptual Model and Taxonomy for Collaborative Augmented Reality. IEEE Trans. Vis. Comput. Gr. 2021, 102, 1. [Google Scholar] [CrossRef]
- Wang, P.; Bai, X.; Billinghurst, M.; Zhang, S.; Zhang, X.; Wang, S.; He, W.; Yan, Y.; Ji, H. AR/MR Remote Collaboration on Physical Tasks: A Review. Robot. Comput.-Integr. Manuf. 2021, 72, 102071. [Google Scholar] [CrossRef]
- Sereno, M.; Wang, X.; Besancon, L.; Mcguffin, M.J.; Isenberg, T. Collaborative Work in Augmented Reality: A Survey. IEEE Trans. Vis. Comput. Gr. 2020, 72, 1. [Google Scholar] [CrossRef]
- Ens, B.; Lanir, J.; Tang, A.; Bateman, S.; Lee, G.; Piumsomboon, T.; Billinghurst, M. Revisiting Collaboration through Mixed Reality: The Evolution of Groupware. Int. J. Hum.-Comput. Stud. 2019, 131, 81–98. [Google Scholar] [CrossRef]
- De Belen, R.A.J.; Nguyen, H.; Filonik, D.; del Favero, D.; Bednarz, T. A Systematic Review of the Current State of Collaborative Mixed Reality Technologies: 2013–2018. AIMS Electron. Electr. Eng. 2019, 3, 181–223. [Google Scholar] [CrossRef]
- Nakamura, K.; Tohashi, K.; Funayama, Y.; Harasawa, H.; Ogawa, J. Dual-Arm Robot Teleoperation Support with the Virtual World. Artif. Life Robot. 2020, 25, 286–293. [Google Scholar] [CrossRef]
- Whitney, D.; Rosen, E.; Ullman, D.; Phillips, E.; Tellex, S. ROS Reality: A Virtual Reality Framework Using Consumer-Grade Hardware for ROS-Enabled Robots. In Proceedings of the IEEE International Conference on Intelligent Robots and Systems, Madrid, Spain, 1–5 October 2018; pp. 5018–5025. [Google Scholar]
- Whitney, D.; Rosen, E.; Phillips, E.; Konidaris, G.; Tellex, S. Comparing Robot Grasping Teleoperation Across Desktop and Virtual Reality with ROS Reality. Springer Proc. Adv. Robot. 2020, 10, 335–350. [Google Scholar] [CrossRef]
- Delpreto, J.; Lipton, J.I.; Sanneman, L.; Fay, A.J.; Fourie, C.; Choi, C.; Rus, D. Helping Robots Learn: A Human-Robot Master-Apprentice Model Using Demonstrations via Virtual Reality Teleoperation. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May–31 August 2020; pp. 10226–10233. [Google Scholar] [CrossRef]
- Britton, N.; Yoshida, K.; Walker, J.; Nagatani, K.; Taylor, G.; Dauphin, L. Lunar Micro Rover Design for Exploration through Virtual Reality Tele-Operation. Springer Tracts Adv. Robot. 2015, 105, 259–272. [Google Scholar] [CrossRef]
- Naceri, A.; Mazzanti, D.; Bimbo, J.; Prattichizzo, D.; Caldwell, D.G.; Mattos, L.S.; Deshpande, N. Towards a Virtual Reality Interface for Remote Robotic Teleoperation. In Proceedings of the 2019 19th International Conference on Advanced Robotics, ICAR 2019, Belo Horizonte, Brazil, 2–6 December 2019; pp. 284–289. [Google Scholar]
- Zhang, T.; McCarthy, Z.; Jowl, O.; Lee, D.; Chen, X.; Goldberg, K.; Abbeel, P. Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018; pp. 5628–5635. [Google Scholar] [CrossRef] [Green Version]
- Concannon, D.; Flynn, R.; Murray, N. A Quality of Experience Evaluation System and Research Challenges for Networked Virtual Reality-Based Teleoperation Applications. In Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019, Amherst, MA, USA, 18–21 June 2019; pp. 10–12. [Google Scholar]
- Stein, C.; Stein, C. Virtual Reality Design: How Head-Mounted Displays Change Design Paradigms of Virtual Reality Worlds. MediaTropes 2016, 6, 52–85. [Google Scholar] [CrossRef]
- Wonsick, M.; Padir, T. A Systematic Review of Virtual Reality Interfaces for Controlling and Interacting with Robots. Appl. Sci. 2020, 10, 9051. [Google Scholar] [CrossRef]
- Wang, Z.; Reed, I.; Fey, A.M. Toward Intuitive Teleoperation in Surgery: Human-Centric Evaluation of Teleoperation Algorithms for Robotic Needle Steering. In Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018; pp. 5799–5806. [Google Scholar] [CrossRef]
- Franzluebbers, A.; Johnsen, K. Remote Robotic Arm Teleoperation through Virtual Reality. In Proceedings of the Symposium on Spatial User Interaction, New Orleans, LA, USA, 19–20 October 2019. [Google Scholar] [CrossRef]
- Pryor, W.; Vagvolgyi, B.P.; Gallagher, W.J.; Deguet, A.; Leonard, S.; Whitcomb, L.L.; Kazanzides, P. Experimental Evaluation of Teleoperation Interfaces for Cutting of Satellite Insulation. In Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Amherst, MA, USA, 18–21 June 2019; pp. 4775–4781. [Google Scholar]
- Baklouti, S.; Gallot, G.; Viaud, J.; Subrin, K. On the Improvement of Ros-Based Control for Teleoperated Yaskawa Robots. Appl. Sci. 2021, 11, 7190. [Google Scholar] [CrossRef]
- Mai, X.; Chen, J.; Wang, Y.; Bi, S.; Cheng, Y.; Xi, N. A Teleoperation Framework of Hot Line Work Robot. In Proceedings of the 2018 IEEE International Conference on Mechatronics and Automation (ICMA), Changchun, China, 5–8 August 2018; pp. 1872–1876. [Google Scholar]
- Rakita, D.; Mutlu, B.; Gleicher, M. A Motion Retargeting Method for Effective Mimicry-Based Teleoperation of Robot Arms. In Proceedings of the Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria, 6–9 March 2017; pp. 361–370. [Google Scholar]
- Quintero, C.P.; Dehghan, M.; Ramirez, O.; Ang, M.H.; Jagersand, M. Flexible Virtual Fixture Interface for Path Specification in Tele-Manipulation. In Proceedings of the IEEE International Conference on Robotics and Automation, Singapore, 29 May–3 June 2017; pp. 5363–5368. [Google Scholar]
- Huang, K.; Chitrakar, D.; Rydén, F.; Chizeck, H.J. Evaluation of Haptic Guidance Virtual Fixtures and 3D Visualization Methods in Telemanipulation—A User Study. Intell. Serv. Robot. 2019, 12, 289–301. [Google Scholar] [CrossRef] [Green Version]
- Livatino, S.; Guastella, D.C.; Muscato, G.; Rinaldi, V.; Cantelli, L.; Melita, C.D.; Caniglia, A.; Mazza, R.; Padula, G. Intuitive Robot Teleoperation through Multi-Sensor Informed Mixed Reality Visual Aids. IEEE Access 2021, 9, 25795–25808. [Google Scholar] [CrossRef]
Post-Hoc Tests | ||||||
---|---|---|---|---|---|---|
Measure | Partial Eta Squared | F | p | MRS-MRD | MRS-B | MRD-B |
Pick-and-place (s) | 0.87 | F(1.382, 31.778) = 148.198 | <0.001 | <0.001 | <0.001 | <0.001 |
Assembly (s) | 0.76 | F(1.553, 35.725) = 74.080 | <0.001 | <0.001 | <0.001 | <0.001 |
Aggregate Time (s) | 0.93 | F(1.875, 43.128) = 303.197 | <0.001 | <0.001 | <0.001 | <0.001 |
Physical Demand | 0.45 | F(1.971, 45.339) = 18.478 | <0.001 | <0.001 | <0.001 | 0.387 |
Mental Demand | 0.59 | F(1.995, 45.874) = 32.638 | <0.001 | <0.001 | <0.001 | 0.002 |
NASA TLX | 0.76 | F(1.663, 38.247) = 74.408 | <0.001 | <0.001 | <0.001 | <0.001 |
Usefulness | 0.41 | F(1.846, 42.449) = 15.794 | <0.001 | <0.001 | <0.001 | 1.000 |
Ease of Use | 0.69 | F(1.832, 42.133) = 50.205 | <0.001 | <0.001 | <0.001 | 0.299 |
Baseline | MRD | MRS | ||||
---|---|---|---|---|---|---|
Measure | Mean | Std.Dev | Mean | Std.Dev | Mean | Std.Dev |
Pick-and-place (s) | 136.06 | 20.72 | 98.27 | 17.00 | 58.52 | 9.75 |
Assembly (s) | 162.60 | 23.61 | 122.88 | 16.84 | 97.51 | 12.18 |
Aggregate Time (s) | 298.66 | 26.42 | 221.15 | 23.39 | 156.03 | 17.19 |
Physical Demand | 76.04 | 11.32 | 70.88 | 11.51 | 55.63 | 12.54 |
Mental Demand | 81.38 | 13.29 | 68.46 | 11.12 | 54.54 | 13.15 |
NASA TLX | 75.81 | 6.23 | 66.89 | 5.39 | 51.92 | 8.15 |
Usefulness | 2.83 | 1.17 | 3.08 | 1.10 | 4.75 | 1.33 |
Ease of Use | 2.67 | 0.87 | 3.25 | 1.39 | 5.50 | 0.89 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Su, Y.-P.; Chen, X.-Q.; Zhou, T.; Pretty, C.; Chase, G. Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System. Appl. Sci. 2022, 12, 4740. https://doi.org/10.3390/app12094740
Su Y-P, Chen X-Q, Zhou T, Pretty C, Chase G. Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System. Applied Sciences. 2022; 12(9):4740. https://doi.org/10.3390/app12094740
Chicago/Turabian StyleSu, Yun-Peng, Xiao-Qi Chen, Tony Zhou, Christopher Pretty, and Geoffrey Chase. 2022. "Mixed-Reality-Enhanced Human–Robot Interaction with an Imitation-Based Mapping Approach for Intuitive Teleoperation of a Robotic Arm-Hand System" Applied Sciences 12, no. 9: 4740. https://doi.org/10.3390/app12094740