Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation
Abstract
:1. Introduction
2. Literature Review
3. Materials and Methods
3.1. Data Collection
3.2. Data Characteristics
3.3. Preprocessing
Algorithm 1 Preprocessing and experimentation pseudocode | |
1: | for each barometer reading do: |
2: | keep closest MARG reading |
3: | Discard the rest of the MARG reading |
4: | for For each angle value do: |
5: | take sensor readings of corresponding timestamp |
6: | take previous sensor readings |
7: | Separate training and test data |
8: | Normalize training and test sensor values using the mean and standard deviation from training data |
9: | Train model using training data |
10: | Obtain performance results using test data |
3.4. The Angle Estimation Model
3.4.1. Model Architecture
3.4.2. Hyperparameters and Window Size Optimization
4. Results
4.1. Model Training
4.2. Window Size
4.3. Comparing this Temporal Deep Learning Method to Ridge Regression
5. Discussion
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Billard, B.; Kragic, D. Trends and challenges in robot manipulation. Science 2019, 364, eaat8414. [Google Scholar] [CrossRef] [PubMed]
- Hammond, F.L.; Weisz, J.; de la Llera Kurth, A.A.; Allen, P.K.; Howe, R.D. Towards a design optimization method for reducing the mechanical complexity of underactuated robotic hands. In Proceedings of the 2012 IEEE International Conference on Robotics and Automation, St. Paul, MN, USA, 14–18 May 2012; pp. 2843–2850. [Google Scholar] [CrossRef]
- Sahin, C.; Garcia-Hernando, G.; Sock, J.; Kim, T.K. A review on object pose recovery: From 3D bounding box detectors to full 6D pose estimators. Image Vis. Comput. 2020, 96, 103898. [Google Scholar] [CrossRef]
- Zimmer, J.; Hellebrekers, T.; Asfour, T.; Majidi, C.; Kroemer, O. Predicting Grasp Success with a Soft Sensing Skin and Shape-Memory Actuated Gripper. In Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, China, 3–8 November 2019; pp. 7120–7127. [Google Scholar] [CrossRef]
- Wettels, N.; Santos, V.J.; Johansson, R.S.; Loeb, G.E. Biomimetic tactile sensor array. Adv. Robot. 2008, 22, 829–849. [Google Scholar] [CrossRef]
- Ward-Cherrier, B.; Pestell, N.; Cramphorn, L.; Winstone, B.; Giannaccini, M.E.; Rossiter, J.; Lepora, N.F. The tactip family: Soft optical tactile sensors with 3d-printed biomimetic morphologies. Soft Robot. 2018, 5, 216–227. [Google Scholar] [CrossRef] [PubMed]
- Lambeta, M.; Chou, P.W.; Tian, S.; Yang, B.; Maloon, B.; Most, V.R.; Stroud, D.; Santos, R.; Byagowi, A.; Kammerer, G.; et al. Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robot. Autom. Lett. 2020, 5, 3838–3845. [Google Scholar] [CrossRef]
- Alspach, A.; Hashimoto, K.; Kuppuswamy, N.; Tedrake, R. Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation. In Proceedings of the 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft), Seoul, Republic of Korea, 14–18 April 2019; pp. 597–604. [Google Scholar] [CrossRef]
- Su, Z.; Hausman, K.; Chebotar, Y.; Molchanov, A.; Loeb, G.E.; Sukhatme, G.S.; Schaal, S. Force estimation and slip detection/classification for grip control using a biomimetic tactile sensor. In Proceedings of the 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), Seoul, Republic of Korea, 3–5 November 2015; pp. 297–303. [Google Scholar] [CrossRef]
- Yoon, S.J.; Choi, M.; Jeong, B.; Park, Y.L. Elongatable Gripper Fingers with Integrated Stretchable Tactile Sensors for Underactuated Grasping and Dexterous Manipulation. IEEE Trans. Robot. 2022, 38, 2179–2193. [Google Scholar] [CrossRef]
- Li, Q.; Kroemer, O.; Su, Z.; Veiga, F.F.; Kaboli, M.; Ritter, H.J. A review of tactile information: Perception and action through touch. IEEE Trans. Robot. 2020, 36, 1619–1634. [Google Scholar] [CrossRef]
- Bandari, N.; Dargahi, J.; Packirisamy, M. Tactile Sensors for Minimally Invasive Surgery: A Review of the State-of-the-Art, Applications, and Perspectives. IEEE Access 2020, 8, 7682–7708. [Google Scholar] [CrossRef]
- She, Y.; Wang, S.; Dong, S.; Sunil, N.; Rodriguez, A.; Adelson, E. Cable manipulation with a tactile-reactive gripper. Int. J. Robot. Res. 2021, 40, 1385–1401. [Google Scholar] [CrossRef]
- Bi, T.; Sferrazza, C.; D’Andrea, R. Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation. IEEE Robot. Autom. Lett. 2021, 6, 5761–5768. [Google Scholar] [CrossRef]
- Zhang, H.; Lu, Z.; Liang, W.; Yu, H.; Mao, Y.; Wu, Y. Interaction Control for Tool Manipulation on Deformable Objects Using Tactile Feedback. IEEE Robot. Autom. Lett. 2023, 8, 2700–2707. [Google Scholar] [CrossRef]
- Drigalski, F.V.; Taniguchi, S.; Lee, R.; Matsubara, T.; Hamaya, M.; Tanaka, K.; Ijiri, Y. Contact-based in-hand pose estimation using Bayesian state estimation and particle filtering. In Proceedings of the IEEE International Conference on Robotics and Automation, Paris, France, 31 May–31 August 2020; pp. 7294–7299. [Google Scholar] [CrossRef]
- Alves de Oliveira, T.E.; Cretu, A.M.; Petriu, E.M. Multimodal Bio-Inspired Tactile Sensing Module. IEEE Sens. J. 2017, 17, 3231–3243. [Google Scholar] [CrossRef]
- Gomes, D.F.; Lin, Z.; Luo, S. GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation. In Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, NV, USA, 25–29 October 2020; pp. 9903–9909. [Google Scholar] [CrossRef]
- Romero, B.; Veiga, F.; Adelson, E. Soft, Round, High Resolution Tactile Fingertip Sensors for Dexterous Robotic Manipulation. In Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 31 May–31 August 2020; pp. 4796–4802. [Google Scholar] [CrossRef]
- Trueeb, C.; Sferrazza, C.; D’Andrea, R. Towards vision-based robotic skins: A data-driven, multi-camera tactile sensor. In Proceedings of the 2020 3rd IEEE International Conference on Soft Robotics (RoboSoft), New Haven, CT, USA, 15 May–15 July 2020; pp. 333–338. [Google Scholar] [CrossRef]
- da Fonseca, V.P.; de Oliveira, T.E.A.; Petriu, E.M. the Orientation of Objects from Tactile Sensing Data Using Machine Learning Methods and Visual Frames of Reference. Sensors 2019, 19, 2285. [Google Scholar] [CrossRef] [PubMed]
- Sipos, A.; Fazeli, N. Simultaneous Contact Location and Object Pose Estimation Using Proprioception and Tactile Feedback. In Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 23–27 October 2022; pp. 3233–3240. [Google Scholar] [CrossRef]
- Lloyd, J.; Lepora, N.F. Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback. IEEE Trans. Robot. 2022, 38, 1201–1212. [Google Scholar] [CrossRef]
- Ji, J.; Liu, Y.; Ma, H. Model-Based 3D Contact Geometry Perception for Visual Tactile Sensor. Sensors 2022, 22, 6470. [Google Scholar] [CrossRef] [PubMed]
- Shah, U.; Muthusamy, R.; Gan, D.; Zweiri, Y.; Seneviratne, L. On the Design and Development of Vision-based Tactile Sensors. J. Intell. Robot. Syst. 2021, 102, 82. [Google Scholar] [CrossRef]
- Suresh, S.; Bauza, M.; Yu, K.T.; Mangelson, J.G.; Rodriguez, A.; Kaess, M. Tactile SLAM: Real-Time Inference of Shape and Pose from Planar Pushing. In Proceedings of the 2021 IEEE International Conference on Robotics and Automation (ICRA), Xi’an, China, 30 May–5 June 2021; pp. 11322–11328. [Google Scholar] [CrossRef]
- Álvarez, D.; Roa, M.A.; Moreno, L. Tactile-Based In-Hand Object Pose Estimation. In Proceedings of the ROBOT 2017: Third Iberian Robotics Conference, Sevilla, Spain, 22–24 November 2017; Ollero, A., Sanfeliu, A., Montano, L., Lau, N., Cardeira, C., Eds.; Springer: Cham, Switzerland, 2018; pp. 716–728. [Google Scholar]
- Azulay, O.; Ben-David, I.; Sintov, A. Learning Haptic-based Object Pose Estimation for In-hand Manipulation with Underactuated Robotic Hands. arXiv 2022, arXiv:2207.02843. [Google Scholar] [CrossRef]
- Funabashi, S.; Isobe, T.; Hongyi, F.; Hiramoto, A.; Schmitz, A.; Sugano, S.; Ogata, T. Multi-Fingered In-Hand Manipulation with Various Object Properties Using Graph Convolutional Networks and Distributed Tactile Sensors. IEEE Robot. Autom. Lett. 2022, 7, 2102–2109. [Google Scholar] [CrossRef]
- Álvarez, D.; Roa, M.A.; Moreno, L. Visual and Tactile Fusion for Estimating the Pose of a Grasped Object. In Proceedings of the Robot 2019: Fourth Iberian Robotics Conference, Porto, Portugal, 20–22 November 2019; Silva, M.F., Luís Lima, J., Reis, L.P., Sanfeliu, A., Tardioli, D., Eds.; Springer: Cham, Switzerland, 2020; pp. 184–198. [Google Scholar]
- Dikhale, S.; Patel, K.; Dhingra, D.; Naramura, I.; Hayashi, A.; Iba, S.; Jamali, N. VisuoTactile 6D Pose Estimation of an In-Hand Object Using Vision and Tactile Sensor Data. IEEE Robot. Autom. Lett. 2022, 7, 2148–2155. [Google Scholar] [CrossRef]
- Park, C.M. Stanford Artificial Intelligence Laboratory. Sci. Technol. 1974, 7, 17–19. [Google Scholar]
- da Fonseca, V.P.; Jiang, X.; Petriu, E.M.; de Oliveira, T.E.A. Tactile object recognition in early phases of grasping using underactuated robotic hands. Intell. Serv. Robot. 2022, 15, 513–525. [Google Scholar] [CrossRef]
- Nigatu, H.; Choi, Y.H.; Kim, D. Analysis of parasitic motion with the constraint embedded Jacobian for a 3-PRS parallel manipulator. Mech. Mach. Theory 2021, 164, 104409. [Google Scholar] [CrossRef]
- Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G.S.; Davis, A.; Dean, J.; Devin, M.; et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. 2015. Available online: tensorflow.org (accessed on 2 May 2023).
Camera | Pressure | MARG Sensor |
---|---|---|
29.95 Hz | 402.19 Hz | 973.50 Hz |
Hyperparameter | Value |
---|---|
Learning rate | 0.00025 |
Batch size | 128 |
Epochs | 400 |
K-folds | 4 |
Iterations | 6 |
Window | MAE | MSE | R | EXP |
---|---|---|---|---|
5 | 0.0422 ± 0.0046 | 0.0042 ± 0.0012 | 0.8710 ± 0.0404 | 0.8732 ± 0.0408 |
10 | 0.0408 ± 0.0040 | 0.0038 ± 0.0009 | 0.8823 ± 0.0292 | 0.8840 ± 0.0297 |
15 | 0.0412 ± 0.0046 | 0.0041 ± 0.0012 | 0.8754 ± 0.0374 | 0.8785 ± 0.0370 |
20 | 0.0392 ± 0.0036 | 0.0036 ± 0.0016 | 0.8873 ± 0.0492 | 0.8894 ± 0.0493 |
25 | 0.0394 ± 0.0040 | 0.0034 ± 0.0007 | 0.8956 ± 0.0237 | 0.8975 ± 0.0246 |
30 | 0.0388 ± 0.0037 | 0.0033 ± 0.0009 | 0.8981 ± 0.0287 | 0.8997 ± 0.0289 |
35 | 0.0392 ± 0.0044 | 0.0033 ± 0.0006 | 0.8981 ± 0.0193 | 0.9005 ± 0.0188 |
40 | 0.0375 ± 0.0028 | 0.0030 ± 0.0004 | 0.9074 ± 0.0153 | 0.9094 ± 0.0148 |
45 | 0.0389 ± 0.0038 | 0.0032 ± 0.0005 | 0.9013 ± 0.0190 | 0.9038 ± 0.0185 |
50 | 0.0380 ± 0.0036 | 0.0031 ± 0.0005 | 0.9053 ± 0.0192 | 0.9069 ± 0.0188 |
55 | 0.0385 ± 0.0037 | 0.0031 ± 0.0005 | 0.9048 ± 0.0153 | 0.9073 ± 0.0149 |
60 | 0.0383 ± 0.0034 | 0.0031 ± 0.0006 | 0.9037 ± 0.0208 | 0.9060 ± 0.0207 |
Model | MAE | MSE | R | EXP |
---|---|---|---|---|
Ridge regressor | 0.0677 | 0.0088 | 0.6875 | 0.7033 |
Linear regressor | 0.0678 | 0.0089 | 0.6862 | 0.7021 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Galaiya, V.R.; Asfour, M.; Alves de Oliveira, T.E.; Jiang , X.; Prado da Fonseca, V. Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation. Sensors 2023, 23, 4535. https://doi.org/10.3390/s23094535
Galaiya VR, Asfour M, Alves de Oliveira TE, Jiang X, Prado da Fonseca V. Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation. Sensors. 2023; 23(9):4535. https://doi.org/10.3390/s23094535
Chicago/Turabian StyleGalaiya, Viral Rasik, Mohammed Asfour, Thiago Eustaquio Alves de Oliveira, Xianta Jiang , and Vinicius Prado da Fonseca. 2023. "Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation" Sensors 23, no. 9: 4535. https://doi.org/10.3390/s23094535
APA StyleGalaiya, V. R., Asfour, M., Alves de Oliveira, T. E., Jiang , X., & Prado da Fonseca, V. (2023). Exploring Tactile Temporal Features for Object Pose Estimation during Robotic Manipulation. Sensors, 23(9), 4535. https://doi.org/10.3390/s23094535