Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings

Zanetti, Matteo; Luchetti, Alessandro; Maheshwari, Sharad; Kalkofen, Denis; Ortega, Manuel Labrador; De Cecco, Mariolino

doi:10.3390/app12115309

Submit to this Journal Review for this Journal Edit a Special Issue

Article Menu

...

Open AccessArticle

Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings

^1,*

²,

Manuel Labrador Ortega

and

Mariolino De Cecco

Department of Industrial Engineering, University of Trento, Via Sommarive 9, 38123 Trento, Italy

Institute of Computer Graphics and Vision, Graz University of Technology, Rechbauerstraße 12, 8010 Graz, Austria

Resources Innovation Center Leoben, Montanuniversität Leoben, Franz Josef Strasse 18, 8700 Leoben, Austria

Author to whom correspondence should be addressed.

Academic Editor: Jorge Martin-Gutierrez

Appl. Sci. 2022, 12(11), 5309; https://doi.org/10.3390/app12115309

Received: 20 April 2022 / Revised: 21 May 2022 / Accepted: 23 May 2022 / Published: 24 May 2022

(This article belongs to the Special Issue Human Empowerment through Mixed Reality)

View Full-Text Download PDF

Browse Figures

Citation Export

Abstract

In this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning experience of students by making it more engaging and immersive due to the possible interaction with 3D virtual models. We developed a general approach usable for any object and shape. The only requirement is to have an accurate CAD model, even without textures of the item, whose pose must be estimated. The developed pipeline has two main steps: vehicle segmentation from the image background and estimation of the vehicle pose. To accomplish the first task, we used deep learning methods, while for the second, we developed a 360° camera simulator in Unity to generate synthetic equirectangular images used for comparison. We conducted our tests using a miniature truck model whose CAD was at our disposal. The developed algorithm was tested using a metrological analysis applied to real data. The results showed a mean difference of 1.5° with a standard deviation of 1° from the ground truth data for rotations, and

1.4

cm with a standard deviation of

1.5

cm for translations over a research range of ±20° and ±20 cm, respectively. View Full-Text

Keywords: image processing; 6DoF pose estimation; mixed reality; human empowerment; educational setting

▼ Show Figures

Figure 1

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Share and Cite

MDPI and ACS Style

Zanetti, M.; Luchetti, A.; Maheshwari, S.; Kalkofen, D.; Ortega, M.L.; De Cecco, M. Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings. Appl. Sci. 2022, 12, 5309. https://doi.org/10.3390/app12115309

AMA Style

Zanetti M, Luchetti A, Maheshwari S, Kalkofen D, Ortega ML, De Cecco M. Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings. Applied Sciences. 2022; 12(11):5309. https://doi.org/10.3390/app12115309

Chicago/Turabian Style

Zanetti, Matteo, Alessandro Luchetti, Sharad Maheshwari, Denis Kalkofen, Manuel Labrador Ortega, and Mariolino De Cecco. 2022. "Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings" Applied Sciences 12, no. 11: 5309. https://doi.org/10.3390/app12115309

Find Other Styles

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Abstract views Pdf views Html views

Article Menu

Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings

Abstract

Share and Cite

Article Metrics

Article Access Statistics

Article Access Map by Country/Region

Further Information

Guidelines

MDPI Initiatives

Follow MDPI