Object Pose Detection to Enable 3D Interaction from 2D Equirectangular Images in Mixed Reality Educational Settings

Abstract

In this paper, we address the challenge of estimating the 6DoF pose of objects in 2D equirectangular images. This solution allows the transition to the objects’ 3D model from their current pose. In particular, it finds application in the educational use of 360° videos, where it enhances the learning experience of students by making it more engaging and immersive due to the possible interaction with 3D virtual models. We developed a general approach usable for any object and shape. The only requirement is to have an accurate CAD model, even without textures of the item, whose pose must be estimated. The developed pipeline has two main steps: vehicle segmentation from the image background and estimation of the vehicle pose. To accomplish the first task, we used deep learning methods, while for the second, we developed a 360° camera simulator in Unity to generate synthetic equirectangular images used for comparison. We conducted our tests using a miniature truck model whose CAD was at our disposal. The developed algorithm was tested using a metrological analysis applied to real data. The results showed a mean difference of 1.5° with a standard deviation of 1° from the ground truth data for rotations, and $1.4$ cm with a standard deviation of $1.5$ cm for translations over a research range of ±20° and ±20 cm, respectively.

1. Introduction

Mixed Reality (MR) has the potential to increase the learning experiences in several scenarios [1] such as industrial maintenance training [2,3]; in Industry 4.0, cyber-physical engineering can be made experienceable and tangible [4]; construction training stressing safety aspects using Augmented Virtuality to enhance hazard recognition [5,6]; and humanities disciplines such as history [7], medicine [8], public speaking [9], and others. In general, MR technologies allow students to “visit” places, such as production plants, power plants, mine sites, and plants for which special permits are mandatory. Moreover, there are often limited resources for practical lessons in the education sector. MR technologies can help to overcome these limits and improve learning. In Europe, there are already education projects that propose this new paradigm, such as MiReBooks [10]. MiReBooks aims to produce a series of Virtual Reality (VR) and Augmented Reality (AR) based interactive mining handbooks as a new digital standard for higher mining education across Europe. Current challenges in mining education are met in an innovative new way, combining classical paper-based teaching materials with MR materials and their transformation into pedagogically and didactically coherent MR handbooks for integrative classroom use [11].

From a technological point of view, it is possible to use immersive solutions that exploit VR gears that are able to display virtual environments augmented with real content, a real environment fully reconstructed in 3D or, enabled by a modern high-resolution whole field of view cameras—360° videos. A great effort is needed to accurately replicate real 3D environments in virtual scenarios: acquiring 360° videos is a much simpler process. On the other hand, 360° videos are not able to provide 3D perception.

For this reason, we propose a method to generate 2D to 3D transitions starting from 360° source equirectangular images: when an interaction with an object is needed, we can propose to the user the 3D virtual model of the latter. To accomplish this, there is the need to develop a robust algorithm capable of estimating the object’s pose inside the 360° frame. 6DoF estimation is one of the main challenging research topics in computer vision [12,13,14].

Specifically, in this study we present a novel pipeline whose aim is to infer the pose of a vehicle given a single 2D equirectangular image. The proposed algorithms are described in Section 5, and the experimental results are provided in Section 6. Although the exploitation is very general, we furthermore demonstrate the application of our approach. Therefore, in Section 2 we show the effects of human neuropsychological empowerment through MR, and we provide an example implementation in Section 3, which demonstrates the proposed pipeline for enabling 2D to 3D transitions in MR environments.

2. Human Neuropsychological Empowerment in Learning Scenarios through Mixed Reality

While MR can enhance the human capabilities in several contexts such as clinics [15], personalized support for mild cognitive impaired users in cooking [16], cognitive training system for mild cognitive impairment [17], can enhance the human-robot interaction while sharing the same virtual-real (mirrored) environment [18]. Regarding the interaction channel, MR can fully exploit a natural interface [19,20] while as input MR can exploit a wide set of channels such as visual text, 2D, 3D, auditory, and kinesthetics [21]. In training, a recent meta-analysis [22] shows that whether one trains in a virtual or a real setting, the results are essentially equivalent. In teaching, MR can enhance the learner’s neuropsychological functions during the comprehension and memorization of the educational concepts. More specifically, in the following, we take into consideration the human perception-action loop (interface and interaction modality) and the whole patterns of neuropsychological functions highlighting the impact of each of them.

Regarding Orientation, immersive MR simulations allow the understanding of complex 3D phenomena that would be difficult to comprehend through other media [23,24]. Collaborative MR can transmit procedural knowledge in manufacturing training almost perfectly, replacing other forms of face-to-face training [25].

Regarding Attention, within MR, information can be spatially and temporally aligned with physical items and the learners’ activities. This solves the problem of the human brain, which has a limited capacity for processing information from sensory channels (too much information results in cognitive overload/poor performances of the selective function and thus poor quality in learning). In fact, in MR it is possible to enhance the spatial and the temporal contiguity effect that makes students learn better when multiple representations of the same information are presented closer in space rather than far apart and/or when multiple representations of the same information are presented at the same time, rather than separated in time [26]. This contextualizes the different information content in an aggregated form within the learning scenario and uses different sensor modalities such as auditory, visual, and tactile, which are processed in parallel within our brain. Furthermore, the “digital augmentation” of reality can direct user attention toward the relevant content. This mechanism effectively enhances learning tasks involving visuospatial information. The system presented in [27] highlights organs to effectively teach students about human anatomy.

Regarding Memory, an enhanced memory encoding is stimulated by the physical immersion of MR experiences, and the fact that users interact with their senses, body, and limbs, which cause learners to encode sensing and proprioceptive information along with the educational content [28].

Regarding Language, MR can leverage current translation technologies to temporally align the verbal translated content to the user.

Regarding Visual perception, recent advances in display technologies are attempting to fill the gap between the natural view and the one mediated by the display. This is not the appropriate place to focus on this topic.

Regarding Motion planning and control, MR can increase depth perception as in [29] where the perception of 3D obstacles is enhanced via 2D projection onto the ground plane with perspective correction based on the subject’s head position. Another way MR can be of benefit is augmenting user exproprioception. Visual exproprioception provides information about the body position in relation to the environment, and it can yield positive effects on position control and gait biomechanics in AR systems. As an example, in [30] an AR application was implemented using Microsoft HoloLens. The experiment revealed that the interface projected in front of the user (instead of the ground), and, from a third-person point of view, improves posture, visual stimuli, and safety.

3. 2D to 3D Objects Transitioning and Its Use in MR Settings

The 360° cameras support capturing dynamic environments from a single point of view. In addition, VR headsets enable viewing the captured environments by supporting head rotations around all three axes. While this enables immersive experiences, the missing translations may cause several perceptual issues [31], and it limits explorations to the pre-defined viewpoint. To enable full 3D object explorations, we propose combining 2D panoramic video data with renderings of registered 3D models. In addition, we propose providing a transitional interface [32,33] for switching between the 2D and 3D data. Thus, an essential requirement for a smooth transition is that the object must first be identified, localized in 6DoF, and augmented with its 3D model when it transitions from video to 3D. Then, after transitioning to the 3D virtual environment, the model can be freely explored (Figure 1).

4. Related Work

6DoF pose estimation using RGB images involves different fields such as bin picking problems [34], robot manipulation [35], autonomous vehicles [36], and MR applications [37].

Usually, to accomplish this task, deep learning methods are used. One of the main approaches to 6DoF pose estimation, as described in [38], is to decouple the translation and the rotation estimation. The translation is estimated by localizing the centre of the object in the image and predicting its distance from the camera. After that, the rotation is estimated by regressing to a quaternion representation. A 6DoF Object detection system with two stages is also proposed in [39]. A single Shot Multibox Detector (SSD) [40] extracts the object bounding boxes and an Augmented AutoEncoder (AAE) estimates the object rotation. Similar to the previous approach, DCS-PoseNet [41] uses a two-step process to estimate 6DoF from 2D object bounding-boxes. First, the framework segments the object from the cropped image, followed by predicting 6DoF pose using DSC-PoseNet, which employs a differential renderer.

Some solutions try to regress rotation and translation simultaneously. For example, 6D-VNet [42] uses an end-to-end deep learning network to estimate the 6DoF pose of vehicles. The network extends the Mask R-CNN object detector and takes its intermediate outputs and further regresses for rotation and translation of the object in 3D space.

Other approaches instead try to solve a Perspective-n-Point problem [43]. For example, the pose estimation method Pix2Pose [44] proposes a deep learning network to supplement a 2D detection pipeline to enable pose estimation. It regresses pixel-wise 3D coordinates from images using texture-less 3D models. The pixel-wise prediction is used to form 2D–3D correspondences. Finally, the PnP algorithm can be applied.

In Ref. [45], the authors propose an extension of the EfficientDet architecture [46] used for 2D object detection to predict the rotation and the translation of the object in the 3D space.

Most current works describe the problem statement and solution for regular RGB images. The application of the algorithms of these works to equirectangular images is tricky. The main reason is that equirectangular images present severe distortion and there is a lack of training data related to these types of images. To the best of our knowledge, some works try to perform 2D object detection in equirectangular images [47,48], but none perform an estimate of the 6DoF pose.

For this reason, our work contributes moving toward objects pose estimation also for this kind of images. Indeed, 360° videos are becoming more popular in the educational context, as they provide students with a more engaging learning activity [49]. The method presented in this paper shows an example of how this media, together with VR technologies, can create contemporary virtual learning environments, which enable students to experience and interact with virtual content [50]. 360° videos, in contrast with 2D movies, offer an immersive experience not only by giving the perception of being physically present in a virtual environment but also allowing interaction with objects. This is only possible when knowing the pose of the object with which we want to interact with inside the scene.

Our pipeline specifically solves the problem of 6DoF pose estimation for objects in equirectangular images. Additionally, while other methods primarily rely on deep learning models to perform the task, ours uses deep learning only for segmentation, which is the first step. It then uses an optimization technique for pose estimation that does not require a trained network and that can be applied with no effort to any type of object. Indeed, a benefit of the proposed method, in comparison to the related works presented in this section, is that there is no need to create a training dataset for the pose estimation, avoiding a task that can be quite a time consuming and difficult in terms of the acquisition of the ground truth pose, scalability, and full coverage of possible poses [51]. Other approaches use synthetic data for the training to avoid the this cumbersome task [52,53]. However, this method has the handicap of the gap domain between real and synthetic data [54]. To resume, in comparison to the traditional methods, the object pose detection phase of our method can be directly tested on the data that must be processed, ensuring a considerable time saving.

5. Algorithm Description

We can subdivide the developed algorithm into two main steps:

• Vehicle segmentation from the equirectangular image;
• Vehicle pose detection with respect to the camera reference frame (6DoF).

To accomplish the first task, we made use of Yolact++ [55], a convolutional model for real-time instance segmentation. Yolact++ proved to be pretty accurate for the segmentation of trucks, also in equirectangular images, in which the distortion is significant. However, to make the vehicle segmentation more robust, we trained Yolact++ with 500 equirectangular images by manually labelling trucks in frames taken from 360° videos of open-pit mining operations. Figure 2 shows the result of the trained Yolact++ model.

The only requirement for the vehicle pose detection is to have an accurate CAD model even without textures of the item, whose pose must be estimated. For the tests described in this paper, we used the CAD model of a Komatsu HD785 truck. We used the CAD model inside Unity 3D, a cross-platform game engine. In this case, we used Unity to create a simulation environment. We implemented a 360° camera simulator in this environment to capture equirectangular images of the CAD model.

Using the output given by Yolact++, we can perform the vehicle pose detection. To accomplish this task, we developed the algorithm schematically described in Figure 3.

Through a connection based on the ZMQ protocol, we allow data exchange between Unity and Matlab. We put the vehicle in a random pose inside Unity. Then we take a picture of the scene using the 360° camera simulation. We will call this picture the synthetic image. In Matlab, we compare this synthetic image with the output of Yolact++ (the segmented real-world image). A score is given for the similarity between the two pictures. A Particle Swarm (PS) optimizer is in charge of finding the optimal solution. At each new iteration, the algorithm sends a new pose to Unity. The 360° simulated camera takes a new picture, and we repeat the previous steps until convergence or the maximum number of iterations are reached.

Hereafter, the expression of the Cost Function (CF):

$$CF=S_E+S_A+S_C+S_V,$$

(1)

where its terms depend on:

• Edges ($S_E$);
• Area ($S_A$);
• Difference in the centroids of the edges ($S_C$);
• Difference in the eigen vectors of the edges ($S_V$);

In the following subsections, we explain the various term of the cost function in detail. It is important to highlight that the terms of the cost function also work for texture-less objects because they mainly rely on shape features than on texture. To show the potential of the developed algorithm, we choose to apply it to a vehicle model whose CAD was at our disposal. The CAD model does not perfectly match the real object in every detail but we will show that the results are accurate enough.

The real-world and synthetic images in Figure 4 are taken as an example to show the computations made for the different terms of the cost function.

5.1. Edges

The first term of the cost function is relative to edges. Using the “Canny” algorithm [56], we computed the images of the edges of the real-world ($E_r$) and synthetic image ($E_s$). Since a perfect correspondence between the CAD model and the actual vehicle is impossible, we smoothed the edges of $E_s$ by applying a Gaussian filter with a standard deviation of 0.5. We will call this last image $E_{sg}$. The two images are then multiplied pixel by pixel, computing $E_m$ as:

$$E_m=E_r·E_{sg}.$$

(2)

Figure 5 shows the images involved in the computation.

The score term is computed as follows:

$$S_E=1.0-\frac{n_m}{n_s},$$

(3)

where $n_m$ and $n_s$ are the number of pixels of $E_m$ and $E_s$, which are greater than zero.

5.2. Areas

The corresponding binary images ($B{W}_r$ and $B{W}_s$) are computed from the real-world and the synthetic ones. We call $A_s$ the area of $B{W}_s$, which is the number of pixels whose value is greater than 0. A dilated version of $B{W}_s$, called $B{W}_{sd}$, is also computed using a disk with a diameter of 7 pixels as the morphological structuring element. Let us indicate with $B{W}_d$ the difference between $B{W}_{sd}$ and $B{W}_s$:

$$B{W}_d=B{W}_{sd}-B{W}_s.$$

(4)

$A_d$ is the area of $B{W}_d$.

Now, it is possible to compute the images $M_a$ and $M_d$, i.e., the result of the pixel-wise multiplication between $B{W}_r$ and $B{W}_s$, and between $B{W}_r$ and $B{W}_d$:

$$\begin{array}{ccc}\hfill M_{rs}& =& B{W}_r·B{W}_s,\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill M_{rd}& =& B{W}_r·B{W}_d.\hfill \end{array}$$

(6)

Figure 6 shows the images involved in the computations.

We can compute the score relative to the areas with the following equation:

$$S_A=1.0-A_{rs}/A_s+A_{rd}/A_d,$$

(7)

where $A_{rs}$ and $A_{rd}$ are the corresponding areas of $M_{rs}$ and $M_{rd}$.

5.3. Difference in the Centroids of the Edges

This part of the cost function is in charge of computing the difference between the centroids of the images $E_r$ and $E_m$. The formula to compute the centroids of the images is as follows:

$$\begin{array}{ccc}\hfill x_c& =& {\displaystyle \frac{{\sum }_{i=1}^{N}I(x_i,y_i)·x_i}{{\sum }_{i=1}^{N}I(x_i,y_i)}},\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill y_c& =& {\displaystyle \frac{{\sum }_{i=1}^{N}I(x_i,y_i)·y_i}{{\sum }_{i=1}^{N}I(x_i,y_i)}},\hfill \end{array}$$

(9)

where $(x_c,y_c)$ are the coordinates of the centroid of the image I, N is the number of pixels whose value is greater than 0, $(x_i,y_i)$ are the general coordinates of the pixel i, and $I(x_i,y_i)$ is the grey value of the pixel in position $(x_i,y_i)$.

The cost function term is computed as:

$$S_C={\displaystyle \frac{\sqrt{{(x_{cr}-x_{cm})}^2+{(y_{cr}-y_{cm})}^2}}{\sqrt{R^2+C^2}}},$$

(10)

where $(x_{cr},y_{cr})$ and $(x_{cm},y_{cm})$ are the coordinates of the centroids of $E_r$ and $E_m$, and R and C are the number of rows and columns of $E_r$.

5.4. Difference in the Eigen Vectors of the Edges

The last term of the cost function can be explained as a constraint for the edge matching to be uniform on all the parts of the edge images, i.e., $E_r$ and $E_{sg}$. To reach this aim, we can arrange the image coordinates $(x_i,y_i)$ of the pixels whose value is greater than 0 in a matrix of dimension $N\times 2$, where N is the number of pixels whose value is greater than 0. Then we can compute the covariance matrices $C_r$ and $C_m$ of this matrix for $E_r$ and $E_m$. Once $C_r$ and $C_m$ are obtained, we can compute the eigenvectors for both. Let us call $\overrightarrow{v_r}$ and $\overrightarrow{v_m}$ the two eigenvectors that correspond to the highest eigenvalue for $C_r$ and $C_m$ (Figure 7).

The cost function term is the dot product between $\overrightarrow{v_r}$ and $\overrightarrow{v_m}$:

$$S_V=1.0-\overrightarrow{v_r}·\overrightarrow{v_m}.$$

(11)

6. Results

Figure 8 shows the experimental setup used to test the developed algorithm. A 360° camera, such as the Insta360 ONE X, is placed on a rotary stage which in turn is placed on a translation stage. The camera frames a miniature model of a Komatsu HD785 truck.

We acquired 10 images by translating the translation stage 8 cm of each new acquisition, and 11 images by rotating the camera 5° of each new acquisition (Figure 9).

Concerning the parameters used for the PS optimization, we set the swarm size to 150 and the maximum number of iterations to 75. The research range was set to ±20° for rotations and to ± 20 cm for translations. The initial pose conditions were set randomly from nominal values within the imposed research ranges. The algorithm ran on an Intel(R) Core(TM) i7-9700KF CPU. The mean computational time to find the optimum was about 20 min.

Table 1. Results obtained by the algorithm, applying a rotation of 5° at each step.

Nominal Angle [°]	Measured Angle [°]	Difference [°]
5.0	6.4	1.4
10.0	10.1	0.1
15.0	17.3	2.3
20.0	19.7	−0.3
25.0	26.3	1.3
30.0	31.0	1.0
35.0	37.0	2.0
40.0	41.9	1.9
45.0	48.2	3.2
50.0	52.1	2.1

and Figure 10 show the results obtained for the imposed rotations.

Table 2. Results obtained by the algorithm applying a translation of 8 cm at each step.

Nominal Translation [cm]	Measured Translation [cm]	Difference [cm]
8.0	8.4	0.4
16.0	16.0	0.0
24.0	24.9	0.9
32.0	32.9	0.9
40.0	40.8	0.8
48.0	49.0	1.0
56.0	57.1	1.1
64.0	67.9	3.9
72.0	76.0	4.0

and Figure 11 show the results obtained for the imposed translations.

Figure 12 shows an example of the results obtained; in this case the camera was rotated by 15 with respect to the initial orientation.

7. Discussion

Results show that the developed algorithm achieved good results for both translations and rotations. In particular, the maximum difference in the rotation estimation was 3.2° for the nominal rotation of 45° (see Table 1). Figure 13 shows the optimization result for this case.

The mean difference for the rotation is 1.5°, while the standard deviation is 1.0°.

The maximum difference in the translation estimation was instead 4 cm for the nominal translation of 72 cm (see Table 2). As shown in Figure 11, it seems that, at the increase of the amount of the translation, the difference increases. Most likely, looking at Figure 14, this is due to how the vehicle appears in the equirectangular image. In this case, the vehicle appears quite far and small translations cannot be appreciated from the image point of view. Indeed, also in this case, at least visually, the difference between the real world and the synthetic image does not seem to be relevant (Figure 14).

The mean difference for the translation is $1.4$ cm, while the standard deviation is $1.5$ cm. The computational time of 20 makes use of the proposed algorithm to be used offline. However, in the case of a video, once the pose is estimated in the first frame, the search field for the next frame is very limited because the vehicle will be in a pose very near to one of the previous frames. This will speed up the elaboration and the pose detection.

8. Conclusions

We presented an innovative method to estimate the 6DoF pose of vehicles in equirectangular images. This method relies on deep learning methods only for the object segmentation, while the pose is estimated through a cost function optimization. Only the CAD model of the object is needed for this step even without textures for the nature of the cost function used. This makes our method quite flexible to be applied to any kind of object and lighting conditions due to the lack of colour-affected terms in the comparison for pose estimation. We tested the results of our algorithm through an experimental setup, comparing them to measured rotations and translations applied to the camera in the real world. We obtained a maximum difference of 3.2° from the ground truth data for rotations, and 4 cm for translations over a research range of ±20° and ± 20 cm, respectively. Future works will try to improve the computational time and reduce the pose detection error.