A 3D Scene Information Enhancement Method Applied in Augmented Reality

Abstract

Aiming at the problem that the detection of small planes with unobvious texture is easy to be missed in augmented reality scene, a 3D scene information enhancement method to grab the planes for augmented reality scene is proposed based on a series of images of a real scene taken by a monocular camera. Firstly, we extract the feature points from the images. Secondly, we match the feature points from different images, and build the three-dimensional sparse point cloud data of the scene based on the feature points and the camera internal parameters. Thirdly, we estimate the position and size of the planes based on the sparse point cloud. The planes can be used to provide extra structural information for augmented reality. In this paper, an optimized feature points extraction and matching algorithm based on Scale Invariant Feature Transform (SIFT) is proposed, and a fast spatial planes recognition method based on a RANdom SAmple Consensus (RANSAC) is established. Experiments show that the method can achieve higher accuracy compared to the Oriented Fast and Rotated Brief (ORB), Binary Robust Invariant Scalable Keypoints (BRISK) and Super Point. The proposed method can effectively solve the problem of missing detection of faces in ARCore, and improve the integration effect between virtual objects and real scenes.

1. Research Background

Augmented reality technology is based on the integration of computer graphics, human–computer interaction and other technologies [1]. With the help of cameras, it displays more information by adding virtual objects to the real scene to enhance people’s understanding of the real world. Augmented reality technology has been widely applied in many areas, such as medical treatment, industry, entertainment and so on [2,3,4,5,6]. In recent years, augmented reality technology has been developed rapidly. Google and Apple, the two largest mobile phone companies in the world, have launched augmented reality development packages ARCore and ARKit based on the Android system and iOS system, respectively, which makes augmented reality technology popular in daily life and able to obtain more opportunities for its application.

The core technology of augmented reality is the seamless integration of virtual objects and real scenes. “Confuse the false with the true” has always been the ultimate goal of researchers in the field of augmented reality. The 3D registration technology is the core technology to improve the realism of augmented reality. Its essence is to properly place the virtual object into the real scene, so that it can be more integrated into the real scene. Therefore, the 3D reconstruction of the real scene structure is very important.

The 3D reconstruction method of a real scene is based on 3D data acquisition equipment or 2D images. The 3D data acquisition equipment includes a RGB-D camera, 3D scanner, LIDAR and so on. The equipment can acquire the depth data of the structure of the scene directly, which can be used to generate the 3D structure of the scene. Kinect is a represent 3D data acquisition equipment from Microsoft. Kinect fusion [7] can get the dense point cloud of the scene in real time and build the 3D structure. KinectFusion has been developed by many researchers [8,9,10]. The 3D reconstruction based on 3D data acquirement equipment has been very efficient and accurate, but cannot be applied widely because of the expensive equipment.

The 3D reconstruction method of a real scene based on images does not need equipment besides cameras, so it has wider application prospects. Many studies have concentrated on this field [11,12,13]. Goesele proposed the Multi-View Stereo (MVS) algorithm [14], which can weaken the influence of illumination to achieve a good reconstruction effect. Furukawa proposed the Patch-Multi-View Stereo (PMVS) algorithm [15], which can build the 3D details in a specific area. Snvaley proposed the Structure from Motion (SFM) algorithm [16], which can build the 3D structure from a serials of unordered images. Many works come from these classical algorithms [17,18,19,20,21,22,23].

Normally, the virtual objects from augmented reality are placed on a plane in the real scene. In most situations, we do not need to build all the 3D structures of a scene. Instead, we only need to recognize the planes where the virtual objects can be put on. Therefore, to recognize planes from the real scene is a very important research topic in the field of augmented reality.

At present, the plane recognition method used in augmented reality has certain recognition accuracy, but there are also some deficiencies. For example, the plane position recognition is not accurate enough, which makes it easy to produce the phenomenon of virtual objects suspension. It is easy to make recognition mistakes for single-color planes, which can result in inaccurate placement of virtual objects. Smaller planes are easy to be missed.

Taking ARCore as an example, ARCore extracts and classifies feature points from the video obtained by mobile camera, then estimates the position and size of the plane. It can detect large planes such as the ground and wall in the scene, but there are some problems. ARCore has a chance to misjudge the position of a small plane with unobvious texture, which can lead to the distortion of the augmented reality scene.

A plane-detection method which can be applied to 3D space in an AR scene is proposed in this paper. Based on a series of monocular images, the plane information of the scene is quickly constructed and embedded into the augmented reality scene, which provides additional supported information for the integration of virtual objects into the video scene, improves the integration of virtual objects into the real scene, and then achieves the purpose of enhancing the real effect of augmented reality.

2. Principle of the Method

As shown in Figure 1, the pinhole imaging model has three coordinate systems: the camera coordinate system, image coordinate system, and the world coordinate system. The origin of the image coordinate system is in the center of the image, which is the intersection point of the vertical line between the camera optical center and the imaging plane. The intersection point of the imaging plane α_i and the connecting line between the camera optical center O_c and a point P (X_C, Y_C, Z_C) in space is the phase point p_i (x₀, y₀) of P on the image. According to the triangular relationship, the relationship can be expressed by a homogeneous equation as follows:

$$Zc\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{ccc}f& 0& \begin{array}{cc}{x}_0& 0\end{array}\\ 0& f& \begin{array}{cc}{y}_0& 0\end{array}\\ 0& 0& \begin{array}{cc}1& 0\end{array}\end{array}\right]\left[\begin{array}{c}Xc\\ Yc\\ \begin{array}{c}Zc\\ 1\end{array}\end{array}\right]$$

(1)

where f is the focal length of the camera.

The point (x, y) in the image coordinate system is transformed into pixels through digital discretization. The pixels are represented by a matrix where the length and width are dx and dy. The relationship between the point (x, y) on the image coordinate system and its corresponding pixel point (U, V) can be expressed by the homogeneous equation as follows:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{dx}& 0& 0\\ 0& \frac{1}{dy}& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(2)

The transformation relationship between the coordinates (Xc, Yc, Zc) of point P in the camera coordinate system and its coordinates (Xw, Yw, Zw) in the world coordinate system can be expressed by the homogeneous equation as follows:

$$\left[\begin{array}{c}\begin{array}{c}Xc\\ Yc\\ Zc\end{array}\\ 1\end{array}\right]=\left[\begin{array}{cc}R& t\\ 0_3^T& 1\end{array}\right]\left[\begin{array}{c}\begin{array}{c}Xw\\ Yw\\ Zw\end{array}\\ 1\end{array}\right]$$

(3)

where $0_3^T=\left[000\right]$, R is the orthogonal rotation matrix, and t is the position of the camera optical center in the world coordinate system.

Substituting Formulas (2) and (3) into Formula (1), we get:

$$Zc\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{dx}& 0& 0\\ 0& \frac{1}{dy}& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}f& 0& \begin{array}{cc}{x}_0& 0\end{array}\\ 0& f& \begin{array}{cc}{y}_0& 0\end{array}\\ 0& 0& \begin{array}{cc}1& 0\end{array}\end{array}\right]\left[\begin{array}{cc}R& t\\ 0_3^T& 1\end{array}\right]\left[\begin{array}{c}\begin{array}{c}Xw\\ Yw\\ Zw\end{array}\\ 1\end{array}\right]$$

(4)

Formula (4) shows the process that a point from the world coordinate system is projected into an image, and then transformed into pixels. The dx, dy, and f are determined by the camera’s own attributes and belong to the internal parameters of the camera. The pose of the camera in the world coordinate system is determined by R and T and belongs to the external parameters of the camera.$\left[\begin{array}{ccc}\frac{1}{dx}& 0& 0\\ 0& \frac{1}{dy}& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}f& 0& \begin{array}{cc}{x}_0& 0\end{array}\\ 0& f& \begin{array}{cc}{y}_0& 0\end{array}\\ 0& 0& \begin{array}{cc}1& 0\end{array}\end{array}\right]\left[\begin{array}{cc}R& t\\ 0_3^T& 1\end{array}\right]$. The content of these three parts is called the projection matrix of the camera.

According to Formula (4), when the camera projection matrix is known, the equation of the line where the projection point P is located can be obtained by obtaining the pixel coordinates of the image. Therefore, the coordinates of P in the world coordinate system can be calculated by calculating the intersection point of multiple lines from more than two images.

Based on this, the position of the same point in different photos can be extracted as a PI point from the photos from different angles of the same scene taken by a monocular camera, and its coordinates in the world coordinate system can be calculated. In order to improve the accuracy of P_i point extraction, the points with significant features are usually selected as PI points, and then the R and t of the camera and the spatial position of point P in the world coordinate system are calculated by using an f matrix and e matrix. As long as the number of PI points extracted is enough, the information of enough spatial points P in the real scene can be obtained, and then the sparse point cloud of the scene can be formed. Based on the sparse point cloud, the plane information in the scene can be reconstructed to provide information support for augmented reality scene. In case the sparse point cloud cannot give enough feature points to describe the basic structure of the scene, a dense point cloud generation method [24] based on a sparse point cloud is used to get enough feature points.

At present, there are many methods to extract feature points from two-dimensional images, such as SIFT [25], the Oriented Fast and Rotated Brief (ORB) [26], Binary Robust Invariant Scalable Keypoints (BRISK) [27], Super Point [28] and so on [29,30,31,32,33,34,35]. The SIFT algorithm has strong robustness to scale and rotation changes. The ORB algorithm has strong rotation robustness, but weak scale robustness. The BRISK algorithm has strong robustness to scale and rotation, but the advantages are obvious in a big data set. The Super Point algorithm is based on a deep neural network. It needs a lot of time and data to train the network. Considering that the images from the scene are taken from different angles and different distances by the camera, which can cause the feature points to be rotated and scaled, the SIFT algorithm, which has strong robustness to scale and rotation changes, was selected for feature points extraction in this paper.

The SIFT algorithm works based on a Gaussian pyramid which consists of images fuzzy down-sampled by a Gaussian function.

Assuming I (x,y) is the image, G (x,y,σ) is the Gaussian function, and σ is the scale factor, then

$$G\left(x,y,\sigma \right)=\frac{1}{2\pi {\sigma }^2}{e}^{-\frac{x^2+y^2}{2{\sigma }^2}}$$

(5)

$L\left(x,y,\sigma \right)$ which is the scale space of an image, defined as

$$L\left(x,y,\sigma \right)=con\left(I\left(x,y\right),G\left(x,y,\sigma \right)\right)$$

(6)

where function con () is the convolution operation on image pixel (x, y), and $G \left(x,y,\sigma \right)$ is used as the convolution kernel.

The difference of Gaussian (DOG) function $D\left(x,y,\sigma \right)$ is defined to describe the differences between images in different scales.

$$D\left(x,y,\sigma \right)=con\left(I\left(x,y\right),G\left(x,y,{\sigma }_1\right)\right)-con\left(I\left(x,y\right),G\left(x,y,{\sigma }_2\right)\right)$$

(7)

The feature points come from the extreme points of the DOGs by comparing eight directional adjacent points in the current DOG image and 9 × 2 adjacent points from adjacent-scale DOGs.

$$D\left(X\right)=D+\frac{\partial D^T}{\partial X}X+0.5X^T\frac{{\partial }^{2}D}{\partial X^2}X$$

(8)

X is the position of the point.

The actual extreme point X_e may, between two extreme points from Formula (8), be calculated when the differential coefficient is 0.

$$X_e=-\frac{{\partial }^2{D}^{-1}\left(X\right)}{\partial X^2}\frac{\partial D\left(X\right)}{\partial X}$$

(9)

By substituting Formula (9) into Formula (8), we can get

$$D\left(X_e\right)=D+0.5\frac{\partial D^T}{\partial X}{X}_e$$

(10)

The descriptor of the feature point is used to identify the feature. It comes from the 16 neighbor pixels of the feature point. The gradient magnitude, m (x, y) and orientation θ (x, y) are calculated as below.

$$m\left(x,y\right)=\sqrt{{\left(I\left(x+1,y\right)+I\left(x-1,y\right)\right)}^2+{\left(I\left(x,y+1\right)+I\left(x,y-1\right)\right)}^2}$$

(11)

$$\theta \left(x,y\right)={\tan }^{-1}\left(I\left(x,y+1\right)-I\left(x,y-1\right)\right)/\left(I\left(x+1,y\right)-I\left(x-1,y\right)\right))$$

(12)

3. Experiments

The position and size of the plane in the scene determine the basic three-dimensional structure of the scene. Our method is to enhance the structure information for the AR scene by embedding invisible planes into the corresponding position in the augmented reality video scene.

In order to verify the information enhancement method proposed in this paper, an experiment scene was built where a sweeper is placed on the floor. For the scene where a sweeper is placed on the floor, 35 images were taken around the scene with a monocular mobile phone at a certain depression angle.

3.1. Feature Points Extraction and Matching Method Based on SIFT

The SIFT algorithm is used to extract and match the feature points of two images with adjacent angles. One of the results is shown in Figure 2. Because of the unobvious texture of the floor and the single color of the sweeper in the scene, the SIFT descriptor cannot effectively distinguish the feature points from two images, which leads to a series of mismatching points.

In order to improve the matching points accuracy rate, an optimized SIFT matching algorithm is proposed in this paper, and the optimization rules are as follows:

• The points of the real scene appear in two different images, which must be one-to-one between two images. It is impossible for one point in one image to be projected to many points in another image. Therefore, the one-to-many matching results must be the polluted results, which can be removed.
• The scene content will not change too much in two images with adjacent angles. Thus, the horizontal angle of the corresponding points in two images will not change too much. Therefore, the matching points with large matching alignment slope must be polluted results, and can be removed.

After the optimization, the accuracy of feature points matching results is significantly improved, and the matching effect is shown in Figure 3. It can be seen that the optimized algorithm significantly improves the accuracy of SIFT matching results, and eliminates a large number of misjudged data.

In order to verify the effectiveness of the method, SIFT, ORB, Super Point and the method proposed in the paper were used to extract and match the feature points of two images with adjacent angles. As shown in Figure 4, the two images at the top in the first row are the original input images. The second row is the SIFT operation result, the third row is the ORB operation result, the fourth row is the Super Point operation result, and the fifth row is the operation result of the proposed method. The results data are shown in

Table 1. Comparison of experimental results for Figure 4.

Algorithm	Feature Points (Left)	Feature Points (Right)	Matched	Accuracy
SIFT	513	377	183	75.41%
ORB	455	468	259	68.34%
Super Point	2023	1470	408	59.31%
Proposed Method	513	377	123	89.43%

. It can be seen that the numbers of matching feature points extracted by SIFT, ORB and the proposed method are smaller than the Super Point algorithm. The results by Sift, ORB and Super Point were doped with a large number of wrong matching results, while the matching accuracy of the proposed method is higher. The locations of planes in the AR scene can be determined based on the sparse point cloud, which has a low requirement for the number of feature points, but high requirement for the accuracy of feature points. Therefore, the method proposed in this paper is better than the other three methods.

The accuracy of feature points detection and matching results by SIFT, ORB, Super Point and our algorithm for 35 pairs of images are shown in Figure 5. It shows that our method has a higher level of accuracy than the other three algorithms.

3.2. Planes Recognition

Based on the effective feature point p_i detected by optimized SIFT, the coordinate of the spatial point P corresponding to the feature point was calculated. After the recursive calculation of 35 photos, a sparse point cloud composed of feature points of the scene can be obtained, as shown in Figure 6.

It can be seen that the sparse point cloud distribution of the scene demonstrates the three-dimensional structural characteristics of the scene. Although it cannot completely show the three-dimensional information of the scene, it can represent the basic structure of the scene, which can be used as an effective support for the augmented reality scene.

We tried to recognize planes by using the classic RANSAC algorithm. As shown in Figure 7, only the floor of the scene could be detected successfully by the RANSAC, while the upper plane was undetected. This is because there were 9963 points in the feature point set of the whole scene, including 9503 points on the lower surface. Due to the unobvious texture, there were only 359 feature points on the upper plane, which were mainly concentrated on the edge, so they were discarded by the classic RANSAC in the detection process.

In order to quickly recognize all possible planes from the point cloud, a plane recognition method has been designed in this paper. The steps are as follows:

• Make statistics and analysis on the height data of the point cloud of the scene. There should be a plane at the height where many points are concentrated. Figure 8 shows the number of points at a certain height. It is shown that most point clouds are concentrated in two height ranges, so it is inferred that there are two planes in the scene. The small point set corresponds to the upper plane of the sweeper, and the large point set corresponds to the ground.
• The points in the two height ranges are extracted to form two point sets. For each point set, the RANSAC algorithm is used to fit a plane. The process can be executed in parallel. The fitting plane effect is shown in Figure 9. Figure 9a shows the fitting effect of the small point set and Figure 9b shows the fitting effect of the large point set.
• After fitting the plane equation, the size of the upper plane of the sweeper in the scene can be determined through the coordinate range of small point set.

In order to verify the effect of the optimized algorithm, comparison experiment was performed on a Windows 10 64 system, with i7-11800H CPU and NVIDIA GeForce RTX 3050 Laptop GPU.

The efficiency of an algorithm describes the time used of a certain mission. The efficiency improvement EI is defined as below:

$$EI=\frac{T_r-T_p}{T_r}\times 100\%$$

(13)

where $T_r$ is the time cost to recognize planes by RANSAC, and $T_p$ is the time cost by the proposed method.

The operational results of the classical RANSAC method and the method proposed in this paper are shown in

Table 2. Comparison of plane recognition results.

	Recognized Planes	Accuracy	Time Cost (s)		Efficiency Improvement
Classical RANSAC	1	50%	0.193		0
Method Proposed	2	100%	Upper plane	0.012	14.5%
			Bottom plane	0.165

.

It can be seen that the proposed method in this paper could correctly identify two planes in the scene, and the recognition rate is 100%, while the classical RANSAC could only recognize one plane. Through parallel execution, the proposed method reduced a lot of the operation time and improved the execution efficiency by 14.5%.

4. Effect Combined with ARCore

Figure 10a,b shows the ARCore operation effect based on a scene where a sweeper is placed on the floor. In the AR scene, two virtual objects were placed on the ground and on the sweeper, respectively. Figure 10a shows the image of the scene when the camera is upon the object, and Figure 10b shows the scene image when the mobile phone was placed on the floor. It can be seen in Figure 10b that the virtual object on the sweeper was not placed on the sweeper, but sunk to the ground, which caused a phenomenon of virtual object suspension in the senses.

After adding supported information to the augmented reality scene, the operation effect of the augmented reality scene is shown in Figure 10c,d. When clicking on the upper surface of the sweeper, the embedded invisible virtual plane generates response information, and the virtual object is placed on the virtual plane on the upper surface of the sweeper, which results in the visual effect where the virtual object is placed on the sweeper, as shown in Figure 10c. When the shooting angle of the camera changes, as shown in Figure 10d, the view of the virtual object on the virtual plane also changes but does not sink to the ground. Obviously, this is much closer to the real scene. The method proposed in the paper corrects the misjudged plane problem in ARCore.

5. Conclusions

With the help of the series of images of a scene taken by a monocular camera, a 3D sparse point cloud of the scene was constructed, and the plane’s position and size in the 3D scene was estimated. The experimental results show that the augmented reality scene information enhancement method proposed in this paper is effective at solving the problem where with ARCore, it is easy to miss the detection of small planes with unobvious texture, and helps to place the virtual object on the undetected plane to improve the integration effect between the virtual object and the real scene. However, at present, this method requires a long calculation time, and needs to complete a series of image acquisitions and point cloud constructions of the scene in advance. Our next study will try to build 3D scene structures in real time in an augmented reality scene.