Hole Concealment Algorithm Using Camera Parameters in Stereo 360 Virtual Reality System

1. Introduction

Virtual reality (VR) has been one of the most important topics in the field of multimedia signals and systems for approximately 10 years. There are many applications for 360 VR systems, such as broadcasting, movies, social media communication, remote education, and virtual tourism [1]. 360 VR images and videos are generated using a 360 VR camera rig or a spinning VR camera, which is usually expensive or consists of heavy equipment.

As the technologies related to VR have been studied, methods that use a general purpose camera or cheap smartphone instead of the technical equipment to generate a 360 VR image have been developed [2,3,4,5]. In [2], the authors explained a method to stitch images using direct alignment or feature-based alignment. When the feature-based alignment approach is employed, feature points are searched in each picture to be stitched using one of the related algorithms [6,7,8,9,10,11]. Then, the feature points are matched by RANSAC [12], which provides homography matrixes to represent the relationship between neighboring pictures with overlapping common regions. Intrinsic and extrinsic matrixes of each picture can be derived from the homography matrix. The intrinsic and extrinsic matrixes of all pictures to be used for generating a VR image are jointly optimized through a bundle adjustment procedure [13]. In [3], a real time algorithm was proposed to make a panoramic image from pictures taken by the camera of a mobile phone, where the resulting image is shown on a cylindrical surface. The pictures taken by a mobile phone have different colors and brightness values compared to other pictures because these depend on the direction and position of the camera used in taking the pictures. This difference produces various artifacts in the VR image. In order to reduce the degradation, an algorithm to find the seams and blend the overlapping regions was proposed in [4]. Automatic Panoramic Image Stitching (APIS) [5] is one of the most efficient methods to make the panorama image or VR pictures. APIS consists of a variety of modules explained in [2,7,8,11,12] which include the feature-based alignment, Scale Invariant Feature Transform (SIFT), Speed-Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), Random Sample Consensus (RANSAC), bundle adjustment, straightening, and multiband blending.

In order to increase the quality of the stitched VR image, we can use high-end specialized equipment, for example GoPro VR cameras. When high-end specialized equipment is used to generate 360° VR images, it is considerably easier to align pictures without notable seam artifacts than that when using conventional cameras. Although VR images obtained from high-end specialized equipment have higher quality than those obtained from conventional cameras, conventional cameras are more affordable and simpler to use than specialized equipment. In fact, specialized equipment is bulky and expensive, thereby hindering its use in daily life.

With the increase in the viewer requirement for a sense of immersion, stereo 360 VR systems have been introduced to meet this need [14,15]. A stereo 360 VR image provides a more realistic and immersive sense to the viewer than a monocular 360 VR image, but the cost to make the contents increases further. The high cost of the equipment to generate 360 VR images may be one of the things preventing the spread of applications based on VR technology. This paper considers a system to generate a stereo 360 VR image using two cheap smartphones or two general purpose cameras attached to a rig.

In our scenario, because the general-purpose cameras are used to capture pictures, the pictures taken in arbitrary directions may produce non-overlapping regions and those become holes in the stitched image. In order to conceal the holes in stereo 360 VR images, various inpainting algorithms [16,17,18,19,20,21,22,23,24] can be considered as solutions, even though these were originally proposed for non-VR images. Bertalmio et al. [16] proposed an inpainting algorithm based on fluid dynamics, where a two-dimensional Navier–Stokes equation is approximated and combined with the Poisson equation for the vorticity. This approach produces an approximate solution for the holes in non-VR images. In the algorithm proposed by Telea et al. [17], each pixel in a hole is progressively concealed from the boundary to the center of the hole, where the value of a pixel to be concealed is set by the weighted average of the neighbouring pixels. Criminisi et al. [18] proposed a method to conceal a hole by filling it using a block. In this algorithm, the most similar patches are searched over a region around the hole. This scheme is a kind of template matching technique and the squared sum of the difference (SSD) is used as a criterion. On other hand, algorithms based on deep learning techniques were proposed in [19,20,21,22,23,24]. Liu et al. [19] proposed a deep learning-based method to conceal a hole in a non-VR image, where convolution weights are normalized by the mask area of the window. This effectively prevents the convolution filters from capturing too many zeros when they traverse over the incomplete region. When the deep learning-based algorithm is applied, the coefficients and variables in the convolutional network should be optimized for the huge quantity of training data. Thus, it requires a long training time and the performance depends on the type of training data. Note that certain conventional methods [16,17,18,19,20,21,22,23,24] use the data of the same camera stream to conceal the holes. In these techniques, the holes are filled based on the neighboring pixels and therefore their performances are limited when the size of the hole is large.

Unlike those techniques [16,17,18,19,20,21,22,23,24], when we fill the holes in the stereoscopic VR images, we use the data from the opposite camera stream. Using the data of the opposite camera stream is more efficient than using the data from the same camera stream, because the left and right views are correlated with each other. In order to compare the performances of two categories (using the data from same camera stream and opposite camera streams), we provide the simulation results obtained by methods in two categories in Section 4.

This paper proposes an efficient algorithm to conceal the holes in stereo 360 VR images, where camera parameters are utilized to estimate the location of the most similar fraction in other view images.

This paper is organized as follows. In Section 2, we formulate the problem, where two general purpose cameras are used to capture pictures for the left and right views. These pictures are used to generate stereo 360 VR images, where some blocks related to a hole are defined. In Section 3, we propose an efficient algorithm to conceal the holes, where the geometrical property of the pixels in VR images and the relationship between them are analyzed. Simulation results that demonstrate the performance of the proposed algorithm are provided in Section 4, where the performances of various techniques are compared subjectively and objectively. In Section 5, we provide a brief conclusion for this paper.

2. Problem Formulation

The Figure 1 shows the process used to make the stereo 360 VR images, which consist of left and right equi-rectangular projections (ERPs). In this configuration, the left and right cameras are mounted on a rig and capture pictures in the same direction, where general-purpose cameras (e.g., built-in cameras in smartphones) are used to capture pictures. The left and right ERPs are generated by applying several techniques to the sets of pictures with the left and right views. For example, the left ERP is made by consecutively applying feature extraction, mapping between features, a calculation of the homography matrix to represent the relationship, bundle adjustment to optimize the camera parameters, seam finding, and blending. The right ERP image is also made by a series of the same techniques. In this paper, APIS [5] is used to make the left and right ERPs, because it is known as one of the most efficient algorithms. APIS is applied independently to the left and right sets of pictures that have been captured by two built-in smartphone cameras. In this scenario, no mechanical equipment is used, other than a tripod. Thus, the pictures are captured in arbitrary directions and positions. Note that APIS does not include a function to conceal the holes in the stitched image. APIS is the basic style algorithm to stitch the pictures.

Figure 2 shows an example of the left and right ERPs that are generated using the procedure shown in Figure 1. As observed in Figure 2, we find some holes in these pictures. In the scenario illustrated in Figure 1, given that general-purpose cameras (instead of specialized equipment) are used, their narrow field-of-view (FOV) and the fact that pictures are often captured in arbitrary directions may produce non-overlapping regions and distortion (e.g., ghost effect or misalignment) in stitched 360° VR images. Non-overlapping regions also show up owing to the limitations of the stitching module, because the derivation of the relationship between spatially neighboring pictures is based on the RANSAC and feature points of each picture. The non-overlapping regions become holes in the stitched image, where the shape of the hole is dependent on the structure of the neighboring warped pictures. The shape of the holes may look like a square or a lozenge because those holes are surrounded by the neighboring warped pictures. Examples of the holes are shown in Section 4, where we can see the various shapes of holes.

In Figure 2, $B_L^c$ is a block overlapping a hole in the left ERP. $B_L^l$ and $B_L^r$ are the left and right sides neighboring blocks of $B_L^c$, respectively. $B_R^c$, $B_R^l$, and $B_R^r$ in the right ERP are blocks corresponding to $B_L^c$, $B_L^l$, and $B_L^r$, respectively. The corresponding blocks in the left and right ERPs are located in the same positions.

This paper proposes an efficient algorithm to conceal the holes, where pixels in the holes are filled with pixels in other ERPs and the location of the pixel used is derived from the camera parameters of the neighboring blocks {$B_R^c$, $B_R^l$, $B_R^r$, $B_L^c$, $B_L^l$, $B_L^r$}.

3. Algorithm Proposed to Conceal a Hole

3.1. Hole Detection

Figure 3 explains the process used to detect a hole in an ERP image. As observed in Figure 2, because a hole is a region that has not been covered by the warped pictures using the stitching algorithm, the color of the hole is black, with a luminance of zero. Thus, in step 1 of Figure 3, the colored ERP image is converted to a black-and-white picture to efficiently detect the holes. In step 2, all of the non-black pixels are replaced by white pixels, as shown in Figure 3c, where the black and white pixels have gray levels of 0 and 255 for an image represented with 8 bits/pixel, respectively. This results in the candidate regions for holes. Note that not all of the pixels in the dark objects have a zero value, even though some of the pixels in the objects may have a zero value. In order to classify the black pixels as hole and non-hole regions, a contour finding algorithm [25] is applied to image Figure 3c in step 3, which provides several contours with various sizes. In image Figure 3d, the perimeter of the contour of the hole is longer than that of a non-hole region, because the non-hole region is part of the dark object and it has the shape of an isolated particle. Therefore, if the length of the contour is shorter than a threshold, the dark region is removed, as shown in image Figure 3e. Finally, the block overlapping the dark region is set to $B_L^c$ in image Figure 3e.

3.2. Camera Parameters of Neighboring Blocks

In Figure 1, when an ERP image is made, the stitching algorithm is applied to the set of pictures that have been taken by a single camera (the left or right camera). As explained in Section 2, each picture in a set of pictures with the left or right views is warped and translated according to the intrinsic matrix K and extrinsic matrix R of each picture. The following equations are the intrinsic and extrinsic matrixes of picture $I_i$.

$${K(I}_i)=\left(\begin{array}{ccc}{f}_x{(I}_i)& {s(I}_i)& c_x{(I}_i)\\ 0& f_y{(I}_i)& c_y{(I}_i)\\ 0& 0& 1\end{array}\right)$$

(1)

$${\mathrm{R}(I}_i)=\left(\begin{array}{ccc}{r}_{11}{(I}_i)& r_{12}{(I}_i)& r_{13}{(I}_i)\\ r_{21}{(I}_i)& r_{22}{(I}_i)& r_{23}{(I}_i)\\ r_{31}{(I}_i)& r_{32}{(I}_i)& r_{33}{(I}_i)\end{array}\right)$$

(2)

where elements $f_x{(I}_i)$ and $f_y{(I}_i)$ are the horizontal and vertical focal lengths of the camera used, respectively. ($c_x(I_i){,c}_y(I_i))$ is the location of the intersection between the Z axis of the world coordinate system and the plane of the image sensor. $s$ is a skew coefficient. In (2), R is a rotation matrix used to represent the direction of camera lens when picture $I_i$ is taken. K and R are derived from a homography matrix calculated by RANSAC. The parameters in the K’s and R’s of all the pictures are optimized by the bundle adjustment based on the Levenberg–Marquardt algorithm [26].

As explained in Section 2, the blocks {$B_R^c$, $B_R^l$, $B_R^r$, $B_L^c$, $B_L^l$, $B_L^r$} have their own K’s and R’s, which can be represented by substituting {$B_R^c$, $B_R^l$, $B_R^r$, $B_L^c$, $B_L^l$, $B_L^r$} into $I_i$ in (1) and (2) as follows. As the equations have the same pattern, we represent those for $B_L^l$ only.

$$K(B_L^l)=\left(\begin{array}{ccc}{f}_x(B_L^l)& s(B_L^l)& c_x(B_L^l)\\ 0& f_y(B_L^l)& c_y(B_L^l)\\ 0& 0& 1\end{array}\right)$$

(3)

$$\mathrm{R}(B_L^l)=\left(\begin{array}{ccc}{r}_{11}(B_L^l)& r_{12}(B_L^l)& r_{13}(B_L^l)\\ r_{21}(B_L^l)& r_{22}(B_L^l)& r_{23}(B_L^l)\\ r_{31}(B_L^l)& r_{32}(B_L^l)& r_{33}(B_L^l)\end{array}\right)$$

(4)

3.3. Concealment Based on Intrinsic and Extrinsic Matrixes

In this subsection, we derive the relationship between the stereo views, where the camera parameters of the warped pictures are utilized. In Figure 4, $I_L$ and $I_R$ are the warped images after they have been taken by the left and right cameras, respectively. Based on the geometric relation between the camera coordinate and world coordinate systems [27], the (X, Y, Z) of point P in the world coordinate system is mapped to the $(x_L,y_L)$ and $(x_R{,y}_R)$ of points $p_L$ and $p_R$ in $I_L$ and $I_R$, respectively. The relation is represented as follows.

$${P=\mathrm{R}(I}_R{)K(I}_R{)}^{-1}{p}_R$$

(5)

$${P=\mathrm{R}(I}_L{)K(I}_L{)}^{-1}{p}_L$$

(6)

Combining (5) and (6) gives

$${\mathrm{R}(I}_R{)K(I}_R{)}^{-1}{p}_R{=\mathrm{R}(I}_L{)K(I}_L{)}^{-1}{p}_L$$

(7)

$$p_R=K(I_R){\mathrm{R}(I}_R{)}^{-1}\mathrm{R}(I_L){K(I}_L{)}^{-1}{p}_L$$

(8)

Therefore, the relation between the pixels in $I_L$ and $I_R$ can be represented with a $3\times 3$ matrix ${H(I}_L\to I_R)$ as follows.

$$p_R{=H(I}_L\to I_R{)p}_L$$

(9)

where

$$H(I_L\to I_R)=K(I_R){\mathrm{R}(I}_R{)}^{-1}\mathrm{R}(I_L){K(I}_L{)}^{-1}$$

(10)

The ${H(I}_L\to I_R)$ can be calculated when ${K(I}_R)$, ${\mathrm{R}(I}_R)$, ${\mathrm{R}(I}_L)$, and ${K(I}_L)$ are known. If $I_L$ and $I_R$ are replaced by $B_L^c$ and $B_R^c$, respectively, then (10) becomes

$$H(B_L^c\to B_R^c)=K(B_R^c){\mathrm{R}(B}_R^c{)}^{-1}\mathrm{R}(B_L^c){K(B}_L^c{)}^{-1}$$

(11)

In the proposed algorithm based on (11), each pixel in a hole is filled with a pixel in the right ERP, where $H(B_L^c\to B_R^c)$ provides the location of the pixel in the right ERP to replace a particular pixel in a hole. This means that we can conceal the hole if ${K(B}_R^c)$, $\mathrm{R}(B_R^c)$, $\mathrm{R}(B_L^c)$, and ${K(B}_L^c)$ are known.

On the other hand, when the hole is found in the right ERP, the following equation is used instead of (11).

$$H(B_L^c\leftarrow B_R^c)=K(B_L^c){\mathrm{R}(B}_L^c{)}^{-1}\mathrm{R}(B_R^c){K(B}_R^c{)}^{-1}$$

(12)

Figure 5 shows an example of concealing each pixel in a hole when the hole is found in the left ERP.

3.4. Prediction for Camera Parameters

This subsection explains the method used to predict ${K(B}_R^c)$, $\mathrm{R}(B_R^c)$, $\mathrm{R}(B_L^c)$, and $K(B_L^c)$ in order to derive ${H(B}_L^c\to B_R^c)$ in (11). As shown in Figure 2, because $B_L^c$ includes a hole, $\mathrm{R}(B_L^c)$ and ${K(B}_L^c)$ are unknown. In addition, because $B_R^c$ consists of fractions of multiple pictures in the right view, ${K(B}_R^c)$ and $\mathrm{R}(B_R^c)$ should be recalculated instead of using those resulting from the process to make the right ERP. In order to predict ${\{K(B}_R^c)$, $\mathrm{R}(B_R^c)$, $\mathrm{R}(B_L^c)$, ${K(B}_L^c)\}$, the following operations are executed.

First, we estimate homography matrixes ${H(B}_L^l\to B_R^l)$ and ${H(B}_L^r\to B_R^r)$ for pairs {$B_L^l$, $B_R^l$} and {$B_L^r$, $B_R^r$}, respectively. ${H(B}_L^l\to B_R^l)$ and ${H(B}_L^r\to B_R^r)$ are derived by applying “feature extraction,” RANSAC, and “bundle adjustment” to those pairs. Figure 6 shows the examples used to estimate ${H(B}_L^l\to B_R^l)$ and ${H(B}_L^r\to B_R^r)$, which have the following relationships.

$$H(B_L^l\to B_R^l)=K(B_R^l)\mathrm{R}{(B_R^l)}^{-1}\mathrm{R}(B_L^l){K(B}_L^l{)}^{-1}$$

(13)

$$H(B_L^r\to B_R^r)=K(B_R^r){\mathrm{R}(B}_R^r{)}^{-1}\mathrm{R}(B_L^r){K(B}_L^r{)}^{-1}$$

(14)

From the homography matrixes in (13) and (14), {${K(B}_R^l)$, $\mathrm{R}(B_R^l)$, $\mathrm{R}(B_L^l)$, ${K(B}_L^l)$} and {${K(B}_R^r)$, $\mathrm{R}(B_R^r)$, $\mathrm{R}(B_L^r)$, ${K(B}_L^r)$} are derived by solving the simultaneous equations related to the focal length.

Second, $\mathrm{R}(B_R^c)$ and $\mathrm{R}(B_L^c)$ are estimated from {$\mathrm{R}(B_R^l)$, $\mathrm{R}(B_L^l)$} and {$\mathrm{R}(B_R^r)$, $\mathrm{R}(B_L^r)$}, respectively, which were obtained in the previous step. In [28], a rotation matrix R can be represented as a combination of rotational components ${\mathrm{R}}_x$, ${\mathrm{R}}_y$, ${\mathrm{R}}_z$ on the x, y, z axes as follows.

$${\mathrm{R}=\mathrm{R}}_z(\theta ){\mathrm{R}}_y(\mathsf{\sigma }){\mathrm{R}}_x(\varphi )=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}\cos \mathsf{\sigma }& 0& \sin \mathsf{\sigma }\\ 0& 1& 0\\ -\sin \mathsf{\sigma }& 0& \cos \mathsf{\sigma }\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \varphi & -\sin \varphi \\ 0& \sin \varphi & \cos \varphi \end{array}\right]$$

(15)

where $\varphi ,$ $\mathsf{\sigma }$, and $\theta$ are the Euler angles about the x, y, and z axes, respectively. Applying the property to {$\mathrm{R}(B_L^l),\mathrm{R}(B_L^c)$, $\mathrm{R}(B_L^r)$, $\mathrm{R}(B_R^l)$, $\mathrm{R}(B_R^c)$, $\mathrm{R}(B_R^r)$} gives

$$\mathrm{R}(B_j^i){=\mathrm{R}}_z({\theta (B}_j^i)){\mathrm{R}}_y({\mathsf{\sigma }(B}_j^i)){\mathrm{R}}_x(\varphi {(B}_j^i))$$

(16)

$${\mathrm{R}}_x(\varphi {(B}_j^i))=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \varphi {(B}_j^i)& -\sin \varphi {(B}_j^i)\\ 0& \sin \varphi {(B}_j^i)& \cos \varphi {(B}_j^i)\end{array}\right]$$

(17)

$${\mathrm{R}}_y({\mathsf{\sigma }(B}_j^i))=\left[\begin{array}{ccc}{\cos \mathsf{\sigma }(B}_j^i)& 0& {\sin \mathsf{\sigma }(B}_j^i)\\ 0& 1& 0\\ {-\sin \mathsf{\sigma }(B}_j^i)& 0& {\cos \mathsf{\sigma }(B}_j^i)\end{array}\right]$$

(18)

$${\mathrm{R}}_z({\theta (B}_j^i))=\left[\begin{array}{ccc}{\cos \theta (B}_j^i)& {-\sin \theta (B}_j^i)& 0\\ {\sin \theta (B}_j^i)& {\cos \theta (B}_j^i)& 0\\ 0& 0& 1\end{array}\right]$$

(19)

where i = {l, c, r} and j = {L, R}. In (16)–(19), $\varphi {(B}_j^i)$, ${\mathsf{\sigma }(B}_j^i)$, and ${\theta (B}_j^i)$ are the Euler angles about the x, y, and z axes in $\mathrm{R}(B_j^i)$, respectively. As observed in Figure 2, because $B_L^c$ and $B_R^c$ are located in the centers of {$B_L^l$ and $B_L^r$} and {$B_R^l$ and $B_R^r$}, respectively, we can assume that their Euler angles have the following relationships.

$$\varphi (B_j^c)=\frac{\varphi (B_j^l)+\varphi {(B}_j^r)}{2}$$

(20)

$$\mathsf{\sigma }(B_j^c)=\frac{\mathsf{\sigma }(B_j^l){+\mathsf{\sigma }(B}_j^r)}{2}$$

(21)

$$\theta (B_j^c)=\frac{\theta (B_j^l){+\theta (B}_j^r)}{2}$$

(22)

where $j=\left\{L,R\right\}$. Substituting (20)–(22) into (16) provides $\mathrm{R}(B_L^c)$ and $\mathrm{R}(B_R^c)$.

Third, ${K(B}_L^c)$ and ${K(B}_R^c)$ are estimated from {${K(B}_R^l)$, ${K(B}_L^l)$} and {${K(B}_R^r)$, ${K(B}_L^r)$}, respectively, which were derived in the first step. As we can see in (1), because the K matrixes consist of the focal length, aspect ratio, and center point, ${K(B}_L^c)$ and ${K(B}_R^c)$ can be derived by averaging the K’s of the neighboring blocks, as follows.

$$K(B_j^c)=\frac{K(B_j^l){+K(B}_j^r)}{2}$$

(23)

where j = {L, R}.

Finally, because we obtained {${K(B}_R^c)$, $\mathrm{R}(B_R^c)$, $\mathrm{R}(B_L^c),{K(B}_L^c)$} in the previous steps, substituting them into (11) results in ${H(B}_L^c\to B_R^c)$, which can be used to conceal each pixel in a hole.

3.5. Summary of the Proposed Algorithm

Figure 7 summarizes the proposed algorithm. In the first step, the hole is found using the algorithm described in Section 3.1, where the hole $B_L^c$ and the neighboring blocks $B_L^l$, $B_L^r$, $B_R^c$, $B_R^l$, and $B_R^r$ are defined according to the relationship between their coordinates. In the second step, we derive ${K(B}_R^c)$, $\mathrm{R}(B_R^c)$, $\mathrm{R}(B_L^c)$, and ${K(B}_L^c)$ using the algorithm proposed in Section 3.4. In the third step, each pixel in the hole $B_L^c$ is filled with the corresponding pixel in the right ERP, where the ${H(B}_L^c$ → $B_R^c)$ of (11) provides the coordinate of the corresponding pixel. When the hole is in the right VR image, $H(B_L^c\leftarrow B_R^c)$ of (12) is used to fill the hole.

The main characteristics of the proposed algorithm are as follows: First, the proposed algorithm conceals the holes in stereoscopic VR images, whereas most conventional algorithms [16,17,18,19,20,21,22,23,24,29] for filling holes were proposed for non-VR images. In particular, certain concealment algorithms [30,31,32] were proposed to fill the holes in depth images, where the holes have arbitrary shapes and some of them are small particles. Second, our algorithm can derive the location of the most similar pixel for each pixel in a hole. In our scenario, we use the data of the camera stream of another view to fill the holes, where the exact location of the most similar part should be calculated to increase the objective and subjective quality of the filled region. The position is estimated by analyzing Euler angles of extrinsic matrices of the related neighbor blocks. In the analysis of Euler angles, we decompose the angles along x, y, and z axes and then merge them to find the exact location of the reference data. The relevant explanation is provided in Section 3.4.

4. Simulation Results

In order to demonstrate the performance of the proposed algorithm, we compare it with various conventional methods, including those of Bertalmio [16], Telea [17], Criminisi [18], Liu [19], GiliSoft Stamp Remover [33], and Theinpaint [34] from the subjective and objective viewpoints. These methods were explained in Section 1.

We captured picture sets using the built-in camera of a Samsung Galaxy 9 smartphone. There were more than 100 pictures in each set. The pictures within a single set were stitched using APIS to create the stereoscopic VR images. Figure 8 shows the left or right VR images of the stereoscopic VR images having holes of various sizes and locations. The VR images including the holes were used as test sets to evaluate the performances of the proposed algorithm and various conventional methods. In Figure 8a–f, the holes have been generated through the stitching algorithm owing to the reason described in Section 2. Those pictures are used only for subjective evaluations because there is no reference VR image without any hole. When we created Figure 8g,h, the stitched VR images did not have any holes. Therefore, we have made an artificial hole in each VR image. Consequently, Figure 8g,h can be used for objective evaluation, because the difference between the reference image (without any hole) and the concealed images (the results of various algorithms) can be calculated numerically.

4.1. Subjective Performance

In order to check the performance of the proposed algorithm, we applied the proposed algorithm for the test sets of Brick Road, Alleyway, Tree, Bench, and Street. In Figure 9, VR images including the holes are compared with the concealed images, respectively. As shown in Figure 9, we can see that the proposed algorithm conceals the various holes effectively in the test images.

To compare the performances of various techniques subjectively, the pictures whose holes have been concealed by the methods are shown in Figure 11, where all of the pictures are a part of the left ERP picture and we made a hole artificially as Figure 10b in order to compare the concealed pictures Figure 11a–h with the reference image Figure 10a.

As observed in Figure 11, the proposed algorithm outperforms the other techniques subjectively. In the picture Figure 11a result from Bertalmio [16], the pixels in the hole are filled by copying the neighboring pixels horizontally or vertically. Thus, the concealed region has artifact lines. When the method of Telea [17] is used to conceal the hole, each pixel in a hole is concealed progressively from the boundary to the center of the hole, where the value of the concealed pixel is set to the weighted sum of the values of the neighboring pixels. Thus, the concealed region of Figure 11b has smoothed and diffused distortion. In Figure 11c, when the method of Criminisi [18] is used, the pixels in the hole are filled by blocks, where the template matching algorithm is used. In this method, the SSD of template part is considered a cost function to estimate the most similar block, and the values of the pixels in the non-template part are not considered. Thus, some pixels in the hole are filled with wrong values. As shown in Figure 11d, the method of Liu [19] based on deep learning has filled the hole with inappropriate blocks, which are generated from the deep learning engine. Figure 11e,f were resulted from Gilisoft stamp remover [33] and Theipaint [34], respectively. As shown in Figure 11e,f, these methods does not remove the holes completely. In Figure 11g, the hole in the left ERP is concealed just by copying a block in the right ERP, where the shapes and coordinates of the blocks, including the hole in the left ERP and the copied block in the right ERP, are same. As observed in Figure 11g, the concealed region has significant mismatches around the hole, because this method does not consider the disparity between the left and right views. Whereas the conventional techniques have serious artifacts in the concealed region, the image Figure 11h produced by the proposed algorithm has a natural and continuous boundary in the concealed region.

Figure 12 and Figure 13 show the simulation results for indoor pictures, where the size of the hole is bigger than that of Figure 10 and Figure 11. As can be seen from Figure 12 and Figure 13 and Figure 10 and Figure 11, the performance tendencies of the various algorithms do not change even if the size of the hole is varied.

Figure 14 shows the simulation results for the naturally generated holes that occurred non-artificially during the stitching procedure. Note that no reference picture was used in this test. In this configuration, both the left and right ERP images had holes at different locations independently. In this test, the concealment algorithms were applied to the left and right ERP pictures independently. The results of the conventional and proposed techniques are shown in Figure 14b–g,h, respectively. When the proposed algorithm was applied, Equations (11) and (12) were used to conceal the holes in the left and right ERPs, respectively. As observed in Figure 14, the proposed method outperformed the other conventional techniques subjectively. In addition, the performance tendencies of the techniques were the same as those of Figure 10 and Figure 11 and Figure 12 and Figure 13.

4.2. Objective Performance

The objective performances of the concealment algorithms were evaluated with common measures like the structural similarity (SSIM) [35], peak-to-peak signal-to-noise ratio (PSNR) [36], and consumed CPU time. All experiments are run on the CPU based on AMD Ryzen 3.40 GHz and 32-GB RAM. The SSIM and PSNR are the average values of the SSIM and PSNR values for the red, green, and blue components. SSIM is in the range of 0–1. As the values of SSIM and PSNR increase, the quality of the resulting image increases. Note that SSIM and PSNR can be evaluated when the reference pictures are given, as in Figure 10 and Figure 11 and Figure 12 and Figure 13.

On the other hand, the CPU times consumed by the methods were checked using a personal computer. The CPU time was measured with the resolution of a second. As the CPU time depended on the complexity of the algorithm, the technique consuming the smallest CPU time was considered the simplest method.

The outdoor pictures (Mozart Hall) shown in Figure 10 and Figure 11 are evaluated in

Table 1. Comparison between performances of various methods for outdoor pictures (Mozart Hall).

Method	SSIM	PSNR (dB)	Time (s)
Bertalmio [16]	0.6483	19.6982	12.9177
Telea [17]	0.6320	16.5155	13.1796
Criminisi [18]	0.4406	17.0804	30346
Liu [19]	0.2381	17.3725	N/A
Gilisoft [33]	0.7427	24.6347	2.1600
Theinpaint [34]	0.7701	25.0058	28.1900
Simple copy	0.6196	21.1793	0.0160
Proposed algorithm	0.7954	25.4479	7.3261

and

Table 2. Performances of proposed method according to the size of the neighboring blocks for outdoor pictures (Mozart Hall).

Width Ratio	SSIM	PSNR (dB)	Time (s)
1.0	0.7491	24.1525	5.4439
1.2	0.7585	24.4355	6.3663
1.5	0.7954	25.4479	7.3261
1.75	0.7614	24.6553	8.0263
2.0	0.5986	19.8681	8.8201

, where the best and 2nd best results are represented with red and blue numbers, respectively. As seen in Table 1, the proposed algorithm produced the best quality from the viewpoints of both the SSIM and PSNR. Theipaint [34] had the 2nd best quality for both SSIM and PSNR. The “simple copy” required the smallest CPU time, because it filled the hole just by copying the same size block of the other view. The proposed algorithm is one of the simplest algorithms in the consumed time. When the method of Liu [19] was used, the consumed CPU time could not be measured because the training time was huge and depended on the quantity of the training data.

Table 2 shows the performances of the proposed algorithm for a variety of sizes of neighboring blocks {$B_R^l$, $B_R^r$, $B_L^l$, $B_L^r$}, where the numbers in the first column denote the ratio between the widths of neighboring block $B_L^l$ and block $B_L^c$ that includes the hole. As observed in this table, the best performance is found when the ratio is 1.5 (i.e., the width of $B_L^l$ is 50% wider than that of $B_L^c$).

When the size of the neighboring blocks is small (the ratio is under 1.5), the performance of the proposed algorithm increases with the size, because the accuracy of ${H(B}_L^c\to B_R^c)$ increases as the number of pixels utilized to derive it increases. However, when the size is large (the ratio is over 1.5), the neighboring blocks include fractions of multiple pictures, which degrades the accuracy of ${H(B}_L^c\to B_R^c)$. In this table, the complexity increases with the width of the neighboring blocks, because the number of pixels to be considered in making ${H(B}_L^l\to B_R^l)$ and ${H(B}_L^r\to B_R^r)$ increases.

Table 3. Comparison between performances of various methods for indoor pictures (Hallway) having a large hole.

Method	SSIM	PSNR (dB)	Time (s)
Bertalmio [16]	0.7153	18.6928	15.7136
Telea [17]	0.7343	18.0315	16.3712
Criminisi [18]	0.5691	18.6113	62791
Liu [19]	0.3511	16.1611	N/A
Gilisoft [33]	0.7487	23.5176	0.5500
Theinpaint [34]	0.7585	22.5666	6.3600
Simple copy	0.6544	16.9261	0.2273
Proposed algorithm	0.7721	23.8020	8.8516

and

Table 4. Performances of proposed method according to size of neighboring blocks for indoor picture (Hallway) having a large hole.

Width Ratio	SSIM	PSNR (dB)	Time (s)
1.0	0.6031	15.8433	6.7485
1.2	0.6476	18.3133	7.6910
1.5	0.7721	23.8020	8.8516
1.75	0.6505	17.6263	9.3063
2.0	0.6656	18.7562	9.9974

show the simulation results for the indoor pictures (Hallway) shown in Figure 12 and Figure 13. Note that the hole in Figure 12 and Figure 13 are larger than that of Figure 10 and Figure 11. As observed in Table 3 and Table 4, the performance tendencies of the methods are similar to those seen in Table 1 and Table 2. As observed in Table 3, the proposed algorithm has the best performance. Gilisoft [33] and Theipaint [34] had the 2nd best quality for PSNR and SSIM, respectively. The 2nd best performance is seen for the different ratios in Table 2 and Table 4 because the pictures of Figure 12 and Figure 13 include more complex regions and larger holes than those of Figure 10 and Figure 11.

From the results in Table 1, Table 2, Table 3 and Table 4, we can see that the proposed algorithm outperforms the conventional methods objectively.

5. Conclusions

We proposed an efficient algorithm to conceal the holes generated in stereo VR pictures, where the positions of the pixels to fill the hole were derived using the camera parameters. The camera parameters were predicted from their relationship with the homography matrix. Whereas conventional inpainting algorithms have constraints in the concealment of the hole because they were invented for non-VR images, the proposed method efficiently fills all of the pixels in the hole using the relationship between the left and right ERPs.