Inter-Frame Based Interpolation for Top–Bottom Packed Frame of 3D Video

Abstract

The frame-compatible packing for 3D contents is the feasible approach to archive the compatibility with the existing monocular broadcasting system. To perceive better 3D quality, the packed 3D frames are expanded to the full size at the decoder. In this paper, an interpolation technique enhancing and comparing the quality of enlarged halt vertical left and right stereo video in the top–bottom frame-compatible packing is presented. To this end, the appropriate interpolation modes from fourteen available modes for each row segment, which exploit the correlation between left and right stereoscopic as well as current and adjacent frames of individual view, are estimated at the encoder. Based on the information received from the encoder, at the decoder, the interpolation scheme can select the most appropriate available original data to find the missing values of to-be-discarded row segments. The proposed method outperformed than the state-of-the-art interpolation methods in terms of subjective visualization and numerical PSNRs and SSMI about 11%, with an execution time of about 12% comparisons.

1. Introduction

Recently, with the increasing popularity of 3D contents, 3DTV broadcasting services becomes a reality. However, compared with convention digital TV (DTV) broadcasting services, 3DTV needs double the transmission bandwidth for the same video resolution since the left and right frame of 3D video are required to perceive 3D contents [1,2]. To utilize existing transmission infrastructure including teh H.264/AVC compression scheme, the compatibility of 3D video with conventional digital television (DTV) transformation format is strongly required [3,4]. The existing broadcasting network can be used by converting the 3D video format into a single DTV frame. For example, a top–bottom packing after the horizontal line sub-sampling can be treated as a single frame of the existing DTV. Other packing methods include side by side, interleaved formats [5]. The existing 3D compatible frame packing formats are shown in Figure 1.

With the results of utilizing the packing methods, we can utilize the existing transmission infrastructure for the stereoscopic 3DTV similar to [6]. However, the reduction in the spatial resolution due to the decimation of frame-compatible packing is a critical problem when the user wants to experience the video’s full quality. One way to solve this problem is that one suitable interpolation method should be applied to recover the decoder’s original resolution.

Interpolation is the problem of image processing for many years. To yield the unknown pixels of the upsampled image, the conventional interpolation methods can be used. The simple bilinear and bicubic interpolations use the linear and non-linear correlations of known surrounding neighbor pixels, and the sophisticated iterative methods use higher-order correlations such as methods [7,8]. NEDI6 [9] and the updated version of NEDI4 [10], which uses six original pixel values in the low-resolution image to estimate coefficients. Computation time is the main advantage of real-time applications. The idea of sending side information based on the horizontal and vertical line to the decoder is considered for an adaptive linear interpolation for the line-pruned image in [11] which proved good for the horizontal and vertical edges. Regarding the stereo images, the relation between the left and right images can play an essential cue for interpolation to full resolution, which has not been considered by conventional methods. The pixel-based stereo matching method [12] produces the horizontal disparities between left image pixels with their right image corresponding pixels and similar for right and left frames case. Because all pixels should be estimated, the disparity value by searching the match point, the burden of computation time and the accuracy of this method lead to motivation to find new methods.

To reduce the computation time of pixel-based stereo matching, in the patent [13], block-based stereo matching is proposed. Each pixel of the left image (or the right image in this block) is assigned the value of the corresponding pixel of another image for each matched block. This leads to the interpolated pixel being less accurate due to a block mismatch problem. Another issue of stereo matching is the accuracy of disparity estimation on the occlusion region where there are no matched regions between the right and left frames. For the top–bottom packing, the disparity estimation for each deleted line can be exploited to determine an appropriate interpolation mode [14]. Although this method significantly improves the video quality, it does not consider the inter-frame relations to improve the quality of interpolated 3D video sequences.

An effective interpolation technique addresing the unpacking top bottom format of stereo sequences was proposed and evaluated. Specificaaly, our proposal’s primary concern is to use the available values of the previous and subsequent frames of the current view to preserve the quality of the interpolated stereo images. To achieve our goal, adding to the modes of paper [14] by the modes which include the previous and subsequent frame of the current frame is taken into account to exploit the high correlation. The best interpolation mode among the proposed interpolation modes is estimated at the encoder (sending side), before sending the decoders to yield the best interpolation results. The proposed method can archive better running time and subjective evaluation comparing to the evaluated interpolation methods.

The paper is organized as follows. The the frame-compatible top–bottom packing is presented in Section 2. The the proposed inter-frame based interpolation is detailed in Section 3. The optimal proposed interpolation strategy is evaluated in Section 4. The conclusion and future works are in the last Section 5.

2. Frame-Compatible Top–Bottom Packing

The proposed optimal mode based interpolation uses the top bottom packing scheme of 3D video during broadcasting 3D contents (see Figure 1). The top bottom packing scheme has a similar configuration with [14], but with the addition of the same packing scheme for the current frame and adjacent frame (previous and subsequent frames) of one view of the stereo sequences. Specifically, each line of two consecutive horizontal rows will be discarded to squeeze the height by half right and left images, and a similar process is applied for the current and previous frames detailed, as shown in Figure 1. To utilize each image’s remaining lines effectively, the horizontal downsized right image $I_r$ and current frame of one view has a left image $I_l$ with one line offset and a previous frame of one view, respectively. Then, the sub-sampled left and right views are combined as an individual frame within the stereoscopic stream for compression and transmission.

3. Proposed Inter-Frame Based Interpolation

In this paper, the idea of exploiting the correlations of inter-frames optimally at the encoder to leverage interpolating the top–bottom packed stereoviews at the decoder is implemented. The first motivation is that the need-to-be-interpolated pixels on the discarded horizontal line of one frame of the current view can be recovered by the corresponding pixels that are not discarded on the other view frame. This is the characteristic of the well-rectified stereo image as the result of epipolar constraint. To obtain the best perception of 3D effects, epipolar constraint states that the corresponding pixel in the right image to the considered pixel in the left image stay on the same row. The second motivation is that the inter-frame correlation is significantly high for frames of a single view. As a result, the current frame’s information can exist in the adjacent frames, such as previous or subsequent frames. As a result, the current frame’s information can exist in the adjacent frames, such as previous or subsequent frames. Figure 2 is an overview of the transmission system.

Conventional stereo matching, which wants to find the matched pixel (or matched block) by using some matching criteria like the sum of absolute differences (SAD), sum of squared differences (SSD) [15,16], to interpolate the right frame from the available left pixel on the left frame (similar for interpolating the right frame). In this paper, with the original available data from not to-be-discarded pixels of the corresponding left pixels and the pixels of previous and after right frames, the combinations of this available information can be estimated at the encoder and exploited to the decoder as shown in Figure 3. To this end, using the biliear, firstly the half vertical resolution right and left views are interpolated to full resolution at the prepossessing step. Each deleted row is divided into S parts with the length L similar to the method [14]. Each part will have the same interpolation modes. To find the best mode for each horizontal part, the mode will be found by the estimation scheme at the encoder. Note that among the total 14 modes in

Table 1. Interpolation modes.

Mode	Interpolation Condition	Mode	Interpolation Mode
1	${\widehat{x}}_{i,j}^{right}(1)=x_{i+1,j}^{right}$	2	${\widehat{x}}_{i,j}^{right}(2)=x_{i-1,j}^{right}$
3	${\widehat{x}}_{i,j}^{right}(3)=\frac{1}{2}(x_{i+1,j}^{right}+x_{i-1,j}^{right})$	4	${\widehat{x}}_{i,j}^{right}(4)=\frac{1}{2}(x_{i-1,j+1}^{right}+x_{i+1,j-1}^{right})$
5	${\widehat{x}}_{i,j}^{right}(5)=\frac{1}{2}(x_{i-1,j-1}^{right}+x_{i+11,j+1}^{right})$	6	${\widehat{x}}_{i,j}^{right}(6)=\frac{1}{4}(x_{i-1,j-1}^{right}+x_{i+1,j+1}^{right}+x_{i-1,j}^{right}+x_{i+1,j}^{right})$
7	${\widehat{x}}_{i,j}^{right}(7)=x_{i,j\ast }^{right-1}$	8	${\widehat{x}}_{i,j}^{right}(8)=x_{i,g\ast }^{right+1}$
9	${\widehat{x}}_{i,j}^{right}(9)=\frac{1}{2}(x_{i,j\ast }^{right-1}+x_{i,g\ast }^{right+1})$	10	${\widehat{x}}_{i,j}^{right}(10)=x_{i,k\ast }^{left}$
11	${\widehat{x}}_{i,j}^{right}(11)=\frac{1}{2}(x_{i,g\ast }^{right+1}+x_{i,k\ast }^{left})$	12	${\widehat{x}}_{i,j}^{right}(12)=\frac{1}{2}(x_{i,j\ast }^{right-1}+x_{i,k\ast }^{left})$
13	${\widehat{x}}_{i,j}^{right}(13)=\frac{1}{3}(x_{i,j\ast }^{right-1}+x_{i,g\ast }^{right+1}+x_{i,k\ast }^{left})$	14	${\widehat{x}}_{i,j}^{right}(14)=\frac{1}{5}(x_{i-1,j}^{right}+x_{i,j+1}^{right}+x_{i,j\ast }^{right-1}+x_{i,g\ast }^{right+1}+x_{i,k\ast }^{left})$

, modes 1 is the equation of parallax added left view $I_l$ data $(x_{i,k\ast }^j\to x_{i,k\ast +L}^j)$ and modes 2 and 3 are the function of adjacent right data $(x_{i,l\ast }^{r-1}\to x_{i,l\ast +L}^{r-1})$, $(x_{i,g\ast }^{r+1}\to x_{i,g\ast +L}^{r+1})$ and mode 4 are the formulas which combine the parallax-compensated left image with the previous and next frame motion estimation. Note that the left $x_i^l$ and right $x_i^r$ are the full resolution frame expanded by bilinear interpolation method and right $x^{r-1}$, right $x^{r+1}$ are the full resolution expanded by the bilinear interpolation of the previous and subsequent right frames, respectively. The horizontal parallax $k\ast$ in interpolation modes can be found as Equation (1).

$$k*=\underset{k\in \mathsf{\Phi }}{argmin}(\sum _{n=1}^L|x_{i,\Delta +n}^r-x_{i,k+n}^l|)$$

(1)

Moreover, the motion displacement l * between the segment of the current right frame and previous right frame can be found as Equation (2):

$$l*=\underset{l\in \mathsf{\Phi }}{argmin}(\sum _{n=1}^L|x_{i,\Delta +n}^r-x_{i,k+n}^{r-1}|)$$

(2)

Similarly, Equation (3) is used to find the motion displacement $g\ast$ between the segment of the right frame and the next right frame:

$$g\ast =\underset{g\in \mathsf{\Phi }}{argmin}(\sum _{n=1}^L|x_{i,\Delta +n}^r-x_{i,k+n}^{r+1}|)$$

(3)

where the parameter $\mathsf{\Phi }=[\Delta -m_d,\Delta +m_d]$ is a set of a horizontals shift, and $\Delta$ is the first pixel of the considered segment, and $m_d$ is the maximum horizontal disparity. Based on the original pixels and the pre-calculated candidate interpolation modes, the best interpolation mode $b_r^{*}(i,s)$ of sequences of deleted row, this can be calculated as Equation (4):

$$b_r^{*}(i,s)=\underset{m=(1toM)}{argmin}(\sum _{n=1}^S|x_{i,\Delta +j}^r(m)-x_{i,\Delta +j}^r|)$$

(4)

where right $x_{i,\Delta +j}^r$ is the pixel in the original right frame at time t and right $x_{i,\Delta +j}^r(m)$ is the interpolated pixel by the mode number m of the proposed interpolated modes. Note that after being estimated, the optimal mode is sent to the decoder to improve the accuracy of the interpolation.

4. Experimental Results

The experiments were done on MATLAB version R2016a environment installed on the machine with the configuration: Intel^®core™i5CPU 4G Ram Windows 10 64 bit operating system. Six stereoscopic sequences named Book, Car, Door, Horse, Moabit, Bullinger are downloaded from the open source database [17] and used to evaluate the efficiencies of the proposed method. For a fair comparison, we selected the 3D sequences as same as the 3D-based frame-compatible interpolation in reference [14]. There are 150 frames for each tested sequence. The 3D frames of tested sequences are down-sampled for the top–bottom packing. The conventional interpolation methods including the Bilinear, NEDI6 [9], pixel-based matching method [12], block-based matching patent [13], the method [14] are implemented to compare with the proposed method in terms of subjective and numerical data.

Note that, in the bilinear method, the interpolated pixel $p_i$ can be estimated by the Equation (5) where $x_l,x_r,x_t,x_b$ are the left right top and bottom undiscarded pixels’ data, and $c_l,c_r,c_t,c_b$ are the corresponding distances from the interpolated pixel. In NEDI6 [9], $I{o}^d$ is the downsample of the original image $I_o$ which is expanded to $I_i$ of size $mxn$ with zero value row at $k_{t}h$ row of $I_o$, the row of $Io$ after the downsample is linked to the odd rows of expanded frame $I_i$ by $I_i(i,2j-1)=I_o(i,j$, and then using the sixth order $h^6$, the discarded row is derived from the undiscarded row as Equation (7). The pixel-based matching method [12] and block-based matching patent [13] can be used to derive the unknown discarded pixels by searching for the minimal value of SAD values within the pixel order or block of pixels order, respectively. The SAD algorithm has the advantage of computational efficiency. The SAD, as Equation (6), finds the disparity $disp$ $ab$ windows of corresponding pixels in the left image $I_l$ and right image $I_r$. The optimal mode-based method [14] only considers the current left and right frame to derive the optimal interpolation modes:

$$p_i=\frac{x_l{c}_l+x_r{c}_r+x_t{c}_t+x_b{c}_b}{c_l+c_r+c_t+c_b}$$

(5)

$$SAD(i,j,disp)=\sum _{k=-a/2}^{a/2}\sum _{l=b/2}^{b/2}|I_r(i+k,j+l)-I_l(i+k-j+l+disp)|$$

(6)

$$I_i(i,2j)=\sum _{k=-1}^1\sum _{l=0}^1(h_{3l+k+1}^6{I}_i(i+k,2j+2l-1))$$

(7)

We applied the strategy by calculating the average measure from one top–bottom frame, and then averaged all 150 frames per one tested video sequence. As a result, the peak signalnoise ratio (PSNR), structural similarity index measure (SSIM) and multiscale structural similarity index measure (MSSSIM) were added in

Table 2. PSNR comparison (dB).

Method	Bilinear	[9]	[12]	[13]	[14]	Proposed
-	(dB)	dB	Method dB	dB	dB	dB
Book	34.66	34.39	35.66	35.82	36.66	37.90
Car	38.93	38.91	39.12	38.87	39.68	41.38
Door	36.16	36.06	37.24	37.12	38.26	39.76
Horse	32.69	32.24	32.72	32.91	33.73	35.34
Moabit	31.93	31.94	33.12	33.01	34.22	36.24
Bullinger	43.33	43.49	43.51	43.42	44.37	45.68
Mean	36.28	36.19	36.90	36.86	37.82	39.38
Standard Deviation	4.27	4.41	4.05	3.96	3.93	3.80

,

Table 3. SSIM comparison.

Method	Bilinear	[9]	[12]	[13]	[14]	Proposed
-						Method
Book	0.872	0.881	0.887	0.891	0.925	0.943
Car	0.901	0.911	0.919	0.922	0.935	0.966
Door	0.891	0.901	0.908	0.921	0.932	0.951
Horse	0.881	0.885	0.896	0.907	0.912	0.933
Moabit	0.903	0.912	0.904	0.911	0.924	0.941
Bullinger	0.908	0.915	0.921	0.926	0.936	0.974
Mean	0.892	0.901	0.906	0.913	0.928	0.951
Standard Deviation	0.0139	0.0146	0.0131	0.0134	0.0090	0.0157

and

Table 4. MS-SSIM comparison.

Method	Bilinear	[9]	[12]	[13]	[14]	Proposed
-						Method
Book	0.842	0.873	0.879	0.882	0.915	0.955
Car	0.866	0.869	0.875	0.873	0.904	0.934
Door	0.887	0.891	0.904	0.911	0.920	0.931
Horse	0.871	0.879	0.882	0.887	0.911	0.926
Moabit	0.893	0.905	0.911	0.919	0.922	0.932
Bullinger	0.896	0.912	0.919	0.921	0.935	0.961
Mean	0.874	0.889	0.895	0.899	0.918	0.939
Standard Deviation	0.0204	0.0140	0.0186	0.0206	0.0106	0.0144

, respectively. Note that, PSNR measures the lossy interpolated images. On another hand, SSMI and MS-SSMI are the objective image quality measures that have a higher correlation with the human visual system. Given original image $I_o$ and interpolated image $I_i$ with the size $M,n$ and standard deviation $\mu$ and mean $\sigma$, Equations (8) and (10) were used to find the PSNR and MMSI, whilst the MS-SSIM is implemented by applied multiple scale SSIM:

$$PSNR(I_o,I_i)=\sum _{i=1}^M\sum _{j=1}^N{(I_o(i,j)-I_i(i,j))}^2$$

(8)

where $MSE$ is calculated as Equation (9):

$$MSE=\frac{{\sum }_{n=1}^N{\sum }_{m=1}^M{(I_o(i,j)-I_i(i,j))}^2}{MN}$$

(9)

$$SSIM(I_o,I_i)=\frac{(2{\mu }_{Io}{\mu }_{Ii}+c_1)(2{\sigma }_{IoIi}+c_2)}{(({\mu }_{Io}^2+{\mu }_{Ii}^2+c_1)({\sigma }_{Io}^2+{\sigma }_{Ii}^2+c_2)}$$

(10)

where $c_1,c_2$ are constants.

The proposed method proved that it archives the better quality when applied to the sequences with complicated textual and many corners like the Alt-moabit sequence. For the sequences in which their left frames and right frames consist of occlusions, the proposed method derived the best results in terms of subjective evaluation than stereo-based matching methods [12,13].

We calculated the ANOVA Fisher’s least significant difference (LSD) procedure the multicomparison post hoc statistical test to check which pairs of means have a significantly different variation from the mean PSNR, SSIM, and MS-SSIM of different interpolation methods. The numerical data are shown in Figure 4 for PSNR, Figure 5 for SSIM, and Figure 6 for MS-SSIM, respectively. As one can observe, the performance of the method [14], which is the second-best method, is considerably lower than proposed optimal mode based interpolation technique. Even though the proposed method’s execution time is slightly higher than the Bilinear and [14], the proposed method outperformed all validated methods in PSNR, SSIM, and MS-SSIM with significant margins and comparable standard deviations.

We zoom in on the interpolated frame of the tested sequences for subjective comparisons in Figure 7, Figure 8, Figure 9 and Figure 10, for the Moabit, Car, Horse, and Book-Arrive sequences, respectively. From the results, the proposed method archives outperforms than others, especially at thin horizontal lines. This better quality caused by deleted pixels on one view still exists in another view, and its value can be almost recovered after interpolation based on feature-based matching. Figure 11 shows the results of the tested method for the zoom in on the vertical edge. The proposed method fixes zigzag artifacts generated by incorrectly estimated disparity. The execution time comparison for encoder and decoder processes of all tested methods is shown in

Table 5. Execution time comparison (s).

Method	Bilinear	[9]	[12]	[13]	[14]	Proposed
-						Method
Book	0.15	10.60	39.09	1.20	1.76	1.81
Car	0.18	5.10	28.78	0.65	1.05	1.12
Door	0.25	9.76	37.09	0.91	1.49	1.15
Horse	0.31	5.28	32.36	0.86	1.55	1.59
Moabit	0.21	10.30	40.77	1.10	2.42	2.62
Bullinger	0.16	2.94	13.82	0.31	0.60	0.69
Mean	0.21	7.33	31.99	0.84	1.47	1.50
Standard Deviation	0.0609	3.2822	9.9408	0.3223	0.6206	0.6755

. Although the bilinear method yeilds the smallest time values, its visual effects are worst among the tested methods. The proposed method takes more time than the bilinear and much less time than the pixel-based stereo matching and NEDI6. Note that the execution time to derive the optimal mode for considered segment at the encoder is method’s primary consumption time [14] and the proposed method.

Furthermore, similar to [18], we add the depth image rendered by the proposed method using the interpolated left and right frame and compare with the method [4]; and as shown in Figure 12, one can observe that the proposed interpolation technique creates the less noise and higher PSRN for the rendered depth images from the interpolated left and right frames.

5. Discussion and Conclusions

Since we need the fast interpolation method to obtain the intermediate full-size resolution at the decoder, we applied the bilinear method so that it could then be refined by the optimal modes send by the encoder. The single image super-resolution such as the tested method yielded a lower quality than the proposed method. The deep learning-based method using the single image might provide better quality but it required heavy computation and advanced hardware to store and load the trained models which is not always available at the encoder of DTV.

In this paper, the matched region between the right and left frames and the adjacent frames of the same view was exploited for the problem of the interpolation frame-compatible top–bottom packing to full resolutions. The correlation of pixels between the view is the right cue to derive the appropriate value of to-be-interpolated pixels. The experimental section shows clearly that the proposed method yields higher PSNR by 1–2 dB and also better visualizes for interpolated images while comparing with the other methods. The future work will focus on (1) reduce mode options; (2) extending to online interpolation; (3) expanding the proposed approach into another frame-compatible packing system; (4) exploring the interpolation mode by AI-aided techniques, testing the method in real hardware devices; and (5) the images quality assessment, which must be studied intensively to provide the appropriate tool to evaluate the interpolated, restored images [19,20,21,22,23]. The method as mentioned in the recommendation ITU-R BT.500-14 [24] will consider as future works since it required setting up the evaluation environment with professional equipment and observers.