Learning Deep CNN Denoiser Priors for Depth Image Inpainting

Abstract

Due to the rapid development of RGB-D sensors, increasing attention is being paid to depth image applications. Depth images play an important role in computer vision research. In this paper, we address the problem of inpainting for single depth images without corresponding color images as a guide. Within the framework of model-based optimization methods for depth image inpainting, the split Bregman iteration algorithm was used to transform depth image inpainting into the corresponding denoising subproblem. Then, we trained a set of efficient convolutional neural network (CNN) denoisers to solve this subproblem. Experimental results demonstrate the effectiveness of the proposed algorithm in comparison with three traditional methods in terms of visual quality and objective metrics.

1. Introduction

Depth images are a key research topic in the field of 2D-to-3D technology, which is widely used in machine vision and 3D reconstruction. However, due to the limitations of hardware and the characteristics of 3D scenes, black holes can easily form in depth images, and they are susceptible to noise. Due to the practical benefits of using depth images, some scholars have begun to investigate the problem of depth image inpainting.

Depth images can be viewed as natural images without texture. Therefore, many natural image inpainting methods can be directly applied for depth image inpainting. In general, inpainting methods can be roughly divided into three categories, as discussed below.

Inpainting methods of the first type rely on spatial information. Several representative algorithms are described as follows. Telea et al. [1] proposed a fast depth image inpainting method based on the mechanism of human vision. This method has a high inpainting speed, but boundaries can easily become blurred. Shen et al. [2] proposed the weighted joint bilateral filter method. Similarly, although the speed of this method is high, the boundaries in the depth images are not well handled. Lee et al. [3] proposed an adaptive edge-oriented smoothing process to solve the depth image inpainting problem. This method achieves a good balance between time consumption and image quality. Methods of this type rely on general methods of image repair.

Inpainting methods of the second type use aligned color images as a guide. Several representative algorithms are described as follows. Liu et al. [4] used a deep-hole inpainting method with color guidance weights combined with bilateral differences. This method shows good robustness. Buyssens et al. [5] proposed a method based on block matching with the edges of aligned color images to guide the direction of inpainting. Deng et al. [6] trained a decision tree to identify edges in depth images and corresponding color images for inpainting. Methods of this type require constrained repair based on aligned color images, and the process is complex.

The third class of inpainting methods consists of model-based optimization methods. Several representative algorithms are described as follows. Lu et al. [7] proposed a low-rank matrix method for depth image enhancement. This method has a poor effect when large-area information is lost. Xue et al. [8] proposed the integration of low gradient regularization with low-rank regularization for depth image inpainting. This method offers improved inpainting quality. Methods of this type are flexible but time consuming. At present, they are attracting considerable attention and have become the mainstream approach to the problem at hand. Therefore, we plugged learned denoiser priors into such a model to solve the problem of depth image inpainting.

Within the framework of model-based optimization methods, we investigated inpainting for single depth images. Depth image inpainting is an ill-posed inverse problem. We can obtain a potential depth image by solving the following formula [9]:

$$\stackrel{\Lambda }{x}=\arg {\min }_x\frac{1}{2}||Hx-y|{|}_2^2+\lambda \cdot \mathsf{\psi }(x)$$

(1)

where $x$ and $y$ are the potential depth image and the degraded depth image, respectively; $||Hx-y|{|}_2^2$ is the data-fidelity term; $\mathsf{\psi }(x)$ is the regularization term; and $\lambda$ is a weight parameter. $H$ is a binary template in which missing information is represented by values of 0 and all other information is represented by values of 1. We hope to obtain a potential depth image $x$ from a degraded depth image $y$.

With the help of variable splitting techniques, many scholars have decoupled the data-fidelity term and the regularization term in Equation (1). Then, they have used various strategies to solve the resulting subproblems. The most commonly used variable splitting techniques are the iterative shrinkage/thresholding (IST) algorithm [10] and the split Bregman iteration (SBI) algorithm [11].

First, we adopt the SBI algorithm for the separation of the terms in Equation (1), and then, we train a set of effective CNN denoisers and integrate the results into the framework of model-based optimization methods to complete the task of depth image inpainting. The structure of the CNN includes rectified exponential linear units (RELUs) [12], batch normalization (BN) [13] and residual learning [14]. The proposed method can be used to more effectively complete the task of depth image inpainting. We chose CNN denoisers for the following reasons:

• The CNN inference process is very efficient due to the parallel computation capability of Graphics Processing Units (GPUs).
• A CNN with a deep architecture exhibits a powerful prior modeling capacity.
• The deep architecture of the CNN facilitates discrimination learning.

Our contributions are summarized as follows: (1). The SBI algorithm is used to separate the data-fidelity term and the regularization term. Therefore, the depth image inpainting problem is transformed into the corresponding denoising subproblem. (2). We trained a set of effective CNN denoisers that are used as important components of a model-based optimization method to solve the denoising subproblem.

The remainder of our paper is organized as follows: Section 2 describes the related work on variable splitting techniques and the application of denoiser priors in image reconstruction. Section 3 introduces the method of training deep CNN denoisers for depth image inpainting. In Section 3, we also present important details of the CNN structure. In Section 4, we report experiments performed on our dataset and compare the results of various algorithms. In Section 5, we discuss future research directions.

2. Related Works

2.1. Variable Splitting Techniques

Variable splitting techniques are widely used in image reconstruction. The variable splitting techniques are usually adopted to decouple the data-fidelity term and the regularization term in Equation (1). This approach offers good convergence and effectiveness.

In Reference [9], Zhang et al. compared the performance of the IST and SBI algorithms. They then used the SBI algorithm to decouple the data-fidelity term and the sparse representation regularization term. Xue et al. used the alternating direction method of multipliers (ADMM) and the SBI algorithm to decouple the data-fidelity term and the regularization term in [8]. In Reference [15], the ADMM was adopted to decouple the data-fidelity term and the regularization term in class-adaptive image prior extraction for image reconstruction. Reports of similar approaches can also be found in References [16,17,18] and others.

Depth image inpainting is a branch of image reconstruction. In this paper, we adopt the SBI algorithm due to its simplicity. We then use the CNN denoiser to solve the resulting subproblems.

2.2. Convolutional Neural Network (CNN) Denoiser Priors

At present, many scholars are committed to using denoiser priors for image reconstruction. This method usually involves the separate processing of the data-fidelity subproblem and the denoising subproblem.

In Reference [14], within the framework of the ADMM, He et al. adopted a class-adaptive denoiser based on a Gaussian mixture model (GMM) to address image deblurring and image compression. In Reference [19], a block matching and 3D filtering (BM3D) denoiser model was proposed to solve the problem of single image super-resolution. In Reference [20], within the framework of spatially adaptive nonlocal patchwise estimation, a BM3D denoiser was used to address the problem of image deblurring. In Reference [21], a set of trained CNN denoisers was developed for image denoising, image super-resolution and image deblurring. Some similar approaches can also be found in References [22,23].

All above methods have shown that the existing denoising models solve many image restoration problems. Considering the performance and effectiveness of the various available algorithms, we chose the CNN-based denoiser approach.

3. Learning Deep CNN Denoiser Priors for Depth Image Inpainting

3.1. Split Bregman Iteration (SBI) Method

By introducing the auxiliary variable $u$, Equation (1) can be reformulated as a constrained optimization problem:

$$\hat{x}=\arg {\min }_x\frac{1}{2}||Hx-y|{|}_2^2+\lambda \cdot \mathsf{\psi }(u)\begin{array}{cc}& s.t.\begin{array}{cc}& u.=x\end{array}\end{array}$$

(2)

Then, by invoking the SBI algorithm, we separate the data-fidelity term and the regularization term. Equation (2) can be solved by solving the following two subproblems:

$$x^{k+1}=\arg {\min }_x\frac{1}{2}||Hx-y|{|}_2^2+\frac{\mu }{2}||x-u^k-b^k|{|}_2^2$$

(3a)

$$u^{k+1}=\arg {\min }_u\lambda \cdot \mathsf{\psi }(u)+\frac{\mu }{2}||x^{k+1}-u-b^k|{|}_2^2$$

(3b)

$$b^{k+1}=b^k-(x^{k+1}-u^{k+1})$$

(3c)

Given $u$, Equation (3a) is converted into a strict minimum problem of a strictly convex quadratic function. We can thus obtain a closed-form solution:

$$x^{k+1}={(H^{T}H+\mu I)}^{-1}[H^{T}y+\mu (u^k+b^k)]$$

(4)

where $I$ is the identity matrix.

Given $x$, Equation (3b), in which the regularization term appears, can be converted into the following formula:

$$u^{k+1}=\arg {\min }_u\mathsf{\psi }(u)+\frac{1}{2{(\sqrt{\lambda /u})}^2}||r^{k+1}-u|{|}_2^2$$

(5)

where $r^{k+1}=x^{k+1}-b^k$.

In Equation (5), we view $r$ as a sort of noisy observation of $u$. Therefore, Equation (5) corresponds to the image $u^{k+1}$ produced by a Gaussian denoiser with noise level $\sqrt{\lambda /\mu }$. Therefore, Equation (5) can be rewritten as follows:

$$u^{k+1}=denoiser(r^{k+1},\sqrt{\lambda /\mu })$$

(6)

In summary, the depth image inpainting problem is converted into the corresponding denoising subproblem.

3.2. CNN Denoiser

A CNN denoiser has the advantages of high efficiency and a strong modeling capability. Therefore, we adopted a CNN denoiser [24] for solving the $u$ subproblem in depth image inpainting.

The structure of the CNN denoiser is illustrated in Figure 1. The basic units are Conv, RELU, and BN types.

The L layers of the CNN denoiser include three types of blocks with different functions.

The first layer is of the first block type: “Conv +RELU”. This layer generates 64 feature maps, and the RELU function enhances the nonlinear relationship between the layers.

The second through (L−1)th layers are of the second block type: “Conv + BN + RELU”. The BN unit is added between the Conv and RELU components in this block type. BN can speed up training and improve the denoising performance.

The Lth layer is of the third block type: “Conv”. This layer is used to reconstruct the output.

The important details of the CNN will be given below.

3.2.1. Network Depth and Receptive Field Size

According to the rules presented in References [25,26], the size of the convolution filter should be set to $3\times 3$. Therefore, the receptive field size, $s$, and the network depth, $d$, satisfy $s=2d+1$. In a CNN, a larger receptive field can better utilize the known information for denoising.

To balance efficiency and effectiveness, it is important to choose a reasonable network depth.

Depth images can be regarded as textureless natural images with low noise levels [21,26]. Therefore, the network depth can be set to 7, and the receptive field size is $15\times 15$.

3.2.2. Integration of Residual Learning and Batch Normalization (BN)

We cite the classic design of the Denoising convolutional neural network (DnCNN), which combines residual learning and Batch Normalization (BN) [27,28]. This integrated approach can achieve the optimal denoising performance.

The residual learning method is used for training to obtain the complete image or residual image from potential information. When the original mapping approaches an identity mapping, this method is more suitable for obtaining the residual image. Therefore, we adopt a single residual unit to predict the residual image.

The BN operation is similar to the regularization operation. BN alleviates internal covariate transfer by means of a normalization operation before the nonlinear operation in each layer. This step can speed up the convergence of the network and reduce the dependence on the initial parameters. BN units have been widely used in deep learning networks [23].

In the proposed network structure, the BN operation is introduced to speed up learning and prevent overfitting.

In summary, this integrated approach not only enables stable training but also boosts the denoising performance.

3.2.3. Zero Padding

In depth image inpainting, the sizes of the input and output need to be consistent. However, this may lead to boundary artifacts. There are two ways to solve this problem: symmetrical padding and zero padding. We chose zero padding to maintain the image size while avoiding boundary artifacts.

3.2.4. Learning of CNN Denoiser Priors

The depth image inpainting problem is reduced to the denoising subproblem in our approach. As the proposed method requires iteration, it is necessary to train denoisers for different image noise levels. In depth images, the noise level is generally low; therefore, we trained a set of denoisers for noise levels in the range (0,30) in increments of two.

Once the denoising subproblem had been solved as described above, we adopted an effective solution to ensure the optimal solution of the overall algorithm.

4. Experiments and Results

It is well known that CNNs benefit from the availability of large training data. To learn the CNN denoiser, we followed Reference [23] in using 2000 images from ImageNet [29] for training. The noisy images were obtained by adding Gaussian noise for noise levels in the range (0,30) in increments of two. The hardware simulation platform was a Lenovo R720 computer (Lenovo company, Haidian district, Beijin, China) with an Nvidia GPU. The software platform used for simulation was MATLAB R2017a with MatConvNet. It takes about one day to train the CNN denoiser. We empirically found that using a larger training database can slightly improve the effect and greatly increase time.

Under the same settings, the inpainting effect should be evaluated in terms of both subjective visual quality and objective metrics. As depth image inpainting is not yet a mature area of research, there are no public datasets or algorithms available for comparison; therefore, we introduced three sets of depth images for comparative experiments: the Mask depth image set, the Teddy depth image set and the Art depth image set. These image sets include both natural areas and artificial areas that need to be repaired. For comparison with our method, we used the total variation (TV) [30], group-based sparse representation (GSR) [9] and low rank with L0 gradient regularization (LRL0) [8] algorithms.

The objective metrics can be assessed on the basis of two objective metrics: the peak signal-to-noise ratio (PSNR) and the feature similarity (FSIM) index [31].

In Experiment (1), we obtained degraded depth images directly from a Kinect sensor, as shown in Figure 2.

In Experiment (2), we obtained degraded depth images by adding artificial text, as shown in Figure 3. The original images were the images repaired with the GSR algorithm in Experiment (1).

Subjectively, all four algorithms meet the visual requirements, as shown in Figure 4 and Figure 5. However, the inpainting effect of the TV algorithm is blurred, and the boundaries are smoothed. The GSR algorithm achieves a better repair effect at the boundaries but can easily lead to blur in areas where the gray values are similar, also resulting in unclear boundaries. The boundary repair effect is improved in both the LRL0 algorithm and the proposed algorithm, and the inpainting effect is better for fine image details.

Objectively, it can be seen from

Table 1. Peak signal-to-noise ratio (PSNR) and feature similarity (FSIM) result in Experiment (1).

Image	Algorithm (PSNR/dB)/FSIM
	TV	GSR	LRL0	Proposed
Mask Depth Image	22.1304/0.9257	23.2678/0.9288	24.2544/0.9993	24.3889/0.9993
Teddy Depth Image	21.3046/0.9314	22.5604/0.9369	23.9979/0.9993	24.0529/0.9993
Art Depth Image	27.2198/0.9683	28.4787/0.9765	30.0326/0.9999	30.4045/0.9999

and

Table 2. PSNR and FSIM result in Experiment (2).

Image	Algorithm (PSNR/dB)/FSIM
	TV	GSR	LRL0	Proposed
Mask Depth Image	25.2308/0.9297	30.7068/0.9864	33.4979/1	33.7880/1
Teddy Depth Image	27.9409/0.9320	32.9311/0.9884	33.8560/1	34.3918/1
Art Depth Image	27.5757/0.9750	31.1104/0.9839	31.4780/1	31.9801/1

that the proposed algorithm is superior to the other three algorithms. For different types of images, the objective metrics of our proposed algorithm are the best.

In summary, the proposed algorithm has advantages in terms of its inpainting effect. Hence, the proposed algorithm is an effective inpainting method for single depth images.

5. Conclusions

The main research issue addressed in this paper is inpainting for single depth images. With the help of the SBI algorithm, we transformed the problem of single depth image inpainting into the corresponding denoising subproblem. Then, we cited and trained a set of effective CNN denoisers for solving the denoising subproblem. Experimental results prove that the proposed algorithm has value for practical applications and that its inpainting effect is superior to that of other algorithms.

However, the proposed approach still has certain shortcomings. On the one hand, it will be interesting to investigate how to balance the efficiency and effect of the CNN denoiser. On the other hand, an effective compression method is needed for the CNN architecture. These issues will be addressed in our future research.