An Efficient Information-Reinforced Lidar Deep Completion Network without RGB Guided

Abstract

Due to the sparsity of point clouds obtained by LIDAR, the depth information is usually not complete and dense. The depth completion task is to recover dense depth information from sparse depth information. However, most of the current deep completion networks use RGB images as guidance, which are more like a processing method of information fusion. They are not valid when there is only sparse depth data and no other color information. Therefore, this paper proposes an information-reinforced completion network for a single sparse depth input. We use a multi-resolution dense progressive fusion structure to maximize the multi-scale information and optimize the global situation by point folding. At the same time, we re-aggregate the confidence and impose another depth constraint on the pixel depth to make the depth estimation closer to the ground trues. Our experimental results on KITTI and NYU Depth v2 datasets show that the proposed network achieves better results than other unguided deep completion methods. And it is excellent in both accuracy and real-time performance.

1. Introduction

With the gradual expansion of computer vision applications, all walks of life have higher requirements for depth information. Accurate depth data is essential for self-driving cars, autonomous navigation systems, and virtual reality [1]. However, depth is often sparse and partially missing due to device and environmental limitations. It is fatal in the reconstruction of 3D information [2]. Therefore, the completion methods of the sparse depth data have been widely studied in recent years [3,4]. However, traditional methods often fail to achieve good results because of the sparsity and lack of sufficient prior knowledge of the missing depth [5,6]. With the development of deep learning, people began to use deep neural networks to complete the depth information and achieve better results than the traditional interpolation methods [7,8].

At present, most depth completion networks supplement the missing depth information through the guidance of RGB images [9]. It can provide effective real edges to distinguish the contours between objects. The mapping relationship between sparse depth and a color image is established image through network learning [10,11]. And then the dense depth is regressed together [12]. Although they achieve a better result, these data are based on the joint action of multiple sensors, which requires the important premise that the image and the point cloud correspond to each other. The problem can be effectively controlled in the datasets. However, the joint calibration of heterogeneous sensors will increase the cost and uncertain error, and the reliability of the function cannot be guaranteed in practical application [13,14]. Therefore, our network no longer uses any RGB image to guide and only relies on a single sparse depth image to complete the depth to be more suitable for real-world scenarios. We use the confidence re-aggregation to consider the depth reliability of the neighborhood and obtain the most accurate local pixel depth estimation. In fact, the depth of the outdoors is more complex and variable than the interior. The overall texture is required to be enhanced, and the range of variation of each target is required to be enlarged. Therefore, we further research the information processing of autonomous driving. We design the structural multi-resolution dense progressive fusion processing and point folding module to further improve the accuracy of global prediction in autonomous driving. We cancel the additional information of RGB images and use only one-fourth of the information of other methods [15,16], which avoids the necessary complex and error operations such as joint calibration and calibration in actual use. It greatly reduces the volume of the network and achieves the requirements of accuracy and speed, which is more suitable for the real-world situation.

In fact, it is difficult to achieve good results by only entering depth information because of the limitation of information content [17,18]. Researchers have come up with several ideas for solving. SI-Net [19] found that ordinary convolution directly used to complete depth could not achieve good results. They proposed a simple and effective sparse convolution layer, which weighted the elements of the convolution kernel according to the validity of the input pixels. Then they transferred the information about the validity of pixels to the subsequent layers of the network. The improvement makes the deep completion task based on a convolutional neural network successful. The relationship between convolution and confidence has become a research hotspot in this field [20,21]. Considering that the use of confidence as a binary value mask to filter out lost measurements can ignore valuable information in the confidence map, N-CNN [22] proposed an algebraic constrained convolutional layer for sparse input CNNS. They used signal confidence as a continuous measure of data uncertainty and convolution constrained by confidence to achieve superior performance. However, Spade [8] believed that sparse convolution is not necessary. They used the ordinary dense CNN architecture and the new sparse training strategy to produce significantly better performance. However, it depends on the change of training mode, and the robustness is poor. According to HMS-NET [23], sparse invariant convolution not only loses numerous spatial information but also cannot be directly integrated into multi-scale structures. Therefore, they proposed three sparse invariant operations for processing sparse inputs and feature maps. And they designed a hierarchical multi-scale network structure that integrates information of different scales to solve the deep completion task.

SPN [24] combined global and local information to learn confidence in an unsupervised manner. The predicted depth maps are weighted by the respective confidence maps. On this basis, PNCNN [25] learned the input confidence estimator in a self-supervised way based on a normalized convolutional neural network to identify the interference measurements in the input. They proposed a probabilistic version of NCNN that produces a statistically meaningful measure of uncertainty for the final prediction. However, the reliability of the effect cannot be guaranteed when the completion result is directly used for the subsequent judgment of depth because the confidence degree is not supervised by the truth value in the learning of these networks. As sparse information provided by spare depth is limited, we should make full use of the local neighborhood information to strengthen the dependence relationship of the neighborhood [26,27].

An affinity is a display form of neighborhood information, which can represent the coherence of color and texture and the similarity between pixels and pixels at the semantic level. But it is not usually seen as part of the learning problem [28]. SPN [24] proposed spatial propagation networks that use learned affinities to guide and disseminate information in images. All modules are differentiable and jointly trained using stochastic gradient descent methods, which are also computationally very efficient. Based on this, CSPN [29] proposed a convolution space propagation network. It no longer scans row/column propagation from four directions like SPN [24] but propagates in parallel through recursive convolution operation. Learning the affinity between adjacent pixels by the deep convolutional neural network can improve the effectiveness and speed of the deep completion task. However, the mode of propagation has a definite limitation, requiring a fixed local neighborhood configuration for propagation. Once the local neighbor is fixed, there will be irrelevant information mixed with the reference information, resulting in the mixing depth problem, especially at the depth boundary. Therefore, NLSPN [30] introduced a learnable affinity normalization method to learn affinity combinations better. They avoided the irrelevant local neighborhood effectively and focused on the relevant non-local neighborhood in the propagation process. However, it has a strong dependence on the neighborhood and a low dependence on confidence. In addition, it takes numerous iterations to receive good results. The complex calculation makes the speed decrease greatly and cannot meet the real-time demand. Based on CSPN [29], DSPN [31] decoupled the neighborhood into different parts according to different distances, generates independent attention graphs recursively, and refines these parts into adaptive affinity matrices. However, it is still based on sparse depth and color image fusion. Therefore, we put forward the idea of confidence reaggregation. We learn neighborhood affinity in the form of spatial propagation and further refine local depth in order to improve the accuracy of depth completion.

In addition, global information is crucial to the complete effect. The multi-scale structure can enhance the reliability of global information, which is very useful for sparse depth of missing information [32]. U-Net [33] designed the contraction path and expansion path following the typical architecture of convolutional networks, and effectively become the main framework for various other tasks to solve the problem of medical image segmentation. Most networks added different residual structures to the U-Net [33] framework with improved points to learn features with different resolutions, which solves the degradation problem and greatly improves the fitting ability of neural networks. However, it only makes up for the loss of resolution caused by convolution and ignores the original information of different resolutions. Therefore, we design a dense progressive fusion multi-scale structure as a complement to the global information to make maximum use of the existing original information. The global completion effect is improved by fusing the global depth information of different scales.

Depth completion is essentially the same as point cloud completion. They are both estimates of 3D information. The difference lies in the way they represent three-dimensional information. Folding-Net [34] wrapped a constant 2D grid into the shape of an input 3D point cloud in the point cloud completion task. Since the network integrates mappings from lower dimensions, it increases the possibility of more points. Similarly, for sparse depth images, the possibility of more pixels requests to be added. Therefore, we designed a point folding module from 1D to 2D to integrate the one-dimensional point information into the missing two-dimensional information through the learning of the network to strengthen the density of the overall depth. It can increase the density of global information.

In summary, we construct a deep completion network with a single sparse depth input that integrates local and global information, which strengthens the known information and does not require any color image guidance (EIR-NET). Compared with other networks, the paper has the following contributions:

• We propose a confidence re-aggregation method, which is re-aggregate the local area, effectively based on the confidence of the local pixel neighborhood to improve the estimation accuracy of local details.
• We designed a dense progressive fusion network structure to further improve the accuracy of global completion by using multi-scale information.
• We propose a 1D to 2D point folding module to increase the density of global depth information.

2. Methods

Our network structure is shown in Figure 1. The sparse depth image is input into the U-shaped network to obtain the initial confidence. The confidence is re-aggregated and optimized by the neighborhood information to obtain deep confidence in the confidence re-aggregation module. The confidence and original sparsity depth are input into the encoding of dense progressive fusion. The convolution results are fused layer by layer from different levels, and the different resolution information of multiple branches is used to supplement the details. We use the point folding module to increase the possibility of points and add it to the final fused feature code in the decoding. After that, the complete depth image is output as the final result.

2.1. Confidence Re-Aggregation Module

We can use convolution to extract features to obtain the confident prediction of depth images. However, there is no real supervision in the network learning, which makes the transmission volatility large. The resulting confidence prediction will have errors. As mentioned in the NLSPN [30], affinity normalization ensures stability and neighborhood correlation during propagation. We used absolute and normalized affinities (AS), regular absolute and normalized affinities (AS*), and hyperbolic tangent functions (TC) to experiment.

AS is restricted to the lines satisfying $\left|{\omega }_1\right|+\left|{\omega }_2\right|=1$ after normalized affinity. It can be expressed as

$$AS({\omega }_{m,n}^{i,j})={\widehat{\omega }}_{m,n}^{i,j}/{\displaystyle \sum _{(i,j)\in {\mathbb{N}}_{m,n}}\left|{\widehat{\omega }}_{m,n}^{i,j}\right|}$$

(1)

where $\widehat{\omega }$ represents the original confidence before normalization and $\omega$ represents the refined confidence after normalization. m and n represent the size of the window. i and j represent the coordinates.

AS* is restricted by

$${\displaystyle {\sum }_d\left|{\omega }_d\right|}>1$$

(2)

where d represents the different dimensions.

TC can be represented as

$${\omega }_d=tanh({\widehat{\omega }}_d)/C$$

(3)

where C represents the normalization factors. $tanh$ are hyperbolic tangent functions.

The essence of spatial a propagation network is to learn a large affinity matrix and transform its diffusion into local linear spatial propagation. It simply and effectively enhances the output. Similarly, we can use it to further enhance unreliable self-monitoring results. The difference is that our specification body is the confidence rather than the convolution. We describe the difference between confidence re-aggregation and normal confidence in Figure 2. The initial confidence is derived from the feature. According to the feature image, the affinity optimization of the initial confidence is carried out to obtain the fine confidence. Then the better feature images are obtained according to the fine confidence and initial feature images. As shown in Figure 1, the module is attached after the feature extraction to generate effective confidence in Step 1. After experiments, we chose absolute and normalized affinities (AS) as the aggregation method in Section 3.3.1.

Finally, we fuse the aggregate result with the original output and receive the new output to avoid degradation. The results of (a) and (b) in Figure 2 are fused on the channel, which can be expressed as

$${\omega }_{m,n}^{i,j}\in {\Phi }^{B,C_1+C_2,H,W}=cat(AS({\omega }_{m,n}^{i,j})\in {\Phi }^{B,C_1,H,W},{\widehat{\omega }}_{m,n}^{i,j}\in {\Phi }^{B,C_2,H,W})$$

(4)

where $\Phi$ represents the representation of dimensions and size. B represents the batchsizes. C represents the channel, H and W represent the height and the weight. The structure of connections increases reliability and makes up for the defects caused by self-supervision.

2.2. Densely Progressive Fusion Module

We request to make the most of the depth map information because our network is not guided by color images. The loss of global information may directly lead to the omission of information. The image pyramid structure will receive different resolutions of depth and receive different scales of information. Making full use of the multi-resolution multi-branch structure can benefit the overall and detailed regression. Therefore, we design a new contraction path based on U-Net [33]. A U-shaped encoder architecture that follows the basic framework of convolution is generated, which is no longer completely symmetric with the expansion path of the decoder.

NCNN [22] is an effective way to fuse confidence and depth features. Similar to the operation of ordinary convolution, the normalized convolution is an operation in the domain of each point pair of the signal. The convolution at each point results in an effective inner product between the kernel of the conjugate and reflection filters and the neighborhood. For a sparse signal $s$, the finite neighborhood of each of its signals is defined in a finite space $n\in {\mathbb{C}}^n$. The confidence $c\in {\mathbb{R}}^n$ of $s$ represents the reliability of the neighborhood, which is usually nonnegative.

The local modeling approach of the signal is to project each sample onto a subspace spanned by basic functions. The signal $s$ can be denoted as

$$S=Br$$

(5)

where $B$ is a $m\times n$ matrix and $r$ is the coordinates of the sample and the basic function concerning $B$. Thus, the least squares estimation of coordinates with weight matrix $w$ can be obtained as

$$\arg \underset{r\in {\mathbb{C}}^n}{\min }{‖Br-s‖}_w$$

(6)

Taking an applicability function $a\in {\mathbb{C}}^n$ as the weight of the basis and a window function as the benchmark, we can receive the following solution:

$$\widehat{r}={(B^{*}WB)}^{-1}{B}^{*}Ws={(B^{*}{D}_a{D}_{c}B)}^{-1}{B}^{*}{D}_a{D}_{c}s$$

(7)

where the weight matrix W is the product of $D_a=Diag(a)$ and $D_c=Diag(c)$. $Diag$ is the diagonal matrix. The equivalent inner product is expressed as follows:

$$\begin{array}{l}r={\left(\begin{array}{ccc}(b_1,b_1)w& \cdots & (b_1,b_m)w\\ ⋮& \ddots & ⋮\\ (b_m,b_1)w& \cdots & (b_m,b_m)w\end{array}\right)}^{-1}\left(\begin{array}{c}(b_1,f)w\\ ⋮\\ (b_m,f)w\end{array}\right)\\ ={\left(\begin{array}{ccc}(a·c·b_1,b_1)& \cdots & (a·c·b_1,b_m)\\ ⋮& \ddots & ⋮\\ (a·c·b_m,b_1)& \cdots & (a·c·b_m,b_m)\end{array}\right)}^{-1}\left(\begin{array}{c}(a·c·b_1,f)\\ ⋮\\ (a·c·b_m,f)\end{array}\right)\end{array}$$

(8)

when choosing a constant function as the basis, we set $B=1$ and can obtain:

$$\widehat{r}={(1^{*}{D}_a{D}_{c}1)}^{-1}{1}^{*}{D}_a{D}_{c}s=\frac{a·(c\odot s)}{a·c}$$

(9)

where $\odot$ is the Hadamard product. Through convolution operation of signal s, we can obtain:

$$\widehat{r}[k]=\frac{{\displaystyle {\sum }_i^{n}a[i]s[k-i]c[k-i]}}{{\displaystyle {\sum }_i^{n}a[i]c[k-i]}}$$

(10)

where k and i are the current coordinate and the unit coordinate in the formula for convolution in the discrete domain. n is the degree. The propagation value between normalized convolutional layers is:

$${\widehat{c}}_i=\frac{〈a|c〉}{〈1_n|a〉}$$

(11)

where $c_i$ represents the confidence value of the $i$ pixel. Although it is obtained by convolution calculation, the relationship between Windows is not tight enough. It hurts spatial transmission.

Aggregation can calculate the neighborhood correlation more accurately and enhance the local effect. The aggregated values are stored in the original space $S$, still in one-to-one correspondence with element positions. The energy function can be expressed as

$$x(c)=x_{original}(c)+x_{neighbor}(c)=\omega$$

(12)

where $x_{original}$ and $x_{neighbor}$ are the original item and the neighborhood aggregation item, respectively.

PNCNN [25] proposed a probabilistic version of a normalized convolutional neural network by deriving the connection between normalized convolutional and statistical least squares methods. Define the signal $s=Br+e$, where $e$ is a random noise variable with mean 0 and variance ${\sigma }^{2}V$, then ${\sigma }^2$ is the global signal and $V$ is a positive definite matrix describing the covariance between observations. It can be expressed as

$$W=V^{-1}$$

(13)

The uncertain value is:

$$\mathrm{cov}(\widehat{s})=\mathrm{cov}(B\widehat{r})=B\mathrm{cov}(\widehat{r})B^{*}$$

(14)

After substituting in the calculation, it can be expressed as

$$\begin{array}{l}\mathrm{cov}(\widehat{s})=B((B^{*}{V}^{-1}B)-1B^{*}{V}^{-1}s)B^{*}\\ ={\sigma }^{2}B{(B^{*}{V}^{-1}B)}^{-1}{B}^{*}\\ ={\sigma }^{2}B{(B^{*}{W}_a{W}_{c}B)}^{-1}{B}^{*}\end{array}$$

(15)

when $B=1$, we can obtain:

$$\widehat{c}={\sigma }^{2}1{(1^{*}{W}_a{W}_{c}1)}^{-1}{1}^{*}=\frac{{\sigma }^2}{〈a|c〉}$$

(16)

where the variance ${\sigma }^2$ has to be estimated. $a$ and $c$ are the applicability and output confidence of the last layer of the normalized convolutional neural network.

As shown in Figure 3, our dense progressive fusion structure is characterized by multi-branching and multi-resolution scales. The contraction path divides into three branches. The expansion path is a gradual step shape continuous fusion. It is more flexible and maneuverable than the ordinary U-shaped structure. We encode features through this hierarchical fusion structure in step 2, which is shown in Figure 1.

The corresponding confidence of the output features of different depths ${\widehat{x}}_{m,n}^i$ can be expressed as

$${\widehat{c}}_{m\prime ,n}^i=\frac{{\sigma }^2}{〈a|c_{m,n-1}^i〉}$$

(17)

where $i$ indicates the branch number. $m$ represents the resolution scale. $n$ represents the number of layers. Half of them are up-sampled and half of them are down-sampled. The transition sampling between $m^{\prime }$ and $m$ is defined as the 2-fold relation. $x_{m,0}^i$ represents the original layer. $i\in 0,\cdots ,i_{\max }-1$ where $i_{\max }=3$. $m\in 0,\cdots ,m_{\max }/2$ where $m_{\max }=8$.

Therefore, the results of the layer $n$ in the down-sampling stage can be expressed as

$$x_{m,n}^i,c_{m,n}^i=nconv(x_{m-1,n-1}^i,c_{m-1,n-1}^i)$$

(18)

The results of the layer $n$ in the down-sampling stage can be obtained by progressive fusion of the main path and branch path and can be expressed as

$$\begin{array}{l}{\widehat{x}}_{m,n},{\widehat{c}}_{m,n}=cat(R_{main},R_{branch})\\ =cat(nconv(x_{m+1,n+1}^0,c_{m+1,n+1}^0),up(nconv(x_{m-1,n-1}^0,c_{m-1,n-1}^0))\\ ,up(nconv(x_{m-1,n}^1,c_{m-1,n}^1)),up(nconv(x_{m-1,n-1}^2,c_{m-1,n-1}^2)))\end{array}$$

(19)

2.3. Point Folding Module

Folding-Net [34] complements spare depth by expanding 2D points into 3D point clouds through folding. The folding operation of reconstructed surfaces from 2D meshes fundamentally establishes the mapping from 2D regular fields to 3D point clouds. It acts as a filling and completion. Similarly, we can consider reducing the dimensionality to complement the 2D pixels with the 1D points. As shown in Figure 4, we expand the dimension of the folded network and use similar ideas to learn the completion depth information. As shown in Figure 1, we arranged it at the front of the decoder to enrich the encoded content and increase plasticity in Step 3. We fold 1D random points into the 2D grid.

The depth image $D$ with size $i\times j$ and $n$ random points folded can be expressed as

$$D=\left.\left\{\widehat{D}\right.,\{P|P_{i,j}\}\right\}=\left.\left\{\widehat{D}\right.,\{P|Trans(p_s)\}\right\}$$

(20)

where $p_s$ is the random point, $s=1,\cdots ,n$. $P_{i,j}$ is the pixel folded into the grid. $\widehat{D}$ is the original depth image.

3. Experimental Evaluation

3.1. Datasets and Setup

KITTI-Depth [35]: The KITTI dataset is an authoritative and widely used deep completion dataset in vehicles. The author collected six hours of real traffic conditions. The dataset consists of calibrated and synchronized images, radar scans, high-precision GPS information, IMU acceleration information, and other modal information. Sparse depth images of the KITTI dataset were obtained by projecting the original LIDAR points onto the camera view. Ground truth semi-dense depth images are generated by projecting cumulative LIDAR scans of multiple time stamps and then removing abnormal depth values from occluded and moving objects. Since there are few LIDAR points at the top of the depth map, the bottom center of the input image is cropped. The remaining depth image has a resolution of 1256 × 352. In the depth completion dataset, a sparse depth image has about 6% valid pixels, while a ground truth depth image has about 14% valid pixels. The dataset contains 86,898 frames for training, 7000 frames for validating, and 1000 frames for testing.

NYU Depth v2 [36]: The NYU dataset is a color and depth camera from Microsoft Kinect, which consists of video sequences of various indoor scenes. According to the official method, we used about 50K images for training and 654 images for testing. We evaluated the effective region of 304 × 228 in order to match the resolution of RGB images and depth maps and compare it with other methods.

Our network runs on the Pytorch framework and is trained end-to-end on a single-stage NVIDIA GTX 2080 Ti GPU. During training, we use the Adam optimizer with ${\beta }_1=0.9$, ${\beta }_2=0.99$. The initial learning rate is 0.01 and decay of ${10}^{-1}$ every 3 epochs. We refer to the network parameters and the evaluation indicators of PNCNN [25] to compare the results fairly. We use the following common evaluation indicators to test our network performance. MAE is Mean Absolute Error (L1 loss). It is the mean of the distance between the model’s predicted value and the true value. Its convergence is fast and its gradient is stable. Therefore, it has a relatively robust solution. MSE is Mean Square Error (L2 Loss), which refers to the Mean squared difference between the predicted value of the model and the real sample value. Because its penalty is squared, it is sensitive to outliers. ABSREL is the Absolute Relative Error. RMSE stands for Root Mean Square Error. IMAE is Inverse Mean Absolute Error. IRMSE is Inverse Root Mean Square Error. DELTA is the percentage of pixels that are satisfied. MAE is shown in Equation (19). Other formulas can be expressed as

$$MSE=\frac{1}{n}{{\displaystyle \sum _{p\in P_v}^n\left(D_p^{gt}-D_p\right)}}^2$$

(21)

$$ABSERL=\frac{1}{n}{\displaystyle \sum _{p\in P_v}^n\left|\frac{D_p^{gt}-D_p}{D_p^{gt}}\right|}$$

(22)

$$RMSE=\sqrt{\frac{1}{n}{{\displaystyle \sum _{p\in P_v}^n\left(D_p^{gt}-D_p\right)}}^2}$$

(23)

$$IMAE=\frac{1}{n}{\displaystyle \sum _{p\in P_v}^n\left|\frac{1}{D_p^{gt}}-\frac{1}{D_p}\right|}$$

(24)

$$IRMSE=\sqrt{\frac{1}{n}{{\displaystyle \sum _{p\in P_v}^n\left(\frac{1}{D_p^{gt}}-\frac{1}{D_p}\right)}}^2}$$

(25)

$$DELT{A}_{\tau }={\delta }^{\tau }:\max (\frac{D_p}{D_p^{gt}}-\frac{D_p^{gt}}{D_p})<\tau ,\tau \in \left\{1.25,{1.25}^2,{1.25}^3\right\}$$

(26)

where $P_v$ represents the set of valid pixels. $D_p^{gt}$ represents the true value of the pixel $p$. $D_p$ represents the predicted value of the pixel $p$, and $n$ represents the number of points.

Although the improvement of the loss function has achieved good results, the improvement effect is relatively small. Moreover, the improvement of the loss function of each network is targeted. It is poor in its extensiveness. While with L2 loss there are more outliers in the data, L2 will bring more errors due to the square operation. Therefore, L1 performs slightly better than L2 [8] in depth prediction. Therefore, we chose the most commonly used mean absolute error MAE (L1 loss) as the loss function of our network, which represents the sum of all absolute differences between the true and predicted values.

3.2. Results of Comparative Experiments

The completed results are shown in Figure 5. As shown in the figure, our network can effectively recover the depth of information and strengthen the density and integrity of 3D information. In some detail, the results are visually clear and obvious. Targets are clearly defined and easier to distinguish. It is very important for the further processing of 3D information.

Our results on KITTI are shown in

Table 1. Comparisons to other methods without RGB image guidance on KITTI. The bold numbers are the best.

	SI-Net [19]	NCNN [22]	IP-Basic [2]	S2D++ [9]	PNCNN [25]	EIR-Net
MAE	481.27	360.28	302.60	288.64	251.77	225.70
IMAE	1.78	1.52	1.29	1.35	1.05	1.02
RMSE	1601.33	1268.22	1288.46	954.36	960.05	1061.75
IRMSE	4.94	4.67	3.78	3.21	3.37	3.77
Time (s)	0.01	0.01	-	0.04	0.02	0.02

, where our network (EIR-Net) achieves the best results with MAE and IMAE compared to other networks with a single depth input. The speed is similar to PNCNN, but all the indicators are better. Compared to the first two network models, our network speed is twice as slow, but the indicators are better than theirs. Compared to S2D++, both RMSE and IRMSE are worse. However, the difference is not much, only increased by 11.25% and 17.45%, but decreased by 27.89% and 32.35% in MAE and IMAE, respectively.

The completed results are shown in Figure 6. The depth of objects in the interior scene is effectively completed. Our results are shown in

Table 2. Comparisons to other methods on NYU. The bold numbers are the best.

	TGV [37]	RGB-d [38]	S2D [9]	NCNN [22]	SPN [24]	PNCNN [25]	EIR-Net
RMSE	0.635	0.228	0.230	0.171	0.162	0.144	0.142
ABSREL	0.123	0.042	0.044	0.026	0.027	0.021	0.020
δ1	81.9	97.1	97.1	98.3	98.5	98.8	98.8
δ2	93.0	99.3	99.4	99.6	99.7	99.8	99.8
δ3	96.8	99.7	99.8	99.9	99.9	99.9	99.9

. Our network (EIR-Net) achieves the best results compared to other networks with a single deep input and some GRB-guided networks.

3.3. Results of the Ablation Experiments

3.3.1. Confidence Re-Aggregation Module

We used different methods to experiment. As can be seen from

Table 3. The ablation experiments of the confidence re-aggregation module. The bold numbers are the best.

Model	Original	+CR(TC)	+CR(AS*)	+CR(AS)
MAE	227.416	227.581	226.786	226.647
MSE	1,274,874.118	1,298,745.842	1,281,756.162	1,270,920.852
RMSE	1065.504	1074.616	1067.330	1063.893
IMAE	1.02	0.98	0.99	0.97
IRMSE	13.821	5.961	13.351	5.763
Time	0.014	0.017	0.017	0.017

, re-aggregation is effective and the different aggregation methods have little impact on the loss function. At the same time, we found that the aggregation effect of AS is better than TC and AS*, and all indicators have been improved to different degrees. Finally, MAE decreased by 0.769, RMSE decreased by 1.611, IMAE decreased by 0.05, and IRMSE decreased by 8.058. In addition, these three methods take the same amount of time because the computation cost is the same. However, the time is only increased by 0.003 s to achieve the effect improvement. This means that the real-time performance of the network can still be guaranteed.

A more intuitive comparison is shown in Figure 7. We can find that the details are strengthened in the completed image after adding this module. Barely visible gaps are more fully displayed. In addition, the line of the back is smoother. Overall, we re-aggregated the confidence to strengthen the local detail effect, so that the outline of the person and the bicycle component is clearer.

3.3.2. Densely Progressive Fusion Module

The effect of our dense progressive fusion Module is shown in Figure 8. From the previous frame, we can know that there is a car in the part marked in the box. However, the network lost the car in learning. The U-shaped structure of the network degrades the resolution in the network without intensive progressive fusion. The original information that can be learned is destroyed. The ideal situation of this position is a plane, and the blank space is output. However, after adding the dense progressive fusion module, we no longer learn features around a single branch but multiple branches in parallel. The combination of multi-resolution images and progressive fusion makes the network not lose any effective information globally. We show exactly where the car is in the image.

We find that the module can be used to compensate for the loss of detail due to the resolution gap. Most current network architectures use convolutional structures with varying resolutions. Therefore, a loss between resolutions can be caused by down-sampling. The module can not only be used in depth completion to fine details but also can be used in other fields to avoid local weakening.

The effect of our intensive progressive fusion is shown in

Table 4. The ablation experiments of the dense progressive fusion module and the point folding module. The bold numbers are the best.

Model	MAE	RMSE	IMAE	IRMSE	δ1	δ2	δ3	Time
None	226.647	1063.893	0.97	5.763	99.594	99.845	99.921	0.017
DPF(1)	226.266	1066.031	1.06	22.622	99.595	99.845	99.921	0.018
DPF(2)	225.871	1063.624	1.07	5.363	99.595	99.846	99.921	0.020
+P-Folding	225.703	1061.745	1.02	3.774	99.600	99.846	99.922	0.020

. DPF represents dense progressive fusion module. (.) represents the number of layers. P-Folding represents the point folding module. Experiments show that the effect is improved. However, increasing the number of layers increases the time cost. In addition, the size of the resolution down-sampling is limited, and the number of layers is limited by the original size. Considering the real-time performance and the complexity of the structure, we choose two layers of DPF to add to our network to improve the global depth. Compared with the network without this module, MAE decreases by 0.776, RMSE decreases by 0.269, and IRMSE decreases by 0.4. IMAE increases, but only by 0.1. The pixel percentage of the second gradient increased by 0.001.

3.3.3. Point Folding Module

We show the point folding module in action in Figure 9. In the beginning, the car surface in the lower left corner does not have a good effect when the output completes the depth because the input depth is too sparse. The surface of the car is very uneven and large areas are missing. The point folding module can effectively supplement the missing part. As we can see, the surface depth of the car becomes flatter and denser with the addition of the point folding module. Further thinking, the shape of the car is more complete and clearer, which will be very conducive to improving the accuracy of 3D target recognition.

We show the effect of our dot fold module in the last row of Table 4. As we can see, the results are significantly improved compared to networks without this module. MAE decreased by 0.168, RMSE decreased by 1.879, IMAE decreased by 0.05, and IRMSE decreased by 1.989. The pixel percentages of the first and third gradients were increased by 0.005 and 0.001, respectively. There is no doubt that our module plays a role in optimizing the overall situation. The results of experiments show that our modules are efficient and portable not only when used separately but also when used in combination. The best results were obtained when we combine them.

4. Conclusions

In this paper, we designed a deep learning network that can directly complete sparse depth without color image. Unlike most existing approaches that require guidance, we re-aggregate confidence to enhance detail. And the global information is improved by dense progressive fusion structure and point folding module. The combination of modules makes the use of effective information to the greatest extent. Through experiments and comparisons on KITTI and NYU Depth v2, we demonstrate the effectiveness of the network.