Self-Supervised Depth Completion Based on Multi-Modal Spatio-Temporal Consistency

Abstract

Due to the low cost and easy deployment, self-supervised depth completion has been widely studied in recent years. In this work, a self-supervised depth completion method is designed based on multi-modal spatio-temporal consistency (MSC). The self-supervised depth completion nowadays faces other problems: moving objects, occluded/dark light/low texture parts, long-distance completion, and cross-modal fusion. In the face of these problems, the most critical novelty of this work lies in that the self-supervised mechanism is designed to train the depth completion network by MSC constraint. It not only makes better use of depth-temporal data, but also plays the advantage of photometric-temporal constraint. With the self-supervised mechanism of MSC constraint, the overall system outperforms many other self-supervised networks, even exceeding partially supervised networks.

1. Introduction

In an intelligent traffic system, it has become a fundamental task to obtain the position of objects around. Especially in autonomous driving, the ability to perceive the environment is the basis for the safe and stable operation of autonomous driving [1,2]. The sparse depth provided by LiDAR can support a object segmentation [3] or a simultaneous localization and mapping (SLAM) system [4], but has a poor performance on the scene topology [5]. Moreover, increasing the density of 3D LiDAR measurements means it is cost-prohibitive. Depth completion has always been a research hotspot, especially on low-cost devices. However, the difficulty of obtaining dense annotations makes this technique difficult to implement [6]. Therefore, the self-supervised depth completion have been widely studied in recent years [7].

At present, many supervised depth completions have been proposed. Uhrig et al. [8] proved that convolution still worked on sparse signals, and Ma et al. [6] proved that image information could help reconstruct dense depth images with higher accuracy. Most of depth completion methods have been proposed to generates dense depth map by LiDAR sparse points cloud and RGB images. Due to the sparsity of input depth, convolutional neural networks have difficulty adapting to spatial pixel information [8]. Uhrig et al. [8] proposed a sparse invariant convolution, which enhanced the adaptability of convolution to sparse signals through adaptive sparse weights. This method uses the sparse mask to avoid the undifferentiated calculation of nonexistent points, while it cannot process occluded points or fuse RGB image information.

However, most of the current depth completion networks use RGB images as guidance, because RGB images can provide edge information of objects [9]. The depth map calibrated by internal parameters can be aligned with the RGB images at pixel level, which provides the theoretical basis for the cross-modal depth completion [7]. Eldesokey et al. [10,11] designed a normalized convolutional layer to introduce RGB image information while completing the depth. Yan and Huang et al. [12,13] proposed a sparse invariant convolution, which can perform feature fusion at each layer.

In terms of fusion methods, it can be divided into signal layer fusion and feature layer fusion [7]. Ma et al. [14] proposed a modular coding network based on ResNet [15] to predict the dense depth map of RGB-D images, and they began to fuse from the signal layer. Jaritz et al. [9,16] chose the strategy of fusion at the feature layer. Another method fuses RGB-D images into multiple-scale feature layers [17,18,19]. Many introduce physical priors based on signal fusion, such as introducing normal surface [20,21,22] and semantic segmentation [9,23,24]. Moreover, some methods guide the propagation of sparse depth, according to the correlation of successive pixels on the RGB map [25,26].

However, all of these full-supervised depth completion methods still faces an unavoidable problem on practical application: how to obtain ground truth labels. To face with the lack of ground truth labels, many studies on self-supervised depth completion have been proposed. The self-supervised network is modelled with the intrinsic consistency within the images and the aligned point clouds. Different from the supervised networks which are modelled relying on a large amount of manual labelled ground truths, the proposed self-supervised network automatically construct a relationship between the multi-modal data and the dense depth, which makes it more robust and have better generalization ability.

Self-supervised depth completion method can be divided into stereo vision and monocular vision [7]. Both take the reprojection error as the major constraint. The stereo method [27,28] converts the depth to parallax and calculates the reprojection error with the help of a pre-calibrated baseline. Compared with the latter, the stereo method does not need to consider the influence of external parameters, but it cannot solve the problems caused by occlusion and cost another camera. The monocular method projected adjacent frames to the current frame through pose and depth. Ma et al. [6] first designed a self-supervising framework, with photometric loss as depth supervision. This method requires pose estimation to provide external parameters. To improve the accuracy of the pose estimation, Feng and Choi et al. [29,30,31] introduced model-based pose estimation module. Among them, Song and Wong et al. [5,32] introduced an odometer to improve their pose network. In a word, existing self-supervised depth completion methods commonly use RGB image reprojection to establish strong constraints for spatial connections. However, different from the dense depth ground truth, the reprojection error is evaluated by photometric value without direct depth supervision. It leads to the difficulty of fusion between image data and the projected sparse depth map [6]. Therefore, Feng et al. [29] generate pseudo dense representations before concatenating the image data and the sparse depth map. Wone et al. [31] pool the sparse depth map while inputting the sparse depth points. It alleviates the problem caused by the photometric evaluation, but the problem of multi-modal information fusion remains.

At present, the self-supervised depth completion have main problems: (1) The reprojection constraint assumes that the scene is static and non-occluded, which reduces the matching success rate of moving objects and occluded regions [6]. Meanwhile, the reprojection error cannot reflect the depth loss between the predicted value and the ground truth, especially in the dark, low-texture parts and distant objects. (2) Depth-temporal information contains a lot of usable information, which should not be discarded in self-supervision. (3) It is not conducive to the fusion of image data and sparse depth map, when the photometric evaluation is used as the depth constraint.

To cope with these challenges, a multi-modal spatio-temporal consistency approach is proposed to help improve the performance of model-based depth completion. We introduce depth-temporal data to reduce the impact of photometric constraints on depth completion, such as dark, low-texture parts and distant objects. Meanwhile, we proposed a depth-temporal consistency constraint to directly supervise depth, so in the whole process of depth completion, RGB image data can significantly improve the effect of depth completion, through effective fusion. However, the sparse depth map are sampled from dynamic scenes, and the temporal depth after simply stacking cannot be directly used to constrain the depth. Therefore, we designed a photometric reprojection auto-mask to remove the occlusion and displace points. Meanwhile, we introduced the photometric-temporal consistency to provide a global constraint for our depth completion. In addition, multi-modal spatio-temporal consistency is also introduced into the pose estimation to improve the accuracy.

2. Methods

In this section, our self-supervised depth completion framework is shown in Figure 1, which takes a single visible image $RG{B}_t$ and a LiDAR image $D_t$ as input and generates a dense depth map, $Pred$.

This methods is divided into two steps: Step 1, spatial translate the depth map of adjacent frames ($D_{t-1}$ and $D_{t+1}$) into the current camera field, to generate the $D_{t-1}^{\prime }$ and $D_{t+1}^{\prime }$. The pose parameter is provided from AFPR-PnP module. After translation, warped depth points can be reflected on $RG{B}_t$. Step 2, self-supervised training procedure based on multi-modal spatio-temporal consistency (MSC) constraint: It requires a sequence of RGB-D images for training. In the inference processing, it needs only a pair of RGB-D images for generating completion depth.

The pose $R,T_{t+i\to t}$ from the current frame $RGB-D_{t+i}$ to the adjacent frame $RGB-D_t$ is calculated by the PnP algorithm, where $i\in \{-1,+1\}$. The depth map of adjacent frames ($D_{t-1}$ and $D_{t+1}$) can be translated to the current camera field ($D_{t-1}^{\prime }$ and $D_{t+1}^{\prime }$) by $R,T_{t-1\to t}$ and $R,T_{t+1\to t}$. This is so that a multi-modal spatio-temporal consistency constraint can be built for the self-supervising framework. At the same time, a photometric reproject auto-mask is designed based on the similarity estimation. This automatic mask reduces the displacement errors caused by the depth-temporal point cloud.

The last, an automatic feature points refinement algorithm, is adopted to improve the performance of PnP pose estimation.

This self-supervised framework does not rely on additional sensors, manual tagging efforts, or other learning-based pose estimation algorithms as building blocks. In the inference, only the current frame is needed as an input to generate depth completion.

2.1. Depth-Temporal Consistency Constraint

This paper proposed a self-supervising constraint method based on depth-temporal consistency. The method only needs to obtain synchronous $RGB-D$ image sequences from monocular cameras and LiDAR. The point clouds of the adjacent frames can be transfered to the current frame to constrain depth completion, as shown in Figure 1-step 1.

As is shown in Figure 2, multi-modal spatio-temporal consistency constraint contains two parts, depth-temporal consistency constraint ($Los{s}_{Depth}$) and photometric similarity evaluation (PSE). The photometric reproject auto-mask generated by PSE was used to assist the depth-temporal consistency constraint on the current frame.

2.1.1. Spatial Translation

We take $RGB-D_t$ at time t, and the adjacent frame as $RGB-D_{t+i}$, where $i\in \{-1,1\}$. Pose parameters are estimated with $RGB-D_t$ and $RGB-D_{t+i}$, containing two groups of the rotation matrix and translation ($R,T_{t+i\to t}$). The detail of pose estimation is introduced in Section 2.3.

The external parameter matrix can be expressed as:

$$T_{t+i\to t}=\left[\begin{array}{cc}{R}_{t+i}& T_{t+i}\\ 0^3& 1\end{array}\right],T_{t\to t+i}=\left[\begin{array}{cc}{R}_{t+i}^{-1}& -T_{t+i}\\ 0^3& 1\end{array}\right].$$

(1)

The information of the LiDAR points at time $t+i$ can be transferred to the camera coordinate system at time t through external parameters, and the process can be expressed as:

$${\left[\begin{array}{cccc}{x}_{t+i}^{\prime }& y_{t+i}^{\prime }& z_{t+i}^{\prime }& 1\end{array}\right]}^T=T_{t+i\to t}·{\left[\begin{array}{cccc}{x}_{t+i}& y_{t+1}& z_{t+1}& 1\end{array}\right]}^T,$$

(2)

where $D_{t+i}^{\prime }=z_{t+i}^{\prime }$. The translated depth map can be calculated by internal parameters:

$$D_{t+i}^{\prime }{\left[\begin{array}{ccc}u& v& 1\end{array}\right]}^T=K·{\left[\begin{array}{cccc}{x}_{t+i}^{\prime }& y_{t+i}^{\prime }& z_{t+i}^{\prime }& 1\end{array}\right]}^T.$$

(3)

We can obtain the sparse depth map $D_{t+i}^{\prime }$ under the camera coordinate system at the time of t.

2.1.2. Depth-Temporal Consistency Module

Sequential frames contain a lot of spatial structure information. For example, adjacent RGB images have most of the same scenes as the current frame, as do depth images. Different from the point cloud of the current frame ($D_t$), it provides a lot of depth information in the blank block. However, it cannot be used directly until it is transferred to the current camera field.

The depth-temporal consistency loss function ($Los{s}_{Depth}$) is built with $D_{t+i}^{\prime }$ and $Pred$. To ground the predictions to a metric scale, we minimize the $L^2$ difference between predictions $pred$ and the sparse warped depth map $D_{t-1}^{\prime }$ and $D_{t+1}^{\prime }$ over its domain (${\Omega }_{t-1}$ and ${\Omega }_{t+1}$):

$$\begin{array}{c}\hfill Los{s}_{Depth}=\frac{1}{|{\Omega }_{t-1}|}\sum _{x\in {\Omega }_{t-1}}\left|\right|D_{t-1}^{\prime }\left(x\right)-pred\left(x\right){\left|\right|}^2\\ \hfill +\frac{1}{|{\Omega }_{t+1}|}\sum _{x\in {\Omega }_{t+1}}\left|\right|D_{t+1}^{\prime }\left(x\right)-pred\left(x\right){\left|\right|}^2.\end{array}$$

(4)

2.2. Photometric-Temporal Consistency Constraint

Because of the object movement and occlusion, $D_{t+i}^{\prime }$ contains many displaced depth points affecting the accuracy of the constraint. Inspired by Godard et al. [33], we select the most similar part of the pixel structure to retain the depth points of this part. The similarity of warped images ($RG{B}_{t+i}^{\prime }$ and $RG{B}_t$) can be used to select the minimum value points in $E_{ph}{|}_{t+i}$ as the automatic photometric reprojection mask ($mas{k}_{t+i}$).

We multiply $mas{k}_{t+i}$ times $D_{t+i}^{\prime }$, and combine their results as a new depth map $D_t^{\prime }$.

2.2.1. Pixal Warp

Similarly with Section 2.1.1, RGB images $RG{B}_{t+i}$ will be mapped to the camera coordinate system at t through the pose and predicted depth, as shown in Figure 3.

Take the pixel point of the predicted depth image $Pred$ at $(u,v)$, then the pixel point coordinates after the predicted depth can be expressed as:

$${\left[\begin{array}{ccc}{u}^{\prime }& v^{\prime }& 1\end{array}\right]}^T=Pred(u,v)K^{-1}{\left[\begin{array}{ccc}u& v& 1\end{array}\right]}^T.$$

(5)

The sampling function of the warped RGB image $RG{B}_{t+i}^{\prime }$ at $(u,v)$ can be expressed as:

$$RG{B}_{t+i}^{\prime }(u,v)={\left(RG{B}_{t+i}\left(K{T}_{t\to t+i}\right)\left[\begin{array}{ccc}{u}^{\prime }& v^{\prime }& 1\end{array}\right]\right)}^T.$$

(6)

Finally, we can obtain the RGB image $RG{B}_{t+i}^{\prime }$ under the camera coordinate system at the time of t.

2.2.2. Photometric Reproject Auto-Mask

We map the adjacent RGB images to the current RGB image for similarity evaluation by referring to the current predicted depth. The evaluation function $E_{ph}$ contains L1 loss and pixel structure loss.

The photometric error of $RG{B}_t$ and $RG{B}_{t+i}^{\prime }$ can be represented as:

$$\begin{array}{c}\hfill E_{ph}{|}_{t+i}=\frac{\omega }{2}(1-SSIM(RG{B}_{t+i}^{\prime },RG{B}_t))\\ \hfill +(1-\omega )\parallel RG{B}_{t+i}^{\prime },RG{B}_t{\parallel }^1,\end{array}$$

(7)

where $\omega$ is the weight between 0 and 1, and the t is time.

We select the adjacent depth point based on the minimum photometric error. Specifically, we took the minimum value of $E_{ph}$ as photometric loss $Los{s}_{ph}$. This photometric loss can overcome the effects of partial occlusion and displacement [33].

It can be expressed as:

$$Los{s}_{ph}=Min\{E_{ph}{|}_{t-1},E_{ph}{|}_{t+1}\},$$

(8)

and the photometric loss function is:

$$Los{s}_{pe}=\frac{1}{|\Omega |}\sum _{(x,y)\in \Omega }Los{s}_{ph}(x,y).$$

(9)

Among them, $Los{s}_{ph}$ generated by selecting the minimum value from $E_{ph}(RG{B}_{t+1}^{\prime },RG{B}_t)$ and $E_{ph}(RG{B}_{t-1}^{\prime },RG{B}_t)$, which means the selected region is the unobstructed part.

The photometric reproject auto-mask according to this characteristics, which represents the most similar part of the warped adjacent RGB images, as shown in Figure 3.

The auto-mask can be represented as:

$$mas{k}_{t+i}(x,y)=\left\{\begin{array}{cc}\hfill 1& \mathrm{if}Los{s}_{ph}(x,y)=E_{ph}{|}_{t+i}(x,y)\hfill \\ \hfill 0& otherwise.\hfill \end{array}\right.$$

(10)

The auto-mask also represents the most similar part of the depth when it overlaps with the depth image. For the occluded part in a adjacent depth, the auto-mask will have a higher response in another adjacent depth image, to complementing the occluded depth. In the selection of adjacent depths, the auto-mask can provide an available reference. The stitched depth still works on moving objects, unlike photometric errors.

The stitched depth $D_t^{\prime }$ is shown in Figure 4, represented as:

$$\begin{array}{c}\hfill D_t^{\prime }=\sum _i^{-1,1}(D_{t+i}^{\prime }\ast mas{k}_{t+i})\end{array}.$$

(11)

The sequence depth loss function can be represented as:

$$Los{s}_{Depth}=\frac{1}{|\Omega |}\sum _{x\in \Omega }\left|\right|D_t^{\prime }\left(x\right)-pred\left(x\right){\left|\right|}^2,$$

(12)

where $\Omega$ is the domain of depth map $D_t^{\prime }$.

Our multi-modal spatio-temporal consistency constraints include depth-temporal consistency constraint (DC) and photometric-temporal consistency constraint (PC). We implement both constraints with $Los{s}_{Depth}$ and $Los{s}_{pe}$.

2.3. Automatic Feature Points Refinement

In this method, the stitched depth comes from the translated adjacent depth. The pose estimation network gives constantly adjusted poses, which will bring a heavy calculation to the preprocessing. Therefore, we made the preprocessing dataset with the PnP algorithm, which including the sparse depth after obtaining the fixed pose and translated depth.

Since there are many moving objects in the traffic scene, the feature points from fast-moving objects will affect the estimation in previous PnP algorithms. The automatic feature points refinement proposed as shown in Figure 5, on the left side, the blue point is the matched pair of feature points, the gray point is the camera coordinate system position of feature points (each pair of feature points maps to the same gray point), and the red point is the fast displacement point in the two imaging. On the right side, we line the matched feature points, the green line is the point pair successfully selected, and the red line is the point pair filtered by AFPR-PnP algorithm (the left column is the PnP algorithm designed by Ma, and the right column is ours).

Our method adopt the evaluation of moving distance in 3d space to remove interference points. For mismatched points, some of these pairs can be filtered in 2D space, while others are indistinguishable in 2D space, as shown in Figure 5. And such problems can be improved by converting to 3D space coordinate. Assuming that the coordinate of point $p_t$ on the visible image is $(u,v)$, and the depth at here is d, the 3D space position can be expressed as $p{s}_t$:

$$p{s}_t=K^{-1}·d·{\left[\begin{array}{ccc}u& v& 1\end{array}\right]}^T.$$

(13)

Transform the feature points into 3D space position coordinates, and calculate their pixel distance $D_{pixel}$ and space distance $D_{space}$.

$$D_{pixel}=\left|\right|p_t-p_{t+i}{\left|\right|}^2,$$

(14)

$$D_{space}=\left|\right|p{s}_t-p{s}_{t+i}{\left|\right|}^2.$$

(15)

We removed the point pairs whose pixel distance $D_{pixel}$ exceeded $t{h}_{d2}$ and space coordinate distance $D_{space}$ exceeded $t{h}_{d3}$.

3. Experimental Evaluation

3.1. Datasets and Setup

This chapter briefly describes the preprocessing of datasets and design experiments to verify the validity of model.

Datasets: We used KITTI https://www.cvlibs.net/datasets/kitti/raw_data.php (accessed on 21 November 2022) as our data set. In the data pre-processing, we deleted the data of the static scene and the camera on the right. It reduces duplicate scenes and detection points. There are 85,342 training sets. For the test, we selected 1000 test set images for comparison.

Before training, we preprocess the data set. We calculate the Pose parameter through the PnP algorithm, translate the point cloud of the adjacent frame to the current camera coordinate system by parameter, and then splice the translated point cloud according to Formula (11) to generate $D_t^{\prime }$. Since the PnP algorithm we use is unchanged for each frame, the input results can be reused after saving, thus reducing a lot of training pressure.

Our preprocessing is given in the following Algorithm 1 flow chart:

Algorithm 1 Data set preprocessing

Input: RGB images and Lidar data set $I_{train}$ For $eachepisode(RG{B}_t,D_t,RG{B}_{t+i},D_{t+i})\in I_{train}$, where $i\in \{-1,1\}$ do:     1. Calculate the $R{t}_{t+i\to t}$ from $RGB-D_{t+i}$ to $RGB-D_t$, through the AFPR-PnP.     2. We transferred the $D_{t+i}$ to the same coordinate system as $D_t$ through $R{t}_{t+i\to t}$ and internal parameters, then obtained $D_{t+i}^{\prime }$ from Equation (3).     3. The generated $D_{t+i}^{\prime }$ and $R{t}_{t+i}$ are saved as the pre-preprocessing set $D_{train}{|}_i$.

Setup: We used PyTorch to train the model and cropped the input image to 352*1216. In the pre-processing stage, color correction and noise superposition are performed on the visible light data. We have a learning rate of ${10}^{-5}$. Set the batch size to 4, learn 30 epochs, and train the model with an NVIDIA TITAN RTX. We tested the validity of proposed module in three test directions. Our depth completion network adopts the network structure of $sparse-to-dense$, and the ResNet18 network experiment verifies the superiority of $Los{s}_{Depth}$ and PnP algorithms. ResNet34 is used to verify the validity of the deep information mining module. Finally, in comparison with other methods, we also use resnet34.

The final loss function:

$$Loss={\omega }_{0}Los{s}_{Depth}+{\omega }_{1}Los{s}_{Smooth}+{\omega }_{2}Los{s}_{pe}$$

(16)

where ${\omega }_0$, ${\omega }_1$ and ${\omega }_2$ are respectively weights of $Los{s}_{Depth}$ (Function (12)), $Los{s}_{Smooth}$, and $Los{s}_{pe}$ (Function (9)).

During the training, the semi-dense depth labels provided by KITTI were not involved in the calculation. We use the KITTI2012 prediction set https://www.cvlibs.net/datasets/kitti/raw_data.php (accessed on 21 November 2022) and our output results to calculate RMSE error as a standard to evaluate the depth completion accuracy because it is the most representative indicator. RMSE can be expressed as:

$$RMSE=(\frac{1}{|\Omega |}\sum _{x\in \Omega }|\widehat{z}\left(x\right)-z_{gt}\left(x\right){{|}^2)}^{\frac{1}{2}},$$

(17)

$\widehat{z}$ denotes the completed depth; $z_{gt}$ denotes the ground-truth depth.

Our training process is given in the following Algorithm 2 flow chart:

Algorithm 2 Flow chart of self-supervised sequence depth constraint training algorithm.

Input: RGB images and Lidar data set $I_{train}$, the pre-preprocessing set $D_{train}{|}_i$ For $eachepisode(RG{B}_t,D_t,RG{B}_{t+i},D_{t+i})\in I_{train}$ and $(D_{t+i}^{\prime },R{t}_{t+i})\in D_{train}{|}_i$ do:     1. Input $RG{B}_t,D_t$ into network, predict output $Pred$.     2. By outputting depth $Pred$, the PnP algorithm estimates pose $R{t}_{t+i}$ and camera internal parameters and maps $RG{B}_{t+i}$ to the camera coordinate system of $RG{B}_t$ to calculate photometric loss $E_{ph}\left(RG{B}_{t+i}^{\prime }\right)$. The formula is given by (7).     3. he photometric loss $E_{ph}\left(RG{B}_{t+i}^{\prime }\right)$ is used to select and splice $D_{t+i}$ to obtain $D_t^{\prime }$, and the relation is given by (11).     4. Calculate the total loss Function (16). For $eachepisode(RG{B}_t,D_t,G{t}_t)\in I_{test}$ do:     1. Input $RG{B}_t,D_t$ into network, predict output $Pred$.     2. Calculate the RMSE errors of A with B by the error Formula (17).

3.2. Results of the Ablation Experiments

Our multi-modal spatio-temporal consistency (MSC) constraint is embodied in using the consistency of sequence multi-modal information to constrain.

In this section, we compare three self-supervised methods: depth-temporal consistency (DC) constraint, photometric-temporal consistency(PC) constraint, and multi-modal spatio-temporal consistency (MSC) constraint.

As shown in

Table 1. Comparison of self-supervised methods for different sequence modality constraints. The bold numbers are the best.

Method	$\mathit{N}\mathit{e}\mathit{t}\mathit{w}\mathit{o}\mathit{r}\mathit{k}$	$\mathit{A}\mathit{u}\mathit{t}\mathit{o}-\mathit{M}\mathit{a}\mathit{s}\mathit{k}$	$\mathit{L}\mathit{o}\mathit{s}{\mathit{s}}_{\mathit{D}\mathit{e}\mathit{p}\mathit{t}\mathit{h}}$	$\mathit{L}\mathit{o}\mathit{s}{\mathit{s}}_{\mathit{S}\mathit{m}\mathit{o}\mathit{o}\mathit{t}\mathit{h}}$	$\mathit{L}\mathit{o}\mathit{s}{\mathit{s}}_{\mathit{p}\mathit{e}}$	$\mathit{L}\mathit{a}\mathit{y}\mathit{e}\mathit{r}\mathit{s}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$
$s2d-DC$	$s2dNet$	√	0.010	0.1		18	4885.72
$s2d-PC$	$s2dNet$			0.1	1.0	18	2299.49
$s2d18-MSC$	$s2dNet$	x	0.010	0.1	1.0	18	1379.70
$s2d18-MSC-auto$	$s2dNet$	√	0.010	0.1	1.0	18	1316.93
$s2d34-MSC$	$s2dNet$	x	0.010	0.1	1.0	34	1241.58
$s2d34-MSC-auto$	$s2dNet$	√	0.010	0.1	1.0	34	1212.69
$kb-MSC$	$kbNet$	x	0.010	0.1	1.0		1527.37
$kb-MSC-auto$	$kbNet$	√	0.010	0.1	1.0		1289.67
$s2d18-MSC1$	$s2dNet$	√	0.001	0.1	1.0	18	1503.27
$s2d18-MSC2$	$s2dNet$	√	0.010	0.1	0.1	18	1577.12

, the depth completion network can converge smoothly with stitched depth $D_t^{\prime }$ participation in the constraint. As seen in Figure 4, mask comes from the part of the minimum photometric loss, which is multiplied by $D_{t+i}^{\prime }$ to filter out most of the wrong displacement points of $D_t^{\prime }$. This further demonstrates the effectiveness of DC constraints.

Since RGB and depth data values in different range, and we only regularized RGB images, and the values of $Los{s}_{Depth}$ and $Los{s}_{Smooth}$ are different from $Los{s}_{pe}$ in scale. We tested different values of ${\omega }_0$ and ${\omega }_2$. There is much space for optimization in the adjusting parameters.

MSC constraint: As can be observed from Table 1, the MSC method has obvious advantages over single-modal constraint in stable convergence: the lack of any modal constraint ($Los{s}_{Depth}$ or $Los{s}_{pe}$) will crimp the network performance.

(1)

Depth-temporal consistency constraint (DC): Many works involve the input depth map in training so that the network will not lose too much depth information. However, due to the displacement between the RGB camera and LiDAR, some of the background depth and the foreground depth are mixed in the same occluded area. Retaining input depth means retaining these erroneous background depth. Our method improved this practice and the stitched sequence depth to replace the current depth.

(2)

Photometric-temporal consistency constraint (PC): The sequence depth constraint is too sparse to provide the global constraint. Thus, we introduce the sequence photometric constraint. As shown in the Table 1, the sequence photometric constraint is also indispensable, and the superiority of the sequence multi-modal constraint is also proved. In the DC constraint, this temporal depth data cannot constrain depth completion directly; it contains displaces depth points. This points can affect the performance of depth completion, which was also shown in subsequent experiments. Therefore, we introduce photometric reproject auto-mask to remove these error points, and the experiment proves that this auto-mask is useful.

(3)

Multi-modal spatio-temporal consistency constraint (MSC): The MSC constraint contains DC and PC, containing three loss functions. We tested the effect of their weights on the two network structure, S2D [6] and KBNet [31]. As can be observed in the Table 1, the network has the best performance when ${\omega }_0=0.01$, ${\omega }_1=0.1$, and ${\omega }_2=1$. We tested several weight ratios of $Los{s}_{Depth}$ and $Los{s}_{pe}$ and finally took ${\omega }_0=0.01$, ${\omega }_1=0.1$, and ${\omega }_2=1$ as the value of the following experiment.

In addition, we change the input for the experiment, which further verifies that the network can not only mine spatial information but also learn the depth completion from RGB information. As shown in the

Table 2. Experiment and result comparison of spatial information matching mask. The bold numbers are the best.

Method	Input	$\mathit{L}\mathit{o}\mathit{s}{\mathit{s}}_{\mathit{D}\mathit{e}\mathit{p}\mathit{t}\mathit{h}}$	$\mathit{L}\mathit{o}\mathit{s}{\mathit{s}}_{\mathit{S}\mathit{m}\mathit{o}\mathit{o}\mathit{t}\mathit{h}}$	$\mathit{L}\mathit{o}\mathit{s}{\mathit{s}}_{\mathit{p}\mathit{e}}$	$\mathit{L}\mathit{a}\mathit{y}\mathit{e}\mathit{r}\mathit{s}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$
1	$RGB+Depth$	0.01	0.1	1	34	1212.69
2	$Depth$	0.01	0.1	1	34	1264.01
3	$Gray+Depth$	0.01	0.1	1	34	1255.51

, input RGB or Gray images can assist depth completion to generate more accurate depth than input depth only.

AFPR-PnP: In this experiment, we demonstrate the reliability of the PnP algorithm based on automatic feature point refinement. Since S2D [6] have the same pose estimation method (PnP algorithm) with ours. We preprocess the same dataset with the PnP algorithm proposed by Ma et al., train the network in the same way, and compare it with our method. As shown in Figure 5, our PnP algorithm filtered error points, including points on moving cars and different objects with repeating textures.

Specifically, we set $t{h}_{d2}=100\left(pixel\right)$, $t{h}_{d2}=3\left(meter\right)$.

As shown in

Table 3. Experiment and result comparison of AFPR-PnP algorithms. The bold numbers are the best.

Method	Network	$\mathit{D}\mathit{e}\mathit{p}\mathit{t}\mathit{h}\mathit{C}\mathit{o}\mathit{n}\mathit{s}\mathit{t}\mathit{r}\mathit{a}\mathit{i}\mathit{n}\mathit{t}$	$\mathit{A}\mathit{u}\mathit{t}\mathit{o}-\mathit{M}\mathit{a}\mathit{s}\mathit{k}$	$\mathit{P}\mathit{o}\mathit{s}\mathit{e}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$
$s2d$	s2dNet	$D_t$	x	PnP(Ma)	1476.76
$s2d+DC$	s2dNet	$D_{t-1}^{\prime }+D_{t+1}^{\prime }$	x	PnP(Ma)	1379.70
$s2d+DC+PC$	s2dNet	$D_t^{\prime }$	√	PnP(Ma)	1322.37
$s2d+MSC+AFPR$	s2dNet	$D_t^{\prime }$	√	PnP(AFPR)	1316.93
$kbnet$	kbNet	$D_t$	x	PnP(Ma)	1495.51
$kbnet+DC$	kbNet	$D_{t-1}^{\prime }+D_{t+1}^{\prime }$	x	PnP(Ma)	1361.9
$kbnet+DC+PC$	kbNet	$D_t^{\prime }$	√	PnP(Ma)	1311.12
$kbnet+MSC+AFPR$	kbNet	$D_t^{\prime }$	√	PnP(AFPR)	1289.67

, we performed ablation experiments with S2D [6] and KBNet [31], the performance for depth completion is enhanced with these methods.

Discussion: In Figure 6, we compare the performance of model-based depth completion with and without MSC constraint (S2D+MSC(ours) and S2D(Ma)) in detail:

(1)

Experiments show that the performance of dark, low-texture parts and distant object is significantly improved(➀ ➁ ➂ ➃).

(2)

Sequence depth replaces the input depth as a constraint, so the network does not fully preserve the input depth with its error points. In addition, our photometric similarity estimation module filters out the occlusion points in adjacent depth. Compared with the scheme without MSC constraints, our method reduces the influence of displace points of sparse depth input on depth completion (➄ ➅).

(3)

For dynamic targets (➃ ➆), our method has better adaptability.

(4)

Our method enhances the fusion ability of the network for multi-modal information, so there will not be a large number of residual sparse points in the image(➀ ➁ ➄ ➅ ➆).

3.3. Results of the Comparative Experiments

As shown in

Table 4. The experimental results compared with other methods. The bold numbers are the best.

Method	$\mathit{S}\mathit{L}\mathit{A}\mathit{M}$	$\mathit{C}\mathit{C}/\mathit{M}\mathit{S}\mathit{C}$	$\mathit{R}\mathit{M}\mathit{S}\mathit{E}$
Kitti2012 Depth Completion Validation Dataset
S2D [6]	PnP	$PC$	1342.33
DepthComp [5]	PnP	$PC$	1330.88
DepthComp	PoseNet	$PC$	1282.81
SelfDeco [30]	PoseNet	$PC$	1212.89
KBNet(withPnP)	PnP	$PC$	1289.67
our	PnP	$MSC$	1212.69
Kitti2012 Depth Completion Test Dataset
S2D	PnP	$PC$	1299.85
IP−Basic [34]	PnP	$PC$	1288.46
KBNet(with~PnP)	PnP	$PC$	1223.59
DFuseNet [28]	Stereo	/	1206.66
DDP [27]	PoseNet	$PC$	1263.19
DepthComp	PoseNet	$PC$	1216.26
VOICED(VGG8) [32]	PoseNet	$PC$	1164.58
VOICED(VGG11)	PoseNet	$PC$	1169.97
ours	PnP	$MSC$	1156.78

, we compared the sequence multi-modal constrain self-supervised algorithm with other algorithms on the validation dataset and test set of KITTI2012, and the lower the indicator, the better. We selected DepthComp [5], VOICED [32], SelfDeco [30] and ddp [27] network pose estimation methods to compare with Ma and other PnP pose estimation methods. At the same time, we compare different SLAM acquisition methods. It can be observed that our PnP position estimation method even exceeds the performance of the pose estimation network methods. The stereo images self-supervised method [28] avoids the effect of pose translation with the cost of one more camera, which exceeds the self-supervised scheme of the partial pose estimation network. But our method still has an advantage over it with only a single camera.

PnP has the advantages of scene generalization and no training but at the cost of reducing accuracy [5]. However, we still obtain good results with AFPR-PnP algorithm. Additionally, we classify existing constraint methods into multi-modal spatio-temporal consistency constraint (MSC) and photometric-temporal constraint (PC). It can be observed that our MSC constraint method contained DC and PC is more advantageous than only PC.

Discussion: From the comparison (Figure 7) of the output results, our depth completion is better for the interior of the object, while for the edge of the object, our depth completion has fewer wrong completions.

By synthesizing all the completion results, it can be observed that the self-supervised completion still has the completion ability for the areas not detected by LiDAR. Compared with the supervised depth completion, it does not rely on manual labelled ground truths and has a better application prospect.

4. Conclusions

To sum up, this paper completed the design of self-supervised depth completion without the ground truth. Among the self-supervised depth completion algorithms, our method is the only one that exploits the MSC constraint. The unique advantage of this method is that it can improve the performance of depth completion on moving objects and occluded/dark light/low texture parts, making the use of the multi-modal spatio-temporal information to the greatest extent. Our experiment demonstrates the effectiveness of our method, even surpassing many supervised methods. However, our method still has much room for improvement. In future work, we will put forward more improvement schemes for the point cloud denoising of sequence LiDAR. Besides, a pose estimation for real-time spatial translation needs to be proposed, and we will focus on improving its estimation accuracy. On this basis, we can still observe the great potential for this kind of self-supervision framework.