Improving Plane Fitting Accuracy with Rigorous Error Models of Structured Light-Based RGB-D Sensors

Abstract

Plane fitting is a fundamental operation for point cloud data processing. Most existing methods for point cloud plane fitting have been developed based on high-quality Lidar data giving equal weight to the point cloud data. In recent years, using low-quality RGB-Depth (RGB-D) sensors to generate 3D models has attracted much attention. However, with low-quality point cloud data, equal weight plane fitting methods are not optimal as the range errors of RGB-D sensors are distance-related. In this paper, we developed an accurate plane fitting method for a structured light (SL)-based RGB-D sensor. First, we derived an error model of a point cloud dataset from the SL-based RGB-D sensor through error propagation from the raw measurement to the point coordinates. A new cost function based on minimizing the radial distances with the derived rigorous error model was then proposed for the random sample consensus (RANSAC)-based plane fitting method. The experimental results demonstrated that our method is robust and practical for different operating ranges and different working conditions. In the experiments, for the operating ranges from 1.23 meters to 4.31 meters, the mean plane angle errors were about one degree, and the mean plane distance errors were less than six centimeters. When the dataset is of a large-depth-measurement scale, the proposed method can significantly improve the plane fitting accuracy, with a plane angle error of 0.5 degrees and a mean distance error of 4.7 cm, compared to 3.8 degrees and 16.8 cm, respectively, from the conventional un-weighted RANSAC method. The experimental results also demonstrate that the proposed method is applicable for different types of SL-based RGB-D sensor. The rigorous error model of the SL-based RGB-D sensor is essential for many applications such as in outlier detection and data authorization. Meanwhile, the precise plane fitting method developed in our research will benefit algorithms based on high-accuracy plane features such as depth calibration, 3D feature-based simultaneous localization and mapping (SLAM), and the generation of indoor building information models (BIMs).

1. Introduction

Point cloud datasets are widely used for 3D model generation [1,2], high definition (HD) map generation [3,4,5], and robot/vehicle navigation [6,7]. Instead of directly using millions of points in the point cloud, 3D modeling is an important step to organize point cloud data [8], and fitting the point cloud to regular shapes (e.g., plane, cylinder, and sphere) is crucial for point cloud classification and segmentation [9,10].

Currently, point cloud datasets are mainly generated by a terrestrial laser scanner (TLS) or an RGB camera. TLS can collect point cloud data with a high accuracy of up to the millimeter-level. However, the high cost and long field surveying time of TLS restrict the broad applications of this technology [11]. Instead, by using an RGB camera, the hardware cost and data collection time can be significantly reduced, but the transformation from color images to a 3D point cloud requires a substantial computation cost and extra scale control. In addition, the output point cloud generally has low precision and reliability [12].

The RGB-D sensor is another choice for point cloud collection, which provides both color images and dense depth maps [13]. According to the principle of range measurements, RGB-D sensors can be divided into two categories: time-of-flight (ToF)-based and structured light (SL)-based sensors [14]. To date, there are numerous RGB-D sensors on the market such as the Xtion Pro from Asus (2012), the second-generation Kinect from Microsoft (2013), Structure Sensor (2014), and the RealSense Series from Intel (2015). Although they were first introduced for the game industry where the accuracy of the measurements is not crucial, RGB-D sensors have been applied for many high accuracy applications recently, for example, indoor 3D modeling [15], simultaneous localization and mapping (SLAM) [16], and augmented reality applications [17], etc., in which the rigorous calibration and error modeling of RGB-D sensor data become increasingly essential [18].

For the 3D modeling of point cloud data, the extraction of regular shapes (i.e., plane, sphere, and cylinder) from point coordinates is an important step for the usage of the point cloud. For example, many studies have tried to develop plane fitting methods for depth calibration [19,20], object segmentation [9,10], and 3D environment reconstruction [21,22,23]. In general, the plane estimation algorithms of the point cloud can be divided into three groups. The first group is based on the classical least-squares (LS) [24,25], which estimates the plane parameters by minimizing the squared offset between the plane and the candidate points. This method performs well for a dataset with few outliers, for example, those collected by the high-performance TLS. However, the performance of this type of method can be significantly degraded if the data quality is low and has a large portion of outliers, or the occlusion region is considerably large [20]. The second group is based on feature space transformations such as principal component analysis (PCA) [26,27] and Hough transform (HT) [14,28]. Nevertheless, PCA is also sensitive to the outliers in a similar way to LS, while HT requires extensive computation and rigorous accumulator design. Therefore, these methods are also difficult to apply to low-quality point cloud data.

The third group is the RANSAC-based fitting methods [29,30], which can provide robust fitting with noisy datasets. This method is a minimization algorithm, as with LS, but the iterations with candidate plane generation by randomly picking a few points dramatically overcome the noise-sensitive problem. Furthermore, M-estimator sample consensus (MLESAC) [31] and progressive sample consensus (PROSAC) [32] have been developed based on the maximum likelihood estimation, which improves the robustness of the RANSAC methods. These plane fitting methods based on RANSAC are useful and practical for high-quality Lidar data. However, there are still some shortcomings in handling datasets collected by a low-cost RGB-D sensor, since the noises of the measurements are significantly larger and distance-related.

Recently, many works have been undertaken to address the plane fitting issue of low-quality point clouds collected by a low-cost RGB-D sensor. Holz et al. [33] proposed smoothing the points with multilateral filtering and model the camera noise with a distance-related empirical model. The experimental results demonstrated that this method achieved excellent performance in the extraction of planes and the image segmentation with a dataset collected by Kinect. However, this method is impractical and ineffective because it requires the generation of an individual model for the particular hardware and applied environment. Fuersattel et al. [34] integrated the fitting of regular shapes with sensor calibration by applying a precise plane fitting method based on the least-squares and improved the accuracy significantly with angle errors of less than a degree and distance errors less than 2 cm. Additionally, their research indicated that the error of the SL-based RGB-D sensor was distance-related, and that the performance of the plane fitting algorithm will decrease as the operating range increases. However, they only tested an algorithm with the dataset in a very close operation range (less than 1.7 meters), where there is relatively little noise. The considerable error in the extended operating range and the vast depth measurement scale in the same frame will be challenging for this method.

In this paper, an improved plane-fitting algorithm, based on the standard RANSAC framework, was proposed to address this issue. First, we derived a rigorous error model for the SL-based RGB-D sensor based on its working principle and error propagation law, from which the weighted coordinates of each point were established. Then, we proposed using the radial distance—that is, the distance in the ray direction—to calculate the residuals for plane estimation by adopting the error characteristics of the SL depth sensors. Next, we modified the cost function of the standard RANSAC with residuals along the radial direction and the weighting information. The experiments and discussions are given in Section 4, while the conclusions are presented in Section 5.

2. Error Distribution of SL Depth Sensors

In this section, we present the working principle of an SL-based RGB-D sensor and discuss the factors that contribute to the error of depth measurements.

An SL-based depth sensor consists of an infrared projector and an infrared camera. The depth for each pixel in the scene is computed from the disparity value, which is the difference between the predesigned pattern emitted from the infrared projector and the actual pattern recorded by the infrared camera. The principle of the SL-based depth sensor is illustrated in Figure 1.

Assume that a pattern $P\left(x_i^{\left(p\right)}\right)$ is projected in the direction of point $x_i^{\left(p\right)}$ onto the reference plane ${\mathsf{\Pi }}_0$ with a distance $Z_0$, and reflected to the infrared camera at the 2D position $x_{0,i}^{\left(c\right)}$; we can obtain

$$x_{0,i}^{\left(c\right)}= x_i^{\left(p\right)}+f·\frac{w}{Z_0} ,$$

(1)

where $w$ is the distance between the camera and the projector; $f$ indicates the focal length of the infrared camera; and $Z_0$ represents the distance between the plane ${\mathsf{\Pi }}_0$ and the sensor.

If the real reflected target is at the point $Q_i$, and the pattern $P\left(x_i^{\left(p\right)}\right)$ is recorded at $x_i^{\left(c\right)}$ on the infrared camera image, the disparity $d_i$ is then defined as

$$d_i= x_i^{\left(c\right)}-x_{0,i}^{\left(c\right)}= x_i^{\left(c\right)}-fw/Z_0,$$

(2)

and the coordinates of the point $Q_i$ can be obtained as [18]

$$\{\begin{array}{l}{X}_i= \frac{x_i^{\left(c\right)}{Z}_i}{f}\\ Y_i= \frac{y_i^{\left(c\right)}{Z}_i}{f}\\ Z_i= \frac{fw}{f\frac{w}{Z_0}+d_i}\end{array}.$$

(3)

For most SL-based depth sensors, the output is a normalized disparity $d_i^{\prime }$ with values from 0 to 2047, and the disparity $d_i$ is calculated using

$$d_i=m·d_i^{\prime }+n,$$

(4)

where $m$ and $n$ are two coefficients. Therefore, the distance $Z_i$ can be rewritten as

$$Z_i=F\left(d_i^{\prime }\right)=\frac{fw}{\frac{fw}{Z_0}+m·d_i^{\prime }+n}=\frac{1}{\alpha d_i^{\prime }+\beta },$$

(5)

where $\alpha =\frac{m}{fw}\mathrm{and}\beta =\frac{1}{Z_0} + \frac{n}{fw}$, which can be estimated through the calibration of the sensor [18]. The quality of the depth $Z_i$ depends on the accuracy of the disparity $d_i$. The error function of disparity can be expressed as:

$$d_i=d_t^p+{\epsilon }_s+{\epsilon }_n,$$

(6)

where $d_i$ is the measurement value of disparity; $d_t^p$ denotes the actual value of disparity; and ${\epsilon }_s$ and ${\epsilon }_n$ represent the system error and the measurement noise, respectively. All the values in Equation (6) reference the pixel unit. Inserting Equation (4) into Equation (6), the normalized disparity can be described as

$$d_i^{\prime }=\frac{d_t^p+{\epsilon }_s+{\epsilon }_n-n}{m}=d_i^n+{\epsilon }_s^n+{\epsilon }_n^n,$$

(7)

where $d_i^{\prime }$ is the measurement of normalized disparity; $d_i^n=\frac{d_t^p-n}{m}$ indicates the actual value of normalized disparity; ${\epsilon }_s^n=\frac{{\epsilon }_s}{m}$ denotes the normalized system error; and ${\epsilon }_n^n=\frac{{\epsilon }_n}{m}$ is the normalized measurement noise.

Based on the Brown model [35] and the work in [18,36], the systematic error ${\epsilon }_s^n$ can therefore be modeled as

$${\epsilon }_s^n=\frac{{\epsilon }_s}{m}=P_1\left(r^2+2x_i{}^2\right)+P_2{x}_i{y}_i+x_i\left(K_1{r}^2+K_2{r}^4+K_3{r}^6\right),$$

(8)

where $P_1,P_2$ are the parameters to model the tangential distortion; $K_1,K_2,K_3$ represent the parameters to model the radial distortion; $(x_i,y_i)$ is the coordinate of the image point; and $r^2=x_i^2+y_i^2$.

Then, the Taylor series is used to present Equation (5), with the linear term as

$$Z_i=\frac{1}{\beta +\alpha d_0}+\frac{\alpha }{{\left(\beta +\alpha d_0\right)}^2}\left(d_i^{\prime }-d_0\right),$$

(9)

where $d_0$ is the expansion point. Considering that the measurement noise is in a normal distribution with a variance ${\sigma }_n^2$, the variance ${\sigma }_z^2$ of $Z_i$ can be calculated based on the error propagation law:

$${\sigma }_z^2=\frac{{\alpha }^2}{{\left(\beta +\alpha d_0\right)}^4}{\sigma }_n^2 .$$

(10)

By combining Equation (3) and Equation (10), the variances of measurement point coordinates ($X_i,Y_i,Z_i$) are

$$\{\begin{array}{l}{\sigma }_x^2={\left(\frac{x_i^{\left(c\right)}}{f}\right)}^2{\sigma }_z^2 \\ {\sigma }_y^2={\left(\frac{y_i^{\left(c\right)}}{f}\right)}^2{\sigma }_z^2 \\ {\sigma }_z^2= \frac{{\alpha }^2}{{\left(\beta +\alpha d_0\right)}^4} {\sigma }_n^2\end{array}.$$

(11)

3. Modified RANSAC Algorithm

In this section, we present the algorithm used in this research and give a detailed description of the proposed method.

As Figure 2 shows, the workflows of the RANSAC plane fitting algorithm can be summarized as follows:

• Initialize the program with input point cloud $P$ and iteration limit $K_0$;
• Select $n$ points randomly from point cloud $P$ to generate the candidate plane model $M_i$;
• Calculate the mean offset $D_i$ between each point in point cloud $P$ and the candidate plane model $M_i$ based on a cost function;
• Update the best fitting plane model $M_i$ and the corresponding mean offset $D$;
• Repeat steps 1–4 until the iteration number $i$ is larger than the iteration limit $K_i$ or the value of $D$ is smaller than the threshold $D_T$ (in this paper, $D_T=0.05m$).

The value of the iteration limit $K_0$ is calculated as [37]

$$K_0=\frac{\log \left(1-p\right)}{\log \left(1-{\omega }^n\right)}$$

(12)

where $p$ is the desired probability for the algorithm to provide an acceptable result; $\omega$ denotes the probability to choose an inlier each time a point is selected; and $n$ is the number of points to generate the candidate plane. Considering the quality of the dataset [37], here, we use $p=0.99,n=3$, and $\omega =0.7$.

Using seriously corrupted data, the methods based on RANSAC provide not only estimation results in a shorter time, but also with higher reliability, compared to the methods based on LS and feature space transformations [38]. However, there are still some problems with handling the dataset from low-cost RGB-D sensors. First, as Figure 3a shows, in the standard RANSAC-based methods, the perpendicular distance is commonly used to evaluate the accuracy of the plane fitting result, which assumes the error distribution of every point in the point cloud is isotropic and comparable [37,38]. This situation is applicable for a high-quality TLS dataset, because the error is ignorable, especially for plane fitting. Nevertheless, this would be problematic for low-cost RGB-D sensors, since the datasets are much noisier, and the error distribution is anisotropic and heterogeneous. Alternatively, the distance along one given coordinate axis such as the normal direction distance shown in Figure 3b, has been widely used in LS-based methods [39]. However, nearly all algorithms based on this criterion strongly depend on the coordinate system. Second, standard RANSAC ignores the fact that the contributions of points with different quality should be different for the candidate plane evaluation. Moreover, this fact happens to be the critical point for the plane estimation of a low-quality dataset, considering that the noisy points frequently deflect the candidate plane from the correct position.

As shown in Equation (3), the $x$ and $y$ coordinates of each measured point are calculated based on the measured depth and the image coordinate $\left(x_i^{\left(c\right)},y_i^{\left(c\right)}\right)$. This means that if $\left(x_i^{\left(c\right)},y_i^{\left(c\right)}\right)$ is fixed, for any depth measurement, the 3D point corresponding to $\left(x_i^{\left(c\right)},y_i^{\left(c\right)}\right)$ will always be located on the radial direction line that starts from the camera origin and passes the image point $\left(x_i^{\left(c\right)},y_i^{\left(c\right)}\right)$. Therefore, in the 3D space, the measurement error of the SL-based RGB-D sensor is in the radial direction, as illustrated in Figure 4, in which the red point is the measured point, the black point is the object point, and the measurement error is the distance between the measured and the object point.

In this paper, we modified the traditional RANSAC algorithm by replacing the perpendicular distance with the radial distance. The candidate plane model in the 3D space is expressed as $ax +by +cz +d =0$, where $\left(a,b,c\right)$ is the normal unit vector of the plane, with the condition as $a^2+b^2+c^2=1$. $d$ is the perpendicular distance between the coordinate system origin and the plane. In this way, the problem of plane fitting could be formulated as the estimation of $a,b,c,$ and $d$. As Figure 5 shows, the radial direction offset between one arbitrary point $P_{}{}_m\left(x_m,y_m,z_m\right)$ and the detected candidate plane can be written as

$$Di{s}_i=\Vert \overrightarrow{P_m{P}_0}+\overrightarrow{P_0{P}_p}\Vert ,$$

(13)

where $P_0$ is the location of the camera center $(x_0,y_0,z_0)$, and $P_p$ illustrates the intersection point between the radial line and the detected plane. If we set $P_0$ to the original point of the coordinate system, $\overrightarrow{P_m{P}_0}\mathrm{and}\overrightarrow{P_0{P}_p}$ can be calculated as

$$\{\begin{array}{l}\overrightarrow{P_m{P}_0}=Z_i\times \left[\begin{array}{ccc}\frac{x_i^{\left(c\right)}}{f_x}& \frac{y_i^{\left(c\right)}}{f_y}& 1\end{array}\right]\\ \overrightarrow{P_0{P}_p}= \frac{d}{a\frac{x_i^{\left(c\right)}}{f_x}+b\frac{y_i^{\left(c\right)}}{f_y}+c}\times \left[\begin{array}{ccc}\frac{x_i^{\left(c\right)}}{f_x}& \frac{y_i^{\left(c\right)}}{f_y}& 1\end{array}\right]\end{array} ,$$

(14)

where $\left(f_y,f_y\right)$ is the focal length of the Infrared (IR) camera and $\left(x_i^{\left(c\right)},y_i^{\left(c\right)}\right)$ is the pixel local of the measurement point. By integrating Equations (10), (13), and (14), we can obtain the variance ${\sigma }_{dis}^2$ of $Dis$:

$${\sigma }_{dis}^2= ({\left(\frac{x_i^{\left(c\right)}}{f_x}\right)}^2+{\left(\frac{y_i^{\left(c\right)}}{f_y}\right)}^2+1)\frac{{\alpha }^2}{{\left(\beta +\alpha d_0\right)}^4}{\sigma }_n^2.$$

(15)

We then set up the cost function based on the points weighted by Equation (15). For each iteration, the weighted mean offset $D_i$ in the radial direction is expressed as

$$D_i=mean({\sum }_i\frac{1}{{\sigma }_{dis}^i{}^2}\times \Vert \overrightarrow{P_m{P}_0}+\overrightarrow{P_0{P}_p}\Vert ),$$

(16)

where $i$ is the index of the point; ${\sigma }_{dis}^i{}^2$ denotes the variance of $Di{s}_i$; $P_m^i$ is the location of point $i$; and $P_p^i$ indicates the intersection point of the detected plane and the radial line.

Similarly, the cost function using an offset in the perpendicular direction can be written as

$$D_i^p=mean({\sum }_i\Vert \overrightarrow{P_{pp}{P}_0}+\overrightarrow{P_0{P}_m}\Vert ),$$

(17)

where $P_{pp}^i$ is the intersection point of the detected plane and its perpendicular line through point $P_m^i$. The cost function using an offset in the direction of the camera plane normal is

$$D_i^n=mean({\sum }_i\Vert \overrightarrow{P_{pn}{P}_0}+\overrightarrow{P_0{P}_m}\Vert ),$$

(18)

where $P_{pn}^i$ is the intersection point of the detected plane and the line through point $P_m^i$, which is parallel to the normal direction of the camera plane. The cost function using an offset without weight information in the radial direction is

$$D_i^r=mean({\sum }_i\Vert \overrightarrow{P_m{P}_0}+\overrightarrow{P_0{P}_p}\Vert ).$$

(19)

The output plane parameters are then determined by minimizing the value of $D_i$ for all iterations, or under the case that the value of $D$ is smaller than the threshold $D_T$.

4. Experiments and Results

In this section, we present the design of the experiments and discuss the results. Three series of experiments have been done to evaluate the performance of the proposed method.

To quantify the accuracy of the planes fitted with different cost functions, the angle error $\left(\mathsf{\Delta }n\right)$, the difference of $d_r\left(\mathsf{\Delta }{d}_r\right)$, and the mean offset of control points $\left(D_c^m\right)$ compared to the “true plane” are used as metrics for comparison:

$$\{\begin{array}{l}\mathsf{\Delta }n={\cos }^{-1}\frac{v_{Gt}\cdot v_M}{\left|v_{Gt}\right|\left|v_M\right|}\\ \mathsf{\Delta }{d}_r=d_r^{Gt}-d_r^M\\ D_c^m=\frac{1}{n}{\displaystyle \sum _i}\left|P_i^{Gt}-P_i^M\right|\end{array}$$

(20)

where $v_{Gt}$ is the normal unit vector of the ground truth plane that equals $\left(a^{gt},b^{gt},c^{gt}\right)$; $v_M$ is the normal unit vector of the estimated plane that equals $\left(a^m,b^m,c^m\right)$; $d_r^{Gt}$ denotes the perpendicular distance between the origin of the camera coordinate system and the ground truth plane; $d_r^M$ represents the perpendicular distance between the origin of the camera coordinate system and the estimated plane; $n$ indicates the number of control points; $P_i^{Gt}$ illustrates the “true” position of control points; and $P_i^M$ is the position of points in the estimated plane corresponding to the control points.

4.1. Experiment for Different Operating Ranges

The first experiment was designed to test whether the proposed method is practical for different operating ranges. As shown in Figure 6, the RGB-D sensor used in this experiment was the Structural Sensor [40], which can be fitted to an iPad, an iPhone, or other mobile instruments. First, the distortion error of the IR camera was calibrated using the method proposed in [36].

Table 1. Parameters for system error calibration.

P1	P2	K1	K2	K3
1.182 × 10−7	−6.864 × 10−13	4.217 × 10−5	1.410 × 10−5	−4.616 × 10−11

gives the calibration parameters for both radial and tangential distortions.

The effect of the calibration is demonstrated in Figure 7, where the projected point clouds of a plane wall for both uncalibrated and calibrated systems are plotted. The improvement in the calibrated depth can be clearly seen as the wall should be a straight line after projection to the floor. Additionally, the quantitative analysis of the depth calibration in the different operating ranges and different areas of the frame is shown in Figure 8. The illustration of the center area and edge area is shown in Figure 9. The accuracy of the depth measurement improved significantly with the calibration operation, especially for the far range and edge area.

The test field and the equipment setup for the data collection are shown in Figure 10. The identifiable control points with known coordinates were distributed on the wall plane with the ground truth obtained by the total station. In our test, the dataset was captured at three different ranges—close range (1.23 m), middle range (2.47 m), and far range (4.31 m)—while the camera was arranged substantially parallel to the wall plane. Then, different cost functions, as discussed in Section 3, were used to fit the wall plane. The “true plane” parameters were obtained from the surveyed coordinates of the control points.

For each operating range, we captured 20 datasets for evaluation.

Table 2. Mean value of metrics for the different operating ranges. LS: least-squares.

Ranges	Distance Type	$\mathsf{\Delta }\mathit{n}\left(\mathit{deg}\right)$	Δdr (cm)	${\mathit{D}}_{\mathit{c}}^{\mathit{m}}\left(\mathit{cm}\right)$
Close (1.23 m)	LS	1.5 ± 0.01	1.9 ± 0.02	2.1 ± 0.02
	Perpendicular	2.3 ± 1.35	0.9 ± 0.65	1.6 ± 0.82
	Normal	2.2 ± 1.57	0.8 ± 0.67	1.5 ± 0.91
	Radial	1.8 ± 0.99	0.7 ± 0.54	1.3 ± 0.62
	Radial Weighted	0.9± 0.05	0.6± 0.16	0.8± 0.04
Middle (2.47 m)	LS	2.9 ± 0.05	2.6 ± 0.13	2.9 ± 0.14
	Perpendicular	2.7 ± 0.95	1.6 ± 1.14	3.7 ± 1.23
	Normal	2.6 ± 1.02	1.8 ± 1.09	3.6 ± 1.13
	Radial	2.4 ± 0.96	1.3 ± 1.07	3.2 ± 1.04
	Radial Weighted	1.1± 0.20	1.2± 0.04	2.7± 0.24
Far (4.31 m)	LS	3.7 ± 0.21	13.3 ± 0.66	14.3 ± 0.70
	Perpendicular	3.6 ± 1.69	8.9 ± 3.18	10.9 ± 4.09
	Normal	2.5 ± 1.39	7.4 ± 2.82	9.6 ± 2.93
	Radial	2.6 ± 1.35	6.5 ± 2.44	8.6 ± 2.62
	Radial Weighted	0.9± 0.46	5.8± 0.64	6.6± 0.69

shows the mean values of the metrics and the standard deviation for the different cost functions and operating ranges. The performances of the perpendicular-based and normal-based method were similar because their values are approximate when the camera plane is parallel to the observed wall plane. For the close range and middle range, when the error of a point in the edge area is not very large, the radial-based method is practical to refine the plane fitting result in terms of both angle and distance, but it is not helpful for the far range because this type of error is distance-related. However, our proposed method, using the radial offset and the weighted cost function, demonstrated higher robustness and is practicable for all three test datasets. In particular, this method reduced the angle error by about one degree. The reason for this is that the weighted cost function mitigates the influence of noisy points. Considering that the noise and error of the depth measurement increases with the operating range, this experiment also demonstrated that the proposed method is effective and practical for datasets with different levels of noise and error.

4.2. Experiment for Large Depth Measurement Scales

The second experiment was designed to test the performance of the proposed method with the dataset in a large depth measurement scale. As above-mentioned, the error of the SL-based RGB-D sensor will increase exponentially with the ranging distance. The points close to the equipment and in the center area of the depth image always exhibit small errors, while the points far from the equipment and in the edge area of the depth image always display a significant error (Figure 11). This issue is challenging for the unweighted method because the error distribution of the dataset is asymmetrical.

As Figure 11 shows, the basic setup of this experiment was like the first experiment. Notably, a flat wall with a height of 2.4 m and a width of 6.4 m was used as the test area, while the coordinates of the control points were surveyed using a total station with an accuracy at the millimeter-level. Then, different cost functions, as discussed in Section 3, were used to fit the wall plane. There were 62 control points on the wall (Figure 12a), and the depth measurements of the control points were in the range of 3.5 m to 7.5 m (Figure 12b).

The conventional plane fitting method minimizing the sum of perpendicular distance was used as the baseline during the comparison. For each cost function, 20 datasets were collected, and the mean values of metrics as well as the standard deviation are given in

Table 3. Mean value of metrics for the second experiment.

	$\mathsf{\Delta }\mathit{n}\left(\mathit{deg}\right)$	Δdr (cm)	${\mathit{D}}_{\mathit{c}}^{\mathit{m}}\left(\mathit{cm}\right)$
LS	3.3 ± 0.5	20.3 ± 0.7	27.4 ± 0.3
Perpendicular	3.7 ± 1.1	16.8 ± 3.8	25.7 ± 7.8
Normal	2.6 ± 1.0	12.4 ± 3.1	20.8 ± 5.6
Radial	2.2 ± 0.7	13.8 ± 1.6	23.0 ± 2.4
Radial Weighted	0.5± 0.2	4.8± 0.7	7.2± 2.2

. We observed that the cost function with weighted radial distance reduced the plane fitting errors dramatically to 0.5 degrees of angle error and 4.7 cm of range error, compared to 2 to 4 degrees of angle error and 12 to 16 cm of range error when the unweighted cost functions were used.

Figure 13 shows the 3D views of the fitted planes with different cost functions. It is clear that the plane, by minimizing the weighted radial residual (red area) proposed in this study, was one closer to the true plane (green area).

The errors of the control points after plane fitting calibration based on different methods are illustrated in Figure 14. Compared with Figure 6b, which gives the range of control points, the errors for the raw data and unweighted cost functions are range-related. The longer the operating range, the larger the error. With the help of the weighted plane fitting method, the range errors were controlled to an accuracy of better than 0.1 m for all of the distance ranges (3.5 m to 7.5 m).

4.3. Experiment with Kinect V2

The third experiment was designed to evaluate the applicability of the proposed method for different types of SL-based RGB-D sensor. In this experiment, we used the Kinect v2 to collect the sample data under a similar test condition to the second experiment. The test results are shown in

Table 4. Mean values of the metrics for the third experiment.

	$\mathsf{\Delta }\mathit{n}\left(\mathbf{deg}\right)$	Δdr (cm)	${\mathit{D}}_{\mathit{c}}^{\mathit{m}}\left(\mathbf{cm}\right)$
LS	4.8 ± 0.2	22.2 ± 0.3	29.6 ± 0.4
Perpendicular	2.7 ± 0.7	15.5 ± 2.2	25.4 ± 3.7
Normal	1.7 ± 1.1	12.1 ± 2.9	20.2 ± 3.0
Radial	1.5 ± 0.7	10.8 ± 2.0	15.6 ± 3.5
Radial Weighted	0.9 ± 0.4	7.1 ± 0.7	9.6 ± 0.9

. When compared to the unweighted perpendicular-based method, the proposed method improved the accuracy of the plane fitting significantly, with the angle error improving from 2.7 degrees to 0.9 degrees and the distance error from 15.5 cm to 7.1 cm. This experiment indicates that the proposed method is practical for different types of SL-based RGB-D sensors.

5. Conclusions

Fitting point cloud data to regular shapes (i.e., plane, sphere, and cylinder) is a fundamental step for point cloud classification, segmentation, and calibration. For low-cost RGB-D sensors, since the depth measurements are range-related, any unweighted fitting methods will introduce significant errors in the fitted shapes.

In this paper, we derived a rigorous error model for a point cloud dataset generated by a low-cost SL-based RGB-D sensor. Based on this error model, we proposed a novel RANSAC-based plane fitting method by replacing the cost function of the perpendicular offset to the weighted radial offset. The experimental results showed that our proposed method is robust and practical for different operating ranges, different working conditions, and different types of SL-based RGB-D sensors. For operating ranges between 1.23 meters and 4.31 meters, the proposed method can obtain a plane fitting result with an angle error of about one degree and a distance error less than 6 cm. Meanwhile, the proposed method improved the angle accuracy from 3.8 degrees to 0.5 degrees and the distance error from 16.8 centimeters to 4.7 centimeters, compared to the unweighted perpendicular offset cost function, with a dataset in a large depth measurement scale.

Further research is still required to apply the proposed error model to the fitting algorithm of more complex shapes (e.g., sphere, cylinder, and curved surfaces). We will also use this point weighting method to identify the confidence of the point matching results of the RGB-D-based SLAM system to improve the robustness and accuracy of the SLAM system. These shape data and precise 3D point clouds will be used to establish an automatic indoor BIM generation system.