A New RGB-D SLAM Method with Moving Object Detection for Dynamic Indoor Scenes

Abstract

Simultaneous localization and mapping (SLAM) methods based on an RGB-D camera have been studied and used in robot navigation and perception. So far, most such SLAM methods have been applied to a static environment. However, these methods are incapable of avoiding the drift errors caused by moving objects such as pedestrians, which limits their practical performance in real-world applications. In this paper, a new RGB-D SLAM with moving object detection for dynamic indoor scenes is proposed. The proposed detection method for moving objects is based on mathematical models and geometric constraints, and it can be incorporated into the SLAM process as a data filtering process. In order to verify the proposed method, we conducted sufficient experiments on the public TUM RGB-D dataset and a sequence image dataset from our Kinect V1 camera; both were acquired in common dynamic indoor scenes. The detailed experimental results of our improved RGB-D SLAM were summarized and demonstrate its effectiveness in dynamic indoor scenes.

1. Introduction

Simultaneous localization and mapping (SLAM) technology has become a fundamental prerequisite in many robotic applications [1,2]. Because of its ability to conduct navigation and perception simultaneously in an unknown environment, SLAM has attracted the attention of many scholars, and has gradually become a research hotspot over the past decades [3]. Current SLAM technology mainly includes two types of systems, distinguished by different data acquisition devices. The first type is based on light detection and ranging (LiDAR) sensors [4]. LiDAR-based SLAM is expensive, and mainly used in driverless cars. The second type, visual SLAM [5] with imaging sensor, has the advantages of low price and small volume; it can provide more abundant texture information for motion estimation. Owing to the ability of autonomous localization and quick environmental perception, visual SLAM plays an important role in global positioning system (GPS)-denied environments, for instance, indoor scenes [6,7].

Depending on the number of sensors, visual SLAM can be classified into monocular-camera- [8,9,10,11], stereo-camera- [12,13,14], and multiple-camera- [15,16] based versions. Monocular camera based visual SLAM cannot obtain scale information of the environment, so it is usually combined with inertial measurement units (IMUs) to obtain three dimensional (3D) information in the scene [17]. As for stereo-camera- and multiple-camera-based SLAM systems, they can obtain 3D coordinates of feature points by photogrammetry. However, extracting feature points and stereo matching takes a lot of time. Subsequently, many scholars have tried to improve the efficiency of SLAM systems by using new algorithms for feature extraction and matching, such as Speeded-Up Robust Features (SURF) [18] and oriented FAST and rotated BRIEF (ORB) [19].

As a new type of depth sensor, an RGB-D camera, such as Microsoft Kinect, has the advantages of directly obtaining 3D coordinates like a LiDAR sensor, and providing texture information as an imaging sensor. Due to its low power consumption and small volume, RGB-D cameras are widely used as data acquisition devices on current robots. Meanwhile, effective RGB-D-based SLAM algorithms have been proposed [20,21]. Some of these algorithms are based on sparse features, such as ORB-SLAM2 [22], which is a representative RGB-D SLAM system. These SLAM systems use the correspondence between current and previous frames to estimate cameras’ ego-motion iteratively. These correspondences can be established by matching feature points, lines, and even planes. Dense-point-cloud-matching-based algorithms have also been proposed, such as the method proposed in literature [23]. These methods are based on the assumption that the grayscale values of corresponding areas are invariant, and adopt an energy minimization function for iterative calculation of the camera’s six degree of freedom (DoFs) motion.

However, most SLAM methods are based on the assumption that the environment is static. In other words, the geometric distribution of objects in the scene is assumed to be stationary in the process. These methods are unable to distinguish the features in static and dynamic objects, which leads to the deterioration of SLAM systems because of erroneous data association and faults in motion estimation. Therefore, the question of how to build a robust SLAM system in a dynamic environment has attracted much attention from scholars in recent years.

The first remarkable SLAM system to combine detection and tracking of moving objects method was proposed by Wang et al. [24]. They used a laser rangefinder to obtain the data and verify their algorithm in urban outdoor environments. After that, one of the basic ideas of SLAM in a dynamic environment is to identify the moving objects and then eliminate their associated data. By means of dense scene flow representation, the visual SLAM system proposed by Alcantarilla et al. [25] can detect moving objects within a distance of about five meters. However, it has distance limitation in moving object detection, and misjudges static points as dynamic ones in some non-artificial and poor texture scenes. The dense RGB-D SLAM proposed by Wang et al. [26] combines moving object segmentation with dense SLAM. It merges regions of which the predominant parts share the same fundamental matrices across the whole SLAM process into a group. Based on the assumption that the largest group corresponds to the static environment, it then removes the largest group and gets the areas of moving objects. Another dense 3D SLAM, proposed by Bakkay et al. [27], assumes a constant motion during a fixed time interval. Those pixels, of which the absolute velocities are higher than the preset threshold, are taken as seed segmentation of moving objects. Starting from these seeds, they use a region growing algorithm to identify dynamic areas. The motion removal approach proposed by Sun et al. [28] roughly determines the moving areas by subtracting the corresponding pixels of the continuous frames. They then apply the maximum-a-posterior (MAP) to precisely moving object determination. This approach is integrated into the front end of dense visual odometry (DVO), and tested in TUM RGB-D datasets [29]. However, only one moving cluster can be segmented, and the performance is not as good as original DVO in a lowly dynamic environment. Recently, with the rapid development of deep learning algorithms, there are some SLAM systems combined with deep learning methods to judge moving objects. Yang et al. [30] proposed an RGB-D SLAM integrated with the Faster R-CNN algorithm [31]. It detects the rectangular areas of humans through Faster R-CNN and removes the feature points in these rectangular areas and the corresponding data, so as to obtain robust results. Zhong et al. proposed a Detect-SLAM [32], which is based on ORB-SLAM and a deep detector to determine and remove dynamic objects. The DRE-SLAM proposed by Yang et al. [33] combines the YOLOv3 algorithm [34] with the multi-view constraint to identify dynamic pixels on moving objects.

Another idea of SLAM in a dynamic environment is to incorporate new, robust data processing mathematical methods into the process of the SLAM system, and then reduce the impact of moving objects. Moratuwage et al. used the random finite set (RFS) model to express features and measurement results in the current sensor field of view (FOV) [35]. They then used a Bayes recursion to estimate the static feature map and obtain the dynamic feature locations. The CoSLAM proposed by Zou et al. [36] studies the problem of visual SLAM in dynamic environments with multiple cameras. This SLAM system uses the re-projection error to distinguish static and dynamic feature points. It works robustly in both highly dynamic environments and static environments, however, the data is processed offline and it has a large drift error in dynamic experiments. The DVO algorithm, proposed by Kerl et al. [37], estimates the camera’s ego-motion by considering the probabilistic model in the error function, which reduces the influence of large residuals and thus obtains robust results. Lee et al. [38] used a pose graph to represent robot poses, and false constraints of the pose graph that were associated with moving objects were pruned according to an error metric based on the grouped nodes. The correct results were then obtained by re-optimizing the new pose graph. In addition to detecting moving objects, there are methods that attempt to find static backgrounds or features to estimate the camera’s ego-motion. The background-model-based dense-visual-odometry (BaMVO) algorithm proposed by Kim et al. [39] estimates the background model from depth scenes and the ego-motion of the camera through eliminating the influence of moving objects. The RGB-D SLAM proposed by Li et al. [40] uses foreground edge points to estimate camera’s ego-motion, and static weights, which indicate the probability that an edge point is static, are combined with intensity assisted iterative closest point (IAICP) algorithm in order to reduce the effect of dynamic moving objects on the motion estimation. Since this method requires depth edges, it is more suitable for geometry rich environments where depth edges abound than the texture-poor environments.

In this paper, a new RGB-D SLAM method with moving object detection for dynamic indoor scenes is presented. The rest of the paper is organized as follows. Section 2 illustrates the proposed RGB-D SLAM in three parts of the flowchart. In Section 3, abounding experiments are reported to analyze the performance of our RGB-D SLAM in common dynamic indoor scenes. Conclusions and future work are discussed in the last section.

2. Methodology

Figure 1 illustrates the proposed RGB-D SLAM in a flowchart, which consists of the following parts: (1) input data and pre-processing; (2) moving object detection and elimination; (3) camera pose estimation. Detailed processes of the algorithms and models are given in the following sub-sections.

2.1. Input Data and Preprocessing

In this part, we mainly introduce the image data used in our RGB-D SLAM and the pre-processing steps. In order to test the proposed method, we recorded RGB-D sequence images at $640\times 480$ resolution with a Microsoft Kinect V1 camera, shown in Figure 2a. The camera was calibrated first, in order to use these sequence images in later high-accuracy processing. Although the RGB-D camera was adjusted in production, it did not satisfy the requirements of the experiment. A flat calibration board, as shown in Figure 2b, was used to calibrate the RGB camera and depth camera. We captured infrared images, RGB images, and depth images of the flat calibration board at different distances and directions. We then used a MATLAB-based calibration toolbox [41] to calculate the calibration parameters of the RGB camera and depth camera. The internal parameters and lens distortion coefficients are shown in

Table 1. Internal parameters and distortion coefficients of the RGB camera and depth camera.

Camera	${\mathit{f}}_{\mathit{x}}$	${\mathit{f}}_{\mathit{y}}$	${\mathit{c}}_{\mathit{x}}$	${\mathit{c}}_{\mathit{y}}$	${\mathit{k}}_1$	${\mathit{k}}_2$	${\mathit{k}}_3$	${\mathit{p}}_1$	${\mathit{p}}_2$
RGB	584.35	584.33	317.97	252.80	−0.10585	0.27096	0.0	0.00504	−0.00166
Depth	519.95	519.55	315.82	238.71	0.04810	−0.19281	0.0	0.00458	−0.00014

¹ The unit of $f_x$, $f_y$, $c_x$, and $c_y$ is the pixel.

, and the external parameters are shown in

Table 2. External parameters of the RGB camera and depth camera.

Rotation Angles (°)	−0.00079	−0.00084	−0.00541
Translation Vector (mm)	−25.59983	0.16700	−0.40571

¹ The unit of rotation angles is the degree and the unit of translation vector is the mm.

. Using these parameters, we can map the depth image to the RGB image.

In order to calculate the camera pose and detect moving objects, we then needed to input the current RGB image, depth image, and previous RGB image. These two RGB images, the current frame and the previous frame, were used for the extraction and matching of point features. In addition to the RGB images, we also needed to use the current depth image. However, due to the influence of optical conditions and imaging modes, some areas of invalid holes existed in the depth images.

Figure 3a shows an original depth image of TUM RGB-D datasets. Owing to the reflection of the computer screen and the optical conditions of some areas, there are invalid holes in Figure 3a. Since the existence of these holes would weaken the effect of next clustering step, we needed to fill them in the pre-processing. Here, we used an algorithm based on morphological reconstruction [42] to fill these holes. As can be seen from Figure 3b, several obvious holes were filled in the depth image. The purpose of this is to better cluster in the subsequent steps, because these invalid holes are sometimes marked as a separate category by the clustering process and may mislead the judgement of moving objects.

2.2. Moving Object Detection and Elimination

In this sub-section, we introduce the principle and pipeline of our moving object detection method using example images taken from the TUM RGB-D dataset. Our RGB-D SLAM takes the proposed moving object detection method as a data processing front-end. This approach is based on two assumptions. The first one is that enough feature points, i.e., more than twenty pairs of matching points, exist in the static environment. These feature points are used to calculate the camera’s pose. Common indoor scenes have many artificial objects, which can basically meet this assumption. The other one is that the parallax between two consecutive frames of the static environment captured by the RGB-D camera is small. The parallax here refers to the difference between the image coordinates of the same feature point on different frames. The reason for this assumption is that we can better find out the inconsistency of feature points caused by the motion of the moving objects in this situation, and then use our method to detect and eliminate the outliers. Large parallax between two consecutive frames will result in tracking loss. Commonly used RGB-D cameras, such as the Microsoft Kinect V1, can provide a high rate of 30 frames per second (FPS), which ensures that the parallax between two consecutive frames is sufficiently small (i.e., less than ten pixels). Therefore, these two assumptions can be satisfied when tested in an indoor environment using a common RGB-D sensor.

The first part of our moving object detection method is shown in Figure 4. We used previous and current RGB images for the extraction and matching of point features. The corresponding feature points are connected with yellow lines. As can be seen from the picture (Figure 4a), dense feature points were extracted from the plaid shirt of the right experimenter, and some points were extracted from the torso of the left experimenter wearing a black T-shirt. Feature points on moving objects will cause errors in the later estimation of camera pose, so we need to eliminate them. Before judging the moving areas, we need to cluster the depth image. In Figure 4b, the k-means clustering algorithm was used to cluster the filled current depth image. The number of clusters was set to ten in this example. We can see that the two experimenters in the clustered filled depth image are almost divided into a single category, and distinguished from the static environment.

Figure 5a shows the position of feature points on the current RGB images. From Figure 5a, we can see that feature points are spread among moving objects and static environments. We then mapped the positions of these feature points onto the clustered depth map, as Figure 5b shows. We can now count the distribution of feature points in the c-th cluster areas. Note that we have not yet excluded outliers in this step, so we use a statistic of the number and percentage of all feature points, $N_{ini}^c$ and $P_{ini}^c$.

We then used the fundamental matrix constraint [43] to identify inconsistency of feature points caused by the motion of the moving objects, and then eliminate the mismatched points. The fundamental matrix $F_{3\times 3}$ describes the relationship between the point $m_{3\times 1}$ and its corresponding epipolar line $l_m^{\prime }$, which can be expressed as follows:

$$l_m^{\prime }=F_{3\times 3}\cdot m_{3\times 1}.$$

(1)

Here, $m_{3\times 1}$ is a homogeneous coordinate by padding with one in the third row. Since the corresponding point $m_{3\times 1}^{\prime }$ of point $m_{3\times 1}$ is on the epipolar line $l_m^{\prime }$, there must be:

$$m_{3\times 1}^{\prime }{}^{\mathrm{T}}\cdot F_{3\times 3}\cdot m_{3\times 1}=0.$$

(2)

That is, if the point $m$ of the first image is obtained, its corresponding point on the second image will be constrained on the epipolar line $l_m^{\prime }$, as Figure 6 shows.

Eight points are randomly selected to calculate a fundamental matrix $F^i$ in the i-th iteration. We can then compute the distance $d_j^i$ of the j-th point according to $F^i$:

$$d_j^i={(m_j^{\prime }{}^{\mathrm{T}}\cdot F^i\cdot m_j)}^2.$$

(3)

Sampson distance weight is used to divide inliers and outliers to improve the robustness of the algorithm. It can be expressed as follows:

$$w_j^i=\frac{1}{{(F^i\cdot m_j)}_1{}^2+{(F^i\cdot m_j)}_2{}^2+{(F^{i\mathrm{T}}\cdot m_j^{\prime })}_1{}^2+{(F^{i\mathrm{T}}\cdot m_j^{\prime })}_2{}^2}.$$

(4)

We can then get the weighted distance:

$$w{d}_j^i=w_j^i\cdot d_j^i.$$

(5)

We took $w{d}^i$ as the array of all distances in the i-th iteration and computed the median of $w{d}^i$. After reaching the maximum number of iterations, the least median of squares method [44] was used to select the best fundamental matrix with the smallest median. The smallest median was taken as the threshold. If the distance $w{d}_j^i$ of the j-th point was less than the threshold, the j-th point was divided as an inlier, otherwise it was divided as an outlier.

In the third step, we used the fundamental matrix constraint mentioned above to determine the inliers and outliers. We then mapped the positions of these inliers and outliers onto the clustered depth map, as Figure 7a shows. Here, the distribution of inliers in the c-th cluster area can be counted. Note that we excluded outliers in this step, so we employed the number $N_{first}^c$ and percentage $P_{first}^c$ here as the statistical properties of the first fundamental matrix constraint. In this process, the eliminated outliers contain a small number of mismatching points and some points located in moving parts. As is shown in Figure 7a, the number of outliers removed from moving cluster areas is larger than that in the static environment. This is because feature points on a moving object are more inconsistent than points on a static area. Next, for the remaining inliers, the fundamental matrix constraint was used again to further eliminate the outliers. The positions of the inliers and outliers on the clustered depth map obtained this time are shown in Figure 7b, and the distribution of inliers in the c-th cluster area can be counted. Note that we have excluded outliers again, so we call the number $N_{second}^c$ and percentage $P_{second}^c$ here the statistical properties of the second fundamental matrix constraint. As is shown in Figure 7, since the number of feature points removed in the moving cluster areas is more than that in the static areas, the proportion of points in the moving areas will decrease.

The last step of our moving object detection method is to input the statistical characteristics obtained from the previous three steps into the moving objects judgment model designed by us. If a cluster area satisfies our model, it will be judged as a moving objects area. Otherwise it will be judged as a static area. The model consists of three modules, and the first module of our model can be expressed as follows:

$$\mathrm{Module}1:(N_{ini}^c-N_{first}^c)/N_{ini}^c\times 100>T_{reduce}.$$

(6)

$T_{reduce}$ is the threshold that we set to find the cluster areas, where a large number of outliers were removed after the first fundamental matrix constraint. However, if a cluster area has a few feature points at the beginning, it will also meet $T_{reduce}$, even if a small number of outliers are removed. Therefore, we needed to set up the second module, which can be expressed as follows:

$$\mathrm{Module}2:P_{ini}^c>T_{ini}.$$

(7)

This threshold $T_{ini}$ is to reduce the interference of small areas with a few feature points, which is easy to satisfy the first module. Normally, because the number of outliers removed from moving cluster areas is larger than that from the static cluster areas, the proportion of points in the moving cluster areas will decrease and the proportion of points in the static cluster areas will increase. Based on this, we set up the third module:

$$\mathrm{Module}3:(N_{ini}^c-N_{second}^c)>0.$$

(8)

If the c-th cluster area satisfies all modules of our moving objects judgement model, the area will be judged to be a moving object. The judgement result of this example is shown in Figure 8a, and the result is a binary image. We can then use this binary image to filter the remaining feature points. As can be seen from Figure 8b, feature points on the moving persons have been eliminated, and the remaining feature points are basically located in the static environment.

In this part, we have illustrated the steps of our moving object detection method through example data from the TUM RGB-D dataset, and the final result in Figure 8 also shows the effectiveness of our method in detecting moving objects. Our RGB-D SLAM takes the proposed moving object detection method as a data filtering process, and estimates the camera pose more robustly and accurately.

2.3. Camera Pose Estimation

After moving object detection and elimination, most of the remaining feature points were located in the static environment. We took the first frame as an initial keyframe and set its position to be the center of a local 3D coordinate system. The relationship between the 3D feature points obtained by the depth sensor, and their image points by the imaging sensor, can be expressed as follows:

$${\lambda }_i\left[\begin{array}{c}{u}_i\\ v_i\\ 1_i\end{array}\right]=K\cdot (R\cdot P_i+\mathrm{t})=K\cdot {(T\cdot \left[\begin{array}{c}{X}_i\\ Y_i\\ Z_i\\ 1\end{array}\right])}_{(1:3)}.$$

(9)

Here, the $R\in SO(3)$ and $\mathrm{t}\in {ℝ}^3$ represent the orientation and position of the camera, respectively. The matrix $T$, containing camera exterior parameters $R$ and $\mathrm{t}$, transforms the homogeneous world coordinate system $P_i{(X_i,Y_i,Z_i,1)}^{\mathrm{T}}$ to the current camera coordinate system $P_i^{\prime }(X_i^{\prime },Y_i^{\prime },Z_i^{\prime })$. $K$ represents the interior parameters of the camera, which was obtained from the camera calibration. The homogeneous pixel coordinate $p_i{(u_i,v_i,1)}^{\mathrm{T}}$ is the corresponding re-projected point of $P_i{(X_i,Y_i,Z_i,1)}^{\mathrm{T}}$. We then used the efficient perspective-n-point method (EPnP) to estimate the camera pose [45]. Due to the unknown camera pose and noise in the observation, there was a re-projection error between world map points and matched feature points, as Figure 9 shows.

We then used bundle adjustment (BA) to optimize the camera pose by minimizing the re-projection error of all matches. It can be defined as follows:

$$\{R,\mathrm{t}\}=\underset{R,\mathrm{t}}{\arg \min }{\displaystyle \sum _{i=1}^n\left({\Vert p_i-\frac{1}{{\lambda }_i}\cdot K\cdot {(T\cdot P_i)}_{(1:3)}\Vert }^2\right)}.$$

(10)

In order to construct the unconstrained error equation, we used a six-dimensional vector $\xi \in {ℝ}^6$ of Lie algebra [46] to represent the camera pose $R$ and $\mathrm{t}$. Because the above equation is a least squares problem, we needed to calculate the derivative of the re-projection error $e_i$ with respect to the optimization variables:

$$\frac{\partial e_i}{\partial \delta \xi }=\frac{\partial e_i}{\partial p_i^{\prime }}\frac{\partial p_i^{\prime }}{\partial P_i^{\prime }}\frac{\partial P_i^{\prime }}{\partial \delta \xi }=-\left[\begin{array}{cccccc}{f}_x\frac{1}{Z_i^{\prime }}& 0& -f_x\frac{X_i^{\prime }}{Z_i^{\prime 2}}& -f_x\frac{X_i^{\prime }{Y}_i^{\prime }}{Z_i^{\prime 2}}& f_x(1+\frac{X_i^{\prime 2}}{Z_i^{\prime 2}})& -f_x\frac{Y_i^{\prime }}{Z_i^{\prime }}\\ 0& f_y\frac{1}{Z_i^{\prime }}& -f_y\frac{Y_i^{\prime }}{Z_i^{\prime 2}}& -f_y(1+\frac{Y_i^{\prime 2}}{Z_i^{\prime 2}})& f_y\frac{X_i^{\prime }{Y}_i^{\prime }}{Z_i^{\prime 2}}& f_y\frac{X_i^{\prime }}{Z_i^{\prime }}\end{array}\right].$$

(11)

Here, the $f_x$ and $f_y$ are the camera interior parameters. In addition to camera pose, we also needed to optimize the 3D world map point $P_i{(X_i,Y_i,Z_i,)}^{\mathrm{T}}$. The partial derivative of the re-projection error $e_i$ with respect to $P_i{(X_i,Y_i,Z_i,)}^{\mathrm{T}}$ can also be obtained via the chain rule of derivative calculation:

$$\frac{\partial e_i}{\partial P_i}=\frac{\partial e_i}{\partial p_i^{\prime }}\frac{\partial p_i^{\prime }}{\partial P_i^{\prime }}\frac{\partial P_i^{\prime }}{\partial P_i}=-\left[\begin{array}{ccc}{f}_x\frac{1}{Z_i^{\prime }}& 0& -f_x\frac{X_i^{\prime }}{Z_i^{\prime 2}}\\ 0& f_y\frac{1}{Z_i^{\prime }}& -f_y\frac{Y_i^{\prime }}{Z_i^{\prime 2}}\end{array}\right]R.$$

(12)

Through Equation (11) and (12), we linearize the 2D pixel re-projection error $e_i$ with respect to camera pose and 3D world map point. We can then use the Levenberg–Marquardt method implemented in g2o [47] to solve the least squares optimization problem described in Equation (10). Finally, a robust result of camera pose $R$ and $\mathrm{t}$ can be obtained. In the following experimental section, we detail experiments we performed with a testing dataset to verify the effectiveness of our RGB-D SLAM system in common dynamic indoor scenes.

3. Experimental Results

In order to evaluate the effectiveness of our new SLAM method, we conducted a series of experiments using two datasets: sequence images captured by our RGB-D camera, and the public TUM RGB-D datasets with ground-truth. In the first experimental part, we qualitatively evaluated the proposed moving object detection method, the data filtering front-end of our improved RGB-D SLAM, with our homemade sequence images in typical dynamic indoor scenes like corridor and office. In the second part, we tested our improved RGB-D SLAM with the public TUM RGB-D datasets. We conducted experiments to analyze the performance of our improved RGB-D SLAM system with moving object detection method and the one without moving object detection method in dynamic environments. For the purpose of comparison, we also compared our SLAM system with the DVO and the BaMVO, both of which are the state-of-the-art RGB-D SLAM methods that can run robustly in a dynamic indoor environment. All the experiments were conducted on a computer with an Intel Core i7-6820HQ CPU with 2.7 GHz and 16 GB RAM. The details of the experimental results are described below.

3.1. Testing with Sequence Images Captured by Our RGB-D Camera

In order to test the effectiveness of the proposed method, we recorded some RGB-D sequence images at $640\times 480$ resolution with a Microsoft Kinect V1 sensor. All the data was recorded at a frame rate of 30 Hz in common scenarios, including the office and the corridor. The experimenter walked at normal speed as a moving object. The RGB-D camera was calibrated using the method described in Section 2.1. Using calibration parameters, we were able to project the depth image onto the RGB image.

The first scenario is an office environment, where we captured two types of sequence images. Frames selected evenly from the experimental results are shown in Figure 10. In the first sequence images of office, the camera was fixed and the experimenter passed through the view with a direction parallel to the camera plane. The experimenter was about 1.5 m away from the camera and walked at a normal speed. In the second sequence images of office, as shown in Figure 10b, the experimenter walked away from the camera with a direction perpendicular to the camera plane at a normal speed, and the interval of presented images is 10 frames. The three rows from top to bottom are RGB images, depth images, and motion detection results. The black parts in the binary images in the third row are the moving object detected by our method, and the white part is the background. As can be seen from Figure 10, our method can detect and remove the moving objects well in the frames with obvious pedestrian characteristics.

The second scenario was a corridor environment. There are two types of sequence images in this scenario. Figure 11 shows some frames evenly selected from the experimental results. In the first sequence images, the camera was fixed. The experimenter approached the camera and turned around. As shown in Figure 11a, the spacing is 20 frames. As for the second sequence images, the camera was unfixed and swung from side to side. The experimenter just walked away from the camera with a normal speed. In Figure 11b, the spacing of the selected frames is 10 frames. According to our moving object detection method, we can get the corresponding binary images as shown in third row of each sub-figure. It is worth mentioning that some of the background parts were treated as moving objects, like Frame 215 in Figure 11a and Frame 120 in Figure 11b. Nonetheless, it can be seen from the Figure 11 that these misclassified parts are mainly white walls and floor tiles. Few feature points will be extracted in these areas, so, even if they are removed, this situation will have little influence on the subsequent camera pose estimation. Overall, our method can detect moving objects well in most frames. After the moving objects are detected, the feature points belonging to the moving objects are eliminated in the subsequent process.

3.2. Evaluation Using TUM RGB-D SLAM Datasets

In this part, the TUM RGB-D SLAM datasets were used to evaluate the proposed RGB-D SLAM method. The RGB and depth images were recorded at frame rate of 30 Hz and a $640\times 480$ resolution. Ground-truth trajectories obtained from a high-accuracy motion-capture system are provided in the TUM datasets. Evaluation tools are available for trajectory accuracy estimation with the ground-truth data. The TUM datasets have a Dynamic Objects category containing three typical dynamic indoor scenes: desk, sitting, and walking. In the desk sequence, a person sits at a desk in an office. In the sitting sequences, two persons sit at a desk with a little gesture. These two scenarios can be considered lowly dynamic environments. In the walking sequences, two persons walk through an office scene. Walking sequences can be used to evaluate the robustness of the proposed method in highly dynamic scenes with quickly moving objects. As is described on the website of the TUM datasets, the RGB-D camera has four different states of motion in the sitting and walking sequences, which are static, xyz, halfsphere, and rpy. The camera is fixed on a position when it is static state, and moves along three directions at the xyz state. For the halfsphere state, the camera moves on a 1 m diameter hemisphere. As for the rpy state, the camera rotates along the principal axes (row-pitch-yaw) at the same position.

Firstly, we conducted a quantitative evaluation of our new SLAM method on dynamic TUM datasets. The ATE, which represents the difference between the estimated trajectory and the ground-truth, was used as the evaluation metric. In particular, an easy-to-use open-source package evo was employed for the evaluation (github.com/MichaelGrupp/evo). Figure 12 shows the accuracy on two highly dynamic datasets. The blue lines represent the ATE of the RGB-D SLAM without moving object detection, and the lines in orange represent the ATE of our new SLAM method with moving object detection. As we can see in blue lines of Figure 12a,b, when there are pedestrians in the FOV of the camera, the ATE values increase dramatically. However, the ATE values of orange lines at those same moments were greatly reduced, because the influence of pedestrians was eliminated with our new RGB-D SLAM.

Table 3. ATE on the sitting and walking dynamic datasets of the proposed RGB-D SLAM with and without the moving object detection. The numbers in bold indicate that these terms are better than those of another method.

Dynamic TUM RGB-D Datasets	Without Moving Object Detection ATE (m)				With Moving Object Detection ATE (m)
	Max.	Avg.	Min.	RMSE	Max.	Avg.	Min.	RMSE
fr3/sitting_static	0.0381	0.0078	0.0007	0.0087	0.0255	0.0059	0.0002	0.0066
fr3/sitting_halfsphere	0.0971	0.0194	0.0018	0.0230	0.0919	0.0152	0.0017	0.0196
fr3/walking_static	0.6391	0.3513	0.0587	0.3843	0.6161	0.2691	0.0231	0.3080
fr3/walking_xyz	1.3661	0.6309	0.0612	0.7328	0.8196	0.2768	0.0430	0.3047
fr3/walking_halfsphere	1.5683	0.5831	0.0690	0.6891	0.6246	0.2972	0.1088	0.3116
fr3/walking_rpy	1.1136	0.4975	0.1118	0.5523	0.9351	0.4637	0.0708	0.4983

¹ The unit of each term is the meter. RMSE = root mean square error.

shows the ATE on the sitting and walking dynamic datasets of RGB-D SLAM with and without the proposed moving object detection. From the root mean square error (RMSE) column, we can see that the proposed RGB-D SLAM system with moving object detection method had a lower RMSE than the one without moving object detection in these dynamic environments. In the first two sitting datasets, the new RGB-D SLAM with moving object detection improved the accuracy slightly. This is because the sitting datasets are lowly dynamic scenes where pedestrians are sitting on chairs, so the pedestrians have little impact on the camera’s ego-motion estimation. The next four walking datasets represent highly dynamic scenes with quick moving objects. By using our moving object detection method, the proposed RGB-D SLAM greatly improved the accuracy in these datasets. The results clearly demonstrate the effectiveness and robustness of our RGB-SLAM in both lowly and highly dynamic indoor scenes.

To demonstrate the results intuitively, several accuracy heat maps of the proposed RGB-D SLAM with and without our moving object detection method are shown in Figure 13. The gray dotted line represents the reference ground-truth. The color solid lines represent the estimated trajectories. The color bar represents the size of the ATE. Each row shows the results of the two methods with the same dataset, and the two color bars of each row have the same maximum error and minimum error. Comparing the three pairs of trajectories, we can observe that the proposed RGB-D SLAM with moving object detection method showed better accuracy in both lowly and highly dynamic indoor scenes.

We then compared our new method with the state-of-the-art RGB-D SLAM methods DVO and BaMVO for comparison. The DVO method proposed by Kerl et al. [37] uses a robust error function that reduces the influence of large residuals caused by moving objects. The BaMVO proposed by Kim et al. [39] detects moving objects through accumulation of the warped difference images between the current and remapped depth images over a period of time. These two RGB-D SLAM methods can run robustly in a dynamic indoor environment. For a fair comparison of the three methods, we closed the loop closure detection of our RGB-D SLAM in the experiment and used the RMSE results of these two reference methods in their papers. We adopted the relative pose error (RPE) as another evaluation metric, which described the error with a distance of one second in the estimated trajectory file. Meanwhile, we calculated both the translation and the rotation part of RPE as an evaluation of six DoFs.

Table 4. Translation RMSE and rotation RMSE of relative pose error (RPE) for TUM RGB-D datasets in the dynamic objects category. The numbers in bold indicate that these terms are better than those of another method.

TUM RGB-D Datasets Category: Dynamic Objects		Translation RMSE (m/s)			Rotation RMSE (°/s)
		DVO	BaMVO	Ours	DVO	BaMVO	Ours
Low Dynamic Environment	fr2/desk_with_person	0.0354	0.0352	0.0069	1.5368	1.2159	0.4380
	fr3/sitting_static	0.0157	0.0248	0.0077	0.6084	0.6977	0.2595
	fr3/sitting_xyz	0.0453	0.0482	0.0117	1.4980	1.3885	0.4997
	fr3/sitting_halfsphere	0.1005	0.0589	0.0245	4.6490	2.8804	0.5643
	fr3/sitting_rpy	0.1735	0.1872	0.0234	6.0164	5.9834	0.7838
High Dynamic Environment	fr3/walking_static	0.3818	0.1339	0.1881	6.3502	2.0833	3.2101
	fr3/walking_xyz	0.4360	0.2326	0.2158	7.6669	4.3911	3.6476
	fr3/walking_halfsphere	0.2628	0.1738	0.1908	5.2179	4.2863	3.3321
	fr3/walking_rpy	0.4038	0.3584	0.3270	7.0662	6.3398	6.3215

¹ Relative pose error for pose pairs with a distance of 1 second.

shows the RPE results of DVO, BaMVO, and our RGB-D SLAM. We calculated the RMSE of three methods in nine scenes, including lowly and highly dynamic indoor environments. As we can see in Table 4, our RGB-D SLAM system performed better in most scenes of TUM dynamic datasets in terms of the translation RMSE and rotation RMSE. In the lowly dynamic environment, compared with the other two methods, our system had a lower RPE in the translation and rotation parts. As for the highly dynamic environment, our system improved the precision greatly when compared with DVO, and had better accuracy in most scenes when compared with BaMVO. This shows that the proposed RGB-D SLAM system with a moving object detection method can yield smaller errors than the reference methods, and performs better in common dynamic environments.

4. Conclusions

In this paper, we present a new RGB-D SLAM method with moving object detection for dynamic indoor scenes. The proposed RGB-D SLAM has two main innovations. First, we developed a new method to detect and eliminate moving objects based on mathematical models and geometric constraints. The proposed moving object detection method can be incorporated into the SLAM system as a data filtering process. Second, we clustered the filled depth images and used them to segment moving objects. Therefore, our improved RGB-D SLAM can detect moving objects and alleviate the impact of them better than those methods for which the moving object detection is based on subtraction between consecutive images.

In the experimental section, the proposed RGB-D SLAM was qualitatively evaluated with our homemade sequence images in some typical dynamic indoor scenes like corridor and office. We then conducted experiments to demonstrate that the proposed RGB-D SLAM can greatly improve the accuracy in dynamic indoor scenes by using our moving object detection method. We also presented the quantitative evaluation results of our improved RGB-D SLAM by comparing with DVO and BaMVO, which both are the state-of-the-art RGB-D SLAM algorithms in dynamic environments by incorporating data processing methods or moving object detection into the SLAM process. These quantitative evaluations were carried out using the TUM RGB-D datasets. All the experimental results show that the proposed RGB-D SLAM was able to run effectively and stably in common dynamic indoor scenes. The attained accuracy of pose estimation was generally higher than that of reference methods.

In the future, we plan to continue improving our RGB-D SLAM method by introducing other clustering methods, such as the Gaussian mixture model. In addition, we intend to combine our moving object detection model with a lightweight deep learning method, so as to achieve robust results of moving object detection in challenging dynamic environments.