A Novel Visual SLAM Based on Multiple Deep Neural Networks

Abstract

The current visual simultaneous localization and mapping (SLAM) systems require the use of matched feature point pairs to estimate camera pose and construct environmental maps. Therefore, they suffer from poor performance of the visual feature matchers. To address this problem, a visual SLAM using deep feature matcher is proposed, which is mainly composed of three parallel threads: Visual Odometry, Backend Optimizer and LoopClosing. In the Visual Odometry, the deep feature extractor with convolutional neural networks is utilized for extracting feature points in each image frame. Then, the deep feature matcher is used for obtaining the corresponding feature landmark pairs. Afterwards, a fusion method based on the last and the reference frame is proposed for camera pose estimation. The Backend Optimizer is designed to execute local bundle adjustment for a part of camera poses and landmarks (map points). While LoopClosing, consisting of a lightweight deep loop closure detector and the same matcher as the one used in Visual Odometry is utilized for loop correction based on pose graph. The proposed system has been tested extensively on most of benchmark KITTI odometry dataset. The experimental results show that our system yields better performance than the existing visual SLAM systems. It can not only run in real-time at a speed of 0.08 s per frame, but also reduce estimation error by at least 0.1 m.

1. Introduction

In the last two decades, simultaneous localization and mapping (SLAM) has become a hot area of research in robotics and computer vision. It is a fundamental module for many applications such as mobile robotics, Micro Aerial Vehicles (MAVs), autonomous driving and augmented reality because they all require real-time localization [1]. SLAM techniques use the information acquired by the sensor to build a map of an unknown environment and localize the sensor in the map. Visual SLAM has been widely studied because of the camera’s low price and its ability to acquire rich information about the environment [2,3,4,5,6,7,8].

Most of the currently used Visual SLAM systems rely on hand-crafted visual features [9,10,11,12,13,14]. For example, ORB-SLAM [9,10] performs feature matching and loop closure detection by extracting ORB [15] features in each image frame, and then estimates the camera pose and constructs a map of the environment. However, hand-crafted features may fail to provide consistent feature detection and accurate matching results in complex environments.

With the development of deep learning, feature extraction based on deep neural network (DNN) has replaced hand-crafted features as the mainstream of relevant research [16,17,18,19,20]. DNN-based features have also been widely used in the design and research of visual SLAM systems. For example, Tang et al. [21] designed the GCN-SLAM based on Geometric Correspondence Network [22], Li et al. [23] proposed the DXSLAM based on HF-Net [24], and Deng et al. [25] used a visual SLAM for autonomous driving based on SuperPoint [26]. Although these works have improved the performance of visual SLAM systems by using DNN-based features instead of hand-crafted features, these systems may not be able to track feature points continuously well. This is because they employ the same feature matching method used for hand-crafted features. Such a design severely limits the performance of the system.

To solve the above problems, this paper proposes a visual SLAM system that uses a deep feature extractor, a deep feature matcher, and a lightweight deep loop closure detector. Figure 1 and Figure 2 show the results of some runs of the proposed visual SLAM on the KITTI odometry dataset. The main contributions are as follows:

• A novel Visual Odometry was proposed by reconstructing SuperPoint [26] and SuperGlue [27] as SG-Detector and SG-Matcher and combining them with a fusion estimation method based on last and reference frame for a camera pose.
• A deep LoopClosing module was designed by combining Seq-CALC [28] and SG-Matcher.
• Compared with many existing SLAM systems, the proposed SLAM system reduces estimation error by at least 0.1 m on the KITTI odometry dataset, and it can run in real time at a speed of 0.08 s per frame.

• The rest of the paper is organized as follows. In Section 2, some related work is presented. In Section 3, the system architecture of our visual SLAM and the design principles of each module are explained in detail. In Section 4, the experiments used to test our system are described and the experimental results are analyzed in detail.

2. Related Work

In this section, we review the existing related work in the following three areas: feature extractions for visual SLAM, deep feature matching, and loop closure detection.

2.1. Feature Extractions for Visual SLAM

Many existing visual SLAM systems generate feature pairs for localization and map building by extracting features in each image frame and matching the extracted features between different image frames. For example, ORB-SLAM2 [10] extracts ORB features [15] in each input image frame, constructs corresponding descriptors, and matches these features in the current frame with reference frames and local maps by the Hamming distance between descriptors. Stereo LSD-SLAM [11] treats high-contrast pixels in input image frames as features and matches the features between image frames based on the photoconsistency of all high-contrast pixels. SuperPoint-SLAM [25] uses SuperPoint (a fully convolutional neural network) [26] to extract features and descriptors, and matches these features across image frames by the Euclidean distance between descriptors.

2.2. Deep Feature Matching

Deep-neural-network-based features have gradually replaced hand-crafted features in many computer vision tasks, since simple feature matching methods based on geometric distances between descriptors can no longer achieve satisfactory matching performance. For example, Han et al. proposed MatchNet [29] to use a neural network consisting of three fully connected layers to learn the learned distance metric for feature matching. Sarlin et al. proposed SuperGlue [27] to match two sets of features by using attention graph neural networks and an optimal matching layer. Sun et al. [30] proposed to match local features of two images directly without feature detection by using Transformers [31] (a special class of graph neural networks [32]).

2.3. Loop Closure Detection

Loop closure detection (LCD) requires to recognize a previous visited place from current camera measurements [33]. Most of the current mainstream visual SLAM systems use bag of words (BoW) for LCD. The BoW builds vocabulary trees is based on different point features. As proposed by Gálvez-López et al., DBoW [34] uses the BoW generated by training with hand-crafted point features (e.g., SIFT [35], SURF [36], ORB [15]) to construct the visual vocabulary tree, and then uses the trained vocabulary tree for LCD in visual SLAM.

Recently, deep-neural-network-based methods have been gradually applied to LCD problems. For example, Nate Merrill and Guoquan Huang [37] proposed a very lightweight and effective autoencoder network. In order to simulate the natural variation caused by robot motion, they also added a random perspective transformation to the training data, which improves the performance of the unlabeled data set. Building on work [37], Xiong et al. [28] applied principal component analysis (PCA) to compress the dimensionality of descriptors and combined linear queries with fast approximate nearest neighbor search to reduce execution time and improve the efficiency of sequence matching.

3. The Proposed SLAM

The main idea of the proposed SLAM system is the combination of deep feature extractor, deep feature matcher and deep loop closure detector in the traditional visual SLAM pipeline. The deep feature extractor (SuperPoint [26]) provides stable feature points to the system. The deep feature matcher (SuperGlue [27]) provides consecutive pairs of correctly matched points required for pose estimation. The deep loop closure detector (Seq-CALC [28]) corrects the accumulated error of the system by detecting correct loop closure in the environment. The design of our system is greatly inspired by ORB-SLAM2 [10]. The system framework is illustrated in Figure 3, which consists of four main modules: Visual Odometry, Backend Optimizer, Map, and Loop Closing. It runs three independent threads: Visual Odometry, Backend Optimizer, and Loop Closing in parallel. These three threads synchronize data through the Map module, which is the database.

In the Visual Odometry, the extracted feature points are used to construct the initial map and calculate the initial camera pose, and then the map points in the reference frame and the temporary map points in the last frame are tracked by feature matching; then, the relative poses are calculated separately, and, finally, the two relative poses are fused and a new keyframe is constructed. The new keyframe is inserted into the Map as the current keyframe.

In the Backend Optimizer, the keyframe sequence and the set of map points in the activation window, intercepted by the current frame, are read to perform the local bundle adjustment optimization and remove the outlier map points.

In the Loop Closing module, a light deep loop closure detector is used to search the Map for the historical keyframes that form a loop closure with the current keyframe; then, the corrected poses of the current keyframe are calculated by feature matching, and finally the poses of the keyframes are corrected for the historical keyframes by the pose graph of the keyframes.

Further, we introduce this system in detail, from feature extraction and matching, $\mathfrak{se}$(3)-based pose fusion estimation ($\mathfrak{se}$(3), which is the Lie algebra corresponding to the special Euclidean group [38]), backend optimization, to loop closure correction.

3.1. Feature Extraction and Matching

SuperPoint [26] and SuperGlue [27] networks were used to build feature detector (extractor) and matcher in this system.

As illustrated in Figure 4, the KeyPoint Encoder module of SuperGlue and SuperPoint are combined to form a SG-Detector, and the remaining modules of SuperGlue are combined to form a SG-Matcher. This allows SG-Matcher the matching with only the input descriptors as traditional matchers used in visual SLAM pipelines.

The SG-Detector is only used to propose keypoints (features) of the input image in Visual Odometry. The SG-Matcher is used to track map points in the reference keyframe and temporary map points from the last frame in Visual Odometry, as well as to calculate corrected poses in Loop Closing.

3.2. $\mathfrak{se}$(3)-Based Pose Fusion Estimation

To further improve the performance of the system, we proposed a novel algorithm for relative positional fusion estimation over $\mathfrak{se}$(3). The reason for choosing to perform pose fusion on $\mathfrak{se}$(3) is that the pose can perform vector addition and scalar–vector multiplication on $\mathfrak{se}$(3). For more details on $\mathfrak{se}$(3), see [38].

To maintain the correlation of feature points across multiple frames and to control the size of the optimization problem, visual SLAM usually calculates the camera pose by tracking the map points of the reference frame, but tracking only the map points of the reference frame results in fewer feature–landmark pairs participating in pose estimation as the camera moves away from the reference frame, which degrades the pose estimation performance. Our $\mathfrak{se}$(3)-based pose fusion algorithm uses the map points generated in the last frame so that there are enough feature–landmark pairs to participate in the pose estimation even when the camera is far away from the reference frame to ensure the quality of the estimation. Moreover, since the map points generated in the last frame are “temporary”, they do not increase the size of the optimization problem.

First, $N_r$ map points are tracked in the reference keyframe and $N_l$ temporary map points are tracked in the last frame using SG-Matcher. Then, we calculate the relative poses ${\check{\mathbf{T}}}_{cr}$ of the current frame relative to the reference keyframe based on the correspondence between the tracked $N_r$ map points and the keypoints of the current frame, and the relative poses ${\mathbf{T}}_{cl}$ of the current frame relative to the last frame based on the correspondence between the tracked $N_l$ temporary map points and the keypoints of the current frame. Lastly, the final relative pose ${\widehat{\mathbf{T}}}_{cr}$ is obtained by weighted fusion of ${\check{\mathbf{T}}}_{cr}$ and ${\mathbf{T}}_{cl}$ (${\mathbf{T}}_{lr}$ is the relative pose of the last frame w.r.t. the reference keyframe) on $\mathfrak{se}$(3), and the absolute pose ${\mathbf{T}}_c$ of the current frame (${\mathbf{T}}_r$ is the absolute pose of the reference keyframe) is calculated, as shown in Equations (1) and (2).

$${\widehat{\mathbf{T}}}_{cr}=exp({(\mathit{\alpha }ln{({\check{\mathbf{T}}}_{cr})}^{\vee }+[1-\mathit{\alpha }]ln{({\mathbf{T}}_{cl}{\mathbf{T}}_{lr})}^{\vee })}^{\wedge }),$$

(1)

$${\mathbf{T}}_c={\widehat{\mathbf{T}}}_{cr}{\mathbf{T}}_r,$$

(2)

where

$$\mathit{\alpha }=\frac{exp(-\frac{R_{{\check{\mathbf{T}}}_{cr}}}{N_r})}{exp(-\frac{R_{{\check{\mathbf{T}}}_{cr}}}{N_r})+exp(-\frac{R_{{\mathbf{T}}_{cl}}}{N_l})},$$

where $R_{{\check{\mathbf{T}}}_{cr}}$ and $R_{{\mathbf{T}}_{cl}}$ are used to indicate the uncertainties of ${\check{\mathbf{T}}}_{cr}$ and ${\mathbf{T}}_{cl}$, respectively, which can be obtained for free in the calculation of ${\check{\mathbf{T}}}_{cr}$ and ${\mathbf{T}}_{cl}$. They are defined as follows:

$$R_{{\check{\mathbf{T}}}_{cr}}=\sum _j^{N_r}{∥{\mathbf{x}}_j^r-\mathsf{\Pi }\left({\check{\mathbf{T}}}_{cr}{\mathbf{T}}_r{\mathbf{p}}_j^r\right)∥}_2^2,$$

$$R_{{\mathbf{T}}_{cl}}=\sum _j^{N_l}{∥{\mathbf{x}}_j^l-\mathsf{\Pi }\left({\mathbf{T}}_{cl}{\mathbf{T}}_{lr}{\mathbf{T}}_r{\mathbf{p}}_j^l\right)∥}_2^2,$$

where ${\mathbf{x}}_j^r$ is the keypoint in the current frame corresponding to map point ${\mathbf{p}}_j^r$, and ${\mathbf{x}}_j^l$ is the keypoint in the current frame corresponding to the temporary map point ${\mathbf{p}}_j^l$. The projection function $\mathsf{\Pi }$ is defined as follows:

$$\mathsf{\Pi }\left(\mathbf{p}\right)=\left[\begin{array}{c}{f}_x\frac{x}{z}+c_x\\ f_y\frac{y}{z}+c_y\end{array}\right],$$

with $\mathbf{p}={[x,y,z]}^{\mathrm{T}}$ as the 3D point, $f_x$ and $f_y$ as the focal lengths of the camera, and $(c_x,c_y)$ as the principal point of the camera.

3.3. Backend Optimization

In the Backend Optimizer thread, the local BA optimization problem is constructed by reading the set ${\mathcal{K}}_a$ of all keyframes in the active window of the Map and the set ${\mathcal{P}}_a$ of all map points seen by these keyframes, as the following:

$$\left\{{\mathbf{T}}_i,{\mathbf{p}}_j\parallel i\in {\mathcal{K}}_a,j\in {\mathcal{P}}_a\right\}=arg\underset{{\mathbf{T}}_i,{\mathbf{p}}_j}{min}\sum _{i\in {\mathcal{K}}_a}\sum _{j\in {\mathcal{P}}_a}\rho \left(e_{ij}\right),$$

(3)

$$e_{ij}={∥{\mathbf{x}}_{ij}-\mathsf{\Pi }\left({\mathbf{T}}_i{\mathbf{p}}_j\right)∥}_2,$$

(4)

where $\rho$ is the huber robust kernel function, ${\mathbf{x}}_{ij}$ is the keypoint corresponding to ${\mathbf{p}}_j$ at ${\mathbf{T}}_i$. Equation (3) is solved by the Levenberg-Marquardt algorithm [39].

3.4. Loop Closure Correction

In the Loop Closing thread, three tasks are completed: the use of Seq-CALC [28] (a lightweight deep loop closure detection network) to detect the historical keyframes that form a loop closure with the current keyframe, the use of SG-Matcher to match the keypoints of the current keyframe and the loop closure keyframe and calculate the corrected absolute pose ${\widehat{\mathbf{T}}}_c$ of the current frame, the optimization of the set $\mathcal{K}$ of all keyframes on the pose graph and correction of the set $\mathcal{P}$ of all map points, as follows:

$${\widehat{\mathbf{T}}}_k=arg\underset{{\mathbf{T}}_k}{min}\sum _{k\in \mathcal{K}}{∥ln{\left({\mathbf{T}}_k{\mathbf{T}}_c^{-1}{\widehat{\mathbf{T}}}_c{\mathbf{T}}_k^{-1}\right)}^{\vee }∥}_2^2,$$

(5)

$${\widehat{\mathbf{p}}}_j={\widehat{\mathbf{T}}}_c^{-1}{\mathbf{T}}_c{\mathbf{p}}_j,j\in \mathcal{P},$$

(6)

where ${\widehat{\mathbf{p}}}_{(·)}$ are the location of corrected map points. Equation (5) is also solved by the Levenberg–Marquardt algorithm [39].

4. Experiments

We tested our proposed system by conducting the following four sets of experiments on the KITTI odometry dataset [40]: Evaluation experiment to evaluate the estimation accuracy, ablation experiment to verify the effectiveness of the $\mathfrak{se}$(3)-based pose fusion estimation algorithm, validation experiment to demonstrate the necessity of the deep feature matcher, and runtime experiment to show the running speed of our proposed system.

The KITTI odometry dataset is an open-source dataset commonly used to evaluate visual localization accuracy. It consists of 22 stereo image sequences all sampled at 10 Hz (i.e., 100 ms sampling period), of which only 11 sequences (00-10) provide ground truth trajectories. We selected 8 of the 11 sequences that are representative and suitable for pure visual SLAM. The root mean square error (RMSE) [41], which is commonly used to represent the accuracy of localization estimation, was used as the evaluation metric for all experiments.

We performed all experiments on a laptop with a 16 Gb RAM, Intel Core i7-11800H CPU and GeForce RTX 3060 Mobile GPU. The SuperPoint, SuperGlue and Seq-CALC networks in the system all use the officially provided pre-trained models without fine-tuning on the KITTI Odometry Dataset. We only ran the SuperPoint, SuperGlue models on a GeForce RTX 3060 Mobile GPU and the rest of the system on an Intel Core i7-11800H CPU. In order to ensure the generality of the experiments, each set of experiments was repeated five times. The average of the results of the five experiments was then taken as the final measure.

4.1. Evaluation Experiment

To evaluate the accuracy of the proposed system, the results are compared with the data on ORB-SLAM2 [10], LSD-SLAM [11] and DSV-SLAM [12]. The main experimental procedure is to connect each image sequence to our system and other comparison systems, respectively. Each system was run independently five times, and then the results of each system were recorded.

Table 1. Comparison of accuracy in the KITTI odometry dataset using RMSE (m).

Sequence	Our	ORB-SLAM2	LSD-SLAM	DSV-SLAM
00	1.0	1.3	1.0	2.6
03	0.5	0.6	1.2	2.5
04	0.2	0.2	0.2	1.1
05	0.6	0.8	1.5	3.8
06	0.7	0.8	1.3	0.8
07	0.4	0.5	0.5	4.4
09	3.1	3.2	5.6	5.3
10	0.8	1.0	1.5	1.8

shows the results on the 8 sequences. Figure 5 shows some examples of the estimated trajectories.

Our system outperforms ORB-SLAM2, LSD-SLAM and DSV-SLAM in most sequences. In addition to Sequence 04, the estimation accuracy of our system is improved by at least 0.1 m (Sequences 03, 06, 07, 09) and at most 0.3 m (Sequence 00) compared to ORB-SLAM2.

Compared to LSD-SLAM, the estimation accuracy of our system is improved by more than 0.1 m in most cases and even more than 1.0 m in some cases, except for Sequences 00 and 04, which are equal. This is mainly due to the presence of many scenes in those sequences (Sequences 03, 05, 06, 07, 09, 10) that cause dramatic changes in light intensity. For example, weaving through a forest, entering a residential area from an open road, etc.

Compared to DSV-SLAM, the estimation accuracy of our system is only more than 0.1 m for Sequence 06, and at least 1.0 m for other sequences. This is because our system, using SuperPoint, extracts features that are less susceptible to changes in light intensity and are robust. The features are then matched continuously and accurately using SuperGlue. Moreover, the use of Seq-CALC, a deep loop closure detector that uses only the original image, avoids the propagation of feature extraction errors in the loop closure detection process.

The accuracy of our system, ORB-SLAM2 and LSD-SLAM, is the same on Sequence 04 because Sequence 04 is a very simple sequence with a slow short-range linear motion on a flat road, as shown in Figure 6.

4.2. Ablation Experiment

To verify the effectiveness of our proposed $\mathfrak{se}$(3)-based pose fusion estimation algorithm, we first run our system normally five times on the selected image sequences and record the experimental results. Then, the $\mathit{\alpha }$ in the fusion method is forced to 1.0 (so that the position estimation depends only on the map points of the tracked reference frames, see Section 3.2 for details), and the same experiment is repeated five more times and the experimental results are recorded. The results are shown in

Table 2. Verifying fusion pose estimation in the KITTI odometry dataset using RMSE (m).

Sequence	Fuse Last and Reference	Reference
00	1.0	1.2
03	0.5	1.5
04	0.2	0.2
05	0.6	0.9
06	0.7	0.8
07	0.4	0.5
09	3.1	3.3
10	0.8	1.3

.

According to Table 2, after fusing the last frame with the reference frame information using our $\mathfrak{se}$(3)-based pose fusion estimation algorithm, the pose estimation error is significantly reduced. This is because as the motion gradually moves away from the reference frame, the tracked landmarks gradually decrease and concentrate to some regions of the image (green feature points in Figure 7). This leads to a larger error in the pose estimation. Our $\mathfrak{se}$(3)-based pose fusion estimation algorithm, by introducing temporary landmarks generated in the last frame (blue feature points in Figure 7), allows the participation of enough landmarks in the pose estimation and ensures that the features corresponding to these landmarks are uniformly distributed over the entire image.

4.3. Validation Experiment

To demonstrate the necessity of using the corresponding deep feature matcher in a visual SLAM system that uses a deep feature extractor, we compare our system with SuperPoint-SLAM [25] on selected sequences. Because SuperPoint-SLAM uses SuperPoint as our system (also using the same pre-trained model) but does not use a deep feature matcher, only the Euclidean distance between descriptors is used to match features.

Table 3. Comparison with SuperPoint-SLAM in the KITTI odometry dataset using RMSE (m).

Sequence	Our	SuperPoint-SLAM
00	1.0	X
03	0.5	1.0
04	0.2	0.4
05	0.6	3.7
06	0.7	14.3
07	0.4	3.0
09	3.1	X
10	0.8	5.3

shows that using the corresponding deep feature matcher in a visual SLAM system employing depth features can significantly improve the estimation accuracy and stability of the system. For example, on Sequences 00 and 09, SuperPoint-SLAM cannot even run the whole sequence, while our system not only runs stably but also achieves good estimation accuracy. This is mainly due to the fact that our system uses SuperGlue which is adapted to SuperPoint. As shown in Figure 8a, the features extracted by SuperPoint can be correctly matched using SuperGlue. Correct matched features are the basis for accurate pose estimation. However, as can be seen in Figure 8b, the matching method used by SuperPoint-SLAM produces more incorrectly matched pairs when matching the features extracted by SuperPoint. This introduces a larger error in the pose estimation.

4.4. Runtime Experiment

In order to show the running speed of our proposed system, we measured the duration of each image frame from reading to deriving the pose of this image frame as the time consumed to process this image frame. The running speed, denoted as $v_t$, is defined by the averaged time by an image sequence:

$$v_t\triangleq \frac{{\displaystyle \sum _{n=1}^{N_f}}{t}_n}{N_f},$$

(7)

where $N_f$ is number of frames in the image sequence, $t_n$ is the time used for processing the nth frame.

From

Table 4. The runtime of our system in the KITTI odometry dataset using time per frame (the sampling period of the sequence is 100 ms).

Sequence	Number of Frames	Time per Frame (ms)
00	4541	81.8
03	801	57.6
04	271	81.5
05	2761	72.1
06	1101	60.4
07	1101	70.8
09	1591	62.7
10	1201	59.8

, it is obvious that our proposed system only takes 81.8 ms to process an image frame with the GeForce RTX 3060 Mobile GPU running the SuperPoint, SuperGlue model and the Intel Core i7-11800H CPU running the rest of the system. Since the sampling period of the sequence is 100 ms, our system can achieve the performance of real-time operation under the above equipment conditions. According to the results published on the corresponding website of [40], the speed of our system (0.08 s per frame) already exceeds that of many existing SLAM systems.

5. Conclusions

In this paper, we proposed a novel Visual Odometry combining a SG-Detector, a SG-Matcher and a $\mathfrak{se}$(3)-based pose fusion estimation algorithms and a deep LoopClosing module based on SG-Matcher and Seq-CALC networks. Then, a complete visual SLAM system was designed using them.

Our proposed system achieves better performance than ORB-SLAM2, LSD-SLAM and DSV-SLAM on most sequences of the KITTI odometry dataset. For example, the estimation error is generally reduced by 0.1 m, and it can run in real time at 0.08 s per frame on GeForce RTX 3060 Mobile GPU and Intel Core i7-11800H CPU.

The experimental results indicated that corresponding feature matchers should be introduced in visual SLAM systems using deep features, since traditional matching methods for hand-crafted features do not match well the deep feature descriptors with the deep feature extractor.

Although our system is currently only tested on the KITTI odometry dataset, the experimental results do yield some indications on the superiority, feasibility and, particularly, the practicality of the proposed SLAM system. This is because the KITTI odometry dataset is real-time data derived from real scenes. In future work, we will focus on the lightweighting of the deep feature matcher and deep feature extractor because these two deep neural network models require a lot of memory and occupy most of the running time of the whole system. If a more lightweight network structure can be designed, it will enable our proposed system to migrate to run in real time in an embedded system. This allows an easier integration of our system into the existing mobile robotics, MAVs, autonomous driving and augmented reality systems.